Mustafa ERBAY

Posted on Sep 27 Originally published at mustafaerbay.com.tr

Reverted in 90 Seconds, Broken for Five Hours

#git #cicd #production #softwarecareer

On the morning of 22 September, at 09:18:46, I published an ordinary post. At 09:20:13 I pushed two Revert commits. In the 87 seconds between them, a security review that was not supposed to be public yet was sitting on the main branch.

This post isn't about those 87 seconds. I handled that part. What's actually interesting is the second half of the same accident: the undo itself produced a new defect, and that one lived five more hours. No test said so. No alarm went off. Across those two days the repository ran 238 workflows, 106 of them jobs whose entire purpose is asking "did something break," and not one went red.

What I pushed

The post that went out that morning was about vm.max_map_count: technical, harmless, minding its own business. Commit 6c168b17. The problem wasn't the post. The problem was what that commit carried on its back.

A day earlier I had finished, locally, a security review that was in a coordinated disclosure process with a vendor. Two commits: the article itself, and a round of corrections. Both were being deliberately held — the publication date had been pushed out, because the other side hadn't closed its response window yet. In my head these were "work set aside." In git's head, they were the tip of main.

With the new post stacked on that tip, the push carried all three. Reverting would delete 374 lines: 185 lines of Turkish text, 185 lines of English translation, and 4 lines added to the source allowlist. Plus two cover images.

So why was that work sitting on local main? I won't defend the answer, but I'll give it: in a one-person repository, main quietly becomes your desk. Branching has a cost — you have to think of a name, switch, come back — and that cost, multiplied by the feeling that nobody's watching anyway, rounds down to zero.

But main is a shared resource even when nobody's watching. The party I share it with isn't another developer; it's my own automation: deploy, generation, social posting, all of them read that tip. Because I couldn't be bothered to name a branch, I left an embargoed document at the mouth of the pipeline.

⚠️ Every gate looked at the change

Source policy validation, 88 policy tests, MDX schema checks, the English translation gate — all green. All of them working correctly, too. Because all of them inspect the change. None of them asks what you built that change on top of.

The record's fingerprint: two dates

The most annoying part is that git had told me from the start. I just never thought to look.

Every commit carries two dates: author date and committer date. Normally they're identical. Not after a rebase. Git's own documentation describes rebase as replaying the commits "one by one, in order," producing a new commit for each — the author date is preserved, the committer date becomes the current time.

That is exactly what's in the record:

5b505d95  author 21 Sep 12:02  |  committer 22 Sep 09:18
6003feb2  author 21 Sep 12:39  |  committer 22 Sep 09:18
6c168b17  author 22 Sep 09:18  |  committer 22 Sep 09:18

A one-day gap, the same committer stamp. In other words: "what I wrote yesterday was replayed today." Work that was being held had picked itself up and become the base underneath a new post. Git didn't hide it; I was always reading git log with %s, never with %cd. Black boxes don't actually spend their worst nights without telling anyone — they just write it in the column nobody reads.

87 seconds

After the push I triggered the deploy by hand, as usual. Shortly after, I cancelled it.

Here's what the record says: the job started at 06:19:03 UTC and was cancelled at 06:20:30, so it ran for 87 seconds. A deploy that completed that same day took 177 seconds. Looking at the step breakdown, the job was cut during step six, "Install dependencies"; Build & atomic swap, Wait for deploy and the Cloudflare purge were all skipped. The build never started.

So "a narrow escape" isn't quite the right framing: it was about halfway through the clock, but the steps that actually publish anything hadn't come up yet. That URL still returns 404 today.

There's something the record doesn't say, and I'll write that down too: I can't reconstruct what I was looking at when I cancelled. No step in the workflow prints the list of changed files, and the run's logs have since been cleared by retention. I probably looked at the push output, or at git log. But I'm not going to invent a scene where I spotted a familiar slug; I don't have evidence for that scene. What I have is a cancel button pressed at second 87.

The loud one ended, the quiet one began

I pushed two Revert commits, redeployed, production was clean. I thought it was done.

It wasn't. The content calendar — content-calendar.json, which holds the pipeline's topic queue — could not be parsed.

The first defect was in 5b505d95 itself. When the new calendar entry was appended, the previous object's closing was forgotten: the }, and { that belonged in between weren't there. Two objects had fused:

      "source": "...the previous entry's long note..."
      "topic": "...the new entry starts right here..."

Python's parser says it plainly: Expecting ',' delimiter: line 6053 column 7. In a file that held 479 entries at the time, two missing characters at the start of a single line.

JSON has no flexibility here; the syntax requires values in an array to be separated by commas. The file wasn't "a bit broken" — either it reads completely or it doesn't read at all. A defect like this blows up on the first line that touches the file. The problem was that nothing was touching it.

Thirty-seven minutes later, the same day, I made a "final audit" correction to that entry: moved the publication date out and updated the source line in place. An in-place edit, in a file that couldn't be parsed. Nobody objected, because the thing that would have objected couldn't read the file in the first place.

Revert is not an undo button

Now the real lesson — and I have to admit that on my first pass at writing this post, I got the facts wrong myself.

Git's documentation is clear: revert records new commits that reverse the effect of earlier ones; it doesn't remove anything from history. It specifically contrasts this with reset. So revert isn't "pretending it never happened," it's "doing the opposite" — and to do the opposite it has to correctly know what was done.

In my file it couldn't. While undoing that "final audit" correction, fd17e0ff put the old source line back — but the new one was still there. The result: two source keys. And because the structure was already fused, the surplus line didn't attach to its own entry but to its neighbour: the calendar entry belonging to the vm.max_map_count post that had carried the whole accident.

So the revert didn't fix the breakage. It added another one, and contaminated an unrelated record. The repair commit in the repository says so in as many words: "the ispmanager entry was added without delimiters + the fd17e0ff revert left a duplicate source line."

Nor was the repair what I first assumed — "putting the missing comma in." At 14:20:09, bcd6c751 made two deletions: it removed the 18-line fused block and the orphan source line. The }, was already there. There was a surplus, not a shortage.

On main, the calendar stayed unparseable from 09:18:46 to 14:20:09 — roughly five hours.

One more detail: what repaired it wasn't "someone opening the file days later." The next generation run, which has to read the queue, failed to open the file, found the problem and fixed it the same day. The defect surfaced the moment its consumer ran — the five hours were simply how long it took for that consumer to run.

Why no gate saw it

From here on it stops being personal incompetence and becomes a design question.

I looked at the deploy workflow. In its trigger path filter sat exactly the line I expected:

- '!scripts/content-calendar.json'

"There's the reason," I thought. It isn't. Look at the positive patterns in the same list: src/**, public/**, astro.config.mjs, tailwind.config.*, package.json, package-lock.json, tsconfig.json. The calendar file matches none of them. GitHub's documentation says a negative pattern only excludes a path that a preceding positive pattern matched; here there is no match to cancel.

So that line does nothing. It excludes a file that was already outside. A line that reads like a guard but is really just a statement of intent — and I, while researching precisely this topic, glanced at it and took it for the cause. The outcome (the deploy doesn't fire on calendar edits) is correct; what produces that outcome isn't the line, it's that scripts/ was never in the list at all.

The numbers show the gap. On 21 and 22 September the repository ran 238 workflows: 179 succeeded, 58 were skipped, 1 was cancelled. Failures: zero. And 106 of those 238 runs — nearly half — were the alert workflow whose job is to notice breakage. It ran, and ran, and ran, and found nothing to say.

💡 A recipe for silent breakage

If a file is critical in production but sits outside your CI's trigger paths, it's a perfect candidate for silent breakage. Keeping it outside may be the right call; what's missing is a separate, cheap check that looks at it. And believing such a check exists is more dangerous than not having one.

Those numbers unsettled me rather than reassuring me. 238 green runs don't show that the pipeline is healthy; they show that the questions it asks had nothing to do with what happened that day. A green dashboard is only as good as its scope, and it paints everything outside that scope the same green.

The one check that looks at the base

I came away with two rules. Both are embarrassingly simple, which is precisely why they went unwritten.

One: work always happens in a clean tree derived from origin/main. Where local main happens to be is none of my business. Run git fetch, branch from that reference, work there.

Two: one question before pushing — what am I sending? git log --oneline origin/main..HEAD answers it in a second. Had I run that command that morning, I'd have seen three lines and known instantly that two of them had no business being there.

And parse the file after every operation that touches the calendar. As cheap as python3 -c "import json;json.load(open(...))". That one line was the price of five hours of silence.

Where do these checks belong? Not in CI, I'd argue. CI runs after the push; the thing that needs stopping here is the push itself. Git's pre-push hook exists for exactly this moment: it receives what's about to be sent on standard input, line by line — local ref, local object name, remote ref, remote object name — and, in the documentation's words, if it exits with a non-zero status git push aborts without pushing anything.

The hook's weakness is documented too: --no-verify bypasses it completely. In a one-person setup I don't mind; if I bypass my own hook deliberately, then I did it deliberately. What I mind is the question that was never asked.

The rule existed, and the trap still worked

To be honest, I'm not in a position to end this post with "lesson learned," because today I stepped into version two of the same trap.

When I opened the repository for production this morning, local main was four days behind. I looked at the calendar: 11 topics appeared to be "pending." When I checked the current origin/main, 7 of them had already been written — and all seven were already live. One of them four days ago, in this very category.

So this time the stale base didn't make me push the wrong post; it would have made me write one. Same root, different output. The difference was that my first action was git fetch.

One more confession: in the first draft of this post I described how the calendar broke, who broke it and how it was repaired — and I got all three wrong. I pinned the duplicate source line on the correction commit; the revert had left it. I assumed the repair was "adding a comma"; it was two deletions. I had written up my own accident from memory instead of from the record. Opening the record line by line, the true story turned out to be both different and better: the undo didn't merely fail to repair the damage, it manufactured more.

"We made a rule, problem solved" doesn't hold up well in this line of work. A rule doesn't end anything; it moves the mistake to where the damage is cheap. On 22 September the error was caught at second 87 of a deploy. Today it was caught before a single line was written. That, it turns out, is what progress looks like.

For anyone who'd rather not repeat this

Before pushing, run git log --oneline origin/main..HEAD — see the load you're carrying, not just your last commit.
Don't park held work on local main. Park it on a branch with a name; the tip of main is not a waiting room.
Add %cd to your git log format. Author and committer dates diverging tells you that commit was replayed.
After a revert, check whether it actually fixed things. A revert is a record of an inverse operation; in a structurally broken file it can create fresh damage.
Read your CI's trigger path list — and test what it actually does. The line you take for a guard may be doing nothing at all.
Parse every machine-edited structured file the moment the editing step finishes. Before schema validation — just the question "is this valid?"

Conclusion

There were two mistakes in this incident, and their costs were exact opposites. The loud one — an embargoed text landing on the main branch — sounds far worse, and it closed in 87 seconds.

The quiet one began with a missing delimiter, multiplied through the undo, and lived five hours. It walked through 238 workflow runs, 106 of which were assigned to look for breakage. What ended it wasn't an alarm; it was the next generation run failing to open a file it had to read.

What stays with me is this: the maturity of an automation pipeline isn't measured by how many gates it has, but by whether you can enumerate the places those gates don't look. Mine had never had that list written down. It's now one line long, it includes a YAML line I mistook for a guard that does nothing at all, and it's probably still incomplete.

If you want a wider accounting of where this blog's automation has broken, the story of 67 repairs I pulled out of the git record is here. And the harmless post that got carried on the back of the accident is over here — perfectly innocent on its own subject, though its calendar entry carried its neighbour's dirt for a while.

DEV Community