On the morning of 13 May I added one line of configuration to this repository.
The intent was innocent enough: when two production runs woke up at the same
time and wrote to the same file, git raised a conflict, the rebase stopped and
the workflow died. So I told git, "you resolve the conflict in this file." It
did.
142 days later I pulled that line back out. Sweeping the repository history
this morning produced this table: over the driver's lifetime, 848 commits
touched the calendar file and 11 of them left it corrupted. Ten of those cost a
production run its record. Six of them raised no error at all.
The part that bothers me most is somewhere else. I missed the root cause of
these accidents in two separate places: in a commit message in May I wrote it
incompletely, and on this blog two weeks ago I wrote it wrong. I will fix both.
So the push would not fail
The production pipeline's memory is a single file: scripts/content-calendar.json.
Which topic has been written, which one is waiting, which post went to which
address, all of it lives there. When publishing frequency went up in mid-May,
two runs landing in the same window became routine. One pushes, the second gets
rejected, the calendar conflicts during the rebase, the rebase halts, the job
ends. The new post's files sit ready in a commit and go nowhere.
The commit message from 13 May at 10:01 lays out that morning's reasoning
flatly: cron at 06:42, manual run at 06:47. Two patches went in together. One
was a strategy flag that leaned the rebase toward the remote side (-X theirs);
the other was this line:
scripts/content-calendar.json merge=union
And next to it I wrote the justification. My own sentence, word for word:
# `merge=union` driver, iki tarafin tum satirlarini KORUR (union) — bizim
# JSON yapimiz icin yeterli (entries array'i bozulmaz, JSON yine parse OK).
Roughly: the union driver KEEPS every line from both sides, which is good
enough for our JSON structure, the entries array stays intact, JSON still
parses OK. What stings about reading that today is not that it is wrong. It is
that I wrote an assumption in the shape of a verified fact. "JSON still
parses OK" reads like a measurement result; nothing was measured that morning.
Five more patches piled up around the same file a week later: four on 21 May,
the fifth on 24 May. I have laid out the pipeline's repair
record before and that
whole family is in there. What that account never names is the pair of
lines the same commit brought in: the driver and the -X theirs flag. Neither
left a visible failure behind.
What the documentation says
Git's gitattributes documentation describes the union driver in three
sentences. It runs a three-way merge on text files but, instead of leaving
conflict markers, it takes "lines from both versions". Then comes the warning:
the added lines tend to end up in random order and "the user should verify the
result". The last sentence looks me straight in the eye: "Do not use this if
you do not understand the implications."
I am in no position to blame the documentation. Where it asked for
verification, I filed an assumption.
The same page has another detail. The merge attribute kicks in when a
file-level merge is needed during git merge, and during "other commands such
as git revert and git cherry-pick". Rebase is not named in that list. But
the git-rebase documentation only goes as far as saying a rebase is "similar
to running git cherry-pick" for each commit; it says nothing about merge
drivers. That the driver also fires during a rebase I know from the third lab
run below, not from the documentation. So the driver was active exactly where
it never crossed my mind: in the rebase that follows a rejected push.
Six runs in an isolated repo
Rather than guess at this, I set it up and ran it. One fixture: an empty
repository, a cal.json holding two older entries, two branches, each appending
its own entry to the end of the array. All six runs below come from that same
fixture; only the .gitattributes line and the command change.
Run 1, driver on, the two entries have identical trailing lines. Git did not
complain:
Auto-merging cal.json
Merge made by the 'ort' strategy.
cal.json | 1 +
The end of the file came out like this:
{
"topic": "run-A-topic",
"topic": "run-B-topic",
"category": "career",
"tags": [
"career"
],
"style": "opinion",
"targetWords": 1500,
"scheduled": "2026-05-14",
"generated": false
}
Valid JSON. The parser sees three entries while the raw text holds four topic
keys. Run A's record is still in the file, and absent from the parsed structure.
Run 2, driver on, trailing lines differ. I made the two entries differ the
way they do in real life: different category, different tags, different word
target, different source note. The second entry's lines start right after the
first entry's last line, with no }, and { in between:
"source": "run A note"
"topic": "run-B-topic",
The parser says so bluntly: Expecting ',' delimiter: line 36 column 7.
Run 3, driver on, a rebase instead of a merge. This is what actually happens
in production. Same damage, same silence:
Rebasing (1/1)Successfully rebased and updated refs/heads/local-side.
git status printed nothing at all and the working tree stayed clean. One more
detail: this rebase finished inside the same second, so the gap between the
commit's author and committer timestamps came out zero. I lean on that gap as a
divider further down, so I am noting it here.
Run 4, the control: driver removed, plain rebase. This time the rebase
stopped:
CONFLICT (content): Merge conflict in cal.json
Automatic merge failed; fix conflicts and then commit the result.
git status reported UU cal.json and three conflict marker lines appeared in
the file. So the driverless behaviour was exactly what I wanted. The thing I had
silenced as a "problem" was the system trying to tell me that two runs were
writing to the same place.
That leaves the second patch from the same commit, -X theirs. I ran that one
twice on the same fixture, and the answer surprised me. With the driver removed,
rebase -X theirs finished without any conflict at all: clean working tree,
valid JSON, zero repeated keys. But the raw text held three topic keys instead
of four. The remote run's entry had been deleted outright, without leaving a
mark. With the driver back on, the same command reproduced run 3: a fused entry
and four keys. A path-specific driver takes precedence over the general strategy
flag.
So there were two silent mechanisms in play, not one. The driver collapses two
records into each other; -X theirs deletes the other side's record completely
and leaves a spotless file behind. Both arrived in the same commit, on the same
morning, with the same justification.
The mechanism fits in one sentence: a union merge only duplicates the lines
that differ and shares the lines that are identical. If two entries have
identical trailing lines, the closing brace is shared too and the file stays
valid. If the trailing lines differ, the brace disappears and the file becomes
unparseable. Either way two records collapse into one object; the only variable
is whether anything raises an error.
Last key wins
To see why the record disappears, look at JSON's own rules. RFC 8259 does not
require names within an object to be unique; it recommends it: "The names
within an object SHOULD be unique." Then it says what follows. When names are
not unique, the behaviour of receiving software is unpredictable; many
implementations report only the last name/value pair, some report an error,
some return all of them.
Everything that reads this calendar sits on the Node side: the generator, the
calendar verifier, the structural reconciler, the tests. Node swallows a
repeated key without argument:
$ node -e 'const o=JSON.parse(`{"topic":"A","topic":"B"}`);
console.log(JSON.stringify(o), Object.keys(o).length)'
{"topic":"B"} 1
Python, which I use in the lab and in the sweep scripts, follows the same rule;
its documentation says repeated names are accepted and only the value of the
last pair is used. The file opens, the schema passes, nothing turns red. One of
the two runs simply has its work ignored.
How many times it actually happened
It would have been tempting to stop here and conclude "so there was a risk."
Risk and incident are different things, and the git history tells you which one
you had.
I wrote a sweep that opens every commit touching the calendar, in order: parse
each version, scan the raw text for repeated keys inside the same object,
compare against the parent's state. For a commit to count as a "damage event"
its parent must be healthy and the commit itself corrupted, otherwise I would
have counted inherited breakage as a fresh accident.
Across the full history, 1,115 commits touched the calendar. The portion inside
the driver's lifetime is 848.
| Metric | Count |
|---|---|
| Commits touching the calendar during the driver's lifetime | 848 |
| Commits whose author and committer timestamps match exactly | 803 |
| Replayed commits (timestamp gap > 0) | 45 |
| Damage events | 11 |
| Damage events among replayed commits | 11 |
| Damage among the 803 same-timestamp commits | 0 |
Eleven out of eleven sit inside the 45 replayed commits. Not one among the 803
with a zero gap. One in every four replayed commits came out corrupted.
The signal is one-way, and that is worth stating. If the committer timestamp is
later than the author timestamp, the commit was replayed; if they are equal, it
was not necessarily left alone, because the replay can land inside the same
second. That is exactly what happened in run 3, where the gap came out zero.
For this repository it is still a safe divider: a plain git commit always
yields a zero gap, and the May-era workflow committed, pushed, and rebased only
when the push was rejected, with no --amend anywhere.
The damage itself is the same every time: two calendar entries collapse into
one object. What varies is whether a syntax error appears alongside it. In five
of the eleven it did; in six it did not.
The alarm rang and the record went anyway
I was ready to end this piece with "loud damage is cheap, quiet damage is
expensive." The history would not allow it.
Take the quiet six first. In each of them the commit's entry count equals its
parent's: a run that added an entry to a 290-entry calendar produced another
290-entry calendar. What happens next is traceable too, because the next
production run parses the file and writes it back out. The calendar that fused
at 09:56 on 14 May came out of the 11:43 run with 291 entries and not a single
repeated key. The fusion launders itself: the parser lets the last key win, the
writer re-serialises the result, the file turns spotless, and the lost record
leaves no trace anywhere outside git history.
Now take the loud five, because the real lesson is there. In the 20 May
accident the place the parser complained about was not the fused entries at all:
}
],
"lastGeneratedAt": "2026-05-20T14:38:04.297Z"
"lastGeneratedAt": "2026-05-20T15:00:40.552Z"
}
The union merge had also duplicated the single-line timestamp at the top level
of the file, and with nobody around to put a comma between them the file blew
up. The 21 May accident broke in exactly the same way. So in those two events
what set off the alarm was a second artifact that landed next to the lost
record, rather than the loss itself.
The next morning I repaired that file. The entire change the repair commit made
to the calendar is this: delete the extra lastGeneratedAt line, add a trailing
newline. The parser went quiet, the pipeline moved, and I thought the job was
done. The fused entry was still sitting there; the repaired file still carries
four repeated keys. Ten minutes later the next production run laundered it along with
all the others.
The record that disappeared that day was called PostgreSQL WAL Bloat Nasıl. That title is not in today's calendar.
Yönetilir: 5 Adımda Etkin Temizlik
The count closed like this: ten of the eleven events cost a calendar record.
The only survivor is the one on 2 October, and the difference there is worth
noticing, because I did not do the repair. The next production run, which has
to read the queue, failed to open the file and put back the two lines git had
swallowed:
+ },
+ {
"topic": "ispmanager'i Uc Tur Test Ettim: ...",
A 509-record calendar became a 511-record one and both runs kept their entry.
The repair that worked was the one that undid what the merge had done, rather
than the one that quieted the parser's complaint.
What comes out of this is simple and still bothers me. The difference between
loud damage and quiet damage is not whether a record is lost, it is whether I
lose time as well. The alarm tells me the file is broken. It does not tell me
the data is short. And I go fix the line the alarm points at and call it
finished.
I missed the diagnosis twice
In May I put the root cause of those accidents into a commit message: the
-X theirs strategy in use during the rebase does a text-level merge and is not
JSON-aware. That diagnosis is not wrong. It is incomplete.
The lab splits this in two. With the driver on, -X theirs changes nothing, so
the damage that morning was the driver's. Without the driver, the same flag
quietly deletes the other side's record, so the flag is not innocent either. There
were two separate silent mechanisms in that one commit. I removed one of them,
left the other in place, and that line lived another 130 days. Writing an
incident's cause to the wrong address makes the second mechanism standing next
to it invisible.
The second miss belongs to the 22 September accident, and that one was properly
wrong. I wrote that
one up on this blog in
detail and
explained it like this: while the new entry was being added, the previous
object's closing was forgotten, and the }, and { that belonged in between
were missing.
The shape of the damage is right, the cause is not. Nobody forgot the closing;
git ate it.
To be honest, you cannot tell those apart by looking at the file. A writer that
appends an entry without its braces and a merge driver that fuses lines leave
the same artifact. The distinction comes from three things outside the file.
First, the timestamps. That commit's author timestamp is 21 September 12:02 and
its committer timestamp is 22 September 09:18. A 21-hour gap, which means the
commit was replayed.
Second, and to my mind this closes the argument: the neighbouring entry it
fused with did not exist yet when the commit was written. The commit that
added that neighbour was authored at 17:18 on 21 September, five hours later. A
writer cannot forget the brace of an object that is not there. Only something
running after both entries existed could have joined them, and that something
was the rebase on the morning of 22 September.
Third, the census: zero damage across 803 commits, eleven across 45 replayed
ones. I had already written down the cause of this one file's breakage twice and
missed it twice. The third time, the count deserved to be trusted ahead of the
story.
What I take from it
I removed the driver on 2 October at 14:55 and put a five-point test in its
place: does the calendar parse, is there a repeated key inside any object, are
the required fields present, are the categories in the known set, does the same
address appear in two entries. This morning I ran that test against three
calendars. On 2 October's broken file all five points failed. On 14 May's quiet
artifact only the second point failed, which makes it the one check that would
have caught the six May events. On today's calendar all five passed.
The test has two honest gaps, and leaving them out would betray the point of
this piece.
The first: it does not run in any workflow today. It sits in package.json as a
command and executes only when I or an agent calls it. The alarm proposed by an
article about damage with no alarm is not automatic yet.
The second is more uncomfortable. I also ran the test against the -X theirs
artifact, the spotless file that came out of the driverless run, and all five
points pass. Of course
they pass: that mechanism leaves no repeated key, it just removes a record. So
the test sees only one of the two silent mechanisms I found. A check that counts
records belongs next to the one that counts keys.
Since the driver came out, 13 commits have touched the calendar with no damage.
A five-day window; I would not offer it as proof, only as the absence of a
signal to the contrary.
Four rules are left. All four are boring, which is exactly why they were never
written down.
Do not let a line-level merge near structured data. JSON, YAML, lock files,
translation catalogues, migration lists: their meaning lives in the order of
the lines and in the braces. When you tell a line-oriented tool to sort it out,
the tool sorts out its own problem. The right fix is structural: fetch the file
again from the remote side and reapply your own change through the schema. The
cron side of this pipeline has done exactly that since 24 May. There is a
cheaper route too, and I should have taken it from the start: one file per
entry instead of a single array file, or a format that keeps one record per
line. If two runs write to different files there is nothing left to merge.
A conflict is a signal, not a fault. What disappears when you silence it is
the signal. And the trade is a bad one: you are swapping a visible, cheap
failure (a rejected push, a job that reruns) for an invisible, expensive one.
Repairing the line the alarm points at is not repairing the damage. On 20
May I fixed the spot the parser flagged, the file opened, and I considered it
finished. I had not asked the next question: what else did the thing that
produced this error do? Today I ask it with one line of code, by looking for
repeated keys inside the same object.
A diagnosis is an artifact too, and an incomplete root cause outlives the
incident. The justification I wrote on 13 May sat at the top of that file for
142 days and told every reader "this was thought through". My commit message on
24 May named one of two mechanisms and made the other one invisible. My 27
September post told the wrong cause for two weeks. All three were ground that
later decisions stood on.
One last thing. In this story the site never went down and no reader noticed
anything. The worst day was 24 May: when the calendar became unparseable the
pipeline stopped, no new post went out for seven and a half hours, and I only
noticed at 21:46 that evening. What was actually lost was
the production pipeline's memory: ten lines out of the ledger that records
which topic has been written and which one is waiting. Resilience in
infrastructure conversations usually means the service staying up. A system can
also stay up and start misremembering its own past, and that failure has no
alarm.
Top comments (0)