There is a sentence at the top of my cover generator that I wrote myself. It
sits there without Turkish characters, the way the code has it: "Existing PNG
idempotent overwrite — same input, same output (sharp is deterministic)." That
line has been there since 7 May 2026, the generator's first commit. I have
trusted it for six months. Tonight I tested it for the first time.
I re-rendered the covers of all 1,386 Turkish articles in the archive with
today's generator and compared before and after, byte for byte. 1,190 of them
didn't match. So I cannot reproduce 86 percent of my site's covers today by
running the same code against the same titles. Chasing it turned up two causes.
One was a decision I made deliberately five months ago; the other I had never
noticed until tonight. Thirteen covers fit neither, and I leave them unsolved
below.
Images I never look at
Let me confess something first: I don't see these covers when I visit my own
site. Post cards don't render the cover — there's even an explicit note inside
the component saying so. The cover image only goes into the og:image and
twitter:image tags, into feed.json and the RSS output, and into the
LinkedIn and Bluesky sharing endpoints.
So these 2,772 PNG files (Turkish and English, two per article) are my blog's
face — but only in the places I never look. The little box that appears when
someone shares my link on WhatsApp, the preview in Slack, the card on X. I have
never wondered how my own house looks from the street.
That's why generating covers has always been a formality for me. When I write a
new article I run two commands, glance at the resulting PNG once — did the title
overflow, are the Turkish letters right — and commit it. That single glance is
enough to validate one cover; it is not enough to validate that 1,386 covers
resemble each other, because I never put two of them side by side.
The measurement: hash, re-render, hash
The method is boring on purpose. I took the SHA-256 digest of every cover, ran
the generator with --all to re-render them all, took the digests again, and
compared the two lists. If the generator is deterministic the two lists must be
identical.
My first reaction was "so sharp isn't deterministic after all." Wrong reaction.
When I render the same cover twice in a row the bytes are identical; determinism
holds. What broke is the stability of the inputs.
Drawing that distinction mattered, because the measurement itself can mislead
you. The cover generator touches the articles' frontmatter while it runs — it
normalises the coverImageAlt field — so an --all run rewrites not just 2,772
PNGs but over a thousand .mdx files too. I reverted all of them with git
afterwards. Had I run this experiment in a dirty working tree, I could not have
told my measurement apart from my own leftovers.
Sorting the changed files by publication date revealed a clean break:
| Publication month | Not reproducible | Byte-identical |
|---|---|---|
| 2024-05 … 2026-03 (four months) | 32 | 0 |
| 2026-04 | 368 | 0 |
| 2026-05 | 274 | 0 |
| 2026-06 | 228 | 3 |
| 2026-07 | 199 | 1 |
| 2026-08 | 78 | 19 |
| 2026-09 | 11 | 85 |
| 2026-10 | 0 | 86 |
| 2026-11 | 0 | 1 |
| Total | 1,190 | 195 |
The rows add up to 1,385; the 1,386th article's cover is not on disk, and I'll
come back to it. All 86 articles I published in October match. Not one of the
368 I published in April does. The break is at the end of August.
First cause: two different machines print my covers
Git does not forget who added what — but it depends which question you ask. My
first reflex was --diff-filter=A, which gives each cover's first adder:
833 for the bot, 553 for me. Then it struck me that the bytes sitting in the
repo today may not have been written by that commit. Ask for the last writer and
the numbers shift:
first adder (--diff-filter=A): BOT 833 | ME 553
last writer (--diff-filter=AM): BOT 712 | ME 674
The 121 covers in between were added by the bot and later overwritten by me. The
second question is the right one, because what I'm trying to reproduce is
today's bytes.
Crossing the last writer with the design era and the outcome clears things up:
| Last writer | Design | Outcome | Count |
|---|---|---|---|
| Me | dark (current) | byte-identical | 195 |
| Me | dark (current) | doesn't match | 13 |
| Me | coloured (retired) | doesn't match | 466 |
| Bot | dark (current) | doesn't match | 399 |
| Bot | coloured (retired) | doesn't match | 312 |
| Bot | coloured (retired) | byte-identical | 1 |
Of the 712 covers the bot wrote last, 711 cannot be reproduced on my
machine. The one that does match is interesting in its own right: no article
points at it, so the generator never opens it. More on that shortly.
Three quarters of my font stack is decoration
Here are the relevant lines from my generator:
// librsvg uses fontconfig; a font family that supports Turkish
// characters (ş, ğ, ı, ç, ü, ö) cross-platform:
// 1) Inter — Linux (apt: fonts-inter)
// 2) Noto Sans — Linux fallback (fonts-noto-core)
// 3) Helvetica — macOS native, supports Turkish
// 4) DejaVu Sans — Linux ultimate fallback
const FONT_FAMILY = "Inter, 'Noto Sans', Helvetica, 'DejaVu Sans', sans-serif";
The intent is clear: Inter on Linux, Helvetica on macOS. That is also what CSS
promises. CSS Fonts Module Level 4 calls
font-family "a prioritized list" and says the user agent "iterates through the
list of family names until it matches an available font that contains a glyph
for the character to be rendered." It even states the reason: because available
fonts differ across platforms.
I checked whether the list really is walked in order. Same text, same SVG
template, changing only the font-family value:
Verdana (reference) 4dace8fe4ffe
Helvetica (reference) c1ac04a63934
ZzQq1, ZzQq2, Verdana 4dace8fe4ffe <- Verdana won
Inter, 'Noto Sans', Verdana 4dace8fe4ffe <- Verdana won
THE REAL STACK c1ac04a63934 <- Helvetica won
The list is walked in order, exactly as the spec says: it skips the two made-up
names in front and settles on Verdana. In my stack Inter and Noto Sans are
skipped, and the winner is Helvetica, in third place.
Apple's macOS font list supports this:
Helvetica sits under "Installed or downloadable fonts," while plain Noto Sans
is absent from the list entirely — only script-specific siblings like Noto Sans
Kannada and Noto Sans Myanmar are there. DejaVu Sans isn't on this machine
either. Inter is here, but in a place that can't help: inside the font folder of
the Java runtime bundled with IntelliJ IDEA and Android Studio. It isn't in a
system font directory, so the rendering engine never sees it.
Let me be precise about what this covers. On this Mac only one of the four
names resolves. What happens on the CI machine I did not measure; all I have
is its output. That machine is my own infrastructure and fc-list is a single
command — not looking was a choice, made because I wanted to close the
measurement notebook tonight.
How different are the letters?
I started with a single cover and nearly wrote down bad evidence: the title line
on the bot's cover is 763 pixels, mine 638. That is n=1, and worse, it compares
an inked width against an advance sum computed from the font file — two
different quantities, with side bearings and letter-spacing in between. If
they match, it's a coincidence.
So I did a paired measurement instead. I picked 60 bot-rendered covers at random
from those carrying the current design, measured the inked width of the widest
line in each title block, then re-rendered the same cover here and measured
again:
60 covers, paired ratio (bot / mine):
48 covers -> median 1.187 (p25 1.165 p75 1.190)
12 covers -> ~1.00
In 48 covers the bot's line is about 19 percent wider than mine, the spread is
narrow, and a gap that size can only come from the letters themselves. In the
remaining 12 the ratio sits at one. I opened one of those and looked: the
letters are the same width, yet the file digest differs. So "the bot's covers
are in a different typeface" is true for the majority, not for all — and I
could not isolate why that minority fails.
So why did nobody complain?
Pango does not actually stay silent when a font can't be loaded. First I had to
establish which engine was even talking. My comment says "librsvg uses
fontconfig"; running the same renders with the PANGOCAIRO_BACKEND variable
settled it:
default = PANGOCAIRO_BACKEND=coretext (hashes IDENTICAL)
PANGOCAIRO_BACKEND=fc (every hash CHANGED)
What's in play is not fontconfig but CoreText. My own comment misdescribes
my own code. I once wrote an article about
my recipes misdescribing my code;
while writing it I had no idea the same mistake was sitting under this line. The
irony is that the fc backend recognises more of the names in my stack — the
engine my code wrongly claims would have behaved better than the one actually
running.
I went to the source
where the warning is emitted (sharp ships version 1.57.0; the function I quote
is unchanged on main). pango_core_text_font_map_load_fontset first tries to
load the font set with the requested description. Failing that, it swaps the
family name for Sans and tries again with the same style. Failing that too, it
resets variant, weight and width to normal — printing the warning before that
third attempt — and tries one last time. The warning's flag is a static
boolean, so it fires once per process.
What the diagram says is this: the warning only fires when the list fails end
to end. My list resolves to something on both machines — to Helvetica on one,
to something else on the other. Both count as success. That is why there isn't a
single line of warning in my terminal or in the CI log. The silence isn't a
malfunction; it is the design.
To see the warning fire, I triggered it on purpose. I picked Impact, whose file
is on my disk and which is on Apple's list:
Impact (no stretch given) fd8b295f25f5 <- fell to the default
Impact font-stretch="condensed" 341c125b00c1 <- actually Impact
Impact font-stretch="semi-condensed" -> Pango-WARNING: couldn't load font
"Impact Semi-Condensed Not-Rotated 58", modified variant/weight/stretch
as fallback, expect ugly output.
The reason is in the font file: Impact's usWidthClass is 3, that is,
"condensed." Pango's CoreText side labels that face PANGO_STRETCH_CONDENSED
and produces no other width step. The default request is normal width, so the
match falls through; the moment I write condensed the font arrives; when I ask
for semi-condensed nothing is left and the warning appears. A font on my disk,
listed by Apple, does not arrive when called by name — because of one metric.
Two more things that fell out while measuring
First: the generate-hero.ts --all command announces "Total posts: 2772,"
meaning it counts English articles as separate posts too. For every .en.mdx
file it renders a cover named <slug>.en.png — 1,386 files nothing points at.
The real name for English covers is -en.png, with a hyphen, not a dot. Not one
file with a .en.png extension exists anywhere in the repository's history, so
that flag has never turned into a commit; had it done so, those files would have
settled in quietly.
Second, that 1,386th cover I kept postponing. Exactly one coverImage has no
file on disk, and the symmetry bothered me: exactly one PNG is pointed at by no
article. One missing, one spare.
The missing one's story is the part that stung. An article I published on 9 June
2026 had a cover; my own cleanup commit deleted it on 9 July. The message
reads: "fix(content-gen): close the orphan fallback cover PNG leak + clean up
173 existing orphans." The command that swept up orphan covers took a
referenced cover along with them. Its English twin came back the next day; the
Turkish one did not. I verified the result on the live site:
page: 200
og:image = .../bootstrap-deadlock-cluster-domain-controller.png
image HTTP: 404
For four months, anyone who shared that article's link saw a blank space where
the picture should be. The page is perfectly healthy; the only broken part is
the side I never look at.
Second cause: a decision I made five months ago
Not all 1,190 failing covers are about typefaces. 778 of them are visibly
different at a glance: taking a 40×40 pixel square from the bottom-left corner
of each cover and reading its average colour gives 779 covers in the colour
scheme I retired and 607 in today's dark one. (Shift the sampling window and the
number moves to 777/609; the boundary is sharp, but where the square sits
matters for a handful of covers.)
The commit message from 9 June 2026 reads: "feat(covers): new dark + neon-blue
cover design (all new posts)." Those three words in parentheses, "all new
posts," were a deliberate decision that day. Re-rendering the old ones would
take a while, the covers weren't visible on the site anyway, so I left them.
Five months later the majority of my archive still carries that retired design.
Every shared link to my blog shows one of two different brands, depending on
when it was written. I can't say "I didn't notice"; it was my decision. What I
didn't notice is how long that decision would live. What I made that day was
really a deferral, and deferrals have a bad habit: they don't make it onto a
calendar.
The two causes also conceal each other. Because the old covers look different to
the eye, the answer to "why don't they match" seemed ready-made. Had I looked at
them and said "right, design change," I would never have opened the 399 bot
covers carrying the same design as today's.
The thirteen left open
And then the row in the table that occupies me most: 13 covers carrying
today's design, last written by me, that still don't match. Same machine, same
design, same stack. Neither of my causes explains them.
There is an alternative I should put on the table honestly: the renderer
version. package.json pins sharp at 0.35.5 today; what is installed on this
Mac is 0.34.5. The pinning history isn't flat either — until 11 August 2026 it
was a range like ^0.34.5, then 0.35.3 that day, then 0.35.5 on 28 September.
So the machines may differ not only in fonts but in drawing libraries.
That does not explain the 19 percent width gap in the 60-cover measurement; that
gap comes from the advance width of letters, and a library upgrade does not move
it. But it could very well explain those 13 covers, and the 12 out of 60 whose
ratio sits at one. I didn't measure it. A number with "I don't know" behind it
beats a number with the wrong reason behind it.
Three minutes' worth of checks for your own setup
Is your stack picking anything other than the default? Render the same text
once more with a font name that doesn't exist. If the outputs match, your stack
lands on the engine's default — which is not the same as "it does nothing,"
because the default may itself be a name from your list. On this Mac Times
behaves exactly like that: installed on disk, on Apple's list, yet its output is
identical to a made-up name's. To sharpen the diagnosis, add a name that does
diverge, like Verdana, as a positive control.
How many machines wrote the bytes in your repo? --diff-filter=A gives you
the first adder, which isn't what you want. Use git log --diff-filter=AM for the last writer and count the commit authors. For me the two
--name-only
questions differed by 121 covers.
Does the archive match what today's generator produces? Hash, re-render,
hash, compare. Those three steps took ten minutes and demolished a six-month-old
assumption. Just make sure your working tree is clean first.
What I'm going to do
Vendor the font into the repo. That was my first reflex: put a .ttf file
in the repository and load it explicitly when rendering. sharp's text API has a
fontfile parameter, and I tried it — on this setup it has no effect at all.
The output is byte-identical with and without the file, and "Fontconfig error:
Cannot load default config file" lands on top. The parameter writes into
fontconfig's application font set, while the engine in play is CoreText. So the
cost isn't "a few hundred kilobytes": I would have to pin the backend and a
fontconfig configuration as well, or abandon the SVG template for a different
drawing path. I learned the price just now.
Render covers on one machine only. I was about to dismiss this with "the
pipeline would publish without covers while the Mac is off," then I looked at
the workflow: runs-on: [self-hosted, itwise-mac]. The pipeline is already tied
to this machine. My objection was void.
Put in a regression gate. This is what was really missing. The whole article
says "no warning, no test told me"; so I should write the thing that tells me. A
CI job that re-renders twenty random covers and compares digests is enough — and
tonight's measurement is its prototype.
I'll do all three, but not with this article. All I did today was measure;
re-rendering the whole archive would change 2,772 files in a single commit. I'm
writing the open items here so I can't duck them: a regression gate,
measuring how the font can actually be pinned, renewing the covers still
on the retired design, and re-rendering that one deleted cover.
Meanwhile the --all flag looks innocent, but in this repo it is a command that
silently rewrites 2,772 files. I reverted everything while measuring; even so, I
won't run it again without checking git status first.
What I take from it
Something similar happened when I
sat down to fix my alt text:
the gap between what the code promises and what the page produces can stay open
a long time, because nobody looks. There the missing thing was text; here it is
the typeface.
But the real lesson of tonight is different. I was awarding the adjective
"deterministic" to my code; that adjective doesn't belong to code. The same code,
with unpinned inputs, is deterministic only on the same machine — and "the same
machine" covers the installed fonts, the drawing library's version, and which
backend happens to be in play. My cover generator is deterministic; on two
separate machines, with two separate results.
Calling something reproducible requires having run it twice. And not both times
in the same place.
Top comments (0)