On May 13, 2026, seven images silently disappeared from TypeVelocity. Not with an error. Not with a warning. They just stopped being deployed — and the two blog posts referencing them served broken image slots to anyone who visited for 17 days straight.
I found out today, May 30. Not from monitoring. Not from an alert. Someone pointed it out while I was working on something else.
The deploy pipeline
TypeVelocity doesn't use git push to deploy. It uses a custom Node.js script that:
- Minifies CSS and JS
- Copies files to a
results/folder - Pushes only the files in a hardcoded
FILESarray to GitHub via the API - Vercel picks it up from there
The FILES list looks like this:
const FILES = [
{ local: 'results/index.html', remote: 'index.html' },
{ local: 'styles.min.css', remote: 'styles.min.css' },
{ local: 'script.min.js', remote: 'script.min.js' },
{ local: 'blogs/assets/some-image.png', remote: 'blogs/assets/some-image.png' },
// ... 50+ more entries
];
Every file that goes live has to be in this list. It's maintained by hand. When you add a blog post or an image, you add it to the array. If you forget, the file doesn't deploy. The script doesn't warn you. It just skips it.
What happened
On May 13, I deployed the Daily Challenge update — a big feature, lots of new files. Somewhere in that deploy, seven image entries got dropped from the FILES list.
The images had been on the site since March:
- 4 Lighthouse before/after screenshots
- 3 UI comparison images
All seven were present in the April 18 deploy. Gone by May 13. The script ran fine. No errors. The images it skipped just quietly stopped being included.
The blogs referencing them — the Lighthouse 100 post and the lives and ranks update — kept loading fine. Text, styles, everything. Just empty <img> boxes where the screenshots should have been.
Nobody reported it. I didn't catch it in review. It was silently broken for 17 days.
Finding the exact commit
I needed to know when it broke. Git history has the answer.
# Check each commit between Mar 20 and May 26
for sha in 4aee7f5b c3da2958 4361a4f6 d4934aa1 9f0bef3a 027ed418 e594fa8d dc6a6e96
do
has=$(gh api repos/mademethedev0/TypeVelocity/git/trees/${sha}?recursive=1 \
| grep -c "typevelocity-100-on-lighthouse")
echo "$sha: images=$has"
done
Output:
4aee7f5b: images=1 # Apr 18 — present
c3da2958: images=1 # Apr 18 — present
4361a4f6: images=0 # May 13 — GONE
d4934aa1: images=0 # May 17 — still gone
...
The images dropped in the May 13 commit. That's the Daily Challenge deploy.
Recovering from git blobs
The images were gone locally too — deleted at some point after the deploy dropped them. But git doesn't delete blobs when you push new commits. They're still there, just not referenced by any current tree.
Here's the recovery script:
const { spawn } = require('child_process');
const fs = require('fs');
const path = require('path');
const REPO = 'mademethedev0/TypeVelocity';
const SHA = '4aee7f5bed1ae9446195ff3b6843e571c6c7a36b'; // Apr 18 commit
const OUT = path.join(__dirname, 'blogs/assets');
const IMGS = [
'typevelocity-100-on-lighthouse-desktop.png',
'typevelocity-100-on-lighthouse-mobile.png',
'typevelocity-before-any-optimization-on-lighthouse-desktop.png',
'typevelocity-before-any-optimization-on-lighthouse-mobile.png',
'typevelocity-update-new-ui-1.png',
'typevelocity-update-new-ui-2.png',
'typevelocity-update-new-ui-3.png',
];
function ghApi(endpoint) {
return new Promise((resolve, reject) => {
const proc = spawn('gh', ['api', `repos/${REPO}/${endpoint}`]);
let out = '';
proc.stdout.on('data', d => out += d);
proc.on('close', code => {
if (code !== 0) reject(new Error(`gh exit ${code}`));
else { try { resolve(JSON.parse(out)); } catch { resolve(out); } }
});
});
}
async function main() {
// Get full tree at that commit
const tree = await ghApi(`git/trees/${SHA}?recursive=1`);
const map = {};
for (const node of tree.tree) {
if (node.path.startsWith('blogs/assets/')) {
map[path.basename(node.path)] = node.sha;
}
}
for (const img of IMGS) {
const blobSha = map[img];
if (!blobSha) { console.log(`NOT FOUND: ${img}`); continue; }
console.log(`Fetching ${img} (${blobSha})...`);
const blob = await ghApi(`git/blobs/${blobSha}`);
// Blob content is base64-encoded
const buf = Buffer.from(blob.content.replace(/\n/g, ''), 'base64');
fs.writeFileSync(path.join(OUT, img), buf);
console.log(` -> ${buf.length} bytes`);
}
console.log('Done.');
}
main().catch(e => { console.error(e); process.exit(1); });
Run it:
$ node recover-images.js
Fetching typevelocity-100-on-lighthouse-desktop.png (a91441b2)...
-> 105956 bytes
Fetching typevelocity-100-on-lighthouse-mobile.png (724f2e40)...
-> 97329 bytes
...
Done.
All seven images recovered in under 2 minutes. Redeployed. Live again.
What I'm doing about it
The real fix is automating the FILES list — scan results/ and blogs/assets/ at deploy time instead of maintaining it by hand. That's an afternoon of work I haven't done yet.
The short-term fix: I added a pre-deploy check that logs a warning for any file referenced in a blog post's src= attributes that isn't in the FILES list. Not automated, but it makes the gap visible instead of silent.
// Pre-deploy check (simplified)
const blogFiles = fs.readdirSync('blogs').filter(f => f.endsWith('.html'));
const missing = [];
for (const file of blogFiles) {
const content = fs.readFileSync(`blogs/${file}`, 'utf8');
const srcs = content.match(/src="([^"]+)"/g) || [];
for (const src of srcs) {
const path = src.match(/src="([^"]+)"/)[1];
if (path.startsWith('assets/')) {
const fullPath = `blogs/${path}`;
const inList = FILES.some(f => f.remote === fullPath);
if (!inList) missing.push(fullPath);
}
}
}
if (missing.length) {
console.warn('Files referenced but not in deploy list:');
missing.forEach(f => console.warn(` - ${f}`));
}
The thing that actually bothers me isn't that it broke. Things break. It's that it broke quietly and stayed broken. A missing image doesn't throw an error anywhere in the pipeline. The build succeeds. The deploy succeeds. Everything looks fine until someone actually reads the blog.
That's the gap worth closing — not "prevent all breakage" but "make breakage loud."
What I should have done (and what it would cost)
Here's the prevention work I didn't do, with rough time estimates:
| Fix | Prevents | Effort |
|---|---|---|
Auto-scan results/ and blogs/assets/ at deploy time |
Files missing from deploy list | 2-3 hours |
Validate all src= and href= references in HTML |
Broken image/link references | 1 hour |
| Add a post-deploy smoke test that checks image load status | Silent 404s on live site | 1-2 hours |
Log warnings for files in results/ not in FILES array |
Deploy list drift | 30 minutes |
| Run Lighthouse CI on every deploy | Performance/accessibility regressions | 1 hour setup |
Total: 6-8 hours of prevention work to avoid 17 days of broken images and 2 hours of recovery scrambling.
The auto-scan is the big one. If the deploy script just walked results/ and blogs/assets/ and pushed everything it found, this entire incident wouldn't have happened. The manual FILES array exists because I wanted explicit control over what deploys — but that control isn't worth the maintenance burden.
The validation check is the quick win. 30 lines of code, runs in under a second, catches broken references before they go live.
The lesson
Solo projects break in ways that only get caught when someone's actually looking. I wasn't looking closely enough at those pages after the initial publish.
If you're maintaining a custom deploy pipeline:
- Automate the file list — don't maintain it by hand
-
Validate references — check that every
src=andhref=points to something that exists - Make failures loud — silent skips are worse than errors
And if you ever accidentally nuke something from production: git history doesn't lie. The blobs are still there. You just have to go get them.
TypeVelocity is a typing test with lives, ranks, a rhythm game mode, and way too many features for something that started as a weekend project. It's live at typevelocity-nu.vercel.app. The images are back. The blogs work again. Go type something.
Top comments (0)