I used JSON files instead of a database for 1,699 records. Here's what actually broke.

Three years ago I'd have called this a bad idea. I still might. But I've been running ryzenstudy.com on flat JSON files for months now — 1,699 exam paper records, 699 distinct subject codes, five academic programmes, zero database servers — and the failures haven't been the ones anyone warned me about.

Nobody warned me about slugs.

The setup

It's a Next.js 15 app on App Router, one VPS, PM2 in fork mode behind Nginx. The entire data layer is this:

export function getAllPapers() {
  const data = fs.readFileSync(DB_FILE, 'utf8');
  return JSON.parse(data || '[]');
}

That's it. That's the ORM. Every query is Array.prototype.filter. Every write is writeFileSync. There is no connection pool because there is no connection.

Why? The data is append-mostly and read-heavy — students download old exam papers, nobody edits them. The whole file is under a megabyte. Postgres would have meant another process to keep alive, another backup story, another thing to be woken up by. fs.readFileSync has never once paged me.

For about eight months it just worked. Then I went looking for something else entirely and found three bugs stacked on top of each other, all of them a direct consequence of not having a database.

Bug 1: 95 papers that existed but had no URL

Slugs are generated from the record:

export function generateSlug(paper) {
  const name = slugify(paper.subjectName);
  const yr = paper.year.replace('/', '-');
  return `aktu-${paper.course}-${paper.semester}-${name}-${yr}`;
}

Course, semester, subject name, year. Looks unique. It isn't.

My university issues the same subject under different codes depending on which branch you're in. "Deep Learning" in seventh semester is KCS078 for computer science, KDS078 for the data science branch, and KOT076 if you take it as an open elective. Three separate exam papers. Three separate PDFs. One slug.

I nearly deleted them. Sitting there in a duplicate report, three rows with the same course, same semester, same year, same subject name — every instinct said dedupe. I checked the files first, mostly out of paranoia:

972a76fc5bb8  148916  KCS078_DEEP_LEARNING.pdf
38febefb87c7  143581  KOT076_DEEP_LEARNING.pdf
ad5608859a7e  145162  KDS078_DEEP_LEARNING.pdf

Different hashes. Different sizes. Three real papers, and I'd been about to throw two of them away because my slug function was too narrow.

A database would have caught this the first week. Not because databases are magic, but because I'd have been forced to declare a unique constraint, and the insert would have blown up in my face while I still remembered what the schema meant. With JSON there's no constraint to violate. The second record just sits there, perfectly valid, permanently unreachable.

95 papers were in that state. Uploaded, stored, served by nothing.

The fix is collision-aware slug generation — build an index of which record "owns" each base slug, and give everyone else a code-qualified URL:

export function generateSlug(paper) {
  const base = baseSlug(paper);
  const primary = slugPrimaryIndex().get(base);

  if (!primary || primary.id === paper.id) return base;

  const code = slugify(paper.subjectCode);
  if (!code || code === primary.code) return base;

  return `aktu-${paper.course}-${paper.semester}-${code}-${slugify(paper.subjectName)}-${yr}`;
}

The code === primary.code check is the part I got wrong on the first deploy, and it's the more interesting half of the story. Keep reading.

The bug I introduced while fixing the bug

First version didn't have that check. Ship it, run a sweep across every URL in the sitemap, and one comes back 308 instead of 200.

One. Out of 1,615.

Turns out my 147 colliding records were two completely different populations wearing the same costume:

• 95 records had a genuinely different subject code. Real distinct papers. Code-qualified slug works.
• 52 records had the identical code as the record they collided with. Duplicate uploads, and a batch of placeholder rows whose IDs literally end in -dummy.

For that second group the code suffix disambiguates nothing. Worse — my slug matcher accepts a legacy code-first URL format for backwards compatibility, so the original record also answers to the new URL, wins the lookup, and the page 301s you back to where you started. My sitemap had started advertising redirects.

If I'd trusted the deploy and skipped the sweep, that ships silently. The section renders. The pages load. Google just quietly gets 52 URLs that bounce.

I now believe a full-sweep is non-optional after any URL-shaped change, and I don't think that belief is specific to flat files.

Bug 2: a homepage section that had been empty for weeks

Different day, same root disease.

The analytics file went through a schema migration a while back. Top-level counters moved down into a legacy object, and a new daily bucket started collecting per-day stats. The admin dashboard got updated. The homepage didn't.

// homepage — reads a key that no longer exists
const downloads = analytics.downloadsBySubject || {};

getAnalytics() returns { legacy, daily, visitedIPs }. There is no top-level downloadsBySubject. So downloads is {}, forever. Every download count resolves to zero, the trending list filters to an empty array, and the component does what any sensible component does with an empty array:

{trending.length > 0 && ( ... )}

It renders nothing. No error. No warning. No log line. The section just stopped existing on the page and I didn't notice, because a missing section looks exactly like a section you forgot you built.

The data was fine the whole time — 1,355 records in the legacy bucket, another 1,003 download events across 31 days in daily. Tracking never broke. Only the read path did.

|| {} is a lovely little footgun. It converts "this key is gone" into "this key is empty," and those two things want very different reactions from you.

And while I was in there: the lookup summed wrong too.

// picks the FIRST matching key and ignores the rest
Object.keys(downloads).find(k => k.startsWith(paper.id))

One paper ID appears under three different key formats depending on which era of the code wrote it — bare ID, id — name, and id — name (code). .find() grabs one and shrugs at the others. My actual top paper had 114 downloads. The old logic reported 55 for a completely different paper.

Schema drift in a typed database gives you a migration error. Schema drift in JSON gives you undefined, and undefined gives you a plausible-looking wrong answer.

Bug 3: the one I saw coming

This one I did anticipate, which is why it's the shortest section.

Two requests writing the file at once will corrupt it. So writes go through a promise chain:

let writeQueue = Promise.resolve();

function enqueue(fn) {
  writeQueue = writeQueue.then(fn).catch(() => {});
  return writeQueue;
}

And the write itself is temp-file-plus-rename, so a crash mid-write leaves the old file intact rather than half a JSON document:

const tmp = ANALYTICS_FILE + '.tmp';
fs.writeFileSync(tmp, JSON.stringify(data, null, 2), 'utf8');
fs.renameSync(tmp, ANALYTICS_FILE);

Plus a daily snapshot to .bak, and a parse-failure path that restores from it. That last part matters more than it sounds: the original code returned empty defaults when the parse failed, which meant one corrupt read would get written back as an empty file and eat the entire history. The recovery path was the bug.

This works because it's one process. Scale to two PM2 instances and the queue is worthless — they don't share it. That's the actual ceiling on this architecture, and it's a hard one. Not row counts. Process counts.

The performance thing I walked into

Making slug generation collision-aware means it needs to know about every other record. getAllPapers() reads and parses the file on every call. The sitemap calls generateSlug() 1,699 times.

You can do that multiplication yourself.

let _slugIndex = null;
let _slugIndexKey = '';

function slugPrimaryIndex() {
  const st = fs.statSync(DB_FILE);
  const key = `${st.mtimeMs}:${st.size}`;
  if (_slugIndex && _slugIndexKey === key) return _slugIndex;
  // ...rebuild
}

Keyed on mtime and size rather than a TTL, so an upload through the admin panel invalidates it instantly and a script editing the file directly does too. statSync is cheap next to read-plus-parse.

This is a query planner. I wrote a bad query planner. That's what "no database" actually costs — not the storage, the accumulated re-implementation of things Postgres finished solving in 1998.

So was it the wrong call?

Honestly, no. Not for this.

The failures above cost me a day to find and fix. A Postgres instance would have cost me a day to set up and then a low background tax forever — migrations, backups, a connection dying at 3am. Read-heavy, append-mostly, sub-megabyte, one writer: flat files are a defensible answer, and I'd pick them again.

But I'd change two things from day one.

Write the uniqueness check even when nothing enforces it. Not a constraint — just a script that greps for records colliding on whatever your URL key is, and run it after every bulk upload. My 95 unreachable papers were detectable in about twelve lines of JavaScript from the very first day they existed.

Never let a read path fail into a plausible default. || {} and || [] hid two bugs from me. If a key you require is missing, that's not empty — that's broken, and it should say so out loud in dev.

The site's live if you want to see what 1,615 JSON records look like served as pages — Ryzenstudy is free, no login, built for students at my own university.

If you're running something similar at a bigger scale, I'd genuinely like to know where your ceiling turned out to be. Mine looks like it's process count. I'm curious whether that's everyone's.

Comments

[Tarun Sharma]

The process count ceiling is real and it's the same wall SQLite hits, one writer at a time is fine until you scale horizontally instead of vertically. If you ever do need a second instance, the cheapest fix isn't necessarily Postgres, it's moving the write queue into something both processes can see, a single Redis instance with a lock would get you multi process without a full relational migration. The slug collision bug is the one that'd worry me most long term though, since it's silent by nature, worth keeping that 12 line collision script running as a cron rather than a one time check.