I build a lot of things I'll probably never ship. Small personal tools, prototypes, local utilities: a genealogy tracker, an expense tracker, a file indexer, the odd mini-game. Most of them start the same way: JavaScript or TypeScript, because nothing beats it for moving fast and staying flexible while an idea is still taking shape.
The one thing that kept getting in the way was storage.
So I built a single-file document database for Node.js: no server, no native bindings, a MongoDB-style API, and append-only writes.
Two options, both wrong for me
Running a real database server for a weekend prototype is overkill, and it gets worse the moment I want to move a tool between machines or share it with a copy-paste. So in practice I had two options.
The first was a single JSON file, loaded into memory and rewritten wholesale on every save. It works, until the dataset grows and rewriting the whole file on every write gets slow, and worse, unreliable. Kill the process mid-write (which happens constantly while debugging) and you can lose the file.
The second was SQLite. Rock solid, but it meant giving up the schemaless flexibility that makes JS/TS so good for fast iteration, exactly the thing I didn't want to trade away while a prototype is still finding its shape.
Neither fit, so I built the thing I actually wanted: a store that never reserializes the whole database on write, survives a process being killed mid-operation, and still feels like working with plain JS objects.
This got more relevant, not less, once AI-assisted coding entered the picture. Prototyping is faster than ever now, which means the storage layer needs to keep up and stay just as disposable and flexible as the rest of the stack.
I end up with my own solution offering: SQLite-like deployment, MongoDB-like data model, no native bindings.
More thinking than prototyping
I didn't cycle through failed prototypes to get here, most of the work happened before I wrote any code.
Earlier in my career I worked on the query engine of a production database: tuning page cache behavior, and optimizing how indexes - kd-trees, in that case, were read and rebuilt. That's where the intuition for this kind of tradeoff comes from: why a write-ahead log beats a full rewrite, what rebuilding an index actually costs, where "durable" starts eating into write throughput. The append-only design was the option that survived once I'd weighed the alternatives against that experience, rather than by trying and discarding them in code.
The rule that came out of it was simple: never reserialize the whole file. Every insert, update, or delete is a sequential append to the end of the file. In-memory state is a set of indexes; on open, it's rebuilt by replaying the log.
That single rule gets you two things for free: writes are fast (no rewriting, no B-tree rebalancing), and a batch of operations is either fully visible or fully discarded if the process dies halfway through it, which matters a lot when the process in question is a prototype you're actively debugging and killing every five minutes.
It also has an obvious cost: reads. If you never rewrite the file, every read has to seek to a document's byte offset and decode it. There's no shortcut around that except caching, which came later.
What the benchmarks actually said
I compared pocket-db against SQLite and a handful of in-memory stores (lowdb, LokiJS). Most of the field is in-memory, so on raw read throughput I lose, no surprise there, they're reading from RAM.
What did surprise me was the write side. Every insert, update, and delete beats every other file-backed store by a wide margin, and lands close to in-memory SQLite, which isn't even durable. That's the part of the story I didn't expect going in.
The part that still bothers me: reads land at roughly 3-4x slower than SQLite on disk. That's the actual gap I'm working to close, and it's the honest headline of where the project stands today, not the "it's fast" story, the "writes are fast, reads are still catching up" story.
The bug that made me name a principle
The most interesting moment in building this wasn't a design decision. It was a bug, and it happened while adding the hot-document cache with AI assistance.
From the start, every read in pocket-db returns an independent copy of a document. It wasn't a deliberate feature at first, just a natural consequence of always decoding from disk — there was no "original object" to hand out a reference to, so every read was necessarily a fresh copy. I kept that property once I noticed it, because it matches how the API is meant to be used: you insert and update documents one at a time, there's no global "save changes," and I wanted it to behave like talking to a real server-side database, not like mutating a shared in-memory blob.
When I added the LRU cache for hot documents, I asked an AI assistant to implement it. It did, but inconsistently. Some code paths returned a fresh clone of the cached object, others handed back the object actually sitting in the cache. The rest of the codebase always cloned, so this wasn't a visible design decision changing; it was a silent regression. Mutate a query result on one of those paths, and you'd corrupt data that was supposed to be internal.
The AI didn't get this wrong because the code was hard. It got it wrong because "every read returns an independent copy" had never been written down anywhere as a rule, it was implicit in how I'd built the rest of the store, and implicit is invisible to a model working one function at a time. I had to name it - "safe reads" - turn it into an explicit guarantee, and audit every code path against it before I trusted the cache.
That's the part worth sitting with: this is exactly the kind of bug a developer without a real grasp of the storage model would ship without noticing. Nothing about "returns a value" versus "returns a shared reference" fails a naive test unless you specifically write one to catch it. Working with AI on a systems-level project doesn't remove the need to understand your own architecture. If anything it raises the bar, because the assistant will happily be locally correct and globally wrong.
Where it's actually running
Pocket-db started four months ago as a quick experiment. I didn't set out to build "a database". I prototyped it fast, and it became clear fast that it solved a real, recurring problem, so it earned a spot as a flagship project on my GitHub instead of staying a throwaway script.
It's now running in about six of my own projects: small personal sites, utility tools, and an upcoming project currently called Devlab internally (it'll need a new name, "Devlab" is already heavily used elsewhere). For a small tool or an early-stage app, it removes a whole category of setup friction. The MongoDB-style API also makes the programming model familiar if the application eventually moves to a server-side document database.
The best stress test so far wasn't synthetic. One of my personal tools - a genealogy tracker - has grown its pocket-db file past a gigabyte of real data. At that scale, the read gap I mentioned earlier isn't a benchmark number anymore, it's something I feel while using my own tool. It's the clearest evidence I have that closing that gap is the right thing to work on next, and a real-world dataset I can keep testing against as I do.
What's next
Indexing and caching for read performance are the priority. Closing the gap with on-disk SQLite without giving up the single-file, append-only model that makes the durability and write story work in the first place. The hot-document cache shipped recently is a first step (roughly 2.9x faster for repeated single-document reads, 1.8x on full scans, opt-in and zero-cost when disabled). Closing the rest of that gap without breaking the append-only guarantees is the hard part, and where most of my attention is going now.
I haven't had real feedback on any of this yet, this is genuinely the point of publishing it. If you've hit the SQLite-native-bindings problem in an Electron app, or you've outgrown a big-JSON-file setup, or you just have opinions on the storage model, I'd like to hear them.
- Would you use something like this instead of SQLite in an Electron app?
- Is the append-only storage model a good tradeoff for your workloads?
- What would make you trust a 0.x database enough to use it in a real project?
Check it out:
- Repo: github.com/AxFab/pocket-db
- Docs & benchmarks: pocket-db.axfab.net
- npm:
@axfab/pocket-db
Top comments (0)