If you've ever built a CLI tool, an Electron app, or a local-first prototype in Node.js, you've faced the same question: where do I store structured data without spinning up a server?
The usual suspects are well known — lowdb, LokiJS, nedb, better-sqlite3. I recently built pocket-db, a new embedded document store for Node.js.
I wanted a database solution that offered me:
- a single-file database
- no native compilation
- Mongo-like API
- easy embedding in CLI or Electron apps
I wanted to know honestly where it stands. So I ran a benchmark across 10 common operations on a collection of 1,000 documents.
Here's what I found — not shying away from the unflattering parts.
The contenders
- pocket-db — append-only single-file document store, MongoDB-like API, indexes in memory
- sqlite (in-memory) — better-sqlite3 with an in-memory database, the fastest SQLite configuration
- sqlite (file) — better-sqlite3 persisted to disk, the most common SQLite setup
- json-file — full JSON file read/write on every operation
- lowdb — JSON-based, keeps everything in memory and flushes to disk on write
- lokijs — in-memory document store, optional persistence
All tests run on Apple M-series hardware. Results are in ops/sec — so higher is better.
The results
Operation pocket-db sqlite (mem) sqlite (file) json-file lowdb lokijs
─────────────────────────────────────────────────────────────────────────────────────────────────
insertOne 198,177 226,278 * 4,421 3,893 2,256 1,004
insertMany (100) 2,345 2,963 * 402 788 628 264
findById 142,776 1,064,774 248,942 12,532,585 * 321,548 4,061,606
findAll 96 361 361 115,774 870,822 * 1,179
findByName (scan) 97 536 524 19,579 24,235 * 8,222
findByRole (index) 277 884 887 18,527 21,796 * 3,215
updateOne 97,889 423,072 * 2,675 784 566 188
deleteOne 454,402 465,026 * 5,551 924 663 220
countAll 12,038 2,233,389 285,285 42,553,191 * 34,914,251 37,348,273
sortByScore (desc) 91 293 293 4,947 5,099 * 1,123
─────────────────────────────────────────────────────────────────────────────────────────────────
All values in ops/sec. * = fastest for this operation.
Before you ask: json-file wins read benchmarks because the entire dataset is already loaded in memory during the benchmark.
Why pocket-db writes so fast
The short answer: every write is a single sequential append to the end of the file. No B-tree rebalancing. No page allocation. No full-file reserialisation.
When you call insertOne, updateOne or deleteOne, pocket-db appends one record to the log and updates the in-memory index pointer. That's it. An update doesn't touch the old record — it writes a replacement which is enough to marks the previous offset as dead. A delete appends a tombstone. This is why deleteOne reaches 454,000 ops/sec, only marginally behind SQLite in-memory.
There's a second reason the numbers look good today: no fsync wait. Writes land in the OS page cache and return immediately. This is fast, but it's a trade-off — a hard crash before the OS flushes to disk can lose the last few writes. Pocket-db is perfectly capable to recover from incomplete operations, so consistency is never compromised. For the target use cases (CLI tools, desktop apps, local prototypes), this is an acceptable trade-off. For anything requiring strict durability guarantees, it's something to be aware of.
Why reads are slower — and what's being done about it
This is the honest part of the benchmark.
pocket-db keeps only indexes in memory, not documents. Every findOne and every cursor step seeks to the document's byte offset in the file and reads from disk. Every result then goes through JSON parsing. This is the fundamental cost of the current design — and it shows clearly in the numbers.
In-memory stores like lowdb and LokiJS serve reads directly from JavaScript objects: no I/O, no parsing. That's why findById on a json-file store reaches 12 million ops/sec while pocket-db sits at 142,000.
Keeping indexes in memory was a necessary first compromise. Without them, reads would require a full file scan for every query — completely unusable in practice. Indexes bring findById to a competitive level for real-world access patterns where you're not reading the same hot document in a tight loop. But the gap on read-heavy workloads is real, and it needs to be closed.
The roadmap for the next version is to address this directly. The first candidate is a document cache: keep recently accessed documents in memory and avoid redundant disk reads for hot-document workloads. This alone should close most of the gap for typical usage patterns. The second candidate is a binary document format to replace JSON serialisation — parsing structured binary is significantly faster than JSON.parse for complex documents. A third option under consideration is memory-mapped I/O with pagination, which would allow the OS to manage the hot-document cache at a lower level.
The memory footprint is also something to watch. Indexes in memory work well at the scales pocket-db is designed for — thousands to low hundreds of thousands of documents. As collections grow, that in-memory index cost grows with them. Load testing at larger scales is to be considered before locked of the architecture.
Reading the numbers in context
The benchmark is deliberately narrow: 1,000 documents, ten operations, one process. That's a fair representation of the workloads pocket-db is built for — a CLI tool tracking state, an Electron app storing user data, a local server caching API responses.
At that scale, the practical difference between 97 and 870,000 read ops/sec is manageable. A CLI command doesn't loop through findByName a million times. But it does insert records frequently, update state on every run, and occasionally compact. For that profile, pocket-db's write throughput is genuinely competitive.
Where the numbers are a real concern is if you're considering pocket-db for a workload that's read-heavy by nature — serving requests in a tight loop, scanning large collections on every query. For those cases today, a pure in-memory store is the honest recommendation.
Where things stand
pocket-db is a young project. These benchmark results are encouraging for what it's designed to do: fast, durable writes in a single-file embedded store with a familiar API. The write story is solid. The read story needs work, and the work ahead is well defined.
The next performance chapter — document cache, binary format, larger-scale index testing — will determine whether pocket-db can compete across the full spectrum of embedded workloads. That's the honest assessment at this first version.
If you want to run the benchmarks yourself:
git clone https://github.com/axfab/pocket-db
npm install
npm run bench
And if a single-file, zero-dependency document store with a MongoDB-style API sounds like what your next project needs:
npm install @axfab/pocket-db
@axfab/pocket-db is MIT licensed. Contributions, issue reports, and benchmark challenges welcome.
Top comments (0)