Facet is a creator platform: one site per creator, with courses, a newsletter, a member community and checkout. Under the hood every creator gets their own SQLite database, and one Go process (built on PocketBase, used as a library) serves many of them. In a load test, one 32 GB box held 500 tenants warm at about 6.6 MB of RAM each.
One database per tenant is lovely for isolation, exports and deletion. It is less lovely for backups. A nightly tarball gives you a day of potential data loss. We wanted "restore this site to 10:42 this morning", for every tenant, on every plan, without the bill scaling with the number of tenants.
This is how that works, what it costs, and the bug we found in it this week.
The shape of it
Each tenant's chain lives under its own prefix in Cloudflare R2:
pitr/v1/<tenant-id>/<epoch>/
snap/<seq>.db.gz a full copy of the database at a clean point
snap/<seq>.manifest.json a signed header for that snapshot
seg/<seq>-<first>-<last>.wal.gz WAL frames since the previous object
Every object is immutable and carries a signed header: tenant id, chain epoch, sequence number, the SHA-256 of its own payload, and the SHA-256 of the object before it. There is no mutable index file. The chain is the index, and a restore verifies every link and signature before it trusts a byte.
A restore is: list the tenant's prefix, pick the newest snapshot at or before the target time, download it, then replay WAL segments in order up to the target, checking each hash against the previous one. The result is staged next to the live database and integrity-checked before anything is swapped.
Why not Litestream?
Litestream is the reference design here, and the shape above borrows from it heavily. We don't run the binary because of one rule in our codebase: no process outside the tenant pool writes to a tenant database. A replicator has to own checkpointing, and checkpointing is a write. A sidecar would also know nothing about our tenant lifecycle: hibernation to cold storage, eviction, renames, moves between hosts and deletion. Any of those racing a replicator is how you lose data.
So the shipper runs inside the same Go process as the pool. It disables SQLite's automatic checkpoint for shipped tenants and does it itself, right after each ship. If a checkpoint ever moves more frames than were shipped, that's a gap in the chain, and the shipper answers it with a fresh base snapshot rather than carrying on silently.
The cost model: idle tenants cost nothing
R2 charges per write operation (Class A, $4.50 per million). A naive shipper that uploads on a timer would cost the same for a dormant hobby site as for a busy course. So the shipper is change-driven:
- A single sweep loop checks each attached tenant's WAL file. No new committed frames, no request. Idle tenants make zero R2 calls.
- When frames exist, it waits for 1 MiB of WAL or 60 seconds, whichever comes first, then ships one segment. That 60-second floor is unconditional; it is what makes the worst case predictable.
- A new base snapshot is taken after 64 MiB of segments or a week, so restores never replay an unbounded tail.
- Every tenant has a hard daily operation budget. Past it, shipping degrades to a five-minute window instead of failing, and an operator gets told.
The model assumes about 17 write operations per tenant per day. At 100,000 tenants that projects to roughly $342 a month in R2, or about $0.0034 per tenant. That's cheap enough to put point-in-time restore on every plan, including the cheapest one.
The bug: a higher number isn't always newer
Each chain has an epoch. A new epoch starts when the chain has to restart, for example after a restore or when the shipper can't prove continuity. To pick a chain, both the restore planner and the retention janitor used one simple rule: the highest epoch is the current one.
This week, while checking our nine public demo sites, we found that rule pointing at the wrong data. Their chains looked like this:
| Demo | Live chain | Stale chain | Stale chain's last write |
|---|---|---|---|
| one site | epoch 1 | epoch 32 | 27 September |
| another | epoch 5 | epoch 48 | 27 September |
The demos had moved from a staging host to production in late September. The staging host kept its local shipper state, a small file holding the epoch and sequence counters, and had been counting epochs up for weeks. Production started fresh at epoch 1. Two chains, one prefix, and the dead one had the bigger number.
A restore on any of those demos would have "succeeded", passed every integrity check, and quietly handed back the site as it was on 27 September. Retention would have treated the live chain as history. Nothing was wrong with any single object; the bug was in what "current" meant.
The fix
Three changes, all now in production:
- Epochs are allocated against remote history, not just local state. When a tenant attaches on a host, the shipper qualifies its namespace: if the remote prefix holds an epoch at or above the local one, it starts a fresh epoch above both. A stale local file can no longer produce a chain that sorts below a dead one.
- Restore picks the epoch by time, not by number. The planner takes the highest epoch that has objects at or before the target time. A target before a move resolves to the chain that was live then.
- Inverted history is refused, not guessed at. If a lower epoch has writes newer than a higher epoch's start, restore stops with "epoch chronology is inverted; a verified new base is required". Retention keeps everything rather than pruning on an ambiguous timeline.
The trade-off is honest: for the affected demos, restore points start from the moment the fix shipped, and that window refills over the retention period. Refusing is better than returning the wrong month.
Then we restored for real
A backup system you haven't restored from is a hypothesis. After the fix deployed, every affected demo started a new epoch above its stale one within minutes, each with its own base snapshot. Then we ran the actual restore code against production R2, staging into a scratch directory:
-
A demo site (new chain: one snapshot plus two segments): restored in 4.9 s;
PRAGMA integrity_checkok; 228 of 228 tables matched the live database's row counts exactly (2,420 rows). - A long-running tenant (one snapshot plus 79 segments): restored in 2 minutes 34 seconds; integrity ok; 228 of 228 tables identical.
- The same demo, asked for yesterday: refused with the inverted-chronology error. Before the fix, that request would have returned September.
The drill also turned up two things we're fixing next: restore planning needs the tenant's local state file, and a brand-new host should refuse to start a tenant on an empty database rather than serving a blank site that then becomes the newest base snapshot. Both are now tracked.
What we'd tell you if you're building this
- Make every object self-describing and chained. A signed header with the previous object's hash turns "is this backup intact?" into a loop you can run anywhere, with no index to corrupt.
- Charge yourself for idleness explicitly. "Zero requests when nothing changed" is the invariant the whole cost model hangs on. Test it.
- "Newest" is a claim about time, so check it against time. Counters, sequence numbers and epochs drift the moment state moves between machines.
- Refuse ambiguity loudly. A restore that errors is an incident. A restore that returns the wrong month is a disaster nobody notices.
- Drill it, and diff the result against live. Our green test suite never saw this bug. Comparing a restored database to the live one table by table would have caught it on day one.
Facet is in early access at get-facet.com. The open-source core is on GitHub at jesposito/Facet.
Top comments (0)