Munchable's app reads a product label and tells you whether that product suits your gut. The part of it that improves most often is not the code, it is the ingredient data: new words, new spellings, new names for the same thing. Shipping that through the app stores would mean a release per data fix, so instead the app downloads a snapshot of it and keeps a copy on the device.
That means I need an answer to a question that comes up more often than you would think: has this phone's copy gone stale?
The answer is one number.
max(updated_at) over every active row of both tables, as an epoch-ms decimal
string. '0' when empty. One number for the whole snapshot.
That string is the version. It is also the ETag. It is also the value of an x-taxonomy-version response header on other endpoints, and the key the server caches under. One value doing four jobs, derived rather than assigned, so nothing has to remember to bump it.
The request is boring, which is the goal
The app sends what it has:
const { data, error, status } = await apiFetch('/api/taxonomy', {
method: 'GET',
headers: { 'If-None-Match': `"${currentVersion}"` },
});
if (status === 304) return { status: 'unchanged' };
and the server answers with the snapshot or with nothing:
if (clientVersion !== null && clientVersion === snapshot.version) {
return new NextResponse(null, { status: 304, headers });
}
Plain conditional HTTP. No sync protocol, no delta format, no "changes since" endpoint. The snapshot is small enough to send whole, so the cheapest correct thing is to send it whole on the rare occasion it moved and send a 304 the rest of the time. Responses carry private, max-age=3600 as well, so a device that asks twice in an hour does not even reach the route.
Two details in there that are easy to get wrong:
Versions are compared numerically, never as text. They are epoch milliseconds in a string, and string comparison puts "9" after "10". There is a function for it and a comment on the function, because this is exactly the kind of thing that works for years and then breaks on a digit boundary.
A newer client is not an error. The app may have just seen a fresher version in a header from a different endpoint, served by a different instance. So a request claiming a version newer than this instance's cache makes the store bypass its caches and read the table. If the client is still ahead after that, clock skew or a retired row, it keeps its own higher version for comparison purposes and takes the server's snapshot. The one outcome that is not allowed is a device being talked backwards into older data by whichever instance happened to answer.
Three caches, cheapest first
module memory (60s per instance)
-> Redis (gzipped, 24h)
-> Postgres
The important design choice is at the write end: a write does not refill the caches. It forgets the cached snapshot and publishes the new version, and the next reader rebuilds from the table. That is slower for exactly one reader and it removes a whole class of bug where the cache holds something the table never agreed to, because there is only one path that turns rows into a snapshot and everything goes through it.
The Redis TTL started at five minutes and is now 24 hours, which sounds backwards for a freshness mechanism until you notice it is not the freshness mechanism. Every write evicts, so the TTL is only a backstop against a lost eviction. At five minutes, with traffic well below one read per five minutes, nearly every read after a quiet spell rebuilt the snapshot from Postgres: a 40,000 row read plus shaping, measured at 36 times a day. The cache existed and was almost never used.
The day the Redis tier silently stopped existing
This is the bug I would like other people to not spend an afternoon on.
The snapshot is cached in Redis only while it fits, because Upstash's REST API takes roughly a megabyte per request. The cap was 512 KB, measured against the uncompressed JSON string. The snapshot passed that cap for a long time, and then the data grew past roughly twenty thousand rows and it never passed again.
The code did what it was written to do. Over the cap, take the else branch, delete the key. On every single read. So:
- the Redis tier no longer held anything,
- every instance whose 60 second memory cache had expired paid a full 32,000 row table read, about seven seconds,
- and nothing logged an error anywhere, because deleting the key on an oversized value was deliberate behaviour.
The fix was to measure the thing that is actually sent. The value is gzipped before storing, and compressed it is a few hundred KB against a 900 KB cap. The comment now says what to do next time:
If this is ever exceeded again the fix is to shard the key, not to raise it
past what one Upstash request can carry.
The key moved to a :v2 name at the same time, because a gzipped-and-base64 value is a different format and reading a v1 value as a v2 one would be a decode error on every read.
While in that neighbourhood I also found this, which is the other half of the same afternoon. The version query restricted to active rows with a FILTER clause instead of a WHERE. Same answer, except FILTER made it scan the whole table for every version check: 815 ms on a Free-plan Postgres with 40,000 rows, on every product lookup whose caches had gone cold. Moving three words fixed it. The aggregate was never the expensive part; the predicate's position was.
One version for two halves, deliberately
The snapshot has two halves. One decides what the engine can recognise on a label, the other decides what it can say about what it recognised. They ship together, under one version, in one response.
Versioning them separately is the obvious optimisation and it is a trap. A device could hold a statement keyed on something its own vocabulary half does not contain, the engine would drop that row when it loaded it, and the result is not an error. It is a phone that quietly knows less than it should, with no symptom to notice.
So they travel as one number. And the newer half is optional on the wire, which means an older app build, or a cache written before that half existed, simply runs on what is compiled into it rather than failing.
Three more guards worth naming, briefly:
- the server validates every row with the engine's own guards when it reads the table, so what lands in the caches is exactly what a device is allowed to load,
- the app validates the payload again before storing it, because trusting your own API is still trusting a network,
- unreadable device storage reads as "nothing stored", which falls back to the copy bundled with the app rather than to an error state.
What the device never sends
Worth stating plainly, since this is a health-adjacent app. This endpoint is condition-blind. The request is an If-None-Match header and nothing else. No conditions, no results, no barcode. The server hands over vocabulary and the device does the rest, which is why the rate limits on this route can fail open: it is a read of Munchable's own data that reveals nothing about anybody.
The plain-English version of how this all adds up for a user is on munchable.app/about, under "How a verdict is decided". The conditions it covers are at munchable.app/conditions, and the question pages at munchable.app/answers are the web-side view of the same curated data, which is why they change without anybody editing a page.
The bit I did not expect going in: the hard parts of shipping data to installed clients were not the protocol. Conditional HTTP handled that in about fifteen lines. The hard parts were a cap measured on the wrong string, a predicate in the wrong clause, and a version that could have been two.
Top comments (0)