DEV Community

Daniel Pertu
Daniel Pertu

Posted on

516 fingerprints in one Redis hash, and the three characters in front of each one

Munchable is a food app: you scan a barcode in the shop and get a straight answer on whether that product suits your gut. Most of its traffic does not start in the app, though. It starts with somebody typing a question into a search engine, which is why the website publishes 516 pages and why I care how fast an engine notices when one of them changes.

Here is the whole set, live, if you want to poke at it: munchable.app/sitemap.xml. It breaks down as 461 question pages (the index is at munchable.app/answers), 38 recipes at munchable.app/recipes, eight condition guides at munchable.app/conditions, one reference list at munchable.app/hidden-gluten-ingredients, and a handful of fixed pages like the homepage and the privacy policy.

Waiting for a crawler to come back round to any one of those takes weeks. IndexNow exists for exactly that: one POST to api.indexnow.org fans your URL list out to Bing, Yandex, Seznam and the rest of the participating engines. It takes about twenty lines to call.

The twenty lines are not the problem. Deciding what to put in the list is.

You are not allowed to announce everything

Bing's guidance is explicit: submit a URL when its content is added, updated or removed. Re-announcing pages that have not changed earns a 429 first and a reputation second. So an hourly job that posts all 516 URLs because the site deployed is not a quick win, it is a slow way to get ignored.

Which means every run has to answer a question: has this page's content moved since the last time we told anyone about it?

The obvious implementation is to fetch the page and hash the HTML. That is simpler and completely useless. Next stamps a fresh build id into every page on every deploy, so byte-identical content hashes differently after a no-op redeploy, and you are back to announcing all 516.

So the fingerprint is taken on the input side instead. Each page's copy comes out of a data module, the module is in the same process as the job, and a page's fingerprint is a hash of the data that page renders:

function fingerprint(value: unknown): string {
  return `v1:${createHash('sha256').update(JSON.stringify(value) ?? 'null').digest('hex').slice(0, 16)}`;
}
Enter fullscreen mode Exit fullscreen mode

Sixteen hex characters is plenty when the comparison is equal-or-not against 516 keys. The whole set lands in one Redis hash, one field per URL, and the hourly job is one hgetall, one diff, and almost always no HTTP request at all.

The three characters that make it work

v1: is not decoration, and it is not really versioning either. It is there because of how the client reads values back.

Upstash's client JSON-parses stored values on read. A bare 16-character hex hash is, about one time in sixteen, all digits. Write "8675309912345678" and read back the number 8675309912345678, which does not compare equal to the string it was written as. That page is now "changed" on every single run, forever, and the diff looks plausible enough that you would never think to question it. Prefix every hash with something that can never parse as a number and the problem cannot occur.

The test for it is one line and says what it is for:

test('fingerprints are prefixed so a hash can never be read back as a number')
Enter fullscreen mode Exit fullscreen mode

That is the whole category of bug I want tests for: not "does sha256 work" but "does this value survive the round trip through the store I chose".

Hash a projection, not an object

The first version of the question pages hashed the whole page object. Each of those pages embeds the condition guide it belongs to, so fixing a typo in the third paragraph of one guide changed the fingerprint of every question page attached to it, and the next run cheerfully announced dozens of URLs because a comma moved.

So each page hashes a projection: the question, the answer line, the short detail, and the identifiers of what it is about. Narrow enough that an unrelated edit elsewhere does not touch it, wide enough that it moves when the page's actual answer moves. That last part is the payoff. When the underlying data changes such that one of those 461 pages now says something different, the next hourly run announces that page, and nobody had to remember to.

The recipe pages get the same treatment for the same reason, hashing the derived page rather than the recipe: a recipe's own words can be untouched while the set of conditions it suits changes, and that is a change worth announcing.

The pages nothing can fingerprint

Two pages have their copy in JSX rather than in a data module: the homepage and /licenses. There is no object to hash. Pretending otherwise would mean hashing the component source, which drifts for formatting reasons and not content reasons.

They get a constant instead:

const STATIC_COPY = fingerprint('static-copy');
Enter fullscreen mode Exit fullscreen mode

Which is to say: announced once, on the first run, when they are new to the engines, and then never again. Editing the homepage copy is what the route's manual POST exists for, with {"all": true} to re-announce everything and {"urls": [...]} for a specific page. The honest version of this design is "two pages are handled by hand", and writing the constant down beats pretending there are no exceptions.

Four guards, each one a sentence

The rest of the module is refusals, and they are more interesting than the submit call.

An empty page set announces nothing. If the content modules ever return zero pages, that is a bug in the content modules, not a site that deleted itself. Without this check the diff sees 516 stored URLs that are no longer live and submits all of them as removals, which asks every engine to drop the entire site.

A failed submission records nothing. Fingerprints are written only after the engines accept them, so a timeout means the next run tries the same URLs again rather than quietly forgetting them.

A failed run answers 502. Not a 200 with an error field in the body. Otherwise the cron dashboard shows a fortnight of green ticks over a key the engines have been rejecting the whole time.

Preview deploys refuse outright. Branch deploys are noindex and their sitemap is empty, so announcing their URLs would hand the engines a competing copy of the site.

The key file is supposed to be public

Ownership is proved by serving a key where the engine can fetch it, and the protocol scopes a submission to the key file's directory, so the key lives at the root of the host. There are two ways to do it: a static <key>.txt at the root, or any filename plus a keyLocation field on every submission.

The second one is worth the extra field. With the key in the filename, rotating it means renaming a committed file and editing the submitter to match. With keyLocation, the key is read from the environment in one place, the file route and the submitter both read it from there, and they cannot disagree. The route is dynamic rather than prerendered so rotation takes effect without a redeploy, and it is cached for five minutes so a burst of submissions does not wake a function each time.

It is checked into the repository, because the protocol requires it to be publicly readable at a well-known URL. That is the entire mechanism by which an engine checks that whoever submitted those URLs controls the host. Here it is: munchable.app/indexnow-key.txt.

What an hour costs

Almost every run submits nothing and costs one Redis read. The runs that do submit are the ones after a deploy that changed copy or moved the data behind a page, and then a new page is in front of Bing within the hour instead of whenever a crawler next wanders past.

That is the trade I wanted: the hourly schedule is cheap precisely because the job is built to decide, 516 times, that there is nothing to say.

If you want to see what it is keeping an eye on, munchable.app/answers is the index of the 461 pages, and each one is a question with a URL of its own.

Top comments (0)