DEV Community: Michael Banigo

One codebase, two install paths: shipping a PWA and an Android app from the same Next.js site

Michael Banigo — Tue, 27 Oct 2026 00:54:51 +0000

The Play build of ANYFREEBOOK is 1.2 MB. Almost none of that is the app. It is an Android shell around the website that already exists, and the interesting engineering is not in the shell — it is in the handshake that lets Android believe the shell owns your domain.

What a Trusted Web Activity actually is

A TWA is an Android activity that loads a URL inside a WebView with the browser chrome stripped away. There is no duplicated application code, no second build of the product, nothing to keep in sync. The Play listing for ANYFREEBOOK is exactly that: package com.anyfreebook.app, host anyfreebook.com, full scope so every route stays inside the app.

The package is built with Bubblewrap over the same Next.js site, and the numbers that matter are small: minSdkVersion 24, targetSdk 36, and a 1.2 MB app bundle. Compare that with shipping the same features natively — a second codebase, a second release train, and a product version that is always behind the website. The TWA gets you a Play listing, a launcher icon, and a native share sheet. Everything else is the website.

Digital Asset Links is the actual handshake

Before Play will open your site in the app shell instead of a browser tab, something has to prove the domain owns the app. That proof is Digital Asset Links, and it has two halves that must agree:

A statement served from your domain at /.well-known/assetlinks.json.
An equivalent statement embedded in the installed app.

If either half is missing or the two disagree, the link degrades to a Custom Tab. The site still opens — inside a browser-chrome shell, which reads as "the app is broken" rather than "the app is being cautious".

Here is what we publish:

[
  {
    "relation": [
      "delegate_permission/common.handle_all_urls"
    ],
    "target": {
      "namespace": "android_app",
      "package_name": "com.anyfreebook.app",
      "sha256_cert_fingerprints": [
        "A6:76:B1:D9:64:9E:4F:5A:34:4D:EB:06:9B:31:64:00:15:F4:6E:76:79:96:C6:FA:13:56:BE:FA:D6:F7:90:C2"
      ]
    }
  }
]

The relation is the part people skip. delegate_permission/common.handle_all_urls says "this app may handle every URL on this host" — broad delegation, rather than a list of individual paths. Scope it narrowly and only the paths you listed open in the app shell, with everything else falling back to the browser.

The bug that cost me an afternoon: fingerprints are uppercase-only

The fingerprint above is uppercase, and that is not a style choice.

Google's Digital Asset Links validator parses certificate fingerprints with the same pattern Bubblewrap uses:

/^([0-9A-F]{2}:){31}[0-9A-F]{2}$/

Look at the character class: [0-9A-F]. Uppercase only — there is no a-f in the range. A lowercase fingerprint, the same 32 bytes and the same certificate, is rejected as malformed cert fingerprint. The validator never gets as far as checking whether the fingerprint matches anything: it refuses to parse the shape.

The fix is one call to .toUpperCase(), but the reliable way to apply it is to make it impossible to forget. Our generator validates and normalises every fingerprint before writing the file:

// scripts/make-assetlinks.mjs
const cleaned = fingerprints.map((f) => f.trim().replace(/^SHA256\s*/i, '').toUpperCase());

const invalid = cleaned.filter((f) => !/^([0-9A-F]{2}:){31}[0-9A-F]{2}$/.test(f));
if (invalid.length > 0) {
  console.error('These do not look like SHA-256 fingerprints (expected 32 uppercase hex octets):');
  for (const f of invalid) console.error('  ' + f);
  process.exit(1);
}

The script accepts more than one fingerprint, and replace(/^SHA256\s*/i, '') handles the SHA256=... prefix some tools emit. Normalising at the point of writing, and failing loudly on a fingerprint that does not match the expected shape, turns a confusing validator error into a build-time message that says exactly what is wrong.

Play App Signing: the fingerprint that matters is not the one you signed with

This is the part that costs people a release. You sign your upload bundle with your own upload key; Google Play then re-signs the app with its own Play App Signing key before delivering it. The device validates the certificate of the APK it actually installed — the Play app-signing certificate, not yours.

So publishing only your upload-key fingerprint means the domain proves it owns a key the installed app does not use. The link opens in a Custom Tab: the app launches, the site loads, and the whole thing looks like a slightly broken browser rather than an app.

The fix is to publish both fingerprints. The upload key's SHA-256 is what you already have locally; the Play app-signing key's SHA-256 lives in the Play Console under Test and release → Testing → App integrity → App signing key → SHA-256, and only exists after your first upload is processed.

node scripts/make-assetlinks.mjs com.anyfreebook.app <upload-fp> <play-signing-fp>

Then verify, rather than guessing. The Digital Asset Links API will tell you whether the domain and the app actually match:

https://digitalassetlinks.googleapis.com/v1/statements:list?source.web.site=https%3A%2F%2Fanyfreebook.com

The site parameter is enough — the response lists every app the domain is associated with. (Adding an source.android.app_id parameter to narrow it returns 400 Unknown name "source.android.app_id", so leave it off.)

A response containing a statement for com.anyfreebook.app and no error means the handshake holds. Getting the JSON to parse is not enough — that only proves the file is syntactically valid. The statements:list call proves the pairing.

There is one more switch in the Play Console: Release → Setup → App integrity → Digital Asset Links verification. Until it is on, the verification is advisory.

The service worker is the same in both installs

The TWA does not have its own offline story. It loads your site, and your site's service worker does the work — the same public/sw.js that serves the PWA install.

That worker precaches a small set of shell assets, serves pages network-first with the cached copy as fallback, serves hashed static assets cache-first, and falls back to /offline.html when the network is gone:

// public/sw.js
const CACHE_NAME = 'anyfreebook-v3';
const OFFLINE_FALLBACK = '/offline.html';
const PRECACHE = ['/', OFFLINE_FALLBACK, '/manifest.json', '/favicon.ico', /* ... */];

self.addEventListener('install', (event) => {
  event.waitUntil(
    caches
      .open(CACHE_NAME)
      // allSettled: one unreachable file must not abort the whole install.
      .then((cache) => Promise.allSettled(PRECACHE.map((url) => cache.add(url))))
      .then(() => self.skipWaiting())
  );
});

Two details there are about living in an app shell rather than a browser tab. The worker returns error statuses untouched, so a genuine 404 stays a genuine 404 rather than being masked by a cached shell. And when an asset is unreachable offline, it returns a 1×1 transparent PNG with a 200 status instead of a failed request — a failed subresource inside a TWA looks like a broken app in a way it never does in a tab.

The Bubblewrap manifest, in essentials

The whole Android project is generated from one JSON file. The parts worth understanding:

{
  "packageId": "com.anyfreebook.app",
  "host": "anyfreebook.com",
  "name": "ANYFREEBOOK",
  "display": "standalone",
  "startUrl": "/",
  "fullScopeUrl": "https://anyfreebook.com/",
  "minSdkVersion": 24,
  "appVersion": "1",
  "appVersionCode": 1,
  "fingerprints": [
    {
      "name": "upload",
      "value": "A6:76:B1:D9:64:9E:4F:5A:34:4D:EB:06:9B:31:64:00:15:F4:6E:76:79:96:C6:FA:13:56:BE:FA:D6:F7:90:C2"
    }
  ]
}

packageId and host are the identity of the app and the domain it claims. fullScopeUrl is what makes it a full-scope TWA rather than a partial one — every route under that origin stays inside the shell. minSdkVersion 24 and targetSdk 36 set the platform floor and the API level the app targets. Changing anything here means re-running npx bubblewrap update --manifest twa-manifest.json, which regenerates the project and bumps appVersionCode.

Note that the fingerprints array here carries the upload key. That is correct for Bubblewrap's signing step, and it is not sufficient for Digital Asset Links — the file the domain serves needs the Play app-signing fingerprint too.

The practical takeaway

A TWA is not an excuse to skip the PWA, because the PWA is what the TWA loads. The manifest, the service worker, the offline page and the icons all have to be right before the app shell has anything to show. If the service worker's cache-first rules are wrong, the wrong version of a page is what the app shows, offline and online alike.

Get the order right and the work is small: build the bundle, upload it, read the Play app-signing fingerprint out of the Play Console, publish it alongside the upload fingerprint in uppercase, verify with the statements:list endpoint, and turn on Digital Asset Links verification. Then the Play icon opens your website instead of a browser tab, and the app you maintain is still the one you already ship.

Your "free" book API might not be free

Michael Banigo — Tue, 27 Oct 2026 00:37:34 +0000

A book-search API hands you a URL, and usually the word "free" printed next to it. That URL is a black box. Behind it there can be a public-domain full text, a borrow-only lending copy, or a catalogue record for a book still in copyright. If your client calls it free, it is making a claim about the law it cannot support — on your users' behalf.

Three tiers, and what each one actually means

ANYFREEBOOK returns a tier with every result. The three tiers are deliberately narrow:

download — a free, legally readable copy of this exact title, resolved from the source's own metadata. This is the only tier that permits the words "free download" anywhere in the product.
borrow — a real Internet Archive copy that the Archive's own metadata says is loan-only. Controlled Digital Lending: a free Internet Archive account plus a loan. There is no file to fetch.
unverified — a catalogue record, or a search-results page for this title. We could not confirm a free, legally readable copy of this exact title.

The tier is not a guess or a heuristic. It is resolved from the source's own metadata.

The evidence has to come from the source

For the Internet Archive, the item's own record answers both questions at once: is this copy open, and which files can be downloaded. src/lib/api/publicRecords.ts reads the lending fields directly:

const restricted =
  metadata['access-restricted-item'] === 'true' ||
  !!metadata['lending_identifier'] ||
  !!metadata['lendinginfo'];

If the metadata call fails, the resolver falls back to unverified rather than assuming the best. A record we could not read is never promoted to download.

For Project Gutenberg, the title and the author both have to match. src/lib/api/resolveSource.ts checks the author, because a title alone is not enough — a book by one Austen must never resolve to a different book that shares the surname:

function authorMatches(author: string, authors: { name: string }[] | undefined): boolean {
  if (!authors || authors.length === 0) return false;
  const tokens = author.toLowerCase().split(/[\s,&]+/).filter(t => t.length >= 3);
  const combined = authors.map(a => a.name.toLowerCase()).join(' | ');
  return tokens.some(t => combined.includes(t));
}

Only when a match survives both checks does the resolver claim a tier:

const match = data.results?.find(
  (r: { title?: string; authors?: { name: string }[] }) =>
    titleMatches(title, r.title) && authorMatches(author, r.authors)
);
if (match) {
  const readUrl =
    match.formats?.['text/html'] ||
    match.formats?.['application/epub+zip'] ||
    `https://www.gutenberg.org/ebooks/${match.id}`;
  return {
    sourceUrl: `https://www.gutenberg.org/ebooks/${match.id}`,
    sourceType: 'gutenberg',
    downloadLinks: [{ label: 'Read on Project Gutenberg', url: readUrl, source: 'Project Gutenberg' }],
    tier: 'download',
  };
}

And the rule that matters most: a catalogue record is not a copy. Open Library holding an entry for a book tells you the book exists and where it can be bought. It says nothing about whether the text is free to read today. Treating "we found the title" as "the title is free" is the single most common bug in this category of API, and the one that ends up in front of a rights holder.

The title has to be checked too, and this is a bug we hit ourselves. Gutendex and Open Library both rank by relevance, so taking the top hit returns whatever scored highest. Our resolver asked Open Library for one result and checked the author; a request for Fairy Circles by William James Hudson came back with a downloadable copy of Alice's Adventures in Wonderland, labelled tier: download. The fix is to fetch several candidates, ask for their title and author fields, and accept the first that is actually the title requested:

const doc = docs.find((d) => {
  if (!titleMatches(title, d.title)) return false;
  if (!author || !d.author_name || d.author_name.length === 0) return true;
  return authorMatches(author, d.author_name.map((name) => ({ name })));
});

titleMatches allows a subtitle ("Fairy Circles: Tales and Legends of …") but not a different book that merely starts with the same words. When nothing matches, the resolver returns unverified with links to real search pages — the honest answer: we cannot confirm a free copy of that book.

Borrow is not download

Controlled Digital Lending is easy to misdescribe. The Internet Archive holds a real copy of the book. The reader gets a real copy of the book. But it is only accessible through the Archive's lending interface: you need a free account, you need to take out a loan, and the loan is time-limited and renewed through a queue.

None of that is a file. A link to a borrow-only item is not something a downloader can fetch; it is a page that requires an account before it gives you anything. If your API says "free download" and links to a borrow-only item, you have shipped a button that leads to a login wall and called it a download.

That is why the tiers are separate values rather than a boolean isFree. borrow is a real answer — this copy is free to read, under conditions — but it differs from "you may fetch this file now".

The rule: never claim a download you cannot point at

Our resolver never returns download for a search-results page. Where the source exposes a file, the API's files[] array carries it. This is a real response, captured from the live API:

{
  "title": "Fairy Circles",
  "author": "Villamaria",
  "tier": "download",
  "tierReason": "Open-access copy on the Internet Archive.",
  "sourceType": "archive",
  "sourceUrl": "https://archive.org/details/fairycirclestale00villiala",
  "license": "Public domain",
  "files": [
    { "format": "epub", "url": "https://archive.org/download/fairycirclestale00villiala/fairycirclestale00villiala.epub", "bytes": 15512987, "label": "fairycirclestale00villiala.epub" },
    { "format": "pdf", "url": "https://archive.org/download/fairycirclestale00villiala/fairycirclestale00villiala.pdf", "bytes": 13373257, "label": "fairycirclestale00villiala.pdf" },
    { "format": "txt", "url": "https://archive.org/download/fairycirclestale00villiala/fairycirclestale00villiala_djvu.txt", "bytes": 514307, "label": "fairycirclestale00villiala_djvu.txt" },
    { "format": "html", "url": "https://archive.org/download/fairycirclestale00villiala/fairycirclestale00villiala_hocr.html", "bytes": 11537829, "label": "fairycirclestale00villiala_hocr.html" }
  ],
  "links": [
    { "label": "Read on Internet Archive", "url": "https://archive.org/details/fairycirclestale00villiala", "source": "Internet Archive" },
    { "label": "View on Open Library", "url": "https://openlibrary.org/works/OL7903107W", "source": "Open Library" }
  ],
  "meta": { "apiVersion": "v1", "generatedAt": "2026-10-24T12:53:48.857Z", "caller": "anonymous" }
}

For Project Gutenberg the same shape comes back with Gutenberg's canonical URLs instead — /ebooks/1342.epub3.images, .txt.utf-8 and .html.images — because those three exist for every ebook. Gutenberg serves no PDF for every title, so none is listed unless the source reported one.

There is one case where download comes back with an empty files[]: the copy is genuinely free to read, but the source exposes no file you can fetch in one request. A DOAJ article or a PubMed Central paper is free full text at the publisher's page; a LibriVox recording is free as chapter files; an Internet Archive item whose only derivatives are DRM-encrypted has no fetchable file. Those records carry links[] instead.

So a machine client should check two things. tier === 'download' says the copy is free without payment or an account. files.length > 0 says you can fetch it right now. A downloader that only checks the tier sometimes finds nothing to download; one that only checks files[] misses the cases where the tier matters.

For borrow and unverified, files[] is empty and links[] points at the real source. That is not an oversight in the payload — it is the payload making the distinction impossible to miss.

When /resolve cannot find a copy it can confirm, the tier is unverified and the links are search URLs for the exact title and author. We would rather send a reader to a real search page than pretend it is a download.

One more wrinkle: the file list contains things that look like documents and are not. Labels are matched exactly against a small allowlist of real formats, and the file name is checked too, because "OCR Search Text" is a gzipped search index and "Single Page Original JP2 Tar" is a several-hundred-megabyte dump of page images. Matching labels loosely is how a downloader ends up fetching a 700 MB tarball thinking it fetched a book.

What a client does with the tier

Here is the client-side consequence, the part most implementations skip. A consumer that downloads each book should check the tier and refuse out loud:

// An example client consuming GET /api/v1/resolve
type Tier = 'download' | 'borrow' | 'unverified';

interface Resolved {
  tier: Tier;
  tierReason: string;
  files: { format: string; url: string }[];
}

async function download(title: string) {
  const res = await fetch(
    `https://anyfreebook.com/api/v1/resolve?title=${encodeURIComponent(title)}`
  );
  const body = (await res.json()) as Resolved;

  if (body.tier !== 'download' || body.files.length === 0) {
    // Refuse out loud. Skipping the title silently would look, to the user
    // watching the output, exactly like the download succeeded.
    throw new Error(`Cannot download "${title}" (${body.tier}): ${body.tierReason}`);
  }

  for (const file of body.files) {
    // fetch file.url
  }
}

The difference between throwing and skipping is the whole point. A silent skip produces a run that looks successful and quietly contains fewer books than it should. A refusal with the tier attached tells the user something true: this title is borrow-only, or it is a catalogue record.

The tier is the single source of truth

Inside our own codebase the discipline is enforced by having exactly one decision point. src/lib/api/resolveSource.ts exports the SourceTier type, and the book page's badge, its <title>, its meta description and its schema.org offer all read from that one value. The green "100% legal, open-access" badge appears only when the tier is download.

When the tier is anything other than download, the page does something better than apologising. It offers a genuinely free alternative — a real open resource for that subject, from OpenStax, MIT OpenCourseWare, Project Gutenberg and Cornell LII, with per-title matches taking priority over per-category ones.

If your API returns "free" URLs, you are making a legal claim

That is the broader point. A "free" URL in an API response is not a convenience field. It is a representation to your users, and indirectly to their employers and their schools, that the thing behind the link costs nothing and can be taken away.

Resolve it properly or don't claim it. The tier is the mechanism: three values, each backed by the source's own metadata, each with a tierReason that a human can read and argue with.

The failure this avoids is concrete. It is shipping a "free download" button that leads to a login wall, which is embarrassing. It is also shipping that same button where a downstream site has filled the gap with a pirated copy, which is not. Between the two, the boring tier system is the cheapest insurance there is.

Fanning out a search across twelve APIs, and telling the user when one of them fails

Michael Banigo — Tue, 27 Oct 2026 00:33:32 +0000

ANYFREEBOOK searches Project Gutenberg, Open Library, the Internet Archive, Google Books, PubMed Central, DOAJ, LibriVox, OpenAlex, Wikisource, Europeana, Zenodo and the Open Textbook Library at the same time. (The public API fans out to all twelve. The website's own search page leaves LibriVox out, because an audiobook is usually not what someone typing a book title is after — that source has its own page.) The parallelism is the easy part. The part that took real thought was deciding what to do when one of the twelve does not answer — because the default behaviour of most multi-source search code is to quietly return a shorter list, and a shorter list is indistinguishable from "that is everything there is".

There is no search index

Start with the constraint, because it dictates everything else. ANYFREEBOOK does not maintain a search index. We keep a curated catalogue of 258 books across 241 categories in src/lib/data.ts, and that catalogue powers the category pages and the per-title detail pages. But a search query is not answered out of that catalogue. It is fanned out, live, to twelve separate upstream APIs — each with its own index, its own rate limits, its own idea of what a result is, and its own bad days.

There is no replica of those indexes in our database and we do not crawl and re-store. Every search is a fan-out of live upstream calls, and the answer is the union of whatever they returned.

That fan-out is bigger than it looks. Twelve sources, but nineteen outbound HTTP requests, because three of them need more than one call to answer: Wikisource is a separate wiki per language and we ask six of them, PubMed Central is a search followed by a summary of the ids it returns, and the Open Textbook Library serves ten books per page so a page of twenty is two requests. Here is the measured breakdown of one search, counting the requests that actually leave the server:

   6  en.wikisource.org, zh, ja, ar, he, fr   (six Wikisource editions)
   2  eutils.ncbi.nlm.nih.gov                  (esearch, then esummary)
   2  open.umn.edu                             (Open Textbook Library, two pages)
   1  openlibrary.org
   1  gutendex.com
   1  archive.org
   1  doaj.org
   1  api.openalex.org
   1  api.europeana.eu
   1  zenodo.org
   1  librivox.org
   1  www.googleapis.com                       (Google Books)

That number is worth knowing before the rest of this, because it is the reason the rate limiter in this system is not optional. Nineteen requests to twelve different services, on somebody else's infrastructure, for one user query.

The design is cheap to start and awkward to operate, because the failure modes of twelve separate services are now our failure modes.

One source failing must not cost you the other six

The obvious version looks like this:

const [gutenberg, openLibrary, googleBooks, archive, pubmed, doaj, librivox, openAlex] =
  await Promise.all([
    searchGutenberg(query, page),
    searchOpenLibrary(query, 20, page),
    // ...and so on
  ]);

Promise.all short-circuits. The first promise that rejects rejects the whole thing, and you throw away eleven responses that already arrived, or would have arrived had the fan-out waited. On a good day nobody notices; when one upstream times out, the search errors out for a query eleven sources could answer perfectly well.

Promise.allSettled waits for every promise to reach a terminal state and hands you each outcome separately. One rejected shelf costs you that shelf, never the page.

How the fan-out actually looks

In src/lib/api/searchAll.ts each source gets its own block, and each block catches its own error:

export interface SearchOutcome {
  books: Book[];
  total: number;
  sources: Record<string, number>;
  failedSources: string[];
}

export async function searchAllSources(
  query: string,
  page: number,
  source: SearchSource,
  options: { includeLibrivox?: boolean } = {}
): Promise<SearchOutcome> {
  const results: SearchOutcome = { books: [], total: 0, sources: {}, failedSources: [] };
  const fetches: Promise<void>[] = [];

  if (source === 'all' || source === 'openlibrary') {
    fetches.push(
      (async () => {
        try {
          const ol = await searchOpenLibrary(query, 20, page);
          results.books.push(...ol.books);
          results.sources['Open Library'] = ol.total;
          results.total += ol.total;
        } catch {
          results.failedSources.push('Open Library');
        }
      })()
    );
  }

  // ...the same block for Gutenberg, Google Books, the Internet Archive,
  // PubMed Central, DOAJ and LibriVox...

  await Promise.all(fetches);
  return results;
}

A note on why this uses Promise.all rather than Promise.allSettled: because every block swallows its own error, none of these promises ever rejects. Promise.all here is safe by construction — it is allSettled semantics, with the outcome recorded in failedSources instead of in a result object.

Where the shape of the data allows it, we write the intent more directly. The category pages do exactly that:

// src/app/category/[slug]/page.tsx
const [ol, archive, gb] = await Promise.allSettled([
  searchOpenLibrary(query, 30, 1),
  searchArchive(query, 1, 24),
  searchGoogleBooks(query, 0, 20),
]);

const olTotal = ol.status === 'fulfilled' ? ol.value.total : 0;
const archiveTotal = archive.status === 'fulfilled' ? archive.value.total : 0;
const gbTotal = gb.status === 'fulfilled' ? gb.value.total : 0;

Both patterns are fine. What matters is that a rejected source produces a smaller list — never an exception, and never a silently truncated list presented as complete.

The other half of the story is timeouts. An upstream that never responds is worse than one that errors, because it holds the request open. Every call to a third-party book API goes through one shared abort signal with a 10-second ceiling (src/lib/api/http.ts), so a hung source costs that one shelf instead of the request.

The contract: failedSources and partial

A failure record is only useful if it reaches the client. The public API (https://anyfreebook.com/api/v1/search) returns it. Here is a real response, captured from the live API:

{
  "query": "python",
  "page": 1,
  "limit": 2,
  "total": 25204850,
  "count": 2,
  "sources": {
    "ANYFREEBOOK": 3,
    "Internet Archive": 23532873,
    "Open Library": 6589,
    "Google Books": 190,
    "DOAJ": 17471,
    "Europeana": 409,
    "Wikisource": 0,
    "OpenAlex": 1253080,
    "Zenodo": 71644,
    "PubMed Central": 300019,
    "Open Textbook Library": 38,
    "Project Gutenberg": 22537,
    "LibriVox": 0
  },
  "failedSources": [],
  "partial": false,
  "results": [ "..." ]
}

All twelve sources answered, so failedSources is empty and partial is false. Two of those counts are zero — Wikisource and LibriVox — and both are real answers: LibriVox has no audiobook whose title starts with "python", and the Wikisource editions we ask have no page titled exactly "python". Zero is a legitimate answer, and it is different from not answering at all.

Here is the same endpoint on a query where three sources genuinely did fail. The query is the word AND, which is a Solr/Lucene operator, so it is a malformed search for the three upstreams that speak that syntax:

{
  "query": "AND",
  "page": 1,
  "limit": 2,
  "total": 20031433,
  "count": 2,
  "sources": {
    "ANYFREEBOOK": 39,
    "OpenAlex": 0,
    "Google Books": 101,
    "LibriVox": 20,
    "Europeana": 7151166,
    "Wikisource": 0,
    "Zenodo": 4541111,
    "PubMed Central": 8337033,
    "Open Textbook Library": 2002,
    "Project Gutenberg": 0
  },
  "failedSources": ["Internet Archive", "DOAJ", "Open Library"],
  "partial": true,
  "results": [ "..." ]
}

This is the interesting case. Ten sources answered and three did not, and the response says which three. Note that Internet Archive and Open Library are simply absent from sources — they are not present with a count of zero, because a zero would be a claim that they answered and found nothing.

Without the flag the same payload would read as a confident "20,031,433 results", and a reader would have no way to know that a slice of the answer was missing. With it, the client can say "here is what we found, and here is what we could not ask". A shorter list is not a wrong list, but an undeclared short list is a misleading one.

Getting there took one more thing that is easy to get wrong: the difference between "no matches" and "did not answer" has to be decided by the adapter, not by the caller. If an upstream returns 503 and the adapter returns an empty list, then the source reports a count of zero — and "Google Books has zero matching books" is a factual claim the code is making without checking anything.

So the adapters throw on failure and return zero only when the upstream genuinely says there is nothing:

// src/lib/api/http.ts
const data = await fetchUpstreamJson<GoogleBooksResponse>(
  url,
  { next: { revalidate: 3600 } },
  { source: 'Google Books', emptyOnStatus: [404] }
);
if (!data) return { books: [], total: 0 };

emptyOnStatus is per upstream, because "no match" is not a universal status code. LibriVox answers a title search that matches nothing with a 404 and a body of {"error":"Audiobooks could not be found"} — that is a real answer, so for LibriVox a 404 is zero matches and not a failure. Every other source here answers no-match with a 200 and an empty list, so for them any error status is a failure.

Transient statuses get retried before the source is written off, because a 503 from a busy API usually clears:

function isTransient(status: number): boolean {
  return status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
}

Two retries, one shared deadline, so a single slow source cannot hold the request open indefinitely. This matters more than it sounds: Next.js caches a fetch only when the response status is 200, so a 503 is never cached and every retry reaches the real upstream rather than a stored failure.

partial is derived on every request, never stored:

// src/app/api/v1/search/route.ts
failedSources: outcome.failedSources,
partial: outcome.failedSources.length > 0,

This is the part I care about most. If a UI renders results without looking at partial, then every time an upstream has a bad afternoon the site presents a truncated list as though it were the complete answer, and the reader concludes there are only a handful of books about Python. That is not a rendering bug — it is a factual claim, and the code is making it.

So the UI has to say something. On our own search page a partial result set gets an explicit banner naming the sources that did not answer, rather than a smaller grid with no explanation. The same contract is documented for API consumers on /developers/api.

Nineteen upstream calls per request forces a real rate limit

The cost model is unforgiving: one API request from us is nineteen requests to somebody else's server. Expose a public API on top of that without a quota and you have turned yourself into a load generator aimed at twelve services you do not control.

The first version of the limiter was a Map of timestamp arrays in process memory (src/lib/rateLimit.ts):

const buckets = new Map<string, number[]>();

It works fine on a single long-lived Node process. On serverless it is decoration. Every cold start gets a fresh empty map, and traffic spread across several instances gives every instance its own private bucket. A caller who exceeds the limit on instance A simply lands on instance B and starts again. You cannot cap a public API with process-local state.

The fix is to keep the counters in Postgres, where every instance sees the same row. One round trip does the counting and the check together:

-- supabase/migrations/012_public_api.sql
insert into public.api_rate_limit as r (bucket, window_started_at, hits)
values (p_bucket, v_now, 1)
on conflict (bucket) do update
  set hits = case
               when r.window_started_at < v_window then 1
               else r.hits + 1
             end,
      window_started_at = case
               when r.window_started_at < v_window then v_now
               else r.window_started_at
             end
returning r.hits, r.window_started_at into v_hits, v_started;

The on conflict ... do update is the load-bearing part. It takes a row lock on the bucket for the duration of the statement, so two concurrent requests cannot both read the old count and both conclude they are under the limit. The naive version — select the count, compare, then update — has a read-modify-write race exactly when it matters most.

Two details around it are easy to get wrong. First, the function is security definer, and Postgres functions are executable by PUBLIC by default; without an explicit revoke, anyone holding the anon key could call it with a limit of a billion and reset everyone's counters. We revoke execute from public, anon and authenticated, granting it to service_role only. Second, it is created with set search_path = public, so a caller who can create objects in a schema they control cannot hijack name resolution inside it.

We apply the per-minute and per-day windows in parallel, so the two counters cost one round trip of latency, and report the window the caller actually has to wait out — the minute window when that is the one that was exceeded, the day window when that is. And if the migration has not been applied yet, the RPC errors; rather than breaking the API outright we fall back to the per-instance limiter and report which scope answered, so the degradation is visible instead of silent.

The part I keep coming back to

A partial result set and a complete result set are different things, and they should look different at every layer above the fan-out: the function signature, the JSON response, the rendered page, and the documentation for anyone consuming the API.

An API that hides partial failure is lying, and the user cannot tell — which is exactly why the failure record has to travel all the way out to the screen instead of stopping at the function boundary.