I paged a table with no ORDER BY and lost 2,797 rows

I ran a verification pass over 22,313 rows, it reported 22,313 rows, and I read that number as proof it had checked all of them. It had checked about 19,500 and looked at some of those twice.

The product is BlockDex, an item-level index of public shadcn registries I'm building under Kynth. Every item page can render a Live preview — an iframe of the component's own docs page — under a caption that says this is the component running, not a screenshot of it. Nothing checked that claim. preview_url was a URL shape learned once per registry from a single sample item name, accepted on status === 200 && text.length > 500, then stamped onto every item in that registry.

Two things went wrong with that. The last docs pattern tried is {origin}/r/{name}, which on a large class of registries is the registry-item JSON route — the same route the install command reads. A 4KB JSON body clears a length test, the request sent accept: 'text/html,*/*' so the */* forced nothing, and an API route sends no X-Frame-Options, so the framability check returned true and marked that registry the most embeddable kind there is. One registry framed its own JSON and Chrome rendered its JSON viewer inside the panel. Separately, a shape learned from one item name does not hold for 7,794 items, so most of the rest resolved to the registry's 404.

I fetched all 22,313 stored preview URLs. 7,382 were not component pages: 4,704 hard 404s, 1,588 JSON documents, 691 soft 404s, 399 server errors.

The count that matched

ops/verify-previews.mjs pulls its work list through a paging helper that walks a table with limit/offset — 23 pages at a thousand rows each. The query had no ORDER BY.

An offset over an unordered query is not a stable window. Postgres is free to return the rows in a different order for each page, so between page 4 and page 5 a row can shift across the offset boundary and be handed to you twice while another row slides the other way and never arrives. The totals still add up, because you asked for 23 slices of a thousand and got them.

I found it the slow way. After the pass finished and wrote its updates, shadcn-io/area-interactive still had a Live preview panel mounted over a 404. Its row was in the set the paging skipped.

// ⛔ `order=id.asc` IS LOAD BEARING. `selectAll` pages with limit/offset, and an offset over a
// query with no ORDER BY is not a stable window — Postgres is free to return the rows in a
// different order for each of the 23 pages, so a row can arrive twice and another never arrive
// at all. The first full run pulled exactly 22,313 rows, which looked like complete coverage
// and was not.
const rows = await store.selectAll(
  'blockdex_items',
  `select=id,registry_slug,name,preview_url,preview_embeddable,docs_url,status&preview_url=not.is.null&order=id.asc${filter}`,
);

The second pass with the sort key found 17,728 rows still carrying a preview URL where 14,931 were expected, and unpublished 1,603 more. A third pass agreed with the second: 16,121 of 16,125 verified, the remaining four a transient 502.

The fan-out that wasn't doing anything

The same file had a second bug I'd have shrugged at if I hadn't been staring at it. Global concurrency is 40. The real limiter is a per-host cap of 3, which is what keeps a registry running on one small deployment from getting hammered. And rows come back from PostgREST grouped by registry.

shadcn-io is 7,794 contiguous rows on one host. So all 40 workers picked up shadcn.io items, 37 of them parked on that host's semaphore, and the other 127 hosts in the corpus sat idle. The run did 1,500 items in eleven minutes and was on course for about three hours.

const byHost = new Map();
for (const row of rows) {
  const host = URL.parse?.(row.preview_url)?.host ?? row.registry_slug;
  if (!byHost.has(host)) byHost.set(host, []);
  byHost.get(host).push(row);
}
const queues = [...byHost.values()];
const work = [];
for (let i = 0; work.length < rows.length; i++) {
  for (const q of queues) if (i < q.length) work.push(q[i]);
}

Round-robin by host, then hand that array to the pool. Wall clock becomes the slowest single host instead of the sum of all of them.

What counts as a missing page

The check itself, in ops/lib/preview.mjs, is deliberately narrow. My first version read the first 4KB of the body looking for not-found markers, and on a 400-URL sample it condemned two real, rendering component pages that happened to carry the string 404 in a nav and an inlined script. Deleting a working item from the index is the worse outcome, because a 404 that slips through is visible on the page and a missing item is not. So the test only reads <title>, which is the one place a framework's not-found page reliably announces itself — fumadocs writes "Component not found", Next writes "404".

A failure clears docs_url alongside preview_url, too. Downgrading a dead frame to a "See it running" link is the same claim in smaller type.

Final state: 11,502 items carry a verified Live preview, 2,369 a verified link, 45,343 neither, 8,985 unpublished. The order=id.asc has a comment sitting on top of it, because the version without it doesn't fail — it returns a number that matches the table.