The limit that looked like a block

The report

An Instagram-profile listing (@neer_mishra21) showed the generic Instagram camera-logo
icon on the board, instead of that account's actual profile photo. The code to prefer a
social profile's own photo (og:image) over the platform's favicon already existed —
built and tested earlier the same week, specifically for this case.

Why it looked like an already-known, already-accepted limitation

This codebase already has an honest, written-down caveat for exactly this symptom: X and
Instagram are documented to serve a login wall / bot-block to a server-side fetch from an
unrecognized user agent, especially from cloud-provider IP ranges. The existing code
already has a fallback for that: if the page itself can't be fetched, try the origin's
bare /favicon.ico directly, since that's often reachable even when the HTML isn't. That
fallback getting used — and landing on Instagram's own generic app icon — was the obvious,
expected explanation. It matched a limitation that had already been explained to the
founder in an earlier conversation. Every reason to stop looking there.

That's what made it dangerous: a plausible, already-accepted explanation is often more
dangerous than no explanation at all, because it stops the investigation instead of
starting it.

How it was found

The instinct was to re-read unfurl.mjs and reason about it. The code looked right —
parseSocialUrl() correctly identifies an Instagram profile, extractOgImage() correctly
prefers og:image over the generic <link rel="icon">, and the fetch/sniff pipeline
around it hadn't changed. Reading the code a second time wasn't going to reveal anything
the first read didn't.

So instead of reasoning about the code, the destination was fetched for real —
curl -A "Kingpin-Unfurl/1" https://www.instagram.com/neer_mishra21 — using the exact
user agent this codebase sends. First surprise: HTTP 200. Not blocked at all. The
"X/Instagram blocks us" explanation, while true in general (it's still real, and still
documented), was not what was happening this time.

Grepping the real response for og:image found it immediately — the correct, real profile
photo URL, with a correct og:title too (Neer Mishra 🧿 (@neer_mishra21) • Instagram
photos and videos). So the data was there. Next, running the actual
extractOgImage()/extractTitle() functions from this codebase against that real,
saved HTML confirmed they parsed it perfectly — the regex-based extraction worked exactly
as designed. Then fetching the extracted photo URL directly confirmed that worked too —
a real 100×100 JPEG came back, HTTP 200.

Every individual piece worked when tested in isolation. That's the moment a bug review
has to stop asking "does each step work" and start asking "what's different about running
them together, in sequence, inside the real function" — because clearly something in the
combination was failing even though no single piece was.

The next check was the actual production Lambda logs for the moderation function, filtered
for the sitemeta capture event across the last few days:

"iconBytes":0        <- capture failed, no icon stored
"iconBytes":0
"iconBytes":9789     <- capture succeeded, real photo bytes
"iconBytes":2214     <- capture succeeded, real photo bytes

Inconsistent — sometimes it worked, sometimes it didn't, for the same kind of listing.
An intermittent bot-block wouldn't be a shocking explanation for that pattern by itself,
but it was worth checking one more concrete number before assuming: how big was the actual
page that had just been fetched? wc -c on the saved response: 726,841 bytes.

The mechanism

unfurl.mjs's page fetch has always been byte-capped, for a real and correct reason
(nothing here should let a slow-loris or a multi-gigabyte response wedge a Lambda):

export const HTML_MAX_BYTES = 256 * 1024; // 256KB
...
const page = await safeFetch(rawUrl, { ...deps, maxBytes: HTML_MAX_BYTES, accept: 'text/html,*/*' });

safeFetch's reader (readCapped) aborts the instant the response — either by a declared
content-length header, or by counting streamed bytes — crosses that cap, and reports
{ ok: false, reason: 'too_large' }. unfurl() treats page.ok === false as "the page
was refused," and runs the same code path it uses for an actual bot-block: try the bare
origin favicon instead, then give up on the title entirely.

Instagram's real profile page is ~727KB — modern SPA pages routinely ship a large inline
JSON/state blob in the initial HTML, well before any human-visible content. The 256KB
cap was rejecting the entire page before the code ever got to read as far as the
<meta property="og:image"> tag that was sitting right there in the document. The
result — favicon fallback, generic Instagram logo — is pixel-for-pixel identical to what
a genuine bot-block produces. There is no error, no distinguishing log line, nothing that
tells the two failure modes apart from the outside. The inconsistency in production
(0 bytes some captures, real bytes others) lines up with this too: page weight isn't
perfectly constant response to response (A/B-tested modules, cache state, logged-out
variance), so some fetches happened to land under 256KB by chance and succeeded, and most
didn't.

This is the same invisible-failure shape this codebase keeps running into (see entries
001 and 005): a defensive limit, correctly placed to guard against one failure mode (an
unbounded/malicious response), silently produces a second, unrelated-looking failure mode
(a legitimate large response) — and that second failure mode is indistinguishable, from
the outside, from a completely different explanation that already has a name. The
existing "sites can block us" narrative absorbed the symptom without anyone needing to
lie or guess; it was just the wrong absorbing explanation.

The fix

Raise the cap to something sized for what a real, modern page actually weighs, not a
generic-website guess from whenever this constant was first written:

export const HTML_MAX_BYTES = 2 * 1024 * 1024; // was 256 * 1024

This fetch only ever runs asynchronously at moderation time — never on a request path,
never anywhere near the read-path invariant that the feed/board must be served from a
materialized artifact with no live work per request. A larger cap here costs nothing on
any hot path; it only changes how much of one background HTML page this function is
willing to read before giving up. 2MB comfortably covers Instagram's ~727KB with room to
spare for other SPA-heavy sites, while still being a real, finite limit — the byte cap
itself was never wrong to have, only the specific number was stale.

A regression test now pads a fake profile page's body past the old 256KB cap (well under
the new one) with an inert block, confirming the title and og:image still get read and
extracted correctly — so a future "let's tighten this cap back down" change gets caught
immediately instead of silently reintroducing this exact bug.

What was considered and rejected

Stream-parse only the <head> instead of raising the cap. Real appeal: bound memory/time
by construction rather than picking a bigger-but-still-arbitrary number, and a page's
<head> is almost always small even when the full document is huge. Rejected for now
because it needs a real (if tiny) incremental HTML tokenizer to know where <head> ends
without just buffering everything anyway — more moving parts than this problem currently
justifies, for a background enrichment step that already tolerates total failure
gracefully. Worth reconsidering if a future site's <head> itself routinely exceeds
whatever the flat cap becomes.

Treat too_large as a distinct case from a real network refusal, with its own fallback
behavior. Also rejected for now: both cases already converge on the same reasonable
fallback (try the origin favicon, don't block the listing), and giving them different
behavior would add branching for a difference that, today, doesn't need to produce a
different outcome — just possibly a different log line for the next person debugging this
class of issue. That's a small enough win to leave for whoever hits it next, not worth
the complexity today.

The general lesson

When a system has an already-known, already-explained failure mode, treat that
explanation as a hypothesis to verify for this specific instance, not a reason to stop
looking. A safety limit that fires silently will always be misread as whatever other
unexplained failure already has a name in the team's head — because a silent limit and a
silent block produce the exact same observable output. The fix isn't to trust the limit
less; it's to make the two failure modes distinguishable (a specific log reason, a
specific metric) before the next person spends an hour re-confirming a hypothesis that
was never actually wrong in general, just wrong this one time.

And, more mechanically: when several individually-tested pieces of a pipeline all pass in
isolation but the pipeline as a whole still fails, the bug is almost always in a boundary
between two pieces, not inside either one — here, specifically, in the size of the buffer
handed from step one (fetch the page) to step two (parse it), which was invisible to any
test of parsing alone, because parsing was never given the chance to run on the real,
full-size input.

---

Originally published on the Kingpin build log.