The 2,000,000-character cap that silently emptied our dataset

We capped a JSON extraction at 2 million characters. Generous, we thought. Nobody embeds more JSON than that in a web page.

Poshmark embeds 4.35 million.

🧨 The bug that returns nothing and raises nothing

Plenty of sites ship their data as a JSON blob inside the HTML — window.__INITIAL_STATE__, __NEXT_DATA__, self.__remixContext, pick your framework. You find the opening brace, walk forward counting braces until the depth returns to zero, and parse what's between.

The brace walk needs a stop condition. A malformed page, or a { inside a string you mishandled, and you scan to the end of a 5 MB document for nothing. So you add a cap:

MAX_BRACE_SCAN_CHARS = 2_000_000

Two million characters is 2 MB of pure JSON. That is an enormous amount of data to inline in a page. It felt like a cap that would never bind.

On a real Poshmark search page, __INITIAL_STATE__ is ~4.35 million characters — the JSON object isn't a fragment of the page, it's essentially the entire page, with the HTML as a thin wrapper around it. The scan hit 2M, gave up, returned None.

And here is the part that matters: that failure is silent. No exception. No 403. No parse error. The fetch succeeded, the page was real, the extractor politely returned nothing, and the run finished green with an empty dataset. Every unit test still passed, because every fixture was a trimmed-down page comfortably under the cap.

It surfaced for exactly one reason: somebody ran the actor against live Poshmark before shipping it.

🧪 Why fixtures couldn't catch this

This is the specific blind spot of fixture-based testing, and it's worth naming precisely.

A fixture is a page you saved because it was representative. But you saved it by hand, which means you probably trimmed it, and even if you didn't, you picked a page that was convenient — a small result set, a simple query. Fixtures systematically under-represent the extremes, and size limits only fail at the extremes.

Every test was green. The parser was correct. The cap was wrong, and no amount of testing the parser would ever have found it, because the cap wasn't in the parser's contract — it was a defensive constant three layers away.

The general shape: any guard you add "just in case" is untested by definition, because you added it for a case you don't have a test for. The two mitigations that actually work:

1. Run against production data before you ship. Not as a smoke test — as a size check. Log the real magnitudes: how big is the blob, how many items, how many pages.
2. Make the guard loud when it trips. Ours now logs the size it bailed at. A cap that returns None silently is indistinguishable from a page with no data, and those two need very different responses.

We raised it to 12M — real headroom over the observed 4.35M, not the "surely enough" figure we picked the first time.

🛒 What the actor does

The Poshmark Listings Scraper pulls listings by search query or category: title, brand, size, condition, price and original price, seller, likes, photos, and the listing URL — one typed row each, Pydantic-validated.

Two things worth knowing if you're building against Poshmark yourself:

Pagination is an opaque cursor, not a page number. Each response carries a max_id you pass to the next request. Don't try to synthesize it or guess offsets — chain it, and stop when it stops coming back.

Original price and current price are different fields, and both matter. For resale-arbitrage work the delta is the signal. Flattening to one price column throws away the reason you pulled the data.

It's live on the Apify Store, pay-per-result, no credit card to try it.

We do the dirty work so your dataset stays clean. 😈