DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Mercari Sold Listings: When 200 OK And 403 Both Lie To You

Mercari sold-comps scraper: "sold" isn't a field, "200 OK" isn't a guarantee, and "403" isn't always a block.

Quick answer

Mercari has five item statuses — on_sale, trading, sold_out, stop, cancel — and none of them is called sold. A "sold" comp is actually status in {trading, sold_out}, and trading means the item is mid-escrow, not that it's still buyable, which is the opposite of what the name suggests. Layer onto that a Cloudflare managed challenge that can arrive with a plain 200 OK status, and a 403 that's sometimes a legitimate "this listing was purged" signal rather than a block — and a naive client either serves stale live listings as "sold" comps, or throws away real data because it treated every non-200 the same way.

Why isn't there a sold: true field on the item?

Mercari's category pages embed a server-rendered GraphQL cache in __NEXT_DATA__ — the same props.pageProps.serverState idiom as other Next.js storefronts — but the item entity's own status field is a five-way enum, not a boolean:

SOLD_STATUSES = frozenset({"trading", "sold_out"})

def status_to_sold(status: str) -> bool:
    """`sold` is derived: status is trading or sold_out."""
    return status in SOLD_STATUSES
Enter fullscreen mode Exit fullscreen mode

trading is the interesting one to get wrong. It means a buyer has committed and the sale is in escrow — not "still on sale," but also not a closed, final sale price the way sold_out is. A reseller pricing tool that only counts sold_out will under-count recent comps; one that treats trading as "still buyable" will double-list an item that's actually gone. We fold both into sold=True, matching what Mercari's own UI calls "Sold."

Why does itemsList break a parser copied from a similar site?

The search entry under ROOT_QUERY looks like a typical Apollo GraphQL connection at first glance, but it isn't one:

# Critical difference: the search entry's itemsList is a flat list of
# refs, not an edges/node connection — callers must never add a
# .get("node") hop here.
def resolve_search_entry(server_state, *, keyword):
    ...
    return entry  # {"count": int, "itemsList": [ref, ref, ...]}
Enter fullscreen mode Exit fullscreen mode

itemsList is a bare list of {"__ref": ...} pointers — no edges, no node wrapper. That's a real trap because it's exactly the shape a different site's Apollo-cache parser (we maintain more than one) would have — copy that pattern verbatim here and every item resolves to an empty dict.

Why does a 200 OK page sometimes contain zero items?

Cloudflare's bot-management challenge doesn't always answer with 403 or 503. It can return the interstitial page with a plain 200 status, so checking status_code == 200 before inspecting the body is backwards:

# Challenge is checked before the 200 fast-path: Cloudflare's managed
# challenge can itself arrive with status 200 (interstitial HTML), not
# just 403/503 — the body-marker check must run regardless of status.
def _is_challenge(status, headers, body):
    return headers.get("cf-mitigated") == "challenge" or "Just a moment" in body
Enter fullscreen mode Exit fullscreen mode

We check the cf-mitigated header and the "Just a moment" body marker on every response, 200 included, before ever trusting the page. A block that hides behind a 200 is worse than one that announces itself with a 403 — it looks like a successful, empty search instead of a failure.

Why does a 403 on an item page sometimes mean nothing is wrong?

Item detail pages can legitimately 403 when Mercari has purged a listing's history — a real, expected outcome, not a block:

ITEM_PURGED_STATUS = 403
ITEM_PURGED_ERROR_CODE = "BundleNotAvailableException"

def _is_item_purged(status, body):
    if status != ITEM_PURGED_STATUS:
        return False
    payload = json.loads(body)
    return payload.get("gqlErrorCode") == ITEM_PURGED_ERROR_CODE
Enter fullscreen mode Exit fullscreen mode

That check runs before the challenge check and before any retry logic, because retrying a purged item forever would just burn attempts on something that will never succeed. The base listing row still gets emitted — only the optional per-item enrichment (exact listing/sold date, seller rating) is skipped, with a warning logged, never a failed run.

This is also why the retry-status set here is deliberately narrower than on other Actors we've built against similar Next.js targets: 408 / 429 / 503 retry, but 403 and 500/502/504 don't — a 403 either means "purged" (handled above) or a real block that a plain retry won't fix.

What does sold_date actually measure?

It's the item detail page's updated timestamp — a Unix epoch integer we convert to ISO 8601 — used as a proxy for "when this sold." It's a last-status-change timestamp, not a guaranteed instant of sale, and the code says so explicitly rather than overselling it:

sold_date=patch.sold_date if (patch and sold) else None,
Enter fullscreen mode Exit fullscreen mode

We only attach it when the row's own derived sold is True, even if the enrichment fetch succeeded — an enrichment patch never overrides the base row's own sold state.

FAQ

Does this scrape active listings, sold comps, or both?
Both — set soldOnly for Mercari's own sold/completed facet, optionally narrowed with soldWithinDays, or leave it off for everything currently listed.

Do prices come back in cents or dollars?
Mercari's wire format is USD cents; we convert to dollars (590059.0) before it reaches your dataset.

Can I filter by condition or price band?
Yes, but as a post-filter applied to already-fetched rows — Mercari's search endpoint has no confirmed server-side parameter for either, so filtering client-side is the honest approach rather than guessing at a param that might silently be ignored.

What happens if one department (category) gets blocked mid-run?
That department is skipped with a logged warning; the other 16 keep scraping. A single Cloudflare wall doesn't zero out a multi-category run.


Packaged and ready to run: Mercari US Sold Listings Scraper — keyword search across all 17 Mercari departments with a first-class sold/completed filter, price, condition, brand, seller and photos per row, plus optional per-item listing/sold-date and seller-rating enrichment.

We do the dirty work so your dataset stays clean. 😈

Top comments (0)