DEV Community

Devil Scrapes
Devil Scrapes

Posted on

No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST

Quick answer: Whatnot's GraphQL endpoint has no persisted-query allowlist and no introspection. It parses and executes whatever query text you send, validated against the live schema. So you can't ask the server what fields exist — but once you know, you can ask for exactly those, with no cookies, no browser and no session. Two of the three queries this Actor uses are not shipped to client JavaScript at all.

If introspection is off, where do the queries come from?

Two different places, because the site gets them two different ways.

GetUser was recovered by decompiling Apollo Client's own DocumentNode AST out of the site's Next.js JS chunks. Apollo compiles a GraphQL document into a structured object tree at build time; the query text is gone, but the tree still names every field, argument and type. Walk the AST and the query reconstitutes exactly.

The other two — Search and ProfileShop — aren't in the client bundle at all, because the search and shop pages render server-side. Those were built by iterative schema-error-driven construction: send a query, read the GraphQL validation error, correct, repeat. The server's own error messages are the schema documentation when introspection is disabled. It is slow and it is not guessing — every field in the final query was confirmed by the server accepting it.

All three were replayed live on 2026-09-16 through curl-cffi over a datacenter proxy, no cookies, no browser, and returned real rows.

Why does a seller lookup take two calls?

Because the inventory query keys on an opaque Relay id, and humans key on usernames.

GET_USER_QUERY = """query GetUser($username: String) {
  getUser(username: $username, excludeBanned: false) {
    id
    username
    displayName
    followerCount
    soldCount
    sellerRating { overall numReviews }
    isLive
    isVerifiedSeller
  }
}
"""
Enter fullscreen mode Exit fullscreen mode

That resolves @usernameid, which ProfileShop($userId: ID, ...) needs. The resolution call isn't pure overhead, though — it carries seller rating, review count, follower and sold counts, and live status. Those are worth having on their own, so the row you were going to pay for anyway comes back richer.

What happens when Whatnot changes its schema?

A clean GraphQL 400, which the client turns into a loud failure:

A future Whatnot schema change breaks these with a clean GraphQL 400 — see client.py's GraphQLSchemaError, which turns that into a loud failure instead of a silently empty dataset.

This is the load-bearing decision in the whole build. Pinned query text is brittle by construction — that's the trade you make for an endpoint with no allowlist. The danger isn't that it breaks; it's that it breaks quietly. A schema drift that returns data: null and gets swallowed produces a SUCCEEDED run with zero rows, which scores 100% on every health dashboard while delivering nothing and still billing the start fee.

So the failure is raised, named, and unmissable.

Why is total > 0 with items: null not an outage?

Because it's almost always the wrong operationName.

Plenty of GraphQL servers key behaviour off the operation name in the request body, not just the query text. Send the right query under the wrong name and you get a 200, a populated total, and a null collection — a response shape that reads exactly like the backend having a bad day. It isn't. Pin the operation name alongside the query:

payload = await post_graphql(
    session, GRAPHQL_URL,
    operation="GetUser", query=gql.GET_USER_QUERY,
    variables={"username": username},
)
Enter fullscreen mode Exit fullscreen mode

Operation, query and variables travel together as one unit. Separating them is how you end up debugging an outage that never happened.

Why is the module split three ways?

graphql_queries.py holds pinned text and nothing else. client.py is pure I/O — it sends a query and hands back raw JSON, with no Pydantic models and no pagination. scraper.py owns the cursor loop.

That boundary exists so the brittle part is small and obvious. When Whatnot ships a schema change, the fix is in a constants file, and the transport and pagination layers that had nothing to do with it stay untouched. A parser tangled into its own fetch loop turns every upstream change into a full rewrite.

What a row looks like

{
  "seller_username": "examplecards",
  "seller_id": "VXNlck5vZGU6MTIzNDU2",
  "is_verified_seller": true,
  "seller_rating": 4.94,
  "seller_num_reviews": 1287,
  "listing_id": "TGlzdGluZ05vZGU6OTg3NjU0",
  "title": "2023 Prizm Rookie PSA 10",
  "price": 189.99,
  "currency": "USD"
}
Enter fullscreen mode Exit fullscreen mode

The rival demand tells you what people do with this: the leading Whatnot seller scraper runs about 6 times per user per day — that is scheduled inventory monitoring, not one-off exports.


😈 Whatnot Seller Inventory Scraper pulls a Whatnot seller's full active shop inventory — listing id, title, price, currency and listing metadata — plus the seller's own rating, review count, follower and sold counts, paginated through Whatnot's Relay cursor. No login, no cookies, no browser. We handle the blocks, the retries, and the schema drift that would otherwise arrive as a silently empty dataset. $2.20 per 1,000 results.

FAQ

Does this need a Whatnot account or API key?
No. The GraphQL endpoint accepts the queries with no cookies and no session — confirmed live through a datacenter proxy.

Does Whatnot allow introspection?
No, and there's no persisted-query allowlist either. The server executes whatever query text you send, validated against its live schema. The queries here were recovered from Apollo's compiled AST and by schema-error-driven construction.

What happens if Whatnot changes its GraphQL schema?
You get a loud failure naming the schema error, not an empty dataset. A run that silently returns zero rows while reporting success is the failure mode this is built to avoid.

Why did my GraphQL call return a total but null items?
Almost always the wrong operationName. Some servers key behaviour off it independently of the query text, and the result looks like an outage. Pin the operation name with the query.

Can it scrape live-stream auctions?
No, and that's a scope decision. This is a seller's active shop inventory plus their public seller stats.

Top comments (0)