DEV Community

Expedia Cars GraphQL Inspector: How I Turned Expedia's Network Traffic into Instant CSV Exports

Hamza on July 01, 2026

Inside Expedia's GraphQL: Building a Tool That Captures 700+ Car Rentals in 30 Seconds. Behind every modern website is an API conversation...

Read full post

Wren Calloway • Jul 3

The replay step has a fragility you're glossing over: you reconstruct the payload from baseVariables plus a swapped cursor, but Expedia's real client almost certainly sends more than an operation name and query string. GraphQL APIs behind CDNs frequently gate on persisted-query hashes, client-version headers, and signed tokens injected by their own bundle. The moment they rotate the query, add an extensions.persistedQuery.sha256Hash, or start validating a header your hardcoded CAR_SEARCH_QUERY doesn't carry, every replayed page 2+ silently returns an error or an empty edges array — while your captured page 1 still works, so it looks like it succeeded. That's the failure mode worth designing around, not the 429.

Concretely: instead of rebuilding the payload from a stored query constant, capture the actual request options from the intercepted first-page fetch (args[1] in your monkey-patch — body, headers, credentials) and reuse them verbatim, only mutating the cursor inside the parsed variables. Replay what the page really sent, not what you think it sent. It also sidesteps the whole "same authentication cookies" hand-wave, because you're not reasoning about which headers matter — you're copying all of them.

Hamza • Jul 3 • Edited

Thanks for the thoughtful write-up @wrencalloway.
Your general concern is valid, any script that reconstructs GraphQL payloads from hardcoded constants is fragile. But in this case, the actual script already follows the exact approach that you are recommending.

The article you read is a simplified conceptual walkthrough, not the production code. The real Expedia Data Extractor.js does not use CAR_SEARCH_QUERY or baseVariables. Those strings do not exist anywhere in the script.

Here is what actually happens:

The fetch hook captures the entire original request, body, variables, query string, and every single header, straight from the intercepted fetch() call:

request: structuredClone(requestBody)       // full request body verbatim
requestHeaders: captureHeaders(request, options)  // all headers, any format
variables: structuredClone(requestBody.variables) // raw variables
query: requestBody.query                     // the full GraphQL query

Then replayLoadMore deep-clones that captured request and reuses it verbatim. It only touches three pagination variables inside the already-existing structure:

const body = structuredClone(entry.request);     // clone the real captured request
const headers = { ...entry.requestHeaders };       // copy all original headers
// only modify pagination values:
setSelection(selections, "selPageIndex", nextIndex);
setSelection(selections, "searchId", entry.searchId);
setSelection(selections, "selPageCount", pageSize);

Nothing is reconstructed. Nothing is hardcoded. The script replays exactly what the page sent, which means:

If Expedia sends a persistedQuery.sha256Hash in the body, it is captured and replayed
If Expedia sends custom auth tokens or client-version headers, they are captured and replayed
The only things stripped are content-length and host, which the browser would reject or miscalculate on replay anyway

So your recommended approach of "capture the actual request options and reuse them verbatim, only mutating the pagination values" is exactly what the code does. The article just did not reflect that because it was written as a simplified explanation of the concept, not the real implementation.

By the way, full source code is available on GitHub. Free, open-source, MIT licensed.

I appreciate you taking the time to write it up though, it is always good to have extra eyes on the architecture.

Nazar Boyko • Jul 1

One thing that might save you a ban when the fast preset runs into a 429: a flat two second retry can actually dig you deeper, since the server is already asking you to slow down and you knock again at the same beat. When a Retry-After header is present, backing off to whatever it says instead of a fixed wait tends to keep the session alive a lot longer. Everything else here is solid, this is just the one edge that tends to bite after you've pointed the tool at a busy endpoint for a while.

Hamza • Jul 2

Good point! The script actually uses exponential backoff (not flat), so on fast preset it's 800ms, 1600ms, 3200ms with jitter. But you're absolutely right that we never check Retry After headers. Quick fix would be to read that header in fetchWithRetry when we get a 429 and back off to whatever it says. I'll add it. Thanks for the heads up.

Vinicius Pereira • Jul 1

Nice writeup, and keying on operationName instead of the URL is the right instinct, that alone survives a lot of what breaks naive scrapers. Since you asked for the sharp questions, here's where this bites in practice, from someone who reaches for the internal GraphQL over HTML every time.

The thing that kills these isn't the scraping logic, it's drift, and it comes in two flavors. One, the operation itself moves, CarSearchV3 becomes V4 or gets swapped for a persisted-query hash, and your interceptor just silently stops matching. Two, and this is the quieter one, the fieldMap. The moment Expedia renames pricing.total or nests it one level deeper, your recursive walker doesn't throw, it just writes a blank column, and now you've got a clean-looking CSV of 700 cars with an empty price field and no idea anything went wrong. That silent blank is worse than a crash, because a crash you actually notice. If I were hardening this I'd have it assert the shape it expects and shout when a mapped path comes back missing, instead of trusting the fieldMap forever.

The other one is the session-borrowing, which is genuinely the elegant part, no tokens to rotate is a real win. But it has a ceiling on a target as defended as Expedia. Your replayed fetch sends Content-Type and nothing else, while the real UI request carries a pile of client-info, traceparent, persisted-query and similar headers. On a heavily fingerprinted endpoint that difference is exactly what flags a replay, so the pattern that works great today can start getting soft-blocked or served degraded results with no obvious error. Cheapest insurance is to clone the original request's full header set when you replay, not just swap the cursor in the body.

None of that takes away from it, for a browser-side tool on your own session it's a clean approach. Just the two spots I'd watch for when it eventually stops working. Good first post.

Hamza • Jul 1

Thanks! I really appreciate this level of feedback. These are exactly the kinds of edge cases that make browser automation so interesting.

You’re absolutely right that long-term maintenance is usually less about the initial extraction logic and more about API drift over time. A few of the points you mentioned are things I’ve already started addressing in version 2.2:

Replay headers: the replay no longer sends only Content-Type. It now clones the original request headers as well, while excluding things like Content-Length and Host, which the browser regenerates. That makes the replay much closer to the browser’s original network behavior and noticeably more reliable.
Session borrowing:I intentionally rely on the user’s authenticated browser session instead of trying to recreate cookies or tokens. That has been one of the biggest wins in terms of simplicity and avoiding authentication headaches.
That said, I completely agree with your two bigger concerns.

Operation drift is inevitable. Right now I match on operationName because it is much more stable than URL matching, but if Expedia eventually moves to persisted queries or renames the operation entirely, the interceptor will need a smarter discovery mechanism rather than a hardcoded name.

Schema drift is probably the more dangerous failure mode. Silent blanks are much worse than exceptions because they produce exports that look believable but are wrong. I’m planning to move toward schema validation with required-field assertions so missing critical fields like pricing or supplier become visible failures instead of quietly generating incomplete CSV files.

In the longer term, I’d also like to make the extractor less Expedia-specific by separating:

network capture
replay engine
schema mapping
exporters

That way, adapting to future API changes OR even other GraphQL-backed sites, becomes mostly a mapping problem rather than a full rewrite of the pipeline.

I really appreciate you taking the time to write such a thoughtful review. Those are exactly the kinds of comments that help move a project from “works today” to “still works next year.”

Vinicius Pereira • Jul 1

Love where you're taking it, and the capture/replay/mapping/exporter split is exactly the seam that makes this survive. One idea for the operation-drift problem you flagged: instead of a smarter way to find the operation by name, discover it by shape. You already know the response you consume looks like data then edges then nodes carrying vehicleId, supplier, pricing, so match on "the response that contains that shape" rather than on the label CarSearchV3. A rename to V4 or a persisted-query hash doesn't touch the shape, so the interceptor keeps working through the exact drift that would break a name match. And it pairs nicely with the required-field assertions you mentioned, since the shape you match on and the shape you validate against are really the same declared contract, one just used to find the response and the other to trust it. Genuinely nice work, following where this goes.

Hamza • Jul 1

Thanks, that shape-matching idea is genuinely clever and I hadn't thought of it. I just checked the actual response structure in the code and it turns out Expedia doesn't use edges/nodes/vehicleId at all. The real shape is a listings[] array where each listing has things like car.vendor, car.priceSummary.total.price.amount, car.vehicle.description, and car.tripsSaveItemWrapper.tripsSaveItem.attributes for the pickup/return details. So matching on the shape by checking for the combo of priceSummary + vendor + listings would survive an operation rename to V4 or even a persisted query hash, no changes needed. And like you said, those same fields are exactly what I'd validate against anyway, so the finder and the validator would share the same contract by default. I'm going to experiment with that approach, appreciate the insight.

Hamza • Jul 1

Thanks for Reading!
If you have questions about GraphQL interception, fetch hooking, pagination, or browser automation, leave a comment. I'll be happy to help.

More developer tools are coming soon.

Frank • Jul 1

How did you handle pagination with the GraphQL API to capture 700+ car rentals in 30 seconds? I'm facing a similar issue and would love to swap ideas on this.

Hamza • Jul 1

Hi there @frank_signorini, Great question! pagination was the trickiest part. The short answer: index-based pagination where I calculate the next page offset and inject it back into the same GraphQL query.

Expedia's CarSearchV3 doesn't use cursors. The response includes a loadMoreAction.searchPagination object with startingIndex, size, and hasNextPage. Each page returns listings at a given starting index, and the next page starts at currentIndex + listingCount.

Here's the stripped-down version of how it actually works:

async function replayLoadMore(previousEntry) {
  const body = structuredClone(previousEntry.request);
  const vars = body.variables ??= {};
  const secondary = vars.secondaryCriteria ??= {};
  const selections = secondary.selections ??= [];

  const pagination = previousEntry.loadMore?.searchPagination;
  const pageSize = pagination?.size ?? 25;
  const currentIndex = pagination?.startingIndex ?? 0;
  const currentCount = previousEntry.listingCount || pageSize;
  const nextIndex = currentIndex + currentCount;

  // Set the next page index
  setSelection(selections, "selPageIndex", nextIndex);
  setSelection(selections, "selPageCount", pageSize);
  setSelection(selections, "searchId", previousEntry.searchId);

  const response = await fetch(previousEntry.url, {
    method: "POST",
    credentials: "include",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body)
  });

  const json = await response.json();

  return {
    listingCount: json.data.carSearchOrRecommendations.carSearchResults.listings.length,
    loadMore: json.data.carSearchOrRecommendations.carSearchResults.loadMoreAction,
    // store the full response too
  };
}

Then the loop:

while (page.loadMore?.searchPagination?.hasNextPage !== false) {
  const next = await replayLoadMore(page);
  // store results, update page tracker
  page = next;

  // delay so you don't overwhelm the API
  await new Promise(r => setTimeout(r, 1500));
}

A couple gotchas I ran into:

Deduplication is necessary. The same car can appear on different pages if results shift. I use a composite key of searchId + startingIndex to skip duplicates.
Speed matters. I landed on 1.5s between pages — fast enough to be useful, slow enough to avoid rate limits. Three presets (4s, 1.5s, 0.6s) let users tune it based on their connection.
Retry on failures. Random 500s happen every ~50 pages. I wrap the fetch with up to 3 retries and exponential backoff.

The key insight: you're replaying the exact same request the browser would send, just with a different page index. Your real session cookies handle auth — no tokens, no proxy, no CAPTCHA.

Happy to dive deeper! What API are you working with?