Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows

Quick answer

Our new TCGPlayer Card Prices scraper passed cloud QA, delivered rows, and would have overcharged every customer by 22%. The deep run that caught it took four minutes and one line of analysis: count distinct IDs, not rows.

Searching charizard at the Actor's own default depth returned 50 rows carrying 39 distinct product IDs. Eleven products were billed twice. Nothing failed, nothing logged an error, and the row count looked exactly right.

Why did pagination produce duplicates at all?

Because TCGPlayer's search endpoint is relevance-sorted, and relevance is not a stable ordering.

The API pages with a plain from/size offset — ask for from=0&size=10, then from=10&size=10, and so on. That contract only holds if the underlying result list stays fixed between requests. On a relevance-ranked search it doesn't: scores shift slightly between calls, items move across the page boundary, and a product that sat at position 10 on your first request sits at position 11 on your second — so you fetch it twice and never see whatever got pushed to 9.

This is the quiet failure mode of offset pagination generally. It is not specific to TCGPlayer, and it is not a bug on their side — it is what happens when you page through a ranked list with an offset instead of a cursor.

The fix is unglamorous: track the IDs you have already emitted, skip repeats, and keep paging until you have the number of distinct items the caller asked for.

before:  50 rows, 39 distinct productIds
after:  100 rows, 100 distinct productIds   (2 queries x 50)

Why "it returned rows" is the wrong thing to check

Because every cheap signal agrees with you when you are wrong.

Our QA harness clamps depth to keep smoke tests cheap — it rewrote maxResultsQuery: 50 down to 3 and said so, in an explicit warning: "the run therefore proves page one only." Three rows come off one page. One page cannot exhibit page-overlap. So the duplicate bug was, by construction, invisible to the test that passed.

That is the general shape worth taking away: a test that clamps the parameter under test proves the thing you were not worried about. The warning was right there in the output and it was advisory — which is a polite way of saying nobody had to act on it.

We also found the same class of problem in a second Actor the same morning: 200 rows carrying 127 distinct filing IDs, because it emitted one row per filing-debtor pair while its documentation promised one row per filing. Same tell, entirely different cause — one was a pagination defect, the other a contract mismatch between the code and its own README. Counting distinct IDs surfaced both.

So the audit is now a script rather than a habit, and it checks two things on a real full-depth run:

1. Distinct identities vs. row count. Duplicates mean either a pagination defect or an undocumented row granularity. Both are worth knowing before a customer finds out.
2. Fields that are empty in every single row. A column that never populates is a promise the product does not keep. We caught one of those the same day: an advertised supplier-website field that was null on 150 of 150 rows.

Under pay-per-event pricing this is not just a data-quality question. Every duplicate row is a billed event. A 22% duplicate rate is a 22% overcharge, and it would have been entirely invisible on our side — the dashboards would have shown a healthy Actor with good volume.

What the Actor gives you

One row per card product matching your search terms, across every game on TCGPlayer — Magic, Pokémon, Yu-Gi-Oh and the rest. Each row carries product_id, product_name, product_line, set_name, set_code, rarity, market_price, lowest_price, lowest_price_with_shipping, total_listings, foil_only and a product_url. Optionally, per-seller listings for the current offers on a product.

Multiple search queries in one run, deduplicated by product ID across pages — which, as above, is the entire point.

Honest limitations 🚧

Search-based: you get the products a query matches, not a full set catalogue dump. Per-seller listings are off by default because they multiply run time and row count — turn them on deliberately. Prices are what TCGPlayer's marketplace reports at fetch time, so for a volatile card, a row is a timestamp and not a standing quote. And a query that genuinely matches nothing succeeds with zero rows, which is the correct answer rather than a fault.

FAQ

Do I need a TCGPlayer API key?
No. No key, no login, no seller account.

What does it cost?
$3.20 per 1,000 products — a $0.20 start fee plus $0.003 per product row. You pay for distinct products, not for our pagination mistakes.

Can I track a card's price over time?
Yes. Run the same queries on a schedule and diff on product_id; market_price and lowest_price are the fields that move.

Which games are covered?
Every product line TCGPlayer carries — the product_line field tells you which one each row came from.