DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Your eBay scraper is billing customers for adverts

#webscraping #python #ecommerce #parsing

Scraping eBay search results looks like a solved problem until you count the rows and find you have paid for advertising.

Sponsored cards sit in the same result grid, in the same markup, as organic listings. If you select every card on the page you will emit them, and a customer paying per result will pay for placements they did not ask for. That is the defect worth engineering against — not the blocking.

Quick answer

Three things to get right on eBay search: filter sponsored cards before you emit anything; warm up a session on a normal page before hitting /sch/, because a cold request straight to search behaves differently from one carrying real cookies; and strip the accessibility suffix — eBay appends Opens in a new window or tab to link text, and it will end up inside your titles if you take .text() naively.

Sponsored cards are the real cost bug

On a pay-per-result Actor, every emitted row is billed. A sponsored card that slips through is a row the customer pays for and did not want, and unlike a crash it never announces itself — the dataset just quietly contains ads.

They are deliberately hard to distinguish: same li.s-card container, same title and price structure, same image treatment. Two signals actually hold up, and the Actor requires neither to be pretty:

SPONSORED_TITLE = "Shop on eBay"
ITEM_ID_SHAPE_REGEX = re.compile(r"^\d{9,}$")

def _is_sponsored(title: str, item_id: str) -> bool:
    """REQ-4: exact sponsored title OR item_id failing the shape check."""
    return title == SPONSORED_TITLE or ITEM_ID_SHAPE_REGEX.match(item_id) is None
Enter fullscreen mode Exit fullscreen mode

The first is eBay's own placeholder title on promoted slots. The second is the load-bearing one: a genuine eBay item has a numeric id of nine or more digits in its /itm/ URL. Ad slots, interstitials and malformed cards do not. Testing the shape of the identifier rather than hunting for a label is the part that survives a redesign — eBay can move the "Sponsored" badge, restyle it, or render it in a shadow root, and an item id is still either a real item id or it is not.

That check runs before the row is constructed:

CARD_SELECTOR = "li.s-card"

for node in tree.css(CARD_SELECTOR):
    row = build_row(node)     # returns None for malformed/sponsored cards
    if row is None:
        continue              # never becomes a ResultRow, never billed
Enter fullscreen mode Exit fullscreen mode

Dropping them at parse time rather than filtering later matters for the same reason: a ResultRow that exists can be pushed by accident. One that was never built cannot.

Note also li.s-card itself. eBay's older search DOM used li.s-item, and a great deal of published eBay-scraping code still targets it. That selector now matches nothing on the live site — and matching nothing looks exactly like being blocked, which sends you debugging your proxy instead of your parser.

The warm-up request

Requesting https://www.ebay.com/sch/i.html?_nkw=... as the very first call in a fresh session is a recognisable pattern: no cookies, no referer, no prior navigation. Real browsers do not arrive at a search results page that way.

So the session does one warm-up request to a normal eBay page first, keeps the cookie jar, and only then issues the search:

# REQ-2: warm-up-then-search cookie flow, one session per run
session = AsyncSession(impersonate=browser_profile)
await session.get(WARMUP_URL)      # collect cookies like a browser would
await session.get(search_url)      # now the search looks like navigation
Enter fullscreen mode Exit fullscreen mode

The session is curl_cffi with browser impersonation, so the TLS and HTTP/2 fingerprints match the User-Agent being claimed. Sending a Chrome UA over Python's default TLS stack is a mismatch that is trivially detectable and is the most common reason a "working" scraper starts failing.

The accessibility suffix

eBay appends screen-reader text to result links:

Apple iPhone 13 128GB UnlockedOpens in a new window or tab
Enter fullscreen mode Exit fullscreen mode

Take .text() and that suffix is now part of the product title in every row. It is not a parsing error — the text really is in the DOM — and it survives any test whose fixture was written by copying the same broken output.

A11Y_SUFFIX = "Opens in a new window or tab"
Enter fullscreen mode Exit fullscreen mode

Strip it explicitly. This is a small thing that shows up in every eBay dataset that was not built carefully, and it is a reliable tell for how much attention a scraper received.

Classifying the buying format

eBay mixes auctions, fixed-price listings and best-offer listings in one result set, and buyers usually want one of them. The format is inferred from card content rather than assumed, and the model makes room for not knowing:

BuyingFormatLabel = Literal["auction", "fixed_price", "best_offer", "unknown"]
Enter fullscreen mode Exit fullscreen mode

unknown is deliberate. When a card does not carry a clear signal, the honest output is unknown, not a guess at fixed_price because it is the common case. A guess here is invisible: it produces a valid value that quietly breaks anyone filtering on format.

Fault isolation per card

One malformed card must not end the run. Each card is parsed inside its own boundary, and a failure drops that card and continues — the single highest-value habit across our fleet, and the most common cause of a low success rate when it is missing. A run that returns 199 of 200 rows is a good run; a run that returns nothing because item 47 had no price is a bug.

eBay Product Listings Scraper — organic eBay search results with sponsored placements filtered out, buying format classified, priced per result.

Built by Devil Scrapes. We publish the traps we hit, because the ones that return 200 OK are the expensive ones.

Top comments (0)