Scraping Best Buy: the 200 OK that quietly lies to you

Quick answer

A geo-random residential exit doesn't fail loudly against bestbuy.com — it succeeds, and that's the trap. Route through a non-US IP and you get an HTTP 200 carrying a 7 KB "Best Buy International: Select your Country" splash page instead of the catalogue, which parses to zero rows and looks exactly like an empty category. The Best Buy Product & Price Scraper pins country_code="US" on every proxy configuration and adds an explicit splash-page guard that treats that 200 as a block, not a green run. Point it at a category URL and get typed price, discount, seller, and availability rows back at $2.42 per 1,000 results.

A 200 OK that quietly lies to you 🌐

Here's the failure mode that should worry anyone building scrapers against geo-sensitive retail sites, because it doesn't announce itself. We ran the same category URL through two Apify residential proxy exits, minutes apart, same build. One exit landed in the US and returned a 2.5 MB payload full of real SearchProduct JSON. The other landed somewhere else and got back a 7 KB HTML page titled "Best Buy International" asking us to pick a country. Both requests returned HTTP 200 — no 403, no CAPTCHA, no rate-limit header.

Downstream, the difference is brutal. The splash page has no SearchProduct block for the extractor to parse, so it silently produces zero rows — which reads exactly like "this category happens to be empty right now," a plausible outcome nobody double-checks. That's the real danger: an unpinned exit doesn't corrupt your data with an error you'd notice, it corrupts it with a shape that looks like a legitimate answer.

We fixed this two ways. Every proxy configuration this Actor builds pins country_code="US" regardless of what the run's proxyConfiguration input otherwise requests. And a splash-page guard fingerprints the "Select your Country" marker text in the response body and treats a match as a blocked fetch — log it, rotate the session and exit IP, retry — never hand that HTML to the parser as if it were valid data.

A local 403 told us nothing about whether the target blocks us 🧱

Before we pinned the country, we hit a second, unrelated wall: every request from our dev sandbox — robots.txt included — came back with a blanket Akamai 403. Stop there and the natural conclusion is "bestbuy.com blocks everything." That conclusion would have been wrong.

The same URLs, issued from Apify's cloud infrastructure instead of our sandbox's outbound IP, came back HTTP 200 with the full 2.4 MB category page. The wall wasn't Best Buy rejecting scrapers categorically — it was Akamai scoring our specific dev egress IP's reputation before the request ever reached logic that cared about our headers or TLS fingerprint. A local 403 tells you about your IP, not about the target's defenses.

Product pages reset the connection; category pages don't 🛒

One more asymmetry worth flagging: direct product-detail-page URLs (/product/.../sku/<id>) don't degrade gracefully the way the geo-splash does — they die with an HTTP/2 stream reset before a status code lands, independent of proxy or country. Rather than let that silently zero-row a batch, the Actor detects the /sku/ marker in a start URL, logs an ERROR, and skips it loudly. Supply category browse URLs (/site/<slug>/<slug>/<categoryId>.c?id=<categoryId>) and the Actor walks pagination, emitting one row per SKU.

What we handle for you 🛡️

• We pin the proxy exit to the US on every run, regardless of what the caller's proxy input otherwise requests — this target is geo-gated, and a random exit produces plausible-looking nothing instead of an error.
• We guard against the geo-splash specifically — a 200 response carrying the "Select your Country" marker is logged, session-rotated, and retried, never parsed as real data.
• We rotate browser fingerprints — curl-cffi impersonating Chrome, Firefox, and Safari TLS handshakes.
• We retry with exponential backoff on 408 / 429 / 5xx, up to 5 attempts per page, honoring Retry-After.
• We fail loud on zero rows — the Actor exits non-zero with a clear status message instead of finishing green with an empty dataset.
• We keep the dataset clean — Pydantic-validated rows, stable sku_id keys, ISO-8601 timestamps.
• You pay only for results that land. No data, no charge, beyond the small run-start warm-up fee.

Full output schema 📦

Nineteen fields per SKU:

Field	Type	Notes
sku_id	string	Best Buy SKU identifier
product_name	string	Short product name
model_number	string | null	Manufacturer model number
product_url	string	Canonical product detail page URL
category_hint	string \	null
condition	string	Product condition, e.g. new
open_box_condition	string | null	Open-box condition grade, if applicable
current_price	float	Current customer-facing USD price
regular_price	float | null	Displayable regular (pre-discount) USD price
discount_percent	float | null	Percent off regular price — sourced from the page, or computed when absent
savings_amount	float | null	Absolute USD savings versus regular price
seller_type	"best_buy" \	"marketplace"
seller_id	string | null	Seller identifier
availability_status	string	Fulfillment button text, e.g. Add to Cart or Sold Out
in_store_pickup_eligible	bool | null	Whether in-store pickup is available for this SKU
rating	float | null	Average customer star rating
review_count	int | null	Number of customer reviews
upc	string | null	Always null in v1 — UPC is absent from Best Buy's wire format
captured_at	string	ISO 8601 UTC scrape-time timestamp (not page-sourced)

Who this is for

Price-drop monitoring — schedule repeat runs and diff current_price / discount_percent across them to catch deals the moment they land.

Competitor price intelligence — track how a category is priced over time against your own catalogue.

Marketplace-vs-Best Buy mix analysis — use seller_type to see which listings are Best Buy-fulfilled (1P) versus third-party marketplace (3P).

Stock/availability alerting — diff availability_status and in_store_pickup_eligible across runs for restock notifications.

Frequently asked questions

Why would a run return zero rows against a non-US proxy exit?
A non-US exit lands on a "Select your Country" splash page instead of the catalogue — HTTP 200, not a block signal, so it parses to nothing. The Actor pins country_code="US" and detects the splash marker explicitly, treating it as a blocked fetch rather than valid HTML.

Can I pass a direct product-page URL instead of a category page?
No. Best Buy's product-detail pages reset the connection on direct fetch, independent of proxy or country — the Actor detects the /sku/ marker, logs an ERROR, and skips that URL. Supply a category browse URL instead.

Does it cover bestbuy.ca or other regional storefronts?
No — bestbuy.com (US) only in v1.

Why is upc always null?
Best Buy's underlying product payload doesn't expose a UPC field on the pages this Actor reads. It's always null rather than a guess.

What does 1,000 results cost, exactly?
$2.42 — $0.02 run-start plus 1,000 × $0.0024 per result.

Can I search by keyword instead of a category URL?
No — Best Buy's free-text keyword search path is disallowed by robots.txt, so the Actor deliberately never uses it. Supply category or product URLs instead.

Try it

Live on the Apify Store: Best Buy Product & Price Scraper.

Point it at one or more category browse URLs and get back typed, SKU-keyed rows with price, discount, seller, and availability — pinned to a real US exit, guarded against the splash page that fools naive scrapers. Pay-per-event: $0.02 per run plus $0.0024 per result, no subscription, no card required to try.

---

Built by Devil Scrapes — we build scrapers for the targets that fight back.