DEV Community

Devil Scrapes
Devil Scrapes

Posted on

The price field that's in two different places on the same site

Quick answer

Walmart doesn't ship one price shape — it ships two, and which one you get depends on which page you're looking at. Product detail pages nest price at priceInfo.currentPrice.price; search-result items carry price as a plain top-level price field, with priceInfo reduced to display strings that only look like the real thing. The Walmart Product & Price Scraper handles both shapes explicitly and returns 15 typed fields per product at $0.002 per result plus a $0.02 run-start charge — $2.02 for 1,000 rows.

Search results and product pages don't share a price shape 💰

Here's the bug that taught us the most, and it's worth walking through because it's the kind of failure that doesn't show up as a failure.

Our first parser assumed one price shape for both entry points into the Actor — search-term lookups and direct product URLs. That assumption came from the product detail page, where the price genuinely does live at priceInfo.currentPrice.price, several levels deep in a nested pricing object alongside wasPrice and formatted display strings. Reasonable enough — that's the richest pricing payload Walmart exposes, so it's the one you study first.

Search results don't use it. When Walmart returns a grid of matching products, each item carries its price as a plain top-level numeric price field, sitting right next to title and the item ID. The nested priceInfo object is still present, but flattened into presentational strings meant for rendering, not for parsing as a price of record.

We shipped the PDP-shaped parser against both entry points. Every field that didn't depend on price parsed correctly for search-mode rows — title, item_id, url, currency, in_stock, rating_average, all populated exactly as expected. price_current came back null on every one of them. Fifty-eight unit tests were green the whole time, because every fixture we'd captured was a static snapshot of one shape or the other — nothing in the suite forced a search-result row through the PDP-shaped path.

That's the framing that matters here: a price scraper that returns null prices is worse than one that crashes. A crash fails loud — you see it in the run log, the Actor exits non-zero, nothing gets billed. A silent null passes every schema check, writes a syntactically valid row, and gets billed like a success. The customer gets a spreadsheet with every column filled in except the one they paid for.

We fixed the parser to branch on entry point rather than assume a shared shape, and we added a structural guard on top: raw payloads now get dumped to the key-value store on every run, and if prices come back null across an entire batch — not one weird row, the whole batch — the run fails loud instead of finishing green with an empty price column.

Walmart blocks datacenter IPs outright, so we pin residential 🌐

Walmart doesn't rate-limit datacenter traffic into submission — it blocks it outright, with a block-interstitial served in place of the page on essentially every request from a shared datacenter pool. There's no ramping up, no soft warning tier. That's why proxyConfiguration defaults to RESIDENTIAL pinned to US: we rotate through residential exit IPs and fresh session IDs on every block so requests keep landing instead of dead-ending on the first hop.

An 11-digit item ID broke our own regex 🔢

Walmart's usItemId used to be reliably 8 or 9 digits, and our first productIds validator matched that pattern and stopped there. Then we hit real Walmart item IDs with 11 digits — 15949610846 for the Nintendo Switch 2 System, confirmed via a cloud run against the live platform — and our own input validator rejected them before a single request went out. The regex now accepts 8-11 digit usItemId values: a scraper that rejects valid input is its own kind of null-price bug, technically correct and functionally useless.

What we handle for you 🛡️

  • We rotate browser fingerprintscurl-cffi impersonation across Chrome, Firefox, and Safari TLS profiles, so requests present as a real browser, not a Python client.
  • We rotate residential proxies through Apify Proxy, pinned to US, with a fresh session ID on every block.
  • We retry with exponential backoff on 408 / 429 / 5xx, up to 5 attempts per page, honoring Retry-After.
  • We branch on payload shape — search-result and product-detail price fields are parsed by the path that matches what Walmart actually sent.
  • We fail loud on all-null batches instead of shipping a green run with an empty price column; raw payloads land in the key-value store for triage.
  • You pay only for results that hit your dataset. No data, no charge, beyond the small run-start warm-up fee.

Full output schema 📦

Fifteen fields per product:

Field Type Notes
item_id string Walmart usItemId
title string Product name
brand string | null Brand name
url string Canonical /ip/{slug}/{itemId} product URL
price_current float | null Current listed USD price
price_was float \ null
currency string Always USD
in_stock bool Derived from the availability field
availability_status string | null Raw Walmart availability status string
rating_average float | null Average star rating
rating_count int | null Review count
seller_name string | null Offer's seller display name
seller_type "walmart" \ "marketplace" | null
image_url string | null Primary product image URL
search_term string | null Populated only for search-mode rows
scraped_at string ISO-8601 row-creation timestamp

Two rows we pulled off the live platform: a "Nintendo Switch w/ Neon Blue & Neon Red Joy-Con" at price_current: 339 USD, in_stock: true, and a "Nintendo Switch 2 System" at price_current: 449 USD, in_stock: true — both resolved via productIds, one of them the 11-digit ID the regex fix now accepts.

Who this is for

Reseller and arbitrage margin checks — pull current price and stock across a candidate SKU list before committing to buy.

Retail price intelligence — track price_current and price_was movement across a competitor catalog by search term or item ID over time.

Marketplace-vs-Walmart mix analysis — use seller_type to separate Walmart-fulfilled listings from third-party marketplace mark-ups at scale.

Stock monitoring — schedule re-runs and diff in_stock / availability_status for restock alerts.

Frequently asked questions

Why did some rows come back with a null price before this fix?
Because the parser assumed every product used the product-detail-page price shape (priceInfo.currentPrice.price). Search-result rows carry price as a plain top-level price field instead — a different shape entirely, not a missing value.

Does the Actor still return every other field when a price is missing?
No — an all-null price batch now fails the run loud rather than shipping a dataset with an empty price column, because a green run with no prices is worse than a run that visibly failed.

What's the difference between price_current and price_was?
price_current is the live listed price. price_was is the strike-through/list price, populated only when Walmart's page actually shows one.

What does 10,000 results cost?
$20.02 — 10,000 × $0.002, plus the $0.02 run-start charge.

Can I mix search terms, product URLs, and item IDs in one run?
Yes — provide any combination of the three input modes. Rows are de-duplicated by item_id, so you're never charged twice for the same product.

Try it

Live on the Apify Store: Walmart Product & Price Scraper.

Point it at search terms, product URLs, or item IDs and get back typed, price-verified rows — no shape assumptions, no silent nulls. Pay-per-event, no subscription.


Built by Devil Scrapes — we build scrapers for the targets that fight back.

Top comments (0)