Three Walmart scraping jobs in Python: product data by URL, ranked search results by keyword, and review bodies by the same product URL. Every request here ran on 27 July 2026 against the Chocodata API, the screenshots are that output, and the field counts are what actually came back rather than what a docs page promises.
Why is it hard to scrape Walmart?
Walmart fronts its pages with a press-and-hold challenge for datacenter traffic, and the challenge is served with HTTP 200, so a scraper branching on r.status_code == 200 logs a clean run while writing empty rows. The product fields live in an embedded JSON blob rather than in HTML attributes, so selectors written against the rendered markup rot on a schedule nobody controls. The costliest layer is the silent one: the 200-with-challenge wastes more debugging time than the JSON reshuffles and the sponsored rows combined.
Prerequisites
Python 3.9 or newer and requests. No browser driver, no proxy pool, no account beyond a free key.
1. A free Chocodata API key. Free, no card, and usable the moment you land on the dashboard.
2. requests in a clean environment.
python -m venv .venv && . .venv/bin/activate
pip install requests
3. The Walmart URL you want. One /ip/ link covers both the product data and the reviews jobs. Tested with Python 3.13.7 and requests 2.34.2 in July 2026. Any language with an HTTP client works identically, so Node, Go and PHP need no special handling.
Scrape Walmart product data
To scrape Walmart product data, send the /ip/ URL as url and read the ten grouped objects that come back.
1. Send the product URL and inspect the shape
Print the top-level keys first, because the payload groups its fields rather than flattening them.
import time
import requests
BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
def call(endpoint, tries=3, **params):
"""One GET, retried, so a batch run survives a dropped response."""
params["api_key"] = API_KEY
for attempt in range(tries):
r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
if r.ok:
break
time.sleep(2)
r.raise_for_status()
data = r.json()
if not data:
raise RuntimeError(f"{endpoint}: empty payload for {params}")
return data
PRODUCT = "https://www.walmart.com/ip/10450114"
product = call("walmart/product", url=PRODUCT)
print(list(product))
print(product["general"]["title"])
print(product["price"]["price"], product["price"]["currency"])
['general', 'price', 'rating', 'seller', 'fulfillment', 'breadcrumbs', 'specifications', 'us_item_id', 'product_id', 'model']
Great Value Whole Vitamin D Milk, Gallon
3.73 USD
HTTP 200 in 4.10 s. There is no envelope, so product["general"] is the first hop and not product["data"]["general"].
2. Flatten the grouped objects into one row
Flatten once in a function so the nesting stops leaking into every call site downstream.
def flatten(p):
return {"us_item_id": p["us_item_id"],
"title": p["general"]["title"],
"brand": p["general"]["brand"],
"gtin": p["general"]["meta"]["gtin"],
"price": p["price"]["price"],
"currency": p["price"]["currency"],
"rating": p["rating"]["rating"],
"rating_count": p["rating"]["count"],
"seller": p["seller"]["name"],
"out_of_stock": p["fulfillment"]["out_of_stock"],
"category": " > ".join(b["category_name"] for b in p["breadcrumbs"])}
for key, value in flatten(product).items():
print(f"{key:<14} {value}")
us_item_id 10450114
title Great Value Whole Vitamin D Milk, Gallon
brand Great Value
gtin 078742351865
price 3.73
currency USD
rating 4.6
rating_count 341232
seller Walmart.com
out_of_stock False
category Food > Dairy & Eggs > Milk > Dairy Milk > Whole Milk
breadcrumbs collapses to a clean category path, which is the best category signal in the payload.
3. Convert the specifications list into a dict
Specifications arrive as a list of {"key": ..., "value": ...} objects, so build the lookup before reading anything out of it.
specs = {s["key"]: s["value"] for s in product["specifications"]}
print(len(specs), "specifications")
for key in ("Milk type", "Shelf life", "Container type", "Food preparation method"):
print(f" {key}: {specs[key]}")
13 specifications
Milk type: Cow Milk
Shelf life: 15 days
Container type: Jug
Food preparation method: Pasteurized
Specification keys are per-category, so a mixed catalogue needs specs.get(...) rather than fixed column names. Search takes a keyword instead of a URL.
Scrape Walmart search results
To scrape Walmart search results, send the keyword as query and read results plus related_queries off the top level.
1. Send the keyword and size the response
Check the row count and the per-row field count before writing a schema against either.
search = call("walmart/search", query="wireless mouse")
print(search["query"], "| page", search["page"], "| rows", len(search["results"]))
print(len(search["results"][0]), "fields per row |",
len(search["related_queries"]), "related queries")
wireless mouse | page 1 | rows 61
22 fields per row | 10 related queries
HTTP 200 in 4.86 s.
2. Filter the sponsored rows out before ranking
Derive rank from the organic rows only, because position counts ads and will overstate every placement you track.
rows = search["results"]
organic = [r for r in rows if not r["sponsored"]]
print(len(rows) - len(organic), "sponsored |", len(organic), "organic")
for rank, r in enumerate(organic[:5], 1):
print(f"organic {rank} page pos {r['position']:>3} "
f"{r['rating']} {r['reviews_count']:>6} {r['title'][:40]}")
21 sponsored | 40 organic
organic 1 page pos 4 4.5 5161 Logitech Silent Wireless Mouse, Blue/Gra
organic 2 page pos 5 4.6 4736 Logitech Compact Wireless Mouse, Blue, W
organic 3 page pos 18 4.5 1584 Wireless Mouse, 2.4GHz with USB Receiver
organic 4 page pos 19 4.7 183 onn Silent Wireless Mouse with 5 Buttons
organic 5 page pos 22 4.5 701 memzuoix Wireless Mouse for Laptop, 5 Bu
Organic 3 sits at page position 18. Storing position as rank would have been wrong by fifteen places on that one row.
3. Harvest related queries and the seller mix
Two free signals ride along in the same response, so pull them rather than paying for a second call.
from collections import Counter
print([q["query"] for q in search["related_queries"]][:5])
print(Counter(r["seller"] for r in rows).most_common(4))
['mouse pad', 'wireless mouse and keyboard', 'wireless mouse logitech', 'laptop holder', 'wireless gaming mouse']
[('Walmart.com', 23), ('Number 8 Store', 4), ('Halokin', 4), ('VIPLIVE', 3)]
Walmart.com held 23 of the 61 rows and the rest were spread thinly across marketplace sellers. Reviews are the third job and the one that returns prose.
Pull Walmart review bodies
To scrape Walmart reviews, send the same /ip/ URL to the reviews endpoint and read both the distribution and the review bodies.
1. Send the product URL and read the distribution
Take the distribution first, since it describes all the reviews while the array describes ten of them.
reviews = call("walmart/reviews", url=PRODUCT)
print("overall", reviews["overall_rating"], "|", reviews["total_reviews"], "reviews")
print(reviews["rating_distribution"])
print(len(reviews["reviews"]), "returned |",
sum(1 for r in reviews["reviews"] if r.get("text")), "with a body")
overall 4.6 | 343892 reviews
{'1': 19375, '2': 5331, '3': 9985, '4': 26701, '5': 282500}
10 returned | 10 with a body
HTTP 200 in 2.88 s. The distribution keys are strings, not ints, so dist["5"] and not dist[5].
2. Read the bodies with their metadata
Each review carries ten fields, and the body is the one the other endpoints cannot give you.
for r in reviews["reviews"][:3]:
print(f"{r['rating']}* {r['review_date']} {r['reviewer_name']} "
f"+{r['positive_feedback']}/-{r['negative_feedback']}")
print(f" {r['title']}")
print(f" {r['text'][:110]}")
5* 11/22/2025 Melissa +3/-0
My Milky Experience
I have been buying great value brand milk from Walmart now for over 20yrs and I have never once gotten a gallo
5* 11/22/2025 Ivan +4/-0
The Absolute Great Value of Vitamin D Whole Milk
Whole Vitamin D Milk is Absolutely Essential for Adults But Even Better For The Grandkids! And We All like It
3* 12/3/2025 Larry +4/-0
Walmart Milk
It always tastes a little bitter for what is supposed to be sweet milk. The lipase activity is very high. I am
Bodies averaged 547 characters and the longest ran to 1,336. All ten were verified_purchase.
3. Count themes across the negative bodies
Filter to the low ratings and count term hits, which is the cheapest useful thing to do with review text.
negative = [r for r in reviews["reviews"] if r["rating"] <= 2]
print(len(negative), "of", len(reviews["reviews"]), "returned reviews are 1-2 star")
for term in ("spoil", "sour", "expiration", "leak"):
hits = [r for r in negative if term in r["text"].lower()]
print(f"{term:<12} {len(hits)}")
7 of 10 returned reviews are 1-2 star
spoil 2
sour 2
expiration 4
leak 1
Four of the seven negative bodies mention the expiration date, which is a product complaint you cannot see in a 4.6 star average. Read the ratio from the distribution and the themes from the bodies, never the other way round.
The part that breaks
Nothing is at the top level except the identifiers. Reaching for a flat field is the first failure.
product["title"]
Traceback (most recent call last):
File "walmart_product.py", line 79, in <module>
print(product["title"])
~~~~~~~^^^^^^^^^
KeyError: 'title'
The title is at product["general"]["title"]. Only us_item_id, product_id and model sit flat.
model is null on plenty of products. It came back as None for this grocery item, so a NOT NULL column on it will reject rows.
The ten returned reviews are not a random sample. The distribution says 82.1% five star, and the ten returned were 7 one-star, 2 five-star and 1 three-star. Compute ratios from rating_distribution and use the bodies for qualitative themes only. Averaging the ten would have produced 2.0 against a real 4.6.
Review titles are optional, bodies are not. Seven of the ten carried a title and all ten carried text, so guard the title and rely on the body.
Search page 2 overlaps page 1. Requesting both pages for the same keyword returned 111 rows containing 90 unique ids, so 21 rows came back twice. Deduplicate on id before ranking.
page 1: 61 rows | page 2: 50 rows
combined: 111 rows, 90 unique ids, 21 repeats
The sponsored share is not a fluke of one keyword. "coffee maker" returned the same shape, 61 rows with 21 sponsored and the first organic row again at page position 4.
Full script
"""Scrape Walmart product data, search results and review bodies.
Tested on Python 3.13.7 with requests 2.34.2, July 2026.
"""
import csv
import time
import requests
from collections import Counter
from datetime import date
BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
FIELDS = ["scraped_on", "us_item_id", "title", "brand", "gtin", "price",
"currency", "rating", "rating_count", "seller", "out_of_stock",
"category"]
def call(endpoint, tries=3, **params):
"""One GET, retried, so a batch run survives a dropped response."""
params["api_key"] = API_KEY
for attempt in range(tries):
r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
if r.ok:
break
time.sleep(2)
r.raise_for_status()
data = r.json()
if not data:
raise RuntimeError(f"{endpoint}: empty payload for {params}")
return data
def flatten(p):
return {"scraped_on": date.today().isoformat(),
"us_item_id": p["us_item_id"],
"title": p["general"]["title"],
"brand": p["general"]["brand"],
"gtin": p["general"]["meta"]["gtin"],
"price": p["price"]["price"],
"currency": p["price"]["currency"],
"rating": p["rating"]["rating"],
"rating_count": p["rating"]["count"],
"seller": p["seller"]["name"],
"out_of_stock": p["fulfillment"]["out_of_stock"],
"category": " > ".join(b["category_name"] for b in p["breadcrumbs"])}
def product(url):
p = call("walmart/product", url=url)
row = flatten(p)
row["specifications"] = {s["key"]: s["value"] for s in p["specifications"]}
return row
def organic_results(query, pages=1):
seen, rows = set(), []
for page in range(1, pages + 1):
data = call("walmart/search", query=query, page=page)
for r in data["results"]:
if r["sponsored"] or r["id"] in seen:
continue
seen.add(r["id"])
rows.append({"rank": len(rows) + 1,
"page_position": r["position"],
"id": r["id"],
"title": r["title"],
"rating": r.get("rating"),
"reviews_count": r.get("reviews_count"),
"seller": r["seller"]})
return rows
def review_themes(url, terms=("spoil", "sour", "expiration", "leak")):
data = call("walmart/reviews", url=url)
bodies = [r for r in data["reviews"] if r.get("text")]
negative = [r for r in bodies if r["rating"] <= 2]
return {"overall": data["overall_rating"],
"total": data["total_reviews"],
"distribution": data["rating_distribution"],
"returned": len(data["reviews"]),
"with_body": len(bodies),
"negative_returned": len(negative),
"themes": {t: sum(1 for r in negative if t in r["text"].lower())
for t in terms}}
if __name__ == "__main__":
URL = "https://www.walmart.com/ip/10450114"
row = product(URL)
print(row["title"], "|", row["price"], row["currency"], "|", row["rating"])
print(len(row["specifications"]), "specifications |", row["category"])
with open("walmart_products.csv", "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=FIELDS)
w.writeheader()
w.writerow({k: row[k] for k in FIELDS})
organic = organic_results("wireless mouse")
print(len(organic), "organic rows | top at page position",
organic[0]["page_position"], "|", organic[0]["title"][:40])
print(Counter(r["seller"] for r in organic).most_common(3))
themes = review_themes(URL)
print(themes["returned"], "reviews,", themes["with_body"], "with a body,",
themes["negative_returned"], "negative")
print(themes["themes"])
Great Value Whole Vitamin D Milk, Gallon | 3.73 USD | 4.6
13 specifications | Food > Dairy & Eggs > Milk > Dairy Milk > Whole Milk
40 organic rows | top at page position 4 | Logitech Silent Wireless Mouse, Blue/Gra
[('Walmart.com', 14), ('Halokin', 2), ('Number 8 Store', 2)]
10 reviews, 10 with a body, 7 negative
{'spoil': 2, 'sour': 2, 'expiration': 4, 'leak': 1}
Summary
All three Walmart jobs are a single GET each: a product URL returned ten grouped objects and thirteen specifications in 4.10 s, a keyword returned 61 rows in 4.86 s, and the same product URL returned ten complete review bodies plus the full star distribution in 2.88 s. What the response will not do is arrive flat or arrive representative, since the product payload nests everything under general, price and rating, and the ten returned reviews skewed 7 to 1 negative against a distribution that is 82.1% five star. Take one habit from this into your own build and make it computing ratios from rating_distribution and rank from the non-sponsored rows, because both raw counts look usable and both are wrong.
FAQ
Is scraping Walmart legal?
Public product pages are not automatically illegal to scrape, but Walmart's terms of use restrict automated collection, which makes it a contract question rather than a criminal one. Not legal advice.
Why does product["title"] raise a KeyError on the Walmart payload?
Because the payload is grouped, so the title lives at product["general"]["title"] while only us_item_id, product_id and model sit at the top level.
Can I average the returned Walmart reviews to get a product score?
No, the returned ten skewed heavily negative here, so read the score from rating_distribution and use the bodies for themes.
Top comments (0)