DEV Community

Bugrahan Saka
Bugrahan Saka

Posted on Fully Autonomous

Scraping Flipkart with Python in 2026: skip the CSS, read the page's JSON state

Copy the CSS selectors from most Flipkart scraping tutorials and you get nothing back. The ._30jeq3 price class and the ._4rR01T name class match zero cards on today's pages. Build fresh ones and they break by category: a title selector that matches 24 of 24 cards on a laptop search matches 0 of 40 on a shoe search, because Flipkart renders two card layouts with different hashed class names.

The surprising part: Flipkart barely blocks anyone. What changes is which page you get.

This post walks through three scrapers (search results, product details, reviews) that read the JSON Flipkart already ships with every page, using only requests, json, re and csv.

Not blocked, sorted by user agent

The same search URL (/search?q=laptop) sent seven ways:

Request Page JSON state Products in state
requests, default user agent 247 KB 0 KB 0
curl/8.5.0 248 KB 0 KB 0
Chrome user agent 655 KB 355 KB 24
Scrape.do, geoCode=in 655 KB 354 KB 24
Scrape.do, render=true 317 KB 0 KB 0

Every request returned a 200 and every page had 24 cards in the HTML. The difference is the data object: non-browser user agents get window.__INITIAL_STATE__ = {}, an empty object. A browser user agent gets the full state with every product, price and rating.

Two other details stand out. Rendering makes things worse: render=true returns the DOM after scripts have run, and that DOM no longer carries the state script. And volume barely mattered in testing: 100 sequential requests from one cloud IP all came back 200.

The data lives in window.__INITIAL_STATE__

Every Flipkart page assigns one JSON object to window.__INITIAL_STATE__ inside a <script> tag. Extracting it takes four lines:

import json
import re


def get_state(html):
    match = re.search(r"window\.__INITIAL_STATE__\s*=\s*", html)
    if not match:
        return {}
    state, _ = json.JSONDecoder().raw_decode(html, match.end())
    return state
Enter fullscreen mode Exit fullscreen mode

raw_decode reads exactly one JSON value starting after the = and stops, so you never need to find where the script ends. A regex that tries to capture the whole object breaks on the first }; inside a product title.

Search results: PRODUCT_SUMMARY widgets

On a search page each product is a PRODUCT_SUMMARY widget under pageDataV4.page.data. The page is a grid of rows keyed by id, so walk every row instead of hard-coding one:

def product_cards(state):
    slots = state.get("pageDataV4", {}).get("page", {}).get("data", {})
    for slot_list in slots.values():
        for slot in slot_list:
            widget = slot.get("widget", {})
            if widget.get("type") == "PRODUCT_SUMMARY":
                yield from widget.get("data", {}).get("products", [])
Enter fullscreen mode Exit fullscreen mode

That yields 24 of 24 cards on a laptop search and 40 of 40 on a shoe search with no code change. Filters and the pagination bar are widgets too; checking the type skips them.

Fetching goes through Scrape.do with geoCode=in, an Indian exit IP (Flipkart sells only in India), at 1 credit per request:

import requests

TOKEN = "<your_token>"
target = "https://www.flipkart.com/search?q=laptop&page=1"

response = requests.get(
    "https://api.scrape.do/",
    params={"token": TOKEN, "url": target, "geoCode": "in"},
)
cards = list(product_cards(get_state(response.text)))
Enter fullscreen mode Exit fullscreen mode

Three things that bite

Prices are picked by flag, not position. pricing.prices holds the struck-out MRP (strikeOff: true) and the selling price:

value = card.get("productInfo", {}).get("value", {})
prices = value.get("pricing", {}).get("prices", [])
price = next((p["value"] for p in prices if not p.get("strikeOff")), "")
mrp = next((p["value"] for p in prices if p.get("strikeOff")), "")
Enter fullscreen mode Exit fullscreen mode

The .get() chain matters: products marked "Currently unavailable" (NO_AVAILABLE_LISTING) have no pricing key at all, and value["pricing"] crashed on 5 of 100 pages in a bulk test.

Sponsored cards repeat. On page two of a laptop search, 10 of 24 cards were products already seen on page one. They carry an adInfo object; keep the first appearance of each product id.

Search stops around page 25. The page bar claimed 57 pages for "laptop"; products stopped at page 41. For "mouse", "headphones" and "shoes" they stopped at page 25. Past that, Flipkart returns a 200 with an empty results area, so stop at the first page with no cards. To get past the ceiling, split the search into price bands with Flipkart's own filter parameters:

low, high = 50000, 70000
target = (
    f"https://www.flipkart.com/search?q=laptop&page={page}"
    f"&p%5B%5D=facets.price_range.from%3D{low}"
    f"&p%5B%5D=facets.price_range.to%3D{high}"
)
Enter fullscreen mode Exit fullscreen mode

Five bands produced 979 unique laptops in 68 requests, against 539 from the unfiltered search. If a band still reaches page 25, halve it.

Product details: a different state shape

Product pages carry the same window.__INITIAL_STATE__, but built as multiWidgetState: a tree of design-system widgets whose text sits in nodes named label_0, label_1 and so on.

The numbers don't need the widget tree. Flipkart's own analytics object holds them as plain JSON:

page = state["multiWidgetState"]["pageDataResponse"]
psi = page["pageContext"]["fdpEventTracking"]["events"]["psi"]

price = psi["ppd"]["finalPrice"]
mrp = psi["ppd"]["mrp"]
rating = psi["pr"]["rating"]
stars = {r["ratingValue"]: r["ratingCount"] for r in psi["pr"]["individualRatingsCount"]}
seller_id = psi["pls"]["sellerId"]
cod = psi["pi"]["isCODAvailable"]
Enter fullscreen mode Exit fullscreen mode

The full spec table sits under widget keys starting with rpd_specifications, as label_0 (name) / label_1 (value) pairs. Collect pairs only under those keys, because the same pattern also holds offer text elsewhere on the page. That returned between 32 and 68 specs per laptop, and without a code change 66 for a phone, 31 for a saree and 23 for a pair of shoes.

Two catches worth knowing:

  • Seller and delivery wording differs by category. Electronics pages say Seller: RetailNet and Delivery by + a date; fashion pages say Fulfilled by RetailNet and Delivery + by 29 Oct, Thu. Check both.
  • The product page price may not match the search card. Open a product at its clean ?pid= URL and you may see Rs 74,990 instead of the card's Rs 79,990. Both are real offers from different sellers. The lid (listing id) parameter decides which one the page shows, so keep it when you follow URLs from search results.

Reviews: 20 pages per sort order

The reviews URL is the product URL with /p/ swapped for /product-reviews/. Each review is a value inside a REVIEWS widget, with the full text, vote counts, city and the exact variant bought.

Pages 1 to 20 return 10 reviews each; page 21 and beyond return a 200 with no reviews. That is 200 per sort order whatever the product has. The four sort orders overlap only partly, so running all of them and deduplicating by review id reaches further:

sortOrder Unique in that order Running total
MOST_HELPFUL 199 199
MOST_RECENT 200 332
POSITIVE_FIRST 196 356
NEGATIVE_FIRST 192 431

80 requests, 431 of 557 reviews (77%). Reviewer names are better left out: a name next to a city is personal data under India's DPDP Act, and it adds nothing to rating or sentiment analysis.

The packaged version

All three scrapers are in an open-source package, flipkart-scraping on GitHub and PyPI:

pip install flipkart-scraping
export SCRAPEDO_TOKEN=your_token

flipkart-scraping search "laptop" --pages 3 --format csv -o flipkart-search.csv
flipkart-scraping reviews "<product-url>" --format csv -o flipkart-reviews.csv
Enter fullscreen mode Exit fullscreen mode
from flipkart_scraping import FlipkartScraper

scraper = FlipkartScraper()
products = scraper.search("laptop", max_pages=3, min_price=50000, max_price=70000)
details = scraper.get_product(products[0]["url"])
print(details["seller"], details["spec_count"], details["delivery_by"])
Enter fullscreen mode Exit fullscreen mode

It removes sponsored repeats, keeps lid so prices match the cards, handles both seller wordings and runs all four review orders.

Before you run it at scale

  • The state is undocumented. It can change without notice. The scripts fail loudly (an empty page stops the loop, a missing key leaves a blank cell) rather than writing wrong data. Re-run them before trusting a new month's CSV.
  • Delivery dates are estimates for Flipkart's default location, not for a PIN code.
  • Follow Flipkart's terms and the laws that apply to you, and keep request rates reasonable. (Not legal advice.)

Source: Scraping Flipkart in 2026: Extract Search Results, Product Details, and Reviews with Python

Top comments (0)