DEV Community: Bugrahan Saka

Scraping Flipkart with Python in 2026: skip the CSS, read the page's JSON state

Bugrahan Saka — Mon, 26 Oct 2026 09:10:06 +0000

Copy the CSS selectors from most Flipkart scraping tutorials and you get nothing back. The ._30jeq3 price class and the ._4rR01T name class match zero cards on today's pages. Build fresh ones and they break by category: a title selector that matches 24 of 24 cards on a laptop search matches 0 of 40 on a shoe search, because Flipkart renders two card layouts with different hashed class names.

The surprising part: Flipkart barely blocks anyone. What changes is which page you get.

This post walks through three scrapers (search results, product details, reviews) that read the JSON Flipkart already ships with every page, using only requests, json, re and csv.

Not blocked, sorted by user agent

The same search URL (/search?q=laptop) sent seven ways:

Request	Page	JSON state	Products in state
`requests`, default user agent	247 KB	0 KB	0
`curl/8.5.0`	248 KB	0 KB	0
Chrome user agent	655 KB	355 KB	24
Scrape.do, `geoCode=in`	655 KB	354 KB	24
Scrape.do, `render=true`	317 KB	0 KB	0

Every request returned a 200 and every page had 24 cards in the HTML. The difference is the data object: non-browser user agents get window.__INITIAL_STATE__ = {}, an empty object. A browser user agent gets the full state with every product, price and rating.

Two other details stand out. Rendering makes things worse: render=true returns the DOM after scripts have run, and that DOM no longer carries the state script. And volume barely mattered in testing: 100 sequential requests from one cloud IP all came back 200.

The data lives in `window.__INITIAL_STATE__`

Every Flipkart page assigns one JSON object to window.__INITIAL_STATE__ inside a <script> tag. Extracting it takes four lines:

import json
import re


def get_state(html):
    match = re.search(r"window\.__INITIAL_STATE__\s*=\s*", html)
    if not match:
        return {}
    state, _ = json.JSONDecoder().raw_decode(html, match.end())
    return state

raw_decode reads exactly one JSON value starting after the = and stops, so you never need to find where the script ends. A regex that tries to capture the whole object breaks on the first }; inside a product title.

Search results: `PRODUCT_SUMMARY` widgets

On a search page each product is a PRODUCT_SUMMARY widget under pageDataV4.page.data. The page is a grid of rows keyed by id, so walk every row instead of hard-coding one:

def product_cards(state):
    slots = state.get("pageDataV4", {}).get("page", {}).get("data", {})
    for slot_list in slots.values():
        for slot in slot_list:
            widget = slot.get("widget", {})
            if widget.get("type") == "PRODUCT_SUMMARY":
                yield from widget.get("data", {}).get("products", [])

That yields 24 of 24 cards on a laptop search and 40 of 40 on a shoe search with no code change. Filters and the pagination bar are widgets too; checking the type skips them.

Fetching goes through Scrape.do with geoCode=in, an Indian exit IP (Flipkart sells only in India), at 1 credit per request:

import requests

TOKEN = "<your_token>"
target = "https://www.flipkart.com/search?q=laptop&page=1"

response = requests.get(
    "https://api.scrape.do/",
    params={"token": TOKEN, "url": target, "geoCode": "in"},
)
cards = list(product_cards(get_state(response.text)))

Three things that bite

Prices are picked by flag, not position. pricing.prices holds the struck-out MRP (strikeOff: true) and the selling price:

value = card.get("productInfo", {}).get("value", {})
prices = value.get("pricing", {}).get("prices", [])
price = next((p["value"] for p in prices if not p.get("strikeOff")), "")
mrp = next((p["value"] for p in prices if p.get("strikeOff")), "")

The .get() chain matters: products marked "Currently unavailable" (NO_AVAILABLE_LISTING) have no pricing key at all, and value["pricing"] crashed on 5 of 100 pages in a bulk test.

Sponsored cards repeat. On page two of a laptop search, 10 of 24 cards were products already seen on page one. They carry an adInfo object; keep the first appearance of each product id.

Search stops around page 25. The page bar claimed 57 pages for "laptop"; products stopped at page 41. For "mouse", "headphones" and "shoes" they stopped at page 25. Past that, Flipkart returns a 200 with an empty results area, so stop at the first page with no cards. To get past the ceiling, split the search into price bands with Flipkart's own filter parameters:

low, high = 50000, 70000
target = (
    f"https://www.flipkart.com/search?q=laptop&page={page}"
    f"&p%5B%5D=facets.price_range.from%3D{low}"
    f"&p%5B%5D=facets.price_range.to%3D{high}"
)

Five bands produced 979 unique laptops in 68 requests, against 539 from the unfiltered search. If a band still reaches page 25, halve it.

Product details: a different state shape

Product pages carry the same window.__INITIAL_STATE__, but built as multiWidgetState: a tree of design-system widgets whose text sits in nodes named label_0, label_1 and so on.

The numbers don't need the widget tree. Flipkart's own analytics object holds them as plain JSON:

page = state["multiWidgetState"]["pageDataResponse"]
psi = page["pageContext"]["fdpEventTracking"]["events"]["psi"]

price = psi["ppd"]["finalPrice"]
mrp = psi["ppd"]["mrp"]
rating = psi["pr"]["rating"]
stars = {r["ratingValue"]: r["ratingCount"] for r in psi["pr"]["individualRatingsCount"]}
seller_id = psi["pls"]["sellerId"]
cod = psi["pi"]["isCODAvailable"]

The full spec table sits under widget keys starting with rpd_specifications, as label_0 (name) / label_1 (value) pairs. Collect pairs only under those keys, because the same pattern also holds offer text elsewhere on the page. That returned between 32 and 68 specs per laptop, and without a code change 66 for a phone, 31 for a saree and 23 for a pair of shoes.

Two catches worth knowing:

Seller and delivery wording differs by category. Electronics pages say Seller: RetailNet and Delivery by + a date; fashion pages say Fulfilled by RetailNet and Delivery + by 29 Oct, Thu. Check both.
The product page price may not match the search card. Open a product at its clean ?pid= URL and you may see Rs 74,990 instead of the card's Rs 79,990. Both are real offers from different sellers. The lid (listing id) parameter decides which one the page shows, so keep it when you follow URLs from search results.

Reviews: 20 pages per sort order

The reviews URL is the product URL with /p/ swapped for /product-reviews/. Each review is a value inside a REVIEWS widget, with the full text, vote counts, city and the exact variant bought.

Pages 1 to 20 return 10 reviews each; page 21 and beyond return a 200 with no reviews. That is 200 per sort order whatever the product has. The four sort orders overlap only partly, so running all of them and deduplicating by review id reaches further:

`sortOrder`	Unique in that order	Running total
`MOST_HELPFUL`	199	199
`MOST_RECENT`	200	332
`POSITIVE_FIRST`	196	356
`NEGATIVE_FIRST`	192	431

80 requests, 431 of 557 reviews (77%). Reviewer names are better left out: a name next to a city is personal data under India's DPDP Act, and it adds nothing to rating or sentiment analysis.

The packaged version

All three scrapers are in an open-source package, flipkart-scraping on GitHub and PyPI:

pip install flipkart-scraping
export SCRAPEDO_TOKEN=your_token

flipkart-scraping search "laptop" --pages 3 --format csv -o flipkart-search.csv
flipkart-scraping reviews "<product-url>" --format csv -o flipkart-reviews.csv

from flipkart_scraping import FlipkartScraper

scraper = FlipkartScraper()
products = scraper.search("laptop", max_pages=3, min_price=50000, max_price=70000)
details = scraper.get_product(products[0]["url"])
print(details["seller"], details["spec_count"], details["delivery_by"])

It removes sponsored repeats, keeps lid so prices match the cards, handles both seller wordings and runs all four review orders.

Before you run it at scale

The state is undocumented. It can change without notice. The scripts fail loudly (an empty page stops the loop, a missing key leaves a blank cell) rather than writing wrong data. Re-run them before trusting a new month's CSV.
Delivery dates are estimates for Flipkart's default location, not for a PIN code.
Follow Flipkart's terms and the laws that apply to you, and keep request rates reasonable. (Not legal advice.)

Source: Scraping Flipkart in 2026: Extract Search Results, Product Details, and Reviews with Python

Scraping TruePeopleSearch with Python: the US geofence, Cloudflare and the JSON-LD shortcut

Bugrahan Saka — Fri, 23 Oct 2026 12:04:03 +0000

TruePeopleSearch is one of the most used free people search sites: search a name and you get an age, addresses, phone numbers and public records, no sign-up. Point a script at it, though, and you rarely get a single byte of profile HTML back.

This post walks through why, how to get through with one request, and how to pull ten structured fields from a profile without fragile HTML parsing.

Every name, phone number and address below is fictional. Swap the placeholder URL for a profile URL you get from a normal search on the site.

Two walls in front of every profile

1. A US-only geofence. Request the site from outside the United States and there is no error page and no loading screen. The connection is simply cut off.

2. Cloudflare. Inside the US, requests and httpx still stall on JavaScript challenges, fingerprint checks and, sometimes, a CAPTCHA. Send requests too quickly and Cloudflare adds rate limiting, then escalates to harder challenges once it sees an automated pattern.

There is no public TruePeopleSearch API, so the request itself has to look like a real visitor from the US.

Getting through: two parameters

Routing the request through Scrape.do with two parameters handles both walls:

Parameter	What it does
`geoCode=us`	keeps the exit IP inside the United States, which clears the geofence
`super=true`	uses a residential IP instead of a datacenter one, so the request looks like a home user

No proxy rotation, no header tuning, no headless browser.

pip install requests beautifulsoup4

import requests
import urllib.parse
from bs4 import BeautifulSoup

token = "<your_token>"
target_url = "https://www.truepeoplesearch.com/find/person/jane-doe"
encoded_url = urllib.parse.quote_plus(target_url)

api_url = f"https://api.scrape.do/?token={token}&url={encoded_url}&super=true&geoCode=us"

response = requests.get(api_url)
soup = BeautifulSoup(response.text, "html.parser")
print(response)  # <Response [200]>

A 200 means the real profile HTML is in soup.

Name and age: read attributes, not text

The profile header is a div#personDetails that carries the values as data attributes. Reading attributes skips every formatting problem, because you get the values the way the site stores them rather than the way it displays them:

person = soup.find("div", id="personDetails")
name = f"{person['data-fn']} {person['data-ln']}"
age = person["data-age"]
# Jane Doe, 23

Address and phone: schema.org item props

The current address sits in a link marked data-link-to-more="address", with street, city and state in their own itemprop spans. The phone works the same way. Targeting itemprop instead of class names keeps the selectors stable through a restyle:

addr = soup.find("a", {"data-link-to-more": "address"})
address = addr.find("span", itemprop="streetAddress").text.strip()
city = addr.find("span", itemprop="addressLocality").text.strip()
state = addr.find("span", itemprop="addressRegion").text.strip()

phone = soup.find("a", {"data-link-to-more": "phone"}).find("span", itemprop="telephone").text.strip()

That gives you the basics. The page holds far more, and the best part isn't in the visible HTML at all.

The shortcut: one JSON-LD block

Every profile embeds a <script type="application/ld+json"> block of type ProfilePage. Its mainEntity is the person, and every field is a dictionary lookup:

import json

person_data = None
for script in soup.find_all("script", type="application/ld+json"):
    try:
        data = json.loads(script.string)
        if isinstance(data, dict) and data.get("@type") == "ProfilePage":
            person_data = data.get("mainEntity", {})
            break
    except (json.JSONDecodeError, TypeError):
        pass

address_data = person_data.get("address", {})
zipcode = address_data.get("postalCode", "")
phones = person_data.get("telephone", [])
emails = person_data.get("email", [])
aliases = person_data.get("alternateName", [])
relatives = [rel.get("name", "") for rel in person_data.get("relatedTo", [])]

On the sample profile:

ZIP: 20001
Phones: 8 found, primary: (202) 555-5555
Emails: 6 found, primary: janedoe@example.com
Aliases: Jane Marie Doe, Jane D Smith, Jane M Johnson
Relatives (24): John Doe, Mary Doe, Robert Smith, Emily Doe, Thomas Doe

Three things worth knowing about this block:

telephone and email are arrays, so you get every number and address on the profile, not only the one shown first. The first entry is usually the most recent.
relatedTo lists every relative as a Person object. Some profiles list dozens.
The JSON-LD keeps full middle names that the visible page abbreviates.

Combined with the name and age attributes, one request returns ten structured fields: name, age, address, city, state, ZIP, phones, emails, aliases and relatives.

Skip the boilerplate: a ready-made package

The same logic is packaged as an open-source library, truepeoplesearch-scraper on GitHub and PyPI:

pip install truepeoplesearch-scraper
export SCRAPEDO_TOKEN=your_token
truepeoplesearch-scraper "https://www.truepeoplesearch.com/find/person/<profile-id>" --format csv -o profiles.csv

from truepeoplesearch_scraper import TruePeopleSearchScraper

person = TruePeopleSearchScraper().get_profile(profile_url)
print(person.name, person.age, person.phones, person.relatives)

It prefers the JSON-LD values, falls back to the visible itemprop spans, retries server errors and returns JSON or CSV.

Before you run it at scale

Space requests out. Cloudflare escalates when traffic arrives too fast. Residential IPs rotate automatically, but pacing still helps.
Use the data responsibly. This is data about real people. Keep it out of hiring, tenant screening, credit and insurance decisions, which the Fair Credit Reporting Act regulates in the US. Honor opt-out requests and follow the privacy laws that apply to you. (Not legal advice.)

Source: TruePeopleSearch.com Scraping: Extract Data Without Blocks

DEV Community: Bugrahan Saka

Scraping Flipkart with Python in 2026: skip the CSS, read the page's JSON state

Not blocked, sorted by user agent

The data lives in window.__INITIAL_STATE__

Search results: PRODUCT_SUMMARY widgets

Three things that bite

Product details: a different state shape

Reviews: 20 pages per sort order

The packaged version

Before you run it at scale

Scraping TruePeopleSearch with Python: the US geofence, Cloudflare and the JSON-LD shortcut

Two walls in front of every profile

Getting through: two parameters

Name and age: read attributes, not text

Address and phone: schema.org item props

The shortcut: one JSON-LD block

Skip the boilerplate: a ready-made package

Before you run it at scale

The data lives in `window.__INITIAL_STATE__`

Search results: `PRODUCT_SUMMARY` widgets