DEV Community

Bugrahan Saka
Bugrahan Saka

Posted on

Myntra Scraper in Python: Listings, Pagination and Stock per Size

#webscraping #python #tutorial #programming

Myntra is India's largest fashion marketplace, and its listing pages are a good source of price data: brand, selling price, MRP, discount, rating and sizes for every product. Getting that into a CSV takes one short Python script, but there are three traps on the way: a request that never answers, a page parameter that is ignored, and a listing price that isn't the price of the size you want.

The block: a request that never answers 

import requests

url = "https://www.myntra.com/men-tshirts"
chrome = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
requests.get(url, headers={"User-Agent": chrome}, timeout=20)
Enter fullscreen mode Exit fullscreen mode
Request Result
requests, default or Chrome user agent connection opens, no bytes, timeout after 20 s
curl (HTTP/2) stream was not closed cleanly: INTERNAL_ERROR
Through Scrape.do, no parameters 200 in about 3 s, full page

Myntra sits behind Akamai Bot Manager (the _abck and bm_sz cookies). There is no error page to detect: the script just hangs. I route every request through Scrape.do with no extra parameters: no rendering, no premium proxies, 1 credit per request. In my tests 6 of 6 plain requests came back with the full page.

The data: one JSON object per page

Myntra renders on the server and puts all the data into one inline script, window.__myx = {...}. Listings keep their products in searchData.results.products, product pages keep everything in pdpData. No CSS selectors needed: find the assignment and let json.JSONDecoder.raw_decode read exactly one object from there.

Sponsored products are separate: organic results are in products, ads in plaProducts (18 of them on the men's T-shirts page), and any ad mixed into the API results carries isPLA. Skip those.

Trap 2: ?p=2 returns page 1

Ask the HTML for page 2, 3 or 50 and you get the first 50 products again. On a listing sorted by newest, page 2 shared 49 of 50 products with page 1. The site pages through its search API, /gateway/v4/search/<slug>, which:

  1. answers 401 without the cookies a page visit sets (at, _abck, bm_sz and others), and
  2. ignores p and o as well unless the request carries a pagination-context header. Every response returns a new one in its headers; send it back to get the next 50.

So the loop is: visit the listing once, take its cookies (Scrape.do returns them in the scrape.do-cookies response header), then call the API page by page with those cookies and the cursor. Scrape.do forwards custom headers when you add extraHeaders=true and prefix them with sd-.

One more catch: roughly one visit in four came back with a single cookie and no at, and the API answered 401 to it. Repeat the visit until at= is there.

import csv
import json
import re
import urllib.parse

import requests

TOKEN = "<your_token>"

def fetch(url, headers=None):
    params = {"token": TOKEN, "url": url}
    if headers:
        params["extraHeaders"] = "true"  # forward the sd- headers to Myntra
        headers = {f"sd-{k}": v for k, v in headers.items()}
    response = requests.get("https://api.scrape.do/", params=params, headers=headers, timeout=120)
    if response.status_code != 200:
        raise RuntimeError(f"HTTP {response.status_code} for {url}")
    return response

MYX_RE = re.compile(r"window\.__myx\s*=\s*")

def extract_myx(html):
    match = MYX_RE.search(html)
    if not match:
        return {}
    data, _ = json.JSONDecoder().raw_decode(html, match.end())
    return data

def to_row(item):
    price, mrp = item.get("price"), item.get("mrp")
    return {
        "product_id": item.get("productId"),
        "brand": item.get("brand"),
        "name": item.get("productName"),
        "price_inr": price,
        "mrp_inr": mrp,
        "discount_pct": round((mrp - price) / mrp * 100) if price and mrp else 0,
        "rating": round(item.get("rating") or 0, 2),
        "rating_count": item.get("ratingCount"),
        "sizes": item.get("sizes"),
        "url": "https://www.myntra.com/" + item.get("landingPageUrl", ""),
    }

LISTING_URL = "https://www.myntra.com/men-tshirts?sort=new"
PAGES = 3

# 1. Visit the listing for its session cookies. Repeat until Myntra hands out "at".
for _ in range(4):
    visit = fetch(LISTING_URL)
    cookies = visit.headers.get("scrape.do-cookies", "")
    if re.search(r"(^|;\s*)at=", cookies):
        break
total = extract_myx(visit.text)["searchData"]["results"]["totalCount"]

# 2. Page through the search API, sending back the cursor each response returns.
parsed = urllib.parse.urlparse(LISTING_URL)
rows, cursor = {}, ""
for page in range(1, PAGES + 1):
    query = urllib.parse.parse_qsl(parsed.query) + [
        ("rows", "50"), ("o", str(max(0, (page - 1) * 50 - 1))), ("plaEnabled", "false"), ("p", str(page)),
    ]
    api_url = f"https://www.myntra.com/gateway/v4/search{parsed.path}?{urllib.parse.urlencode(query)}"
    headers = {"Accept": "application/json", "x-meta-app": "channel=web", "app": "web",
               "X-myntraweb": "Yes", "Cookie": cookies}
    if cursor:
        headers["pagination-context"] = cursor
    response = fetch(api_url, headers)
    data = response.json()
    for item in data.get("products", []):
        if not item.get("isPLA"):  # skip sponsored cards
            rows.setdefault(item["productId"], to_row(item))
    cursor = response.headers.get("pagination-context", "")
    if not data.get("hasNextPage") or not cursor:
        break

rows = list(rows.values())
with open("myntra-listing.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    writer.writeheader()
    writer.writerows(rows)
print(f"{total} products listed, saved {len(rows)} to myntra-listing.csv")
Enter fullscreen mode Exit fullscreen mode

Three API pages plus the visit: 4 credits, about 15 seconds, 150 distinct products. In my runs it came back with 150 every time.

Trap 3: the listing price is not the size price

A listing card has one price. On the product page every size lists its sellers, each with its own price and exact stock in pdpData.sizes[].sizeSellerData:

def size_rows(url):
    pdp = extract_myx(fetch(url).text)["pdpData"]
    sellers = {s["sellerPartnerId"]: s["sellerName"] for s in pdp.get("sellers", [])}
    rows = []
    for size in pdp["sizes"]:
        for offer in size.get("sizeSellerData") or [{}]:
            rows.append({
                "size": size["label"],
                "seller": sellers.get(offer.get("sellerPartnerId")),
                "price_inr": offer.get("discountedPrice"),
                "mrp_inr": offer.get("mrp"),
                "units": offer.get("availableCount", 0) if size["available"] else 0,
            })
    return pdp["name"], rows

name, rows = size_rows("https://www.myntra.com/12027436")
print(name)
for row in rows:
    print(row["size"], row["seller"], row["price_inr"], row["units"])
Enter fullscreen mode Exit fullscreen mode 
Levis Men Soft Pure Cotton Round Neck Half Sleeve Tshirt
S Vision Star 402 398
M MODENIK LIFESTYLE PRIVATE LIMITED 389 83
M Vision Star 402 1261
L Vision Star 402 927
XL MODENIK LIFESTYLE PRIVATE LIMITED 389 303
Enter fullscreen mode Exit fullscreen mode

This T-shirt is listed at ₹389. That price only exists in M and XL, from one seller; in S and L the only seller charges ₹402. If you track prices for a specific size, read the product page.

Skip the boilerplate

The same logic is packaged as an open-source library, myntra-scraper on GitHub and PyPI, with session refresh on 401, retries and CSV output:

pip install myntra-scraper
export SCRAPEDO_TOKEN=your_token
myntra-scraper listing "https://www.myntra.com/men-tshirts?sort=new" --pages 3 -f csv -o myntra-listing.csv
myntra-scraper product -i myntra-listing.csv --limit 10 --sizes -f csv -o myntra-sizes.csv
Enter fullscreen mode Exit fullscreen mode

There is also a no-code version on Apify (Myntra Scraper) and a browser demo on Hugging Face.

Run it at scale with Scrape.do

Every request here is a plain Scrape.do call: no browser, no proxy setup, 1 credit each. Five listing pages (250 products) cost 6 credits, so the free plan's 1,000 monthly credits go a long way. Get 1000 free credits and start scraping with Scrape.do.

The scripts read listing, search and product pages, which Myntra's robots.txt allows (Allow: /), and never touch cart, checkout or account paths. Product data is not personal data; read Myntra's terms for your use case and keep request rates modest.

Top comments (1)

Collapse
 
alexcodebytes profile image
Oleksandr •

The fact that plain requests just hang while Scrape.do gets a 200 in 3 seconds is a good argument for using a scraping service. If you're doing this at any scale, the time saved on debugging bot protection is worth the credits.