DEV Community: Coco

How to scrape eBay in 2026?

Coco — Tue, 04 Aug 2026 16:30:00 +0000

Three eBay scraping jobs in Python: keyword search results, one listing by URL, and a seller profile by store URL. Every request here ran on 27 July 2026 against the Chocodata API, the screenshots are that output, and the field counts below are what actually came back rather than what the docs promise.

Why is it hard to scrape eBay?

eBay serves three selling formats through one result set, and a single "nintendo switch" search returned 38 buy-it-now listings, 17 accepting offers and 2 live auctions, each with a different set of populated fields. Datacenter IPs get an interruption page in place of the listing, which parses to nothing rather than raising. The expensive problem is the optional fields: watchers appeared on 12 of 57 rows and bid counts on 4, so a parser written against the top of the page throws or silently writes nulls further down.

Prerequisites

Python 3.9 or newer and requests. No browser driver, no proxy pool, no account beyond a free key.

1. A free Chocodata API key. Free, no card, and usable the moment you land on the dashboard.

2. requests in a clean environment.

python -m venv .venv && . .venv/bin/activate
pip install requests

3. The eBay URL you want, either an /itm/ listing link or an /str/ store link. Tested with Python 3.13.7 and requests 2.34.2 in July 2026. Any language with an HTTP client works identically, so Node, Go and PHP need no special handling.

Scrape eBay search results

To scrape eBay search results, send the keyword as query to the search endpoint and read the results array off the top level of the response.

1. Send the keyword and check the payload

Wrap the call so an empty body raises instead of flowing into your parser as a zero-row run.

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(endpoint, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
        if r.ok:
            break
        time.sleep(2)
    r.raise_for_status()
    data = r.json()
    if not data:
        raise RuntimeError(f"{endpoint}: empty payload for {params}")
    return data


search = call("ebay/search", query="nintendo switch")
print(search["query"], "| page", search["page"], "| rows", len(search["results"]))
print(len(search["results"][0]), "fields on the first listing")

nintendo switch | page 1 | rows 57
26 fields on the first listing

HTTP 200 in 2.69 s. There is no envelope around the payload, so it is search["results"] and not search["data"]["results"].

2. Count how many rows carry each optional field

Count the populated rows per field before writing any parsing code, because the eBay search response defines far more fields than any single listing fills.

from collections import Counter

rows = search["results"]

for field in ("rating", "old_price", "watchers", "quantity_sold", "bids"):
    print(f"{field:<16} {sum(1 for r in rows if r.get(field))}/{len(rows)}")

print(Counter(r["buying_format"] for r in rows))

rating           22/57
old_price        16/57
watchers         12/57
quantity_sold    8/57
bids             4/57
Counter({'buy_it_now': 38, 'accepts_offers': 17, 'auction': 2})

Nothing below rating clears half the rows. Run this against your own keyword before you decide which fields your schema can require.

3. Normalise the rows and write a CSV

Coalesce the optional fields to zero at write time so downstream aggregation does not have to special-case None.

import csv
from datetime import date

FIELDS = ["scraped_on", "position", "id", "title", "price", "condition",
          "buying_format", "watchers", "quantity_sold"]


def normalise(row):
    return {"scraped_on": date.today().isoformat(),
            "position": row["position"],
            "id": row["id"],
            "title": row["title"],
            "price": row["price"],
            "condition": row["condition"],
            "buying_format": row["buying_format"],
            "watchers": row.get("extracted_watchers") or 0,
            "quantity_sold": row.get("extracted_quantity_sold") or 0}


with open("ebay_search.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=FIELDS)
    w.writeheader()
    w.writerows(normalise(r) for r in rows)

print(len(rows), "rows written")

57 rows written

Use extracted_watchers and extracted_quantity_sold rather than their raw twins, since the extracted pair are already integers. The raw pair are display strings, which matters later. A single listing fills in much more than a search row does.

Pull a single eBay listing

To scrape one eBay listing, pass its URL as url to the product endpoint, with no item ID lookup in between.

1. Send the listing URL

The /itm/ link from the address bar is the whole input.

LISTING = "https://www.ebay.com/itm/327168915787"

item = call("ebay/product", url=LISTING)
print(len(item), "fields")
print(item["title"])
print(item["price"], item["currency"], "|", item["condition"])

16 fields
New Nintendo Switch OLED Model HEG-001 Handheld Console - White - New Other
294.99 USD | Open box

HTTP 200 in 2.08 s for 16 top-level fields.

2. Compute the discount and the stock position

Guard the discount maths, because old_price is absent on most listings and present only when eBay is showing a strikethrough.

def discount_pct(item):
    old, new = item.get("old_price"), item.get("price")
    if not old or not new or old <= new:
        return 0.0
    return round((old - new) / old * 100, 1)


print("was", item["old_price"], "now", item["price"],
      "=>", discount_pct(item), "% off")
print("sold", item["quantity_sold"], "| available", item["quantity_available"],
      "| in stock", item["availability"])

was 319.99 now 294.99 => 7.8 % off
sold 1 | available 1 | in stock True

On the listing endpoint quantity_sold is an integer, unlike its namesake on the search endpoint.

3. Flatten the nested identifiers

Collapse ids into a flat dict, because it arrives as a list of single-key objects rather than one object.

ids = {k: v for entry in item.get("ids", []) for k, v in entry.items()}

print(ids)
print("item_id", item["item_id"], "| brand", item["brand"])
print("description", len(item["description"]), "chars")

{'mpn': '115461'}
item_id 327168915787 | brand Nintendo
description 177 chars

additional_properties came back as an empty list on this listing, so treat it as optional too. Seller reputation is the last of the three jobs.

Profile an eBay seller

To scrape an eBay seller, send the store URL as url and read the reputation fields off the top level.

1. Send the store URL

Count the populated fields on arrival, since three of the fourteen come back empty for most stores.

STORE = "https://www.ebay.com/str/OfficialBestBuy"

seller = call("ebay/seller", url=STORE)
populated = sum(1 for v in seller.values() if v not in (None, [], {}))

print(len(seller), "fields |", populated, "populated")
print(seller["store_name"], "/", seller["username"])
print("feedback", seller["feedback_score"], "|",
      seller["positive_feedback_percent"], "% positive")

14 fields | 11 populated
Best Buy / best_buy
feedback 983317 | 99 % positive

HTTP 200 in 4.81 s. top_rated_seller, categories and active_items_count were the three empty ones.

2. Compute the real positive rate from the raw counts

Recompute the positive rate from feedback_ratings rather than trusting the rounded headline percentage.

fb = seller["feedback_ratings"]
total = sum(fb.values())

print(fb)
print(f"{fb['positive'] / total * 100:.2f}% positive across {total} rated transactions")
print("negative", fb["negative"], "| neutral", fb["neutral"])

{'positive': 132670, 'neutral': 426, 'negative': 1389}
98.65% positive across 134485 rated transactions
negative 1389 | neutral 426

The headline said 99%, the raw counts say 98.65%. Over 134,485 transactions that gap is 1,389 unhappy buyers, which is the number you actually want in a supplier scorecard.

3. Score the four detailed ratings

Sort the ratings object ascending so the weakest dimension is the first thing printed.

ratings = seller["ratings"]

for name, score in sorted(ratings.items(), key=lambda kv: kv[1]):
    print(f"{name:<18} {score:<5} {'#' * int(score * 4)}")

print("mean", round(sum(ratings.values()) / len(ratings), 2))
print("member since", seller["member_since"], "|", seller["items_sold"], "sold")

communication      4.6   ##################
item_as_described  4.7   ##################
shipping_speed     4.9   ###################
shipping_charges   5     ####################
mean 4.8
member since Jul 26, 2012 | 6100000 sold

Communication is the weak dimension at 4.6 against a 4.8 mean.

The part that breaks

quantity_sold on search rows is a display string. Casting it directly is the first thing that fails.

row = next(r for r in rows if r.get("quantity_sold"))
print(repr(row["quantity_sold"]))
int(row["quantity_sold"])

'2,595 sold'
Traceback (most recent call last):
  File "ebay_search.py", line 81, in <module>
    int(row["quantity_sold"])
    ~~~^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '2,595 sold'

Use extracted_quantity_sold, which is already 2595.

bids is an object, not a count. It arrives as {'count': 0}, so arithmetic on it raises TypeError: unsupported operand type(s) for +: 'dict' and 'int'. Read row["bids"]["count"].

Page 2 repeats page 1. Requesting both pages for the same keyword returned 114 rows containing 97 unique ids, so 17 listings came back twice. Deduplicate on id before counting anything.

page 1: 57 rows
page 2: 57 rows
combined: 114 rows, 97 unique ids, 17 repeats

The seller slug is not the username. Passing /str/OfficialBestBuy returned username 'best_buy' and a canonical url of https://www.ebay.com/str/best_buy, so key your storage on username rather than on the URL you sent.

Result sets move between runs. The same two-page search returned 97 unique listings on one run and 102 twenty minutes later. Stamp every row with a date and compare snapshots rather than expecting a stable count.

Full script

"""Scrape eBay search results, one listing and a seller profile.

Tested on Python 3.13.7 with requests 2.34.2, July 2026.
"""
import csv
import time

import requests
from collections import Counter
from datetime import date

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

FIELDS = ["scraped_on", "position", "id", "title", "price", "condition",
          "buying_format", "watchers", "quantity_sold"]


def call(endpoint, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
        if r.ok:
            break
        time.sleep(2)
    r.raise_for_status()
    data = r.json()
    if not data:
        raise RuntimeError(f"{endpoint}: empty payload for {params}")
    return data


def normalise(row):
    return {"scraped_on": date.today().isoformat(),
            "position": row["position"],
            "id": row["id"],
            "title": row["title"],
            "price": row["price"],
            "condition": row["condition"],
            "buying_format": row["buying_format"],
            "watchers": row.get("extracted_watchers") or 0,
            "quantity_sold": row.get("extracted_quantity_sold") or 0}


def search_listings(query, pages=2):
    seen, rows = set(), []
    for page in range(1, pages + 1):
        data = call("ebay/search", query=query, page=page)
        for row in data["results"]:
            if row["id"] in seen:
                continue
            seen.add(row["id"])
            rows.append(normalise(row))
    return rows


def listing(url):
    item = call("ebay/product", url=url)
    old, new = item.get("old_price"), item.get("price")
    item["discount_pct"] = (round((old - new) / old * 100, 1)
                            if old and new and old > new else 0.0)
    return item


def seller_profile(url):
    data = call("ebay/seller", url=url)
    fb = data["feedback_ratings"]
    total = sum(fb.values())
    data["positive_rate"] = round(fb["positive"] / total * 100, 2)
    data["rated_transactions"] = total
    return data


if __name__ == "__main__":
    rows = search_listings("nintendo switch")
    with open("ebay_search.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        w.writeheader()
        w.writerows(rows)
    print(len(rows), "unique listings written")
    print(Counter(r["condition"] for r in rows).most_common(3))

    item = listing("https://www.ebay.com/itm/327168915787")
    print(item["title"])
    print(item["price"], item["currency"], "|", item["discount_pct"], "% off")

    store = seller_profile("https://www.ebay.com/str/OfficialBestBuy")
    print(store["store_name"], "|", store["positive_rate"], "% of",
          store["rated_transactions"])

102 unique listings written
[('Pre-Owned', 79), ('Very Good - Refurbished', 6), ('Open Box', 5)]
New Nintendo Switch OLED Model HEG-001 Handheld Console - White - New Other
294.99 USD | 7.8 % off
Best Buy | 98.65 % of 134485

Summary

All three eBay jobs are a single GET with everything at the top level of the response: 57 search rows in 2.69 s, 16 listing fields in 2.08 s, and 14 seller fields in 4.81 s. What the API cannot give you is a populated field on every row, because eBay itself only publishes watchers, bids and sold counts on a minority of listings, and the search endpoint mirrors that faithfully. Take one habit from this into your own build and make it the extracted_* fields plus or 0, because the display strings look numeric right up to the point where int() raises on a comma.

FAQ

Is scraping eBay legal?

Public listing pages are not automatically illegal to scrape, but eBay's user agreement restricts automated collection, which makes it a contract question rather than a criminal one. Not legal advice.

Can I scrape an eBay listing without the item ID?

Yes, the /itm/ URL is the only input the product endpoint needs, so no ID lookup step is required.

Why does `int(row["quantity_sold"])` fail on eBay search results?

Because that field is the display string '2,595 sold', and the integer you want is in extracted_quantity_sold.

How to scrape Bing in 2026?

Coco — Mon, 03 Aug 2026 16:30:00 +0000

Scraping Bing search results in Python: ranking position, title, link, displayed link, snippet and publication date for any query, using the Chocodata search endpoint. Every request below was executed on 27 July 2026 and the screenshots are that output.

Here is what the finished script prints:

query                        realpython.com
python web scraping          [2, 4]
python asyncio tutorial      [1]
fastapi tutorial             [5]

wrote bing_ranks.csv (15 rows)
9 of 15 results carried a date

Those two counts move between runs, and that is a property of the live result page rather than a bug in the script. The measurement is in the failures section.

TL;DR

One GET returns four top-level fields and five organic results, each with seven fields of its own. No wrapper, so r.json()["organic_results"] is the whole access path.

position arrives pre-assigned, so you never count nodes and never inherit the off-by-one that ad and answer modules cause in the raw markup.

date was None on 6 of 15 results. Pass one to strptime and you get TypeError: strptime() argument 1 must be str, not None.

A domain can hold more than one position. realpython.com came back at 2 and 4 for the same query, so next() silently reports half the footprint.

Repeat calls are usually identical but not guaranteed: across 30 calls, 27 returned their query's modal top five and 3 returned a different set. Tested on Python 3.13.7 and requests 2.34.2, July 2026.

Why is it hard to scrape Bing?

Scraping Bing is hard because the result page is assembled per visitor, so the market and language inferred from the connection change both which results appear and their order. Ads, answer panels and video modules sit in the same container as organic results, which means node-counting in raw markup assigns positions that are quietly wrong. The layer that wastes the most time is the consent and verification interstitial served to automated clients, because it arrives shaped like a normal successful response rather than an error.

Prerequisites

To scrape Bing search results with the code below you need three things, and the first is free.

A free Chocodata API key. Sign up, copy the key from the dashboard, and pass it as api_key on every request. Free to start, no card required.

Python 3.9 or newer and requests. No browser driver, no proxy, no Microsoft account.

python -m venv .venv
pip install requests

The queries you want to track, for example python web scraping.

Tested on Python 3.13.7 with requests 2.34.2 in July 2026.

How to scrape Bing search results?

One GET to the search endpoint returns the ranked organic results as JSON, and the three steps below take it from a query string to a CSV of dated ranks.

1. Define the query

Bing search scraping takes the query itself as input, sent as q alongside your key, so there is no URL to construct beyond the endpoint.

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

url = f"{BASE}/bing/search"
params = {"q": "python web scraping", "api_key": API_KEY}

r = requests.get(url, params=params, timeout=60)
print(r.status_code, r.headers.get("content-type"))

200 application/json

2. Call the search endpoint and guard on the payload

The Bing search response has four top-level fields, so check for organic_results before you index into it rather than trusting the status code.

r.raise_for_status()
data = r.json()

if "organic_results" not in data:
    raise RuntimeError(f"no results in response for {params['q']!r}")

print(list(data.keys()))
print(data["query"], "|", data["engine"], "|",
      data["results_count"], "|", len(data["organic_results"]))

['query', 'engine', 'results_count', 'organic_results']
python web scraping | bing | 5 | 5

Indexing straight into a response that came back short gives you KeyError: 'organic_results' at whatever hour the job runs, which is why the guard is three lines up rather than in a wrapper you add later.

3. Extract position, link and date

Each organic result carries seven fields, and position, link and date are the three that make a row comparable across runs.

for item in data["organic_results"]:
    print(f"{item['position']:>2}  {item['date'] or '-':>12}  {item['link']}")

 1   Dec 8, 2025  https://www.geeksforgeeks.org/python/python-web-scraping-tutorial/
 2   Dec 1, 2024  https://realpython.com/beautiful-soup-web-scraper-python/
 3  May 28, 2026  https://www.scrapingbee.com/blog/web-scraping-101-with-python/
 4  Dec 21, 2024  https://realpython.com/python-web-scraping-practical-introduction/
 5             -  https://github.com/luminati-io/Python-web-scraping

The remaining fields are title, displayed_link, snippet and source. Note the or '-' on date, which is doing real work rather than cosmetic work, for the reason in the failures section. The same request works from any language with an HTTP client, so Node, Go and PHP need nothing beyond their standard library.

Compare rankings across a query set

Comparing Bing rankings across a query set is the same search call in a loop with a domain filter on top, and it is the only version of this that answers a question. One query gives a number. A set gives you which topics a domain owns.

def positions(data, domain):
    """Every position the domain holds, not just the best one."""
    return [i["position"] for i in data["organic_results"] if domain in i["link"]]


QUERIES = ["python web scraping", "python asyncio tutorial", "fastapi tutorial"]
DOMAIN = "realpython.com"

for q in QUERIES:
    resp = requests.get(f"{BASE}/bing/search",
                        params={"q": q, "api_key": API_KEY}, timeout=60)
    resp.raise_for_status()
    data = resp.json()
    hits = positions(data, DOMAIN)
    print(f"{q:28} {hits if hits else 'not in top 5'}")

python web scraping          [2, 4]
python asyncio tutorial      [1]
fastapi tutorial             [5]

Returning a list rather than a single rank is the design decision that matters here. The tracked domain holds two of the five slots on the first query, which a best-rank-only tracker would report as a flat #2 and lose half the picture.

Write every result to a dated CSV rather than only the matches, because the rows you did not care about this month are the competitor set you will want next month:

import csv
from datetime import date

FIELDS = ["captured", "query", "position", "link", "title", "date"]

def rows_for(data):
    captured = date.today().isoformat()
    return [{"captured": captured, "query": data["query"],
             "position": i["position"], "link": i["link"],
             "title": i["title"], "date": i.get("date")}
            for i in data["organic_results"]]

captured,query,position,link,title,date
2026-07-27,python web scraping,1,https://www.geeksforgeeks.org/python/python-web-scraping-tutorial/,Python Web Scraping Tutorial - GeeksforGeeks,"Dec 8, 2025"
2026-07-27,python web scraping,2,https://realpython.com/beautiful-soup-web-scraper-python/,Beautiful Soup: Build a Web Scraper With Python,"Dec 1, 2024"

The part that breaks

Four failures, all hit while writing this.

date is None on a lot of results. Nine of the fifteen rows in the run above carried a date and six did not, so any parser reached without a guard dies on the first undated result.

from datetime import datetime

resp = requests.get(f"{BASE}/bing/search",
                    params={"q": "python web scraping", "api_key": API_KEY}, timeout=60)
resp.raise_for_status()
results = resp.json()["organic_results"]

undated = [i for i in results if not i.get("date")]
print(len(undated), "of", len(results), "results carried no date")

# datetime.strptime(item["date"], "%b %d, %Y")
# TypeError: strptime() argument 1 must be str, not None


def parse_date(value):
    if not value:
        return None
    return datetime.strptime(value, "%b %d, %Y").date()


print([parse_date(i.get("date")) for i in results])

next() hides a domain's second position. A domain can occupy more than one of the five slots, and the obvious one-liner reports only the first.

print(next((i["position"] for i in results if "realpython.com" in i["link"]), None))
# 2      <- correct, and incomplete

print([i["position"] for i in results if "realpython.com" in i["link"]])
# [2, 4]  <- the actual footprint

Five results is the ceiling, so "absent" means "not in the top five". results_count was 5 on every query I sent. A domain at position six is indistinguishable from a domain that does not rank at all, which is fine for a comparison and wrong in a report that claims coverage.

Repeat calls are usually identical, not always. I sent five queries six times each, 30 calls in total. Twenty-seven returned their query's modal top five in the same order. Three returned a different set for the same query minutes apart, because the live page is assembled per request. Two of the five queries were stable across all six calls and three were not, so do not assume the query you tested is one of the stable ones.

assert [i["link"] for i in results] == EXPECTED_LINKS   # goes red eventually
assert len(results) == 5 and all(i["position"] for i in results)   # what to assert

Pin the shape, not the contents, and let the stored captures carry the history.

Full script

"""Bing rank tracking across a query set via the Chocodata search endpoint.
Tested: Python 3.13.7, requests 2.34.2, 27 July 2026.
"""
import csv
import time
from datetime import date, datetime

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
CSV_PATH = "bing_ranks.csv"

FIELDS = ["captured", "query", "position", "link", "title", "date"]


def search(query, timeout=60):
    """Return the parsed response for one query, or raise."""
    r = requests.get(f"{BASE}/bing/search",
                     params={"q": query, "api_key": API_KEY}, timeout=timeout)
    r.raise_for_status()
    data = r.json()
    if "organic_results" not in data:
        raise RuntimeError(f"no results in response for {query!r}")
    return data


def rows_for(data):
    """One dated row per organic result. .get() on date because it is often None."""
    captured = date.today().isoformat()
    return [{"captured": captured, "query": data["query"],
             "position": i["position"], "link": i["link"],
             "title": i["title"], "date": i.get("date")}
            for i in data["organic_results"]]


def positions(data, domain):
    """Every position the domain holds, not just the best one."""
    return [i["position"] for i in data["organic_results"] if domain in i["link"]]


def parse_date(value):
    """Bing dates look like 'Dec 8, 2025'. Many results carry none at all."""
    if not value:
        return None
    return datetime.strptime(value, "%b %d, %Y").date()


def track(queries, domain, path=CSV_PATH, delay=2.0):
    all_rows = []
    print(f"{'query':28} {domain}")
    for n, query in enumerate(queries):
        data = search(query)
        all_rows.extend(rows_for(data))
        hits = positions(data, domain)
        print(f"{query:28} {hits if hits else 'not in top 5'}")
        if n < len(queries) - 1:
            time.sleep(delay)

    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        w.writeheader()
        w.writerows(all_rows)

    print(f"\nwrote {path} ({len(all_rows)} rows)")
    return all_rows


if __name__ == "__main__":
    rows = track(["python web scraping", "python asyncio tutorial", "fastapi tutorial"],
                 "realpython.com")
    dated = [parse_date(r["date"]) for r in rows]
    print(f"{sum(d is not None for d in dated)} of {len(dated)} results carried a date")

Switch open(path, "w") to open(path, "a") and drop the header write once you start running it on a schedule, since the value is entirely in the comparison between captures.

Summary

One GET to the search endpoint returns four top-level fields and the top five organic Bing results with position already assigned, which removes the node-counting step that puts an off-by-one into most hand-rolled parsers. What does not work is treating the response as complete or fixed, since five results is the ceiling, six of fifteen rows came back with no date, and two of sixteen repeat calls returned a different set for the same query. The one thing to carry away is to assert on the shape of the response and store dated captures, because the rank you fetched once is worth nothing next to the diff between two of them.

FAQ

Is scraping Bing search results legal?

Search results are public pages and the constraint is contractual rather than criminal, since Microsoft's terms restrict automated querying, and none of this is legal advice.

Why do my Bing results differ from what my browser shows?

Bing assembles the page per request using the market and language it infers from the connection, so the results reflect that inference rather than your own browser session.

How many results does one Bing request return?

Five organic results, reported in results_count, which is why a domain at position six reads as absent rather than ranked.

How to scrape Amazon in 2026?

Coco — Sun, 02 Aug 2026 16:30:00 +0000

Three Amazon scraping jobs in Python, each driven by a URL: product data, search results, and the star distribution behind a listing. Every request below was executed on 27 July 2026 against Chocodata and the numbers are that run.

HTTP 200 in 3.17s
fields: 60
title: Amazon Kindle Paperwhite 16GB (newest model)
price: 159.99 USD

TL;DR

amazon/product takes a product URL and returns 60 top-level fields in about 3 seconds, with no data envelope to unwrap.

amazon/search takes a search URL and returned 22 rows, 16 organic and 6 sponsored, for mechanical keyboard.

rating_stars_distribution returns 5 bands. Recomputing the mean for the Fire TV Stick gave 4.66 against a listed 4.6.

Sponsored rows carry organic_position: None and some rows carry price: None, so sorted() raises TypeError unless you filter first.

Why is it hard to scrape Amazon?

Amazon layers its defences: datacenter IPs are scored before your headers are parsed, product markup is A/B tested so the same URL returns different DOM shapes minutes apart, and sequential access patterns get profiled. The expensive one is the robot check, which arrives as HTTP 200, so any scraper branching on r.status_code == 200 logs a clean run while writing empty rows. That single behaviour causes more wasted debugging than the other three combined.

Prerequisites

A free Chocodata API key, taken from the dashboard after sign-up. Free tier, no card.

Python 3.9+ and requests. Tested here on Python 3.13.7 with requests 2.34.2, July 2026. Other languages hit the same endpoints, but every snippet below is Python.

python -m venv .venv && . .venv/bin/activate
pip install requests

The Amazon URL you want to scrape. Product page or search page, copied from the address bar.

Fetch product data from a product URL

To scrape Amazon product data, send the product URL as url and read the fields straight off the response. There is no wrapper object.

1. Send the URL and assert on a field, not the status

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
URL = "https://www.amazon.com/dp/B0CFPJYX7P"

r = requests.get(f"{BASE}/amazon/product",
                 params={"url": URL, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
product = r.json()

assert product.get("asin"), "empty payload behind a 200"
print("fields:", len(product))
print("title:", product["title"][:44])
print("price:", product["price"], product["currency"])

fields: 60
title: Amazon Kindle Paperwhite 16GB (newest model)
price: 159.99 USD

2. Map the fields you actually store

KEEP = ("asin", "parent_asin", "brand", "price", "price_buybox", "currency",
        "rating", "reviews_count", "max_quantity", "has_videos")

for key in KEEP:
    print(f"{key:>16}: {product[key]}")

            asin: B0CFPJYX7P
     parent_asin: B0DFYWDGW6
           brand: Amazon
           price: 159.99
    price_buybox: 159.99
        currency: USD
          rating: 4.7
   reviews_count: 21493
    max_quantity: 5
      has_videos: True

3. Flatten the variation list to CSV

import csv

with open("variations.csv", "w", newline="", encoding="utf-8") as fh:
    w = csv.writer(fh)
    w.writerow(["asin", "color", "style", "configuration"])
    for v in product["variations"]:
        d = v["dimensions"]
        w.writerow([v["asin"], d.get("Color"), d.get("Style"), d.get("Configuration")])

print("rows:", len(product["variations"]))

asin,color,style,configuration
B0CTMS23DZ,Jade,With 3 months of Kindle Unlimited,Without Lockscreen Ads
B0CTMR865D,Raspberry,With 3 months of Kindle Unlimited,Ad-supported
B0CFPTK5JG,Raspberry,Without Kindle Unlimited,Ad-supported
B0CFPJYX7P,Black,Without Kindle Unlimited,Ad-supported
B0DDZJS3SB,Black,Without Kindle Unlimited,Without Lockscreen Ads
rows: 12

dimensions keys vary per product line, which is why .get() beats [] here. The search endpoint returns a flatter shape and is covered next.

Fetch search results from a search URL

To scrape Amazon search results, send the search page URL unchanged. The keyword stays inside the URL, so nothing needs parsing out of it.

1. Request the search URL unchanged

SEARCH = "https://www.amazon.com/s?k=mechanical+keyboard"

r = requests.get(f"{BASE}/amazon/search",
                 params={"url": SEARCH, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
search = r.json()

print("products:", len(search["products"]))
print("keys:", sorted(search["products"][0])[:6])

products: 22
keys: ['asin', 'badges', 'best_seller', 'brand', 'climate_pledge_friendly', 'currency']

2. Split sponsored rows from organic rows

organic = [p for p in search["products"] if not p["is_sponsored"]]
print(len(organic), "organic of", len(search["products"]))

for p in organic[:5]:
    print(f'{p["organic_position"]:>2} {p["asin"]} {str(p["price"]):>7} {p["rating"]} {p["title"][:42]}')

16 organic of 22
 1 B0CF3VGQFL   29.99 4.3 Redragon Mechanical Gaming Keyboard Wired,
 2 B0CDWP1D58   36.99 4.5 Redragon K668 108-Key Hot-Swap Wired RGB G
 3 B09LK1P1RD   156.9 4.2 Logitech MX Mechanical Wireless Illuminate
 4 B09JG7KRC7   43.31 4.4 Keychron C2 Full Size Wired Mechanical Key
 5 B0C8QYB8W6   89.99 4.5 Razer BlackWidow V4 X Mechanical Gaming Ke

3. Page the URL and dedupe by ASIN

def fetch_page(keyword, page):
    url = f"https://www.amazon.com/s?k={keyword}&page={page}"
    r = requests.get(f"{BASE}/amazon/search",
                     params={"url": url, "api_key": API_KEY}, timeout=90)
    r.raise_for_status()
    return r.json()["products"]

rows = {}
for page in (1, 2):
    got = fetch_page("mechanical+keyboard", page)
    print(f"page {page}: {len(got)} rows")
    for p in got:
        rows[p["asin"]] = p

print("unique asins:", len(rows))

page 1: 22 rows
page 2: 22 rows
unique asins: 37

Pages overlapped by 7 ASINs on that run, so keying the dict on asin matters more than it looks. Rating data needs no extra call, which the next section uses.

Fetch ratings and the star distribution

To scrape Amazon reviews data, read the rating fields off the same product response. One call per listing covers the score, the volume and the five bands.

1. Read rating and reviews_count

FIRE = "https://www.amazon.com/dp/B0BP9MDCQZ"

r = requests.get(f"{BASE}/amazon/product",
                 params={"url": FIRE, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
item = r.json()

print("rating:", item["rating"])
print("reviews_count:", item["reviews_count"])

rating: 4.6
reviews_count: 111766

2. Expand rating_stars_distribution

for b in item["rating_stars_distribution"]:
    bar = "#" * (b["percentage"] // 2)
    print(f'{b["rating"]} star {b["percentage"]:>3}% {bar}')

5 star  83% #########################################
4 star   9% ####
3 star   3% #
2 star   1%
1 star   4% ##

3. Convert percentages to counts and recheck the mean

total = item["reviews_count"]
counts = {b["rating"]: round(total * b["percentage"] / 100)
          for b in item["rating_stars_distribution"]}
mean = sum(k * v for k, v in counts.items()) / sum(counts.values())

for k in sorted(counts, reverse=True):
    print(f"{k} star: {counts[k]:>7,}")
print(f"weighted mean: {mean:.2f} (listed {item['rating']})")
print(f"negative share: {(counts[1] + counts[2]) / sum(counts.values()) * 100:.1f}%")

5 star:  92,766
4 star:  10,059
3 star:   3,353
2 star:   1,118
1 star:   4,471
weighted mean: 4.66 (listed 4.6)
negative share: 5.0%

The recomputed mean lands 0.06 above the listed rating because the band percentages are rounded to whole numbers, so treat it as a cross-check rather than a replacement. The reviews list on the same response carries the eight most recent reviewers with author, rating, timestamp, is_verified and helpful_count, so reviewer-level tracking works without the written text.

The part that breaks

Nulls in the search rows, not the requests. On the 22-row mechanical keyboard response, 6 rows had organic_position: None because they were sponsored, 1 row had price: None, 1 had rating: None and 4 had sales_volume: None. Sorting straight off those keys raises:

Traceback (most recent call last):
  File "scrape.py", line 32, in <module>
    cheapest = sorted(prods, key=lambda p: p["price"])
TypeError: '<' not supported between instances of 'NoneType' and 'float'

Traceback (most recent call last):
  File "scrape.py", line 38, in <module>
    ranked = sorted(prods, key=lambda p: p["organic_position"])
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

Filter before you sort, and use a sentinel for the rank so sponsored rows sort last instead of blowing up:

prods = search["products"]

priced = [p for p in prods if p["price"] is not None]
cheapest = sorted(priced, key=lambda p: p["price"])
ranked = sorted(prods, key=lambda p: p["organic_position"] or 999)

Two more things worth knowing before you build on this. Search row counts move between runs, 16 on one wireless mouse call and 22 on the keyboard call, so never hard-code a slice length. And the entries in reviews come back with content and title unset, so any pipeline expecting review sentences needs to run on rating_stars_distribution instead.

Full script

"""Amazon product, search and rating scraper. Python 3.13.7, requests 2.34.2."""
import csv
import sys
import time
import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(path, url, timeout=90, tries=3):
    """One GET, retried, so a batch run survives a dropped response."""
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}",
                         params={"url": url, "api_key": API_KEY}, timeout=timeout)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def product(url):
    p = call("amazon/product", url)
    if not p.get("asin"):
        raise RuntimeError(f"empty payload for {url}")
    return p


def star_counts(p):
    total = p["reviews_count"]
    return {b["rating"]: round(total * b["percentage"] / 100)
            for b in p.get("rating_stars_distribution") or []}


def search(keyword, pages=2):
    rows = {}
    for page in range(1, pages + 1):
        url = f"https://www.amazon.com/s?k={keyword}&page={page}"
        for row in call("amazon/search", url)["products"]:
            rows[row["asin"]] = row
    return list(rows.values())


def main():
    p = product("https://www.amazon.com/dp/B0CFPJYX7P")
    print(f'{p["title"][:44]} | {p["price"]} {p["currency"]} | {p["rating"]}')

    for star, n in sorted(star_counts(p).items(), reverse=True):
        print(f"  {star} star: {n:>7,}")

    rows = search("mechanical+keyboard", pages=2)
    priced = [r for r in rows if r["price"] is not None]
    priced.sort(key=lambda r: r["price"])
    print(f"{len(rows)} unique asins, {len(priced)} priced")

    with open("amazon.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["asin", "title", "price", "rating", "reviews_count", "sponsored"])
        for r in priced:
            w.writerow([r["asin"], r["title"], r["price"], r["rating"],
                        r["reviews_count"], r["is_sponsored"]])
    print("wrote amazon.csv")


if __name__ == "__main__":
    sys.exit(main())

Summary

Three Amazon jobs run off one request shape: a product URL returns 60 top-level fields including the rating and the five star bands, and a search URL returns product rows with sponsored flags and organic positions. What the payload does not carry is review body text, so this covers price tracking, rank tracking and score monitoring rather than sentiment reading. The habit that saves the most time is asserting on asin rather than on the status code, and filtering None out of every key you intend to sort on.

FAQ

How many fields does the Amazon product response return?

Sixty top-level fields on both products tested, covering price, rating, images, variations and the star distribution.

Why does sorting search rows raise TypeError?

Sponsored rows carry organic_position: None and a few listings carry price: None, so filter the nulls out or supply a sentinel before sorting.

Does the star distribution match the listed rating?

Close but not exact: recomputing the mean from the Fire TV Stick bands gave 4.66 against a listed 4.6, because the band percentages are rounded to whole numbers.

How to scrape YouTube in 2026?

Coco — Sat, 01 Aug 2026 16:30:00 +0000

Three Python functions that scrape YouTube video metadata, a channel's uploads and a full timed transcript, using the Chocodata API. Every snippet below was executed on 27 July 2026 and the output blocks are copied from that run.

Here is what the finished script prints:

NASA  15,000,000 subscribers  6,100 videos  more=True
pulled 30 rows from page 1
newest: Moon Base: June 2026 Update  (199,184 views)
  199,192 views | 8,156 likes | 65s | Science & Technology
  transcript: 13 segments, 94 words, 1 languages
  [Dramatic music] [Jared Isaacman] People are looking up again, believi
wrote youtube_channel_videos.csv

TL;DR

The video endpoint returns 26 top-level fields. There is no data wrapper, so r.json()["title"] is correct and r.json()["data"] raises KeyError.

A channel request returns 30 videos per page plus has_more and a 2,372-character continuation token.

Transcripts come back as timestamped segments. The Rick Astley video gives 61 segments, 487 words, 2,089 characters.

comment_count is None and keywords is [] on plenty of videos. Both crash naive code.

Why is it hard to scrape YouTube?

YouTube ships the watch page as a shell and builds the visible values in the browser from a JSON blob, so a plain requests.get plus an HTML parser returns markup with no title, no view count and no captions. Channel uploads compound it by paginating with continuation tokens instead of page numbers, and transcripts are not in the page at all. The part that eats the most time is the transcript, because captions are served from a separate call whose parameters are signed and short lived.

Prerequisites

To scrape YouTube with this code you need three things.

1. A free Chocodata API key. Sign up and copy it from the dashboard. Free tier, no card.

2. Python 3.9+ and requests. Tested on Python 3.13.7 with requests 2.34.2 in July 2026.

pip install requests

3. A target URL. The watch URL for videos and transcripts, the channel URL for uploads.

Scrape one video's metadata

To scrape a YouTube video, send a single GET with the watch URL and read the fields off the top level of the response.

1. Send the request and check the status

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

r = requests.get(f"{BASE}/youtube/video",
                 params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
                         "api_key": API_KEY}, timeout=30)
r.raise_for_status()
v = r.json()
print(r.status_code, len(v), "fields")
print(v["video_id"], v["title"])

Output:

200 26 fields
dQw4w9WgXcQ Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)

2. Map the fields you actually need

Pick the fields explicitly rather than storing the whole payload. The counts are real integers, not display strings.

row = {k: v[k] for k in ("video_id", "title", "channel_name", "view_count",
                         "like_count", "duration_seconds", "category",
                         "publish_date")}
print(row["view_count"], row["like_count"], row["duration_seconds"])
print(row["channel_name"], "|", row["category"], "|", row["publish_date"][:10])

Output:

1797072750 19279452 213
Rick Astley | Music | 2009-10-24

3. Handle the null and empty fields

comment_count came back None on this video and keywords is an empty list on many others, so coalesce before doing arithmetic or indexing.

print("comment_count:", repr(v["comment_count"]))
print("keywords:", len(v["keywords"]), "| thumbnails:", len(v["thumbnails"]),
      "| related:", v["related_count"])
engagement = (v["like_count"] or 0) + (v["comment_count"] or 0)
print("engagement:", engagement)

Output:

comment_count: None
keywords: 27 | thumbnails: 5 | related: 12
engagement: 19279452

Pull a channel's video list

To scrape a YouTube channel's uploads, use the channel parameter, which accepts the channel URL as its value.

1. Request the channel

r = requests.get(f"{BASE}/youtube/channel",
                 params={"channel": "https://www.youtube.com/@NASA",
                         "api_key": API_KEY}, timeout=60)
r.raise_for_status()
ch = r.json()
print(ch["channel_name"], ch["channel_id"],
      ch["subscriber_count"], ch["video_count"])

Output:

NASA UCLA_DiR1FfKNvjuUpBHmylQ 15000000 6100

2. Flatten the videos list into rows

ch["videos"] is a list of dicts with position, id, title, url, thumbnail, channel, views and published. Note that views is a display string and published is relative, so keep id as your key.

rows = [{"position": x["position"], "video_id": x["id"], "title": x["title"],
         "views": x["views"], "published": x["published"], "url": x["url"]}
        for x in ch["videos"]]

print(len(rows), "rows")
for x in rows[:3]:
    print(x["position"], x["video_id"], x["views"], "|", x["title"][:44])

Output:

30 rows
1 aRSBZN2UF0Q 199,184 views | Moon Base: June 2026 Update
2 75-H-i9ctkE 39,288 views | Independence Day Wishes from the Space Stati
3 84id2dzpwU0 45,248 views | Astronauts on the Space Station Celebrate th

3. Follow has_more for the next page

Pagination is a token, not an offset, so check has_more and carry next_page_token forward instead of incrementing a number.

print("videos_count:", ch["videos_count"], "| has_more:", ch["has_more"],
      "| token chars:", len(ch["next_page_token"]))

if ch["has_more"]:
    r = requests.get(f"{BASE}/youtube/channel",
                     params={"page_token": ch["next_page_token"],
                             "api_key": API_KEY}, timeout=60)
    r.raise_for_status()
    nxt = r.json()
    print("page", nxt["page"], "->", nxt["videos_count"], "more videos")

Output:

videos_count: 30 | has_more: True | token chars: 2372
page 2 -> 30 more videos

Positions restart at 1 on each page, so build your key from video_id and not from position.

Extract a video transcript

To scrape a YouTube transcript, send the watch URL to the transcript endpoint and read the segments list.

1. Fetch the transcript for a video URL

r = requests.get(f"{BASE}/youtube/transcript",
                 params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
                         "api_key": API_KEY}, timeout=60)
r.raise_for_status()
tr = r.json()

if not tr["transcript_available"]:
    raise SystemExit("no captions for this video")

print(tr["source"], tr["language"], tr["segment_count"],
      tr["word_count"], tr["char_count"])

Output:

native en 61 487 2089

source is native when the captions were authored rather than machine generated, which is_generated also reports as False.

2. Filter segments by timestamp

Every segment carries start and duration in seconds, so slicing a time window is a list comprehension.

window = [s for s in tr["segments"] if 40 <= s["start"] < 60]
print(len(window), "segments between 40s and 60s")
for s in window[:4]:
    print(f"{s['start']:>6.2f} +{s['duration']:.2f}  {s['text']}")

Output:

7 segments between 40s and 60s
 40.52 +2.40  â™ª Gotta make you understand â™ª
 43.00 +2.12  â™ª Never gonna give you up â™ª
 45.20 +1.88  â™ª Never gonna let you down â™ª
 47.32 +3.80  â™ª Never gonna run around and desert you â™ª

3. Join the segments into one text block

For search or summarisation you want the flat text, which is a join away.

text = " ".join(s["text"] for s in tr["segments"])
print(len(text), "chars |", len(text.split()), "words")
print(text[:70])

Output:

2089 chars | 487 words
[â™ªâ™ªâ™ª] â™ª We're no strangers to love â™ª â™ª You know the rules and so

The part that breaks

Three failures, all hit while writing this, none of them in the network layer.

Assuming a data wrapper. The fields are at the top level. Reaching for ["data"] is the single most common mistake porting from other APIs:

Traceback (most recent call last):
  File "scrape_youtube.py", line 12, in <module>
    title = r.json()["data"]["title"]
            ~~~~~~~~^^^^^^^^
KeyError: 'data'

comment_count is None, not 0. Summing it straight is a TypeError that only fires on the videos where it is missing:

TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

keywords is empty on many videos. The Rick Astley video has 27, the NASA clip has zero, and indexing the empty one raises:

IndexError: list index out of range

One more that is easy to miss: available_languages can repeat a code. The TEDx talk lists en twice inside 28 entries, so wrap it in sorted(set(...)) before you count languages.

Full script

"""YouTube scraper: video metadata, channel uploads, transcripts.
Tested: Python 3.13.7, requests 2.34.2, 27 July 2026."""
import csv
import sys
import time

import requests

sys.stdout.reconfigure(encoding="utf-8")

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(job, timeout=60, tries=3, **params):
    """One GET against the API. Fields come back at the top level, no wrapper.

    Retried, because a long-running batch should not die on one slow response.
    """
    params["api_key"] = API_KEY
    for attempt in range(tries):
        try:
            r = requests.get(f"{BASE}/{job}", params=params, timeout=timeout)
            if r.ok:
                return r.json()
        except requests.exceptions.RequestException:
            if attempt == tries - 1:
                raise
        time.sleep(2)
    r.raise_for_status()


def get_video(url):
    v = call("youtube/video", url=url)
    return {
        "video_id": v["video_id"],
        "title": v["title"],
        "channel": v["channel_name"],
        "views": v["view_count"],
        "likes": v["like_count"],
        "comments": v["comment_count"] or 0,      # null on many videos
        "seconds": v["duration_seconds"],
        "category": v["category"],
        "published": v["publish_date"][:10],
        "keywords": len(v["keywords"]),           # empty list on many videos
    }


def get_channel_videos(channel_url, limit=30):
    c = call("youtube/channel", channel=channel_url)
    rows = [{"position": v["position"], "video_id": v["id"], "title": v["title"],
             "views": v["views"], "published": v["published"], "url": v["url"]}
            for v in c["videos"][:limit]]
    return {"name": c["channel_name"], "channel_id": c["channel_id"],
            "subscribers": c["subscriber_count"], "total_videos": c["video_count"],
            "has_more": c["has_more"]}, rows


def get_transcript(url):
    t = call("youtube/transcript", url=url)
    if not t["transcript_available"]:
        return None
    return {
        "video_id": t["video_id"],
        "language": t["language"],
        "generated": t["is_generated"],
        "segments": t["segment_count"],
        "words": t["word_count"],
        "languages": sorted(set(t["available_languages"])),   # duplicates appear
        "text": " ".join(s["text"] for s in t["segments"]),
    }


def to_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)
    return path


if __name__ == "__main__":
    meta, rows = get_channel_videos("https://www.youtube.com/@NASA")
    print(f"{meta['name']}  {meta['subscribers']:,} subscribers  "
          f"{meta['total_videos']:,} videos  more={meta['has_more']}")
    print(f"pulled {len(rows)} rows from page 1")

    newest = rows[0]
    print(f"newest: {newest['title']}  ({newest['views']})")

    video = get_video(newest["url"])
    print(f"  {video['views']:,} views | {video['likes']:,} likes | "
          f"{video['seconds']}s | {video['category']}")

    tr = get_transcript(newest["url"])
    if tr:
        print(f"  transcript: {tr['segments']} segments, {tr['words']} words, "
              f"{len(tr['languages'])} languages")
        print(f"  {tr['text'][:70]}")

    to_csv(rows, "youtube_channel_videos.csv")
    print("wrote youtube_channel_videos.csv")

The chain is the useful part: one channel call produces 30 URLs, and each URL feeds both the video and transcript endpoints without any ID extraction in between. The CSV it wrote has a header plus 30 rows.

Summary

Three endpoints cover the YouTube jobs most pipelines need, and all three take the URL you already have: the video endpoint returns 26 top-level fields with integer view and like counts, the channel endpoint returns 30 uploads per page with a continuation token for the rest, and the transcript endpoint returns timestamped segments plus a language list. The thing to carry away is that the failures are in the data shape rather than the request, so coalesce comment_count, guard keywords, and never reach for a data wrapper that does not exist. Python is used throughout here, though any language that can send a GET works the same way.

FAQ

Is it legal to scrape YouTube?

Public video metadata and captions sit in a different position from private or logged-in data, so stay on publicly visible pages and check YouTube's terms for your specific use.

How many videos does one channel request return?

Thirty per page, with has_more and a next_page_token telling you whether to request another page.

Why is my view count a string instead of a number?

You are reading view_count_text or the channel list's views field, both of which are display strings, while view_count on the video endpoint is a real integer.

How to scrape Walmart in 2026?

Coco — Fri, 31 Jul 2026 16:30:00 +0000

Three Walmart scraping jobs in Python: product data by URL, ranked search results by keyword, and review bodies by the same product URL. Every request here ran on 27 July 2026 against the Chocodata API, the screenshots are that output, and the field counts are what actually came back rather than what a docs page promises.

Why is it hard to scrape Walmart?

Walmart fronts its pages with a press-and-hold challenge for datacenter traffic, and the challenge is served with HTTP 200, so a scraper branching on r.status_code == 200 logs a clean run while writing empty rows. The product fields live in an embedded JSON blob rather than in HTML attributes, so selectors written against the rendered markup rot on a schedule nobody controls. The costliest layer is the silent one: the 200-with-challenge wastes more debugging time than the JSON reshuffles and the sponsored rows combined.

Prerequisites

Python 3.9 or newer and requests. No browser driver, no proxy pool, no account beyond a free key.

1. A free Chocodata API key. Free, no card, and usable the moment you land on the dashboard.

2. requests in a clean environment.

python -m venv .venv && . .venv/bin/activate
pip install requests

3. The Walmart URL you want. One /ip/ link covers both the product data and the reviews jobs. Tested with Python 3.13.7 and requests 2.34.2 in July 2026. Any language with an HTTP client works identically, so Node, Go and PHP need no special handling.

Scrape Walmart product data

To scrape Walmart product data, send the /ip/ URL as url and read the ten grouped objects that come back.

1. Send the product URL and inspect the shape

Print the top-level keys first, because the payload groups its fields rather than flattening them.

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(endpoint, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
        if r.ok:
            break
        time.sleep(2)
    r.raise_for_status()
    data = r.json()
    if not data:
        raise RuntimeError(f"{endpoint}: empty payload for {params}")
    return data


PRODUCT = "https://www.walmart.com/ip/10450114"

product = call("walmart/product", url=PRODUCT)
print(list(product))
print(product["general"]["title"])
print(product["price"]["price"], product["price"]["currency"])

['general', 'price', 'rating', 'seller', 'fulfillment', 'breadcrumbs', 'specifications', 'us_item_id', 'product_id', 'model']
Great Value Whole Vitamin D Milk, Gallon
3.73 USD

HTTP 200 in 4.10 s. There is no envelope, so product["general"] is the first hop and not product["data"]["general"].

2. Flatten the grouped objects into one row

Flatten once in a function so the nesting stops leaking into every call site downstream.

def flatten(p):
    return {"us_item_id": p["us_item_id"],
            "title": p["general"]["title"],
            "brand": p["general"]["brand"],
            "gtin": p["general"]["meta"]["gtin"],
            "price": p["price"]["price"],
            "currency": p["price"]["currency"],
            "rating": p["rating"]["rating"],
            "rating_count": p["rating"]["count"],
            "seller": p["seller"]["name"],
            "out_of_stock": p["fulfillment"]["out_of_stock"],
            "category": " > ".join(b["category_name"] for b in p["breadcrumbs"])}


for key, value in flatten(product).items():
    print(f"{key:<14} {value}")

us_item_id     10450114
title          Great Value Whole Vitamin D Milk, Gallon
brand          Great Value
gtin           078742351865
price          3.73
currency       USD
rating         4.6
rating_count   341232
seller         Walmart.com
out_of_stock   False
category       Food > Dairy & Eggs > Milk > Dairy Milk > Whole Milk

breadcrumbs collapses to a clean category path, which is the best category signal in the payload.

3. Convert the specifications list into a dict

Specifications arrive as a list of {"key": ..., "value": ...} objects, so build the lookup before reading anything out of it.

specs = {s["key"]: s["value"] for s in product["specifications"]}

print(len(specs), "specifications")
for key in ("Milk type", "Shelf life", "Container type", "Food preparation method"):
    print(f"  {key}: {specs[key]}")

13 specifications
  Milk type: Cow Milk
  Shelf life: 15 days
  Container type: Jug
  Food preparation method: Pasteurized

Specification keys are per-category, so a mixed catalogue needs specs.get(...) rather than fixed column names. Search takes a keyword instead of a URL.

Scrape Walmart search results

To scrape Walmart search results, send the keyword as query and read results plus related_queries off the top level.

1. Send the keyword and size the response

Check the row count and the per-row field count before writing a schema against either.

search = call("walmart/search", query="wireless mouse")

print(search["query"], "| page", search["page"], "| rows", len(search["results"]))
print(len(search["results"][0]), "fields per row |",
      len(search["related_queries"]), "related queries")

wireless mouse | page 1 | rows 61
22 fields per row | 10 related queries

HTTP 200 in 4.86 s.

2. Filter the sponsored rows out before ranking

Derive rank from the organic rows only, because position counts ads and will overstate every placement you track.

rows = search["results"]
organic = [r for r in rows if not r["sponsored"]]

print(len(rows) - len(organic), "sponsored |", len(organic), "organic")

for rank, r in enumerate(organic[:5], 1):
    print(f"organic {rank}  page pos {r['position']:>3}  "
          f"{r['rating']}  {r['reviews_count']:>6}  {r['title'][:40]}")

21 sponsored | 40 organic
organic 1  page pos   4  4.5    5161  Logitech Silent Wireless Mouse, Blue/Gra
organic 2  page pos   5  4.6    4736  Logitech Compact Wireless Mouse, Blue, W
organic 3  page pos  18  4.5    1584  Wireless Mouse, 2.4GHz with USB Receiver
organic 4  page pos  19  4.7     183  onn Silent Wireless Mouse with 5 Buttons
organic 5  page pos  22  4.5     701  memzuoix Wireless Mouse for Laptop, 5 Bu

Organic 3 sits at page position 18. Storing position as rank would have been wrong by fifteen places on that one row.

3. Harvest related queries and the seller mix

Two free signals ride along in the same response, so pull them rather than paying for a second call.

from collections import Counter

print([q["query"] for q in search["related_queries"]][:5])
print(Counter(r["seller"] for r in rows).most_common(4))

['mouse pad', 'wireless mouse and keyboard', 'wireless mouse logitech', 'laptop holder', 'wireless gaming mouse']
[('Walmart.com', 23), ('Number 8 Store', 4), ('Halokin', 4), ('VIPLIVE', 3)]

Walmart.com held 23 of the 61 rows and the rest were spread thinly across marketplace sellers. Reviews are the third job and the one that returns prose.

Pull Walmart review bodies

To scrape Walmart reviews, send the same /ip/ URL to the reviews endpoint and read both the distribution and the review bodies.

1. Send the product URL and read the distribution

Take the distribution first, since it describes all the reviews while the array describes ten of them.

reviews = call("walmart/reviews", url=PRODUCT)

print("overall", reviews["overall_rating"], "|", reviews["total_reviews"], "reviews")
print(reviews["rating_distribution"])
print(len(reviews["reviews"]), "returned |",
      sum(1 for r in reviews["reviews"] if r.get("text")), "with a body")

overall 4.6 | 343892 reviews
{'1': 19375, '2': 5331, '3': 9985, '4': 26701, '5': 282500}
10 returned | 10 with a body

HTTP 200 in 2.88 s. The distribution keys are strings, not ints, so dist["5"] and not dist[5].

2. Read the bodies with their metadata

Each review carries ten fields, and the body is the one the other endpoints cannot give you.

for r in reviews["reviews"][:3]:
    print(f"{r['rating']}*  {r['review_date']}  {r['reviewer_name']}  "
          f"+{r['positive_feedback']}/-{r['negative_feedback']}")
    print(f"    {r['title']}")
    print(f"    {r['text'][:110]}")

5*  11/22/2025  Melissa  +3/-0
    My Milky Experience
    I have been buying great value brand milk from Walmart now for over 20yrs and I have never once gotten a gallo
5*  11/22/2025  Ivan  +4/-0
    The Absolute Great Value of Vitamin D Whole Milk
    Whole Vitamin D Milk is Absolutely Essential for Adults But Even Better For The Grandkids! And We All like It
3*  12/3/2025  Larry  +4/-0
    Walmart Milk
    It always tastes a little bitter for what is supposed to be sweet milk. The lipase activity is very high. I am

Bodies averaged 547 characters and the longest ran to 1,336. All ten were verified_purchase.

3. Count themes across the negative bodies

Filter to the low ratings and count term hits, which is the cheapest useful thing to do with review text.

negative = [r for r in reviews["reviews"] if r["rating"] <= 2]
print(len(negative), "of", len(reviews["reviews"]), "returned reviews are 1-2 star")

for term in ("spoil", "sour", "expiration", "leak"):
    hits = [r for r in negative if term in r["text"].lower()]
    print(f"{term:<12} {len(hits)}")

7 of 10 returned reviews are 1-2 star
spoil        2
sour         2
expiration   4
leak         1

Four of the seven negative bodies mention the expiration date, which is a product complaint you cannot see in a 4.6 star average. Read the ratio from the distribution and the themes from the bodies, never the other way round.

The part that breaks

Nothing is at the top level except the identifiers. Reaching for a flat field is the first failure.

product["title"]

Traceback (most recent call last):
  File "walmart_product.py", line 79, in <module>
    print(product["title"])
          ~~~~~~~^^^^^^^^^
KeyError: 'title'

The title is at product["general"]["title"]. Only us_item_id, product_id and model sit flat.

model is null on plenty of products. It came back as None for this grocery item, so a NOT NULL column on it will reject rows.

The ten returned reviews are not a random sample. The distribution says 82.1% five star, and the ten returned were 7 one-star, 2 five-star and 1 three-star. Compute ratios from rating_distribution and use the bodies for qualitative themes only. Averaging the ten would have produced 2.0 against a real 4.6.

Review titles are optional, bodies are not. Seven of the ten carried a title and all ten carried text, so guard the title and rely on the body.

Search page 2 overlaps page 1. Requesting both pages for the same keyword returned 111 rows containing 90 unique ids, so 21 rows came back twice. Deduplicate on id before ranking.

page 1: 61 rows | page 2: 50 rows
combined: 111 rows, 90 unique ids, 21 repeats

The sponsored share is not a fluke of one keyword. "coffee maker" returned the same shape, 61 rows with 21 sponsored and the first organic row again at page position 4.

Full script

"""Scrape Walmart product data, search results and review bodies.

Tested on Python 3.13.7 with requests 2.34.2, July 2026.
"""
import csv
import time

import requests
from collections import Counter
from datetime import date

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

FIELDS = ["scraped_on", "us_item_id", "title", "brand", "gtin", "price",
          "currency", "rating", "rating_count", "seller", "out_of_stock",
          "category"]


def call(endpoint, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{endpoint}", params=params, timeout=60)
        if r.ok:
            break
        time.sleep(2)
    r.raise_for_status()
    data = r.json()
    if not data:
        raise RuntimeError(f"{endpoint}: empty payload for {params}")
    return data


def flatten(p):
    return {"scraped_on": date.today().isoformat(),
            "us_item_id": p["us_item_id"],
            "title": p["general"]["title"],
            "brand": p["general"]["brand"],
            "gtin": p["general"]["meta"]["gtin"],
            "price": p["price"]["price"],
            "currency": p["price"]["currency"],
            "rating": p["rating"]["rating"],
            "rating_count": p["rating"]["count"],
            "seller": p["seller"]["name"],
            "out_of_stock": p["fulfillment"]["out_of_stock"],
            "category": " > ".join(b["category_name"] for b in p["breadcrumbs"])}


def product(url):
    p = call("walmart/product", url=url)
    row = flatten(p)
    row["specifications"] = {s["key"]: s["value"] for s in p["specifications"]}
    return row


def organic_results(query, pages=1):
    seen, rows = set(), []
    for page in range(1, pages + 1):
        data = call("walmart/search", query=query, page=page)
        for r in data["results"]:
            if r["sponsored"] or r["id"] in seen:
                continue
            seen.add(r["id"])
            rows.append({"rank": len(rows) + 1,
                         "page_position": r["position"],
                         "id": r["id"],
                         "title": r["title"],
                         "rating": r.get("rating"),
                         "reviews_count": r.get("reviews_count"),
                         "seller": r["seller"]})
    return rows


def review_themes(url, terms=("spoil", "sour", "expiration", "leak")):
    data = call("walmart/reviews", url=url)
    bodies = [r for r in data["reviews"] if r.get("text")]
    negative = [r for r in bodies if r["rating"] <= 2]
    return {"overall": data["overall_rating"],
            "total": data["total_reviews"],
            "distribution": data["rating_distribution"],
            "returned": len(data["reviews"]),
            "with_body": len(bodies),
            "negative_returned": len(negative),
            "themes": {t: sum(1 for r in negative if t in r["text"].lower())
                       for t in terms}}


if __name__ == "__main__":
    URL = "https://www.walmart.com/ip/10450114"

    row = product(URL)
    print(row["title"], "|", row["price"], row["currency"], "|", row["rating"])
    print(len(row["specifications"]), "specifications |", row["category"])

    with open("walmart_products.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        w.writeheader()
        w.writerow({k: row[k] for k in FIELDS})

    organic = organic_results("wireless mouse")
    print(len(organic), "organic rows | top at page position",
          organic[0]["page_position"], "|", organic[0]["title"][:40])
    print(Counter(r["seller"] for r in organic).most_common(3))

    themes = review_themes(URL)
    print(themes["returned"], "reviews,", themes["with_body"], "with a body,",
          themes["negative_returned"], "negative")
    print(themes["themes"])

Great Value Whole Vitamin D Milk, Gallon | 3.73 USD | 4.6
13 specifications | Food > Dairy & Eggs > Milk > Dairy Milk > Whole Milk
40 organic rows | top at page position 4 | Logitech Silent Wireless Mouse, Blue/Gra
[('Walmart.com', 14), ('Halokin', 2), ('Number 8 Store', 2)]
10 reviews, 10 with a body, 7 negative
{'spoil': 2, 'sour': 2, 'expiration': 4, 'leak': 1}

Summary

All three Walmart jobs are a single GET each: a product URL returned ten grouped objects and thirteen specifications in 4.10 s, a keyword returned 61 rows in 4.86 s, and the same product URL returned ten complete review bodies plus the full star distribution in 2.88 s. What the response will not do is arrive flat or arrive representative, since the product payload nests everything under general, price and rating, and the ten returned reviews skewed 7 to 1 negative against a distribution that is 82.1% five star. Take one habit from this into your own build and make it computing ratios from rating_distribution and rank from the non-sponsored rows, because both raw counts look usable and both are wrong.

FAQ

Is scraping Walmart legal?

Public product pages are not automatically illegal to scrape, but Walmart's terms of use restrict automated collection, which makes it a contract question rather than a criminal one. Not legal advice.

Why does `product["title"]` raise a KeyError on the Walmart payload?

Because the payload is grouped, so the title lives at product["general"]["title"] while only us_item_id, product_id and model sit at the top level.

Can I average the returned Walmart reviews to get a product score?

No, the returned ten skewed heavily negative here, so read the score from rating_distribution and use the bodies for themes.

How to scrape TikTok in 2026?

Coco — Thu, 30 Jul 2026 16:30:00 +0000

Three TikTok scraping jobs in Python: a profile with its stats, the aggregate shape of an account's audience, and metadata for a single video. Every request below ran against the Chocodata API on 27 July 2026 with Python 3.13.7 and requests 2.34.2, and every output block is what my terminal actually printed.

Why is it hard to scrape TikTok?

TikTok renders profile and video pages from JavaScript, so the HTML a plain client downloads holds no follower count and no caption, and its internal endpoints expect signed parameters that are generated in the browser and expire. The layer that wastes the most time is fingerprinting, because TLS and device signals are scored ahead of your headers, so a datacenter IP with a convincing User-Agent still lands on a verification wall instead of the data.

Prerequisites

A free Chocodata API key. Sign up, confirm the email, copy the key from the dashboard. Free, no card.

Python 3.9+ and requests.

python -m venv .venv && source .venv/bin/activate
pip install requests

The TikTok URL you want to scrape, copied from the address bar.

Tested with Python 3.13.7 and requests 2.34.2 in July 2026.

How to scrape TikTok profiles?

Scraping a TikTok profile takes the profile URL and returns identity plus the follower, like and video totals in one call.

1. GET the profile endpoint with the profile URL

Pass the profile URL as url and read the fields off the top level, because there is no envelope around them.

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
PROFILE_URL = "https://www.tiktok.com/@nasa"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


profile = get_json("tiktok/profile", url=PROFILE_URL)
print(len(profile), "fields")
print(f"@{profile['uniqueId']} | {profile['nickname']} | verified={profile['verified']}")
print(profile["signature"])

13 fields
@nasa | NASA | verified=True
Making the seemingly impossible, possible.✨

HTTP 200 in 1.07s, 13 top-level fields.

2. Read statsV2 for the exact counts

The profile returns two stats objects, and statsV2 is the one carrying exact numbers rather than display-rounded ones.

print("stats  :", profile["stats"])
print("statsV2:", profile["statsV2"])

exact = int(profile["statsV2"]["followerCount"])
print(f"rounded {profile['stats']['followerCount']:,} vs exact {exact:,}")

stats  : {'followerCount': 194300, 'followingCount': 20, 'heartCount': 745200, 'videoCount': 15}
statsV2: {'followerCount': '194286', 'followingCount': '20', 'heartCount': '745186', 'videoCount': '15'}
rounded 194,300 vs exact 194,286

A 14-follower gap here, and it scales with account size. The statsV2 values are strings, so cast them at the parse boundary rather than at every comparison.

3. Append a timestamped row to CSV

Write one dated row per run, since the absolute count means little and the delta between runs is the whole point.

import csv
from datetime import datetime, timezone


def snapshot(profile, path="tiktok_profile_history.csv"):
    row = {"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
           "username": profile["uniqueId"],
           "followers": int(profile["statsV2"]["followerCount"]),
           "hearts": int(profile["statsV2"]["heartCount"]),
           "videos": int(profile["statsV2"]["videoCount"])}
    with open(path, "a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(row))
        if f.tell() == 0:
            w.writeheader()
        w.writerow(row)
    return row


print(snapshot(profile))

{'ts': '2026-07-27T13:11:42+00:00', 'username': 'nasa', 'followers': 194286, 'hearts': 745186, 'videos': 15}

Counts move between calls, so a row without a timestamp cannot be diffed later. Who those followers are is a separate endpoint, and it is best treated as an aggregate.

How to scrape TikTok followers?

Scraping TikTok followers returns pages of the public follower list, and the output worth keeping is the aggregate shape of the audience rather than a roster of accounts.

1. Fetch the first follower page

Send the username and the response returns one page plus the handles for continuing.

page = get_json("tiktok/follower", username="nasa")

print(f"{page['followers_count']} on this page | has_more={page['has_more']} "
      f"| total={page['total']:,}")
print("fields per follower:", len(page["followers"][0]))

30 on this page | has_more=True | total=194,289
fields per follower: 11

HTTP 200 in 1.39s. Thirty entries per page against a reported total of 194,289, with followerCount and followingCount on each entry.

2. Follow next_cursor while has_more is true

Loop on next_cursor with a page cap, because an uncapped loop against a six-figure follower list runs for hours and collects far more than any aggregate needs.

import time


def sample_followers(username, pages=2, delay=1.0):
    cursor, users = None, []
    for _ in range(pages):
        params = {"username": username}
        if cursor:
            params["cursor"] = cursor
        data = get_json("tiktok/follower", **params)
        users.extend(data["followers"])
        cursor = data["next_cursor"]
        if not data["has_more"] or not cursor:
            break
        time.sleep(delay)
    return users


users = sample_followers("nasa", pages=2)
print(len(users), "sampled |", len({u["uniqueId"] for u in users}), "unique")

60 sampled | 60 unique

Sixty entries and sixty distinct accounts, so the cursor advances with no overlap between pages.

3. Reduce to aggregates and drop the identity fields

Collapse the sample to counts in the same function that collects it, so nothing downstream ever holds per-account rows.

import statistics
from collections import Counter


def audience(users):
    sizes = [u["followerCount"] for u in users]
    buckets = Counter("10k+" if c >= 10000 else "100-9999" if c >= 100 else "<100"
                      for c in sizes)
    return {"sample": len(sizes),
            "median_followers": statistics.median(sizes),
            "median_following": statistics.median(u["followingCount"] for u in users),
            "verified": sum(1 for u in users if u["verified"]),
            "with_bio": sum(1 for u in users if u["signature"]),
            "buckets": dict(buckets)}


print(audience(users))

{'sample': 60, 'median_followers': 150.5, 'median_following': 468.5, 'verified': 0,
 'with_bio': 33, 'buckets': {'100-9999': 37, '<100': 23}}

Thirty-seven accounts in the 100 to 9,999 band against 23 below 100, a median of 150.5 followers, and no verified accounts in the sample. That reads as a consumer audience rather than an industry one, and it is the whole answer without keeping a single row about a person. Individual videos are the other half of the picture.

How to scrape TikTok video metadata?

Scraping TikTok video metadata takes the video URL and returns the caption, creator, thumbnail and embed markup for that clip.

1. Call oembed with the video URL

Pass the video URL and the metadata comes back in a single request.

VIDEO_URL = "https://www.tiktok.com/@scout2015/video/6718335390845095173"

video = get_json("tiktok/oembed", url=VIDEO_URL)

print(len(video), "fields |", video["type"], "|", video["embed_type"])
print(video["title"])
print(video["author_name"], "| @" + video["author_unique_id"])

17 fields | video | video
Scramble up ur name & I’ll try to guess it😍❤️ #foryoupage #petsoftiktok #aesthetic
Scout, Suki & Stella | @scout2015

HTTP 200 in 2.07s with 17 fields. title holds the caption, and type confirms the URL resolved to a video rather than a creator page.

2. Parse hashtags out of the title

Split title into caption text and hashtags, because those two go into different columns in every downstream job.

import re


def split_caption(title):
    tags = re.findall(r"#\w+", title)
    return re.sub(r"\s*#\w+", "", title).strip(), tags


caption, tags = split_caption(video["title"])
print("caption:", caption)
print("hashtags:", tags)

caption: Scramble up ur name & I’ll try to guess it😍❤️
hashtags: ['#foryoupage', '#petsoftiktok', '#aesthetic']

3. Batch a URL list with error handling

Wrap the call so one unreachable URL does not end the run, and collect rows rather than raw payloads.

def video_rows(urls, delay=1.0):
    rows = []
    for u in urls:
        try:
            d = get_json("tiktok/oembed", url=u)
        except requests.HTTPError as e:
            print(f"[skip] {u} -> {e.response.status_code}")
            continue
        caption, tags = split_caption(d["title"])
        rows.append({"id": d["id"], "author": d["author_unique_id"],
                     "caption": caption[:40], "tags": " ".join(tags),
                     "thumb": f"{d['thumbnail_width']}x{d['thumbnail_height']}",
                     "embed_chars": len(d["html"])})
        time.sleep(delay)
    return rows


for row in video_rows([VIDEO_URL]):
    print(row)

{'id': '6718335390845095173', 'author': 'scout2015', 'caption': 'Scramble up ur name & I’ll try to guess ',
 'tags': '#foryoupage #petsoftiktok #aesthetic', 'thumb': '576x1024', 'embed_chars': 957}

A 576 by 1024 thumbnail and 957 characters of embed markup per video. Download the thumbnail bytes rather than storing its URL, for the reason in the next section.

The part that breaks

There is no data envelope. Fields sit at the top level, so the reflex from other APIs fails on the first line:

KeyError: 'data'

statsV2 values are strings. Comparing them without a cast raises:

TypeError: '>' not supported between instances of 'str' and 'int'

And stats is rounded, so 194,300 there against 194,286 in statsV2. Cast statsV2 with int() once and never read stats.

oEmbed carries no playback metrics. The stats field on that response is null, so reaching into it raises:

TypeError: 'NoneType' object is not subscriptable

Treat oEmbed as the identity and embed layer. Caption, creator, thumbnail and embed HTML are what it returns.

Media URLs are signed and expire. The avatar URL I pulled carried an expiry 47.8 hours out and the thumbnail URL is signed the same way, so links stored in a database go dead within days. Mirror the image bytes at collection time.

Full script

"""Scrape TikTok profiles, follower audience shape and video metadata.
Python 3.9+, requests. Tested 27 July 2026 with 3.13.7 / requests 2.34.2.
"""
import csv
import re
import statistics
import time
from collections import Counter
from datetime import datetime, timezone

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def snapshot_profile(profile_url, path="tiktok_profile_history.csv"):
    p = get_json("tiktok/profile", url=profile_url)
    row = {"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
           "username": p["uniqueId"],
           "followers": int(p["statsV2"]["followerCount"]),
           "hearts": int(p["statsV2"]["heartCount"]),
           "videos": int(p["statsV2"]["videoCount"])}
    with open(path, "a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(row))
        if f.tell() == 0:
            w.writeheader()
        w.writerow(row)
    return row


def audience_profile(username, pages=2, delay=1.0):
    cursor, users = None, []
    for _ in range(pages):
        params = {"username": username}
        if cursor:
            params["cursor"] = cursor
        data = get_json("tiktok/follower", **params)
        users.extend(data["followers"])
        cursor = data["next_cursor"]
        if not data["has_more"] or not cursor:
            break
        time.sleep(delay)

    sizes = [u["followerCount"] for u in users]
    buckets = Counter("10k+" if c >= 10000 else "100-9999" if c >= 100 else "<100"
                      for c in sizes)
    return {"sample": len(sizes),
            "median_followers": statistics.median(sizes),
            "median_following": statistics.median(u["followingCount"] for u in users),
            "verified": sum(1 for u in users if u["verified"]),
            "with_bio": sum(1 for u in users if u["signature"]),
            "buckets": dict(buckets)}


def split_caption(title):
    tags = re.findall(r"#\w+", title)
    return re.sub(r"\s*#\w+", "", title).strip(), tags


def video_rows(urls, delay=1.0):
    rows = []
    for u in urls:
        try:
            d = get_json("tiktok/oembed", url=u)
        except requests.HTTPError as e:
            print(f"[skip] {u} -> {e.response.status_code}")
            continue
        caption, tags = split_caption(d["title"])
        rows.append({"id": d["id"], "author": d["author_unique_id"],
                     "caption": caption[:40], "tags": " ".join(tags),
                     "thumb": f"{d['thumbnail_width']}x{d['thumbnail_height']}",
                     "embed_chars": len(d["html"])})
        time.sleep(delay)
    return rows


if __name__ == "__main__":
    print(snapshot_profile("https://www.tiktok.com/@nasa"))
    print(audience_profile("nasa", pages=2))
    for row in video_rows(["https://www.tiktok.com/@scout2015/video/6718335390845095173"]):
        print(row)

Summary

One request each covers the three jobs: a profile URL returns 13 fields with exact totals in about a second, a username returns 30 followers per page with a cursor that reaches 60 unique accounts across two calls, and a video URL returns 17 fields of caption, creator, thumbnail and embed markup. Any language with an HTTP client works the same way, so Node, Go and Ruby need no special handling. The habit to build in from the start is casting statsV2 and ignoring stats, because the rounded block silently merges accounts that differ by hundreds of followers.

FAQ

Can I get a TikTok video's view count from the oEmbed response?

No, that response carries the caption, creator, thumbnail and embed HTML rather than playback metrics.

Why do TikTok avatar and thumbnail URLs stop working after a while?

They are signed with an expiry, and the avatar I pulled expired 47.8 hours after the request, so mirror the image instead of storing the link.

Is scraping TikTok legal?

Public data is not automatically illegal to collect, but TikTok's terms restrict automated access, which makes it a contract question rather than a criminal one, and this is not legal advice.

How to scrape Reddit in 2026?

Coco — Wed, 29 Jul 2026 16:30:00 +0000

Three Reddit scraping jobs in Python: a single post with its comment tree, a subreddit listing, and search results. Every request below ran against the Chocodata API on 27 July 2026 with Python 3.13.7 and requests 2.34.2, and every output block is what my terminal actually printed.

Why is it hard to scrape Reddit?

Reddit blocks datacenter IPs on the public JSON views, renders the logged-out site from JavaScript so the HTML you download holds no scores, and serves the same content through several front ends that do not all expose vote counts. The one that costs the most time is comment pagination, because a thread returns a slice of its discussion by default and a partial tree is indistinguishable from a complete one unless you check the count.

Prerequisites

A free Chocodata API key. Sign up, confirm the email, copy the key from the dashboard. Free, no card.

Python 3.9+ and requests.

python -m venv .venv && source .venv/bin/activate
pip install requests

The Reddit URL you want to scrape, copied from the address bar.

Tested with Python 3.13.7 and requests 2.34.2 in July 2026.

How to scrape Reddit posts?

Scraping a Reddit post takes the thread URL and returns the post plus its comment tree in one call.

1. GET the post endpoint with the thread URL

Pass the post URL as url and read the fields off the top level of the response, because there is no envelope around them.

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
POST_URL = "https://www.reddit.com/r/webscraping/comments/p64rqq/linkedin_scraping/"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


data = get_json("reddit/post", url=POST_URL)
print(list(data.keys()))
print(data["post"]["title"])

['post', 'comments', 'comments_returned', '_meta']
LinkedIn Scraping

HTTP 200 in 5.04s. Four top-level keys, and post is where the thread lives.

2. Pull the post object off the top level

The post object carries score, ratio, comment count, author and body, so one request answers the engagement questions.

post = data["post"]

print(f"{post['score']} points at {post['upvote_ratio']:.2f} ratio")
print(f"{post['num_comments']} comments | u/{post['author']['username']} | {post['created'][:10]}")
print(f"{post['domain']} | locked={post['is_locked']} | body {len(post['body'])} chars")

12 points at 0.93 ratio
17 comments | u/pizzihut | 2021-08-17
self.webscraping | locked=None | body 224 chars

author is a nested object, not a string. That trips code written against the subreddit endpoint, where the same field is a plain username.

3. Flatten the nested reply tree

Comments come back as a tree where each comment holds its own replies list, so recursion turns it into rows you can store.

def flatten(comments, depth=0, out=None):
    out = [] if out is None else out
    for c in comments:
        out.append({"depth": depth, "author": c["author"]["username"],
                    "score": c["score"], "body": c["body"]})
        flatten(c["replies"], depth + 1, out)
    return out


rows = flatten(data["comments"])
for c in rows[:4]:
    print("  " * c["depth"], f"[{c['score']:>2}] {c['author']}: {c['body'][:46]}")

print(f"{len(rows)} flattened / {data['comments_returned']} returned "
      f"/ {post['num_comments']} reported")
print("truncated:", data["_meta"]["truncated"])

 [ 3] [deleted]: I need to scan profiles of generel information
   [ 3] pizzihut: I need to scan profiles of generel information
 [ 2] nubela: Try Proxycurl! It has a dedicated Linkedin Scr
   [ 1] pizzihut: Hi! Just tried the free subscription, but it s
8 flattened / 8 returned / 17 reported
truncated: True

Four top-level threads flattened to 8 comments against the 17 Reddit reports, and _meta.truncated says so explicitly. Assert on comments_returned before you treat a thread as complete. Listings behave differently, because there the pagination handle is a cursor.

How to scrape Reddit subreddits?

Scraping a subreddit returns the community listing as an array of posts plus a cursor for the next batch.

1. Fetch the first 25 posts

Send the subreddit name and the response returns the listing under posts with the sort it used.

sub = get_json("reddit/subreddit", subreddit="webscraping")

print(sub["sort"], "|", sub["total_results"], "posts |", sub["after_cursor"])
for p in sub["posts"][:2]:
    print(f"{p['score']:>4} {p['num_comments']:>4} {p['upvote_ratio']:.2f}  {p['title'][:44]}")

hot | 25 posts | dDNfMXV3YWxodg==
   7   12 1.00  I own a wood factory and want to use webscra
   2    8 0.67  Vinted cloudfare detection

HTTP 200 in 1.87s. Each post carries 13 fields including awards, domain, external_url and permalink. Adding sort="top" with t="week" returned 16 posts for the week, led by an 83-point thread.

2. Page with after_cursor

Feed after_cursor back as after to walk deeper, and stop when the cursor comes back empty.

import time


def fetch_pages(subreddit, n_pages=2, delay=1.0):
    cursor, seen = None, []
    for _ in range(n_pages):
        params = {"subreddit": subreddit}
        if cursor:
            params["after"] = cursor
        page = get_json("reddit/subreddit", **params)
        seen.extend(page["posts"])
        cursor = page["after_cursor"]
        if not cursor:
            break
        time.sleep(delay)
    return seen


allp = fetch_pages("webscraping", 2)
print(len(allp), "posts |", len({p["id"] for p in allp}), "unique ids")

50 posts | 50 unique ids

Fifty rows and fifty distinct ids, so the cursor advances cleanly with no overlap between pages. The id set is the check worth keeping, because a cursor that stalls silently returns duplicates rather than an error.

3. Write the listing to CSV

Select the fields you need and write rows, since keeping the whole payload per post buys nothing for trend work.

import csv

FIELDS = ["id", "title", "score", "num_comments", "upvote_ratio",
          "author", "created", "permalink"]

with open("subreddit_posts.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=FIELDS)
    w.writeheader()
    for p in allp:
        w.writerow({k: p.get(k) for k in FIELDS})

print(f"wrote {len(allp)} rows to subreddit_posts.csv")
print("mean score:", round(sum(p["score"] for p in allp) / len(allp), 1))

wrote 50 rows to subreddit_posts.csv
mean score: 11.3

A mean of 11.3 across 50 posts is the baseline that makes a 83-point thread readable as an outlier. Search covers the same content across communities rather than inside one.

How to scrape Reddit search results?

Scraping Reddit search results returns a ranked mix of communities and posts for a keyword.

1. Query the search endpoint

Send the query and the response returns positioned results with the type of each hit.

search = get_json("reddit/search", q="web scraping")

print(search["query"], "|", search["sort"], "|", search["total_results"], "results")
for r in search["results"][:4]:
    print(f"{r['position']:>3}  {r['result_type']:<9} {r['title'][:44]}")

web scraping | relevance | 25 results
  1  subreddit webscraping
  2  subreddit WebScrapingInsider
  3  subreddit Scraping the web
  4  post      The Complete Web Scraping & Anti-Bot Bypass

HTTP 200 in 2.55s under the default relevance sort.

2. Filter on result_type

Split the results by result_type so community hits and post hits go into different pipelines.

from collections import Counter

posts = [r for r in search["results"] if r["result_type"] == "post"]
subs = [r for r in search["results"] if r["result_type"] == "subreddit"]

print(len(subs), "communities |", len(posts), "posts")
for name, n in Counter(p["subreddit"] for p in posts).most_common(5):
    print(f"r/{name:<20} {n}")

3 communities | 22 posts
r/webscraping          2
r/scrapingtheweb       2
r/programming          2
r/automation           1
r/PythonLearning       1

Three communities and 22 posts, with the topic spread thin past the top three subreddits.

3. Sort by new for monitoring

Pass sort="new" when the job is monitoring rather than research, because relevance ordering hides anything posted this morning.

newest = get_json("reddit/search", q="web scraping", sort="new")
fresh = [r for r in newest["results"] if r["result_type"] == "post"]

for r in fresh[:4]:
    print(r["created"][:16], "|", f"r/{r['subreddit']}", "|", r["title"][:40])

2026-07-27T12:00 | r/LocalLLaMA | XYZAILab/XYZ-Aquila-mini · Hugging Face
2026-07-27T10:23 | r/jobsearch | Roast my resume
2026-07-27T09:14 | r/BotNation | What's the best way to learn web scrapin
2026-07-27T08:38 | r/hireforgigs | [for hire] I'm available: Web Designer/D

Same-day results, so a cron job on this call is a working keyword alert.

The part that breaks

There is no data envelope. Every field sits at the top level, so the reflex from other APIs fails immediately:

KeyError: 'data'

author changes shape between endpoints. On a post it is an object, on a listing it is a string:

AttributeError: 'dict' object has no attribute 'lower'

Normalise it once at the parse boundary with a["username"] if isinstance(a, dict) else a.

Vote counts are not on every surface. Search results can return score as null, and sorting on it raises:

TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

Use key=lambda r: r["score"] or 0 or filter the nulls out before sorting.

Comment trees arrive truncated by default. comments_returned was 8 against a num_comments of 17, with _meta.truncated set to True. Treat a thread as partial unless those two numbers agree.

Full script

"""Scrape Reddit posts, subreddit listings and search results.
Python 3.9+, requests. Tested 27 July 2026 with 3.13.7 / requests 2.34.2.
"""
import csv
import time
from collections import Counter

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def get_json(path, tries=3, **params):
    """One GET, retried, so a batch run survives a dropped response."""
    params["api_key"] = API_KEY
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def author_name(a):
    return a["username"] if isinstance(a, dict) else a


def scrape_post(post_url):
    data = get_json("reddit/post", url=post_url)
    post = data["post"]

    def flatten(comments, depth=0, out=None):
        out = [] if out is None else out
        for c in comments:
            out.append({"depth": depth, "author": author_name(c["author"]),
                        "score": c["score"], "body": c["body"]})
            flatten(c["replies"], depth + 1, out)
        return out

    comments = flatten(data["comments"])
    if data["comments_returned"] < post["num_comments"]:
        print(f"[warn] {data['comments_returned']}/{post['num_comments']} comments returned")
    return post, comments


def scrape_subreddit(subreddit, n_pages=2, delay=1.0, path="subreddit_posts.csv"):
    cursor, seen = None, []
    for _ in range(n_pages):
        params = {"subreddit": subreddit}
        if cursor:
            params["after"] = cursor
        page = get_json("reddit/subreddit", **params)
        seen.extend(page["posts"])
        cursor = page["after_cursor"]
        if not cursor:
            break
        time.sleep(delay)

    fields = ["id", "title", "score", "num_comments", "upvote_ratio",
              "author", "created", "permalink"]
    with open(path, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=fields)
        w.writeheader()
        for p in seen:
            w.writerow({k: p.get(k) for k in fields})
    return seen


def search_reddit(query, sort=None):
    params = {"q": query}
    if sort:
        params["sort"] = sort
    results = get_json("reddit/search", **params)["results"]
    posts = [r for r in results if r["result_type"] == "post"]
    return posts, Counter(p["subreddit"] for p in posts)


if __name__ == "__main__":
    post, comments = scrape_post(
        "https://www.reddit.com/r/webscraping/comments/p64rqq/linkedin_scraping/")
    print(post["title"], "|", post["score"], "points |", len(comments), "comments")

    listing = scrape_subreddit("webscraping", n_pages=2)
    print(len(listing), "posts written |",
          "mean score", round(sum(p["score"] for p in listing) / len(listing), 1))

    posts, where = search_reddit("web scraping")
    print(len(posts), "search posts |", where.most_common(3))

Summary

One request each covers the three jobs: a thread URL returns the post and its comment tree in about five seconds, a subreddit name returns 25 posts with a cursor that pages cleanly to 50 unique ids, and a keyword returns 25 ranked results split across communities and posts. Any language with an HTTP client works the same way, so Node, Go and Ruby need no special handling. The thing to build in from the start is the comments_returned check, because a truncated thread parses perfectly and quietly loses most of the discussion.

FAQ

Do I need a Reddit account or OAuth token?

No, the requests above authenticate with a Chocodata key and never touch Reddit credentials or an app registration.

Why is score null on some Reddit results?

Not every Reddit surface exposes vote counts, so search results return identity, subreddit and timestamps with score left null rather than guessed.

Is scraping Reddit legal?

Public pages are not automatically illegal to scrape, but Reddit's user terms restrict automated collection, which makes it a contract question rather than a criminal one, and this is not legal advice.

How to scrape Instagram in 2026?

Coco — Tue, 28 Jul 2026 09:00:00 +0000

Scraping Instagram public profile analytics in Python: follower count, following count, post count, bio, display name and verification status for business and creator accounts, using the Chocodata profile endpoint. Every request below was executed on 27 July 2026 and the screenshots are that output.

Here is what the finished script prints:

canva     2,578,854 followers   2,819 posts  verified=True
figma       938,129 followers   1,040 posts  verified=True

wrote instagram_profiles.csv (2 rows, captured=2026-07-27)

TL;DR

One GET returns 18 top-level fields for a public profile. No wrapper object, so r.json()["follower_count"] is the whole access path.

Four of the 18 are None on public profiles: external_url, category, is_business, posts. Use .get() or you will ship a NoneType traceback.

og_description disagrees with the numeric fields. For canva it says 1,450 Following while following_count is 1405. Trust the integers.

follower_count moves between calls. Canva returned 2,578,851 and 2,578,854 minutes apart, so assert on a range, never on equality.

Tested on Python 3.13.7 and requests 2.34.2, July 2026.

Why is it hard to scrape Instagram?

Scraping Instagram is hard because the public profile page hydrates its counts from an embedded JSON payload whose key names are hashed and rotate between deploys, so selector-based parsers return zero rows instead of raising. On top of that a logged-out client gets a small number of views before the page is replaced by a sign-in interstitial that still looks like a successful response. The layer that wastes the most time is IP reputation, because the same code that works from a home connection hits a checkpoint challenge from any cloud range.

Prerequisites

To scrape Instagram profile data with the code below you need three things, and the first is free.

A free Chocodata API key. Sign up, copy the key from the dashboard, and pass it as api_key on every request. Free to start, no card required.

Python 3.9 or newer and requests. No browser driver, no proxy, no Instagram account.

python -m venv .venv
pip install requests

The handles you want to pull, for example canva and figma.

Tested on Python 3.13.7 with requests 2.34.2 in July 2026.

How to scrape Instagram profiles?

One GET to the profile endpoint returns the full public profile as JSON, and the three steps below take it from a URL to a CSV row.

1. Build the request URL

To scrape an Instagram profile, the endpoint is instagram/profile and it takes two query parameters: the handle as username and your key as api_key.

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"

url = f"{BASE}/instagram/profile"
params = {"username": "canva", "api_key": API_KEY}

r = requests.get(url, params=params, timeout=60)
print(r.status_code, r.headers.get("content-type"))

200 application/json

2. Fetch the profile JSON and check a field, not the status

The Instagram profile payload puts all 18 fields at the top level, so guard on a field you expect rather than on the status code, because a logged-out interstitial upstream can still produce a well-formed response.

r.raise_for_status()
profile = r.json()

if "username" not in profile:
    raise RuntimeError("no profile in response - check the handle")

print(len(profile), "fields")
for k in ["username", "full_name", "follower_count", "following_count",
          "posts_count", "is_verified", "is_private", "biography"]:
    print(f"{k:16} {profile[k]}")

18 fields
username         canva
full_name        Canva
follower_count   2578854
following_count  1405
posts_count      2819
is_verified      True
is_private       False
biography        What will you design today?

The remaining populated fields are id, url, profile_pic_url, og_description, source and data_source.

3. Flatten the profile into a CSV row

Flattening the profile into one dated row is what turns a lookup into a dataset, so the capture date goes in before anything else.

import csv
from datetime import date

FIELDS = ["captured", "username", "full_name", "follower_count",
          "following_count", "posts_count", "is_verified", "url"]

def to_row(p):
    row = {k: p.get(k) for k in FIELDS[1:]}
    row["captured"] = date.today().isoformat()
    return row

with open("instagram_profiles.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=FIELDS)
    w.writeheader()
    w.writerow(to_row(profile))

captured,username,full_name,follower_count,following_count,posts_count,is_verified,url
2026-07-27,canva,Canva,2578854,1405,2819,True,https://www.instagram.com/canva/

Note the .get() in to_row. That is not defensive style for its own sake, and the reason is in the failures section below. The same request works from any language with an HTTP client, so Node, Go and PHP need nothing beyond their standard library.

Track a competitor set over time

Tracking an Instagram competitor set is the same profile call in a loop, with the capture date doing the real work. Absolute follower counts tell you who is bigger. The diff between two captures tells you who is growing.

import time

HANDLES = ["canva", "figma"]

def fetch(handle, tries=3):
    """One handle, retried, so a batch run survives a dropped response."""
    for attempt in range(tries):
        r = requests.get(f"{BASE}/instagram/profile",
                         params={"username": handle, "api_key": API_KEY},
                         timeout=60)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()

rows = []
for h in HANDLES:
    p = fetch(h)
    rows.append(to_row(p))
    print(f"{p['username']:8} {p['follower_count']:>10,} followers  "
          f"{p['posts_count']:>6,} posts  verified={p['is_verified']}")
    time.sleep(2)

canva     2,578,854 followers   2,819 posts  verified=True
figma       938,129 followers   1,040 posts  verified=True

Append rather than overwrite, run it on a schedule, and compute the delta at read time:

import collections

history = collections.defaultdict(list)
with open("instagram_profiles.csv", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        history[row["username"]].append((row["captured"], int(row["follower_count"])))

for handle, points in history.items():
    points.sort()
    if len(points) > 1:
        delta = points[-1][1] - points[0][1]
        print(f"{handle:8} {points[0][0]} -> {points[-1][0]}  {delta:+,}")

One capture gives you nothing from that loop, which is the point. Take the first snapshot before you have a reason to.

The part that breaks

Four failures, all hit while writing this.

Four fields are None on public profiles. external_url, category, is_business and posts come back empty, and bracket access stores the None silently until something calls a method on it.

print({k: profile[k] for k in ["external_url", "category", "is_business", "posts"]})

profile["category"].lower()

{'external_url': None, 'category': None, 'is_business': None, 'posts': None}
Traceback (most recent call last):
  File "instagram_profiles.py", line 24, in <module>
    profile["category"].lower()
AttributeError: 'NoneType' object has no attribute 'lower'

Build the schema around the fourteen populated fields and read the rest with .get().

There is no wrapper object. The fields are at the top level, so reaching for one costs you a lookup that never resolves.

profile["data"]

Traceback (most recent call last):
  File "instagram_profiles.py", line 31, in <module>
    profile["data"]
KeyError: 'data'

og_description disagrees with the numeric fields. It is a display string, rounded and formatted, and it does not match the integers beside it.

print(profile["following_count"])   # 1405
print(profile["og_description"])
# 3M Followers, 1,450 Following, 2,819 Posts - See Instagram photos and videos from Canva (@canva)

Figma shows the same split: following_count is 87 while its og_description reads 93 Following. Parse the integers, never the sentence.

Counts move between calls. Canva returned 2578851 and then 2578854 within the same session, and Figma moved by one in the other direction.

assert profile["follower_count"] == 2578851        # fails on the next run

Traceback (most recent call last):
  File "instagram_profiles.py", line 44, in <module>
    assert profile["follower_count"] == 2578851
AssertionError

assert profile["follower_count"] > 1_000_000       # what you actually want

Any regression test that pins an exact follower count goes red on its second run.

Full script

"""Instagram public profile analytics via the Chocodata profile endpoint.
Tested: Python 3.13.7, requests 2.34.2, 27 July 2026.
Public business and creator profiles only.
"""
import collections
import csv
import time
from datetime import date

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
CSV_PATH = "instagram_profiles.csv"

FIELDS = ["captured", "username", "full_name", "follower_count",
          "following_count", "posts_count", "is_verified", "url"]


def fetch_profile(handle, timeout=60):
    """Return the profile dict for one public handle."""
    r = requests.get(f"{BASE}/instagram/profile",
                     params={"username": handle, "api_key": API_KEY},
                     timeout=timeout)
    r.raise_for_status()
    profile = r.json()
    if "username" not in profile:
        raise RuntimeError(f"no profile returned for {handle!r}")
    if profile.get("is_private"):
        raise RuntimeError(f"{handle!r} is private - counts are withheld")
    return profile


def to_row(profile):
    """Flatten a profile into one dated CSV row. .get() because four
    fields come back empty on public profiles."""
    row = {k: profile.get(k) for k in FIELDS[1:]}
    row["captured"] = date.today().isoformat()
    return row


def capture(handles, path=CSV_PATH, delay=2.0):
    """Append one dated row per handle. Safe to run on a schedule."""
    try:
        with open(path, encoding="utf-8") as f:
            existing = bool(f.readline())
    except FileNotFoundError:
        existing = False

    rows = []
    for i, handle in enumerate(handles):
        profile = fetch_profile(handle)
        rows.append(to_row(profile))
        print(f"{profile['username']:8} {profile['follower_count']:>10,} followers  "
              f"{profile['posts_count']:>6,} posts  verified={profile['is_verified']}")
        if i < len(handles) - 1:
            time.sleep(delay)

    with open(path, "a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        if not existing:
            w.writeheader()
        w.writerows(rows)

    print(f"\nwrote {path} ({len(rows)} rows, captured={rows[0]['captured']})")
    return rows


def deltas(path=CSV_PATH):
    """Follower change per handle between the first and latest capture."""
    history = collections.defaultdict(list)
    with open(path, encoding="utf-8") as f:
        for row in csv.DictReader(f):
            history[row["username"]].append(
                (row["captured"], int(row["follower_count"])))

    out = {}
    for handle, points in history.items():
        points.sort()
        if len(points) > 1:
            out[handle] = points[-1][1] - points[0][1]
            print(f"{handle:8} {points[0][0]} -> {points[-1][0]}  {out[handle]:+,}")
    return out


if __name__ == "__main__":
    capture(["canva", "figma"])
    deltas()

First run writes the header plus two rows and prints nothing from deltas(), because one capture has nothing to compare against. That is expected and it is the whole reason to start capturing early.

Summary

One GET to the profile endpoint returns 18 top-level fields for a public Instagram profile, and the fourteen populated ones cover the analytics people actually chart: display name, bio, follower count, following count, post count and verification status. What does not work is treating any single reading as exact, since follower counts drift between calls, og_description rounds and disagrees with the integers next to it, and four fields are always empty on public profiles. The one thing to carry away is that this is a time-series job, so write the capture date into the row on the first run rather than the tenth.

FAQ

Is scraping Instagram profiles legal?

Public profile pages are not automatically off limits, but Instagram's terms prohibit automated collection, which makes this a contract question rather than a criminal one, and none of this is legal advice.

Can I scrape a private Instagram account?

No, and the script above raises on is_private rather than trying, because a private account's counts are withheld from anyone it has not approved.

Why do my follower counts change between two runs minutes apart?

Because the count is live and moves on large accounts, which is why you compare dated captures instead of asserting on an exact number.

How to scrape LinkedIn in 2026?

Coco — Mon, 27 Jul 2026 18:35:32 +0000

Three LinkedIn scraping jobs in Python with no session cookie anywhere: profiles, company pages and job listings. Every request below was executed on 27 July 2026 against Chocodata and the numbers are that run.

HTTP 200 in 3.07s
fields: 12
             name: Reid Hoffman
         headline: Co-Founder, Executive Board Chair
  current_company: Manas AI
        followers: 2782185

TL;DR

linkedin/profile takes a profile URL and returns 12 top-level fields in about 3 seconds, with no data envelope to unwrap.

linkedin/company returns 13 fields of firmographics, including employee_count at 16,850 for Stripe against a banded company_size of 5,001-10,000.

linkedin/jobsearch returned 10 live postings for backend engineer in the United Kingdom in 1.47s, each with a direct url.

specialties is a comma string on some company pages and None on others, so .split(",") raises AttributeError unless you guard it.

Why is it hard to scrape LinkedIn?

LinkedIn treats anonymous traffic as hostile: most of a profile sits behind an auth wall, class names are hashed and rotate so saved selectors rot within weeks, and rate limits attach to the account rather than the IP. The expensive one is HTTP 999, the non-standard code LinkedIn returns to suspected automation, because retry logic written for 429 does not recognise it and keeps hammering until the identity is burned. That mismatch costs more debugging time than the markup churn.

Prerequisites

A free Chocodata API key, taken from the dashboard after sign-up. Free tier, no card.

Python 3.9+ and requests. Tested here on Python 3.13.7 with requests 2.34.2, July 2026. Other languages hit the same endpoints, but every snippet below is Python.

python -m venv .venv && . .venv/bin/activate
pip install requests

The LinkedIn URL you want to scrape. A profile URL copied from the address bar is enough to start. No LinkedIn account is involved at any point.

Fetch a profile from its URL

To scrape a LinkedIn profile, send the profile URL as url and read the fields straight off the response. There is no wrapper object.

1. Send the URL and assert on a field, not the status

import time

import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"
URL = "https://www.linkedin.com/in/reidhoffman"

r = requests.get(f"{BASE}/linkedin/profile",
                 params={"url": URL, "api_key": API_KEY}, timeout=90)
r.raise_for_status()
profile = r.json()

assert profile.get("name"), "empty payload behind a 200"
print("fields:", len(profile))
for k in ("id", "name", "headline", "location", "current_company", "followers", "connections"):
    print(f"{k:>16}: {profile[k]}")

fields: 12
              id: reidhoffman
            name: Reid Hoffman
        headline: Co-Founder, Executive Board Chair
        location: United States
 current_company: Manas AI
       followers: 2782185
     connections: 500+

2. Iterate the experience and education lists

print("experience:", len(profile["experience"]), "| education:", len(profile["education"]))

for e in profile["experience"][:6]:
    print("  ", e["company"])
for s in profile["education"][:4]:
    print("  ", s["school"])

experience: 13 | education: 6
   Manas AI
   Inflection AI
   Greylock
   Village Global
   Microsoft
   Aurora
   Università degli Studi di Perugia
   University of Oulu
   Babson College
   Oxford University

3. Flatten the nested lists into one CSV row

import csv

row = {
    "id": profile["id"],
    "name": profile["name"],
    "headline": profile["headline"],
    "location": profile["location"],
    "current_company": profile["current_company"],
    "followers": profile["followers"],
    "companies": "|".join(e["company"] for e in profile["experience"]),
    "schools": "|".join(s["school"] for s in profile["education"]),
}

with open("profiles.csv", "w", newline="", encoding="utf-8") as fh:
    w = csv.DictWriter(fh, fieldnames=list(row))
    w.writeheader()
    w.writerow(row)

print(len(row["companies"].split("|")), "companies flattened")

13 companies flattened

Pipe-joining keeps one profile on one row, which matters once you enrich thousands of them. Company names pulled out of experience are the handles the next section takes.

Fetch a company page by handle

To scrape a LinkedIn company page, pass the handle as company. The response carries the firmographics a profile does not.

1. Request the page and print the identity fields

r = requests.get(f"{BASE}/linkedin/company",
                 params={"company": "stripe", "api_key": API_KEY}, timeout=90)
r.raise_for_status()
company = r.json()

print("fields:", len(company))
for k in ("id", "name", "industry", "founded", "headquarters"):
    print(f"{k:>14}: {company[k]}")

fields: 13
            id: stripe
          name: Stripe
      industry: Technology, Information and Internet
       founded: 2010
  headquarters: South San Francisco, California

2. Read the size and reach signals

print(f'employee_count : {company["employee_count"]:,}')
print(f'company_size   : {company["company_size"]}')
print(f'followers      : {company["followers"]:,}')
print(f'website        : {company["website"]}')

employee_count : 16,850
company_size   : 5,001-10,000 employees
followers      : 1,591,183
website        : https://stripe.com

employee_count is the number of profiles listing that page as their employer, so it drifts a few people between calls and is not the same measure as the banded company_size.

3. Batch several handles into one table

def company_row(handle):
    r = requests.get(f"{BASE}/linkedin/company",
                     params={"company": handle, "api_key": API_KEY}, timeout=90)
    r.raise_for_status()
    return r.json()

for handle in ("stripe", "spotify", "microsoft"):
    d = company_row(handle)
    print(f'{d["name"]:<10}{d["employee_count"]:>9,}{d["followers"]:>12,}  {d["industry"][:34]}')

Stripe       16,850   1,591,183  Technology, Information and Intern
Spotify      19,179   4,628,644  Musicians
Microsoft   233,255  28,719,471  Software Development

Note Spotify's industry reading as Musicians, which is the value on the page rather than a parsing artefact, so normalise industry yourself if you plan to group on it. Hiring volume is the other size signal, and that comes from the jobs endpoint.

Search jobs by keyword and location

To scrape LinkedIn jobs, pass keywords and location. The response is a ranked list of live postings, each with a direct URL.

1. Run the search and inspect the row shape

r = requests.get(f"{BASE}/linkedin/jobsearch",
                 params={"keywords": "backend engineer",
                         "location": "United Kingdom",
                         "api_key": API_KEY}, timeout=90)
r.raise_for_status()
jobs = r.json()

print("results:", len(jobs["results"]))
print("keys:", [k for k in jobs["results"][0] if k != "company_logo"])

results: 10
keys: ['position', 'id', 'job_id', 'title', 'url', 'company', 'company_url', 'location', 'posted_date', 'posted_label', 'salary']

2. Print the listings as a fixed-width table

for j in jobs["results"][:6]:
    print(f'{j["position"]:>2} {j["posted_date"]} {j["company"][:22]:<24}{j["title"][:34]}')

 1 2026-07-08 Happl                   Backend Engineer
 2 2026-07-19 Deliveroo               Software Engineer
 3 2026-06-17 Conduct                 Backend Engineer
 4 2026-07-19 MAI UK                  Backend Engineer
 5 2026-07-24 Midjourney              Backend Engineer, Core Systems
 6 2026-07-16 nPlan                   Backend Engineer

3. Filter to fresh postings by date

CUTOFF = "2026-07-01"

fresh = [j for j in jobs["results"] if j["posted_date"] >= CUTOFF]
fresh.sort(key=lambda j: j["posted_date"], reverse=True)

print(f"{len(fresh)} of {len(jobs['results'])} posted since {CUTOFF}")
for j in fresh[:5]:
    print(f'  {j["posted_date"]}  {j["posted_label"]:<12}{j["company"][:20]}')

6 of 10 posted since 2026-07-01
  2026-07-24  2 days ago  Midjourney
  2026-07-21  6 days ago  Thought Machine
  2026-07-19  1 week ago  Deliveroo
  2026-07-19  1 week ago  MAI UK
  2026-07-16  1 week ago  nPlan

posted_date is an ISO string, so string comparison sorts it correctly with no datetime parsing. Position 1 in the raw response was dated 2026-07-08 while position 5 was 2026-07-24, so rank is not recency.

The part that breaks

Optional nested fields, not the requests. specialties came back as a 23-item comma string for microsoft and as None for stripe, and founded did the opposite, None for Microsoft and 2010 for Stripe. Calling a string method on either one raises:

Traceback (most recent call last):
  File "scrape.py", line 57, in <module>
    tags = company["specialties"].split(",")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

Coalesce before you touch it, and the same guard covers every optional field on the page:

raw = company.get("specialties") or ""
tags = [t.strip() for t in raw.split(",") if t.strip()]

Three more shapes worth knowing. Inside experience, only company was populated on both profiles tested, with title and date_range unset, so build on company names rather than job titles. connections came back 500+ for reidhoffman and None for williamhgates, so it is a display string and not a number. And salary was None on all 10 United Kingdom job rows, because most public postings do not publish a figure, which makes posted_date and company the only fields worth filtering on.

Full script

"""LinkedIn profile, company and job scraper. Python 3.13.7, requests 2.34.2."""
import csv
import sys
import time
import requests

BASE = "https://api.chocodata.com/api/v1"
API_KEY = "YOUR_API_KEY"


def call(path, params, timeout=90, tries=3):
    """One GET, retried, so a batch run survives a dropped response."""
    params = dict(params, api_key=API_KEY)
    for attempt in range(tries):
        r = requests.get(f"{BASE}/{path}", params=params, timeout=timeout)
        if r.ok:
            return r.json()
        time.sleep(2)
    r.raise_for_status()


def profile(url):
    p = call("linkedin/profile", {"url": url})
    if not p.get("name"):
        raise RuntimeError(f"empty payload for {url}")
    return {
        "id": p["id"],
        "name": p["name"],
        "headline": p["headline"],
        "location": p["location"],
        "current_company": p["current_company"],
        "followers": p["followers"],
        "companies": "|".join(e["company"] for e in p.get("experience") or []),
        "schools": "|".join(s["school"] for s in p.get("education") or []),
    }


def company(handle):
    c = call("linkedin/company", {"company": handle})
    raw = c.get("specialties") or ""
    return {
        "id": c["id"],
        "name": c["name"],
        "industry": c["industry"],
        "founded": c.get("founded"),
        "headquarters": c["headquarters"],
        "employee_count": c["employee_count"],
        "company_size": c["company_size"],
        "followers": c["followers"],
        "specialties": [t.strip() for t in raw.split(",") if t.strip()],
    }


def jobs(keywords, location, since=None):
    rows = call("linkedin/jobsearch",
                {"keywords": keywords, "location": location})["results"]
    if since:
        rows = [j for j in rows if j["posted_date"] >= since]
    rows.sort(key=lambda j: j["posted_date"], reverse=True)
    return rows


def main():
    p = profile("https://www.linkedin.com/in/reidhoffman")
    print(f'{p["name"]} | {p["current_company"]} | {p["followers"]:,} followers')

    with open("profiles.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=list(p))
        w.writeheader()
        w.writerow(p)

    for handle in ("stripe", "spotify", "microsoft"):
        c = company(handle)
        print(f'{c["name"]:<10}{c["employee_count"]:>9,}  {len(c["specialties"])} specialties')

    rows = jobs("backend engineer", "United Kingdom", since="2026-07-01")
    print(f"{len(rows)} fresh postings")
    for j in rows[:5]:
        print(f'  {j["posted_date"]}  {j["company"][:20]:<22}{j["url"]}')


if __name__ == "__main__":
    sys.exit(main())

Summary

Three LinkedIn jobs run off one request shape and no login: a profile URL returns 12 top-level fields with experience and education as nested lists, a company handle returns 13 fields of firmographics, and a keyword plus a location returns ranked live postings with direct URLs. What the payload does not carry is anything gated behind a session, so this covers public research and lead enrichment rather than connection graphs. The habit that saves the most time is coalescing every optional field with or "" before you touch it, because specialties, founded, connections, salary and the nested job titles are all populated inconsistently from page to page.

FAQ

Do I need a LinkedIn login to run this?

No, every request here sends a URL or a handle plus an API key, with no session cookie and no account involved.

Why is `specialties` sometimes None?

Not every company page fills that section, so Stripe returned None while Microsoft returned a 23-item comma string, and calling .split(",") unguarded raises AttributeError.

Does the jobs endpoint return salary?

Salary came back None on all ten United Kingdom rows, because most public LinkedIn postings do not publish a figure.

DEV Community: Coco

How to scrape eBay in 2026?

Why is it hard to scrape eBay?

Prerequisites

Scrape eBay search results

1. Send the keyword and check the payload

2. Count how many rows carry each optional field

3. Normalise the rows and write a CSV

Pull a single eBay listing

1. Send the listing URL

2. Compute the discount and the stock position

3. Flatten the nested identifiers

Profile an eBay seller

1. Send the store URL

2. Compute the real positive rate from the raw counts

3. Score the four detailed ratings

The part that breaks

Full script

Summary

FAQ

Is scraping eBay legal?

Can I scrape an eBay listing without the item ID?

Why does int(row["quantity_sold"]) fail on eBay search results?

How to scrape Bing in 2026?

Why is it hard to scrape Bing?

Prerequisites

How to scrape Bing search results?

1. Define the query

2. Call the search endpoint and guard on the payload

3. Extract position, link and date

Compare rankings across a query set

The part that breaks

Full script

Summary

FAQ

Is scraping Bing search results legal?

Why do my Bing results differ from what my browser shows?

How many results does one Bing request return?

How to scrape Amazon in 2026?

Why is it hard to scrape Amazon?

Prerequisites

Fetch product data from a product URL

1. Send the URL and assert on a field, not the status

2. Map the fields you actually store

3. Flatten the variation list to CSV

Fetch search results from a search URL

1. Request the search URL unchanged

2. Split sponsored rows from organic rows

3. Page the URL and dedupe by ASIN

Fetch ratings and the star distribution

1. Read rating and reviews_count

2. Expand rating_stars_distribution

3. Convert percentages to counts and recheck the mean

The part that breaks

Full script

Summary

FAQ

How many fields does the Amazon product response return?

Why does sorting search rows raise TypeError?

Does the star distribution match the listed rating?

How to scrape YouTube in 2026?

Why is it hard to scrape YouTube?

Prerequisites

Scrape one video's metadata

1. Send the request and check the status

2. Map the fields you actually need

3. Handle the null and empty fields

Pull a channel's video list

1. Request the channel

2. Flatten the videos list into rows

3. Follow has_more for the next page

Extract a video transcript

1. Fetch the transcript for a video URL

2. Filter segments by timestamp

3. Join the segments into one text block

The part that breaks

Full script

Summary

FAQ

Is it legal to scrape YouTube?

Why does `int(row["quantity_sold"])` fail on eBay search results?

Why does `product["title"]` raise a KeyError on the Walmart payload?