DEV Community: Gio Rich

Mining Capterra reviews with Python for competitor research

Gio Rich — Thu, 15 Oct 2026 15:01:42 +0000

Capterra reviews are one of the most honest sources of "what do people actually hate about this product". Every review has separate pros and cons fields, sub-ratings for ease of use, support, features and value, and sometimes a "reasons for switching" answer that tells you exactly why someone left a competitor. The catch is that capterra.com sits behind an aggressive Cloudflare challenge, so scraping it directly needs a real browser and fails a lot.

I'm an 18-year-old engineering student, and I built Capterra Reviews Scraper around a simpler route. This post shows how to call it, what you get back, and a small competitor-research script.

How it works

Capterra, GetApp and Software Advice are all owned by Gartner and share one review database. Software Advice serves those same reviews without the Cloudflare challenge, so the actor reads them there, over plain HTTP with no headless browser. Reviews keep their original Capterra IDs (like Capterra___7209621), so you can match them against what you see on capterra.com. It's about 25 reviews per request; 600 reviews took under 5 seconds in my tests.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/capterra-reviews-scraper").call(run_input={
    "products": ["https://www.capterra.com/p/135003/Slack/", "Asana"],
    "maxReviewsPerProduct": 200,
    "sortBy": "newest",
})

for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(r["productName"], r["rating"], r["title"])

You can pass Capterra, GetApp or Software Advice URLs, or a product name. Names are matched exactly, never guessed, so URLs are the safest input.

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/capterra-reviews-scraper").call({
  products: ["https://www.capterra.com/p/19319/Jira/"],
  maxReviewsPerProduct: 0, // every review matching the filters
  starRatings: ["1", "2"],
  companySizes: ["D", "E", "F"], // 51-1,000 employees
  reviewedAfter: "2025-01-01",
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, "negative reviews from mid-size companies");

What comes back

A real row from a Slack run (newest first), trimmed:

{
  "productName": "Slack",
  "reviewId": "Capterra___7209621",
  "title": "Can’t work without",
  "rating": 4,
  "pros": "Its ease of use and flexibility. The way you can open a new chat and a channel to share with others is the best",
  "cons": "Just as a personal point of view, the UI is not as modern as other tools are made. Some changes could help improve.",
  "reasonsForSwitching": "Many features were under expensive plans",
  "easeOfUseRating": 5,
  "customerServiceRating": 3,
  "featuresRating": 5,
  "valueForMoneyRating": 4,
  "date": "2026-09-05",
  "reviewerName": "Lorenzo F.",
  "reviewerIndustry": "Computer Software",
  "reviewerCompanySize": "11-50 employees",
  "timeUsed": "More than 2 years",
  "usageFrequency": "Daily",
  "incentive": "Nominal gift (e.g. gift card)",
  "productOverallRating": 4.66,
  "productTotalReviews": 24209,
  "productRatingBreakdown": { "1star": 48, "2star": 127, "3star": 912, "4star": 5731, "5star": 17391 }
}

The product summary (overall rating, total reviews, star breakdown, average sub-ratings) is repeated on every row, so you don't need a second table.

Use case: turn a competitor's bad reviews into a feature-gap list

Say you're building a project management tool and want to know what Jira and Asana users complain about. Pull the 1- and 2-star reviews, then count the words that show up most in the cons field.

import re
from collections import Counter
from apify_client import ApifyClient

STOP = set("the a an and or to of in is it for on with that this be are was not but you i my we they have has its at as can very too".split())

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/capterra-reviews-scraper").call(run_input={
    "products": ["https://www.capterra.com/p/19319/Jira/", "Asana"],
    "maxReviewsPerProduct": 0,
    "starRatings": ["1", "2"],
    "reviewedAfter": "2025-01-01",
})

words = Counter()
switch_reasons = []
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    text = (r.get("cons") or "").lower()
    words.update(w for w in re.findall(r"[a-z']+", text) if w not in STOP and len(w) > 2)
    if r.get("reasonsForSwitching"):
        switch_reasons.append((r["productName"], r["reasonsForSwitching"]))

print("Most common words in cons:")
for word, n in words.most_common(25):
    print(f"{n:>4}  {word}")

print("\nWhy people switched:")
for product, reason in switch_reasons[:20]:
    print(f"- [{product}] {reason}")

Word counts are crude, but they point you at the clusters ("slow", "pricing", "notifications", "learning curve") worth reading in full. For something better, feed the cons text into an LLM or a topic model. The reviewerCompanySize and reviewerIndustry fields let you split complaints by who's making them.

For ongoing monitoring of your own product, schedule a daily run with sortBy: "newest" and reviewedAfter set to yesterday. With newest-first sorting the scraper stops as soon as it reaches older reviews.

What it costs

$1.50 per 1,000 reviews. No start fee. Reviews skipped by your filters, duplicates, and products that aren't found cost nothing.

The Jira + Asana negative-review pull is usually a few hundred reviews: well under $1
10,000 reviews = $15
Apify's free plan gives $5 of monthly credit, about 3,300 reviews

Limits

Coverage is the Software Advice catalog. Most Capterra products are there (Slack, Asana, Jira, monday.com, Notion, QuickBooks Online, Docusign, Canva, Shopify all worked in testing). A few big vendors aren't, for example Salesforce Sales Cloud, HubSpot CRM, Zendesk and Mailchimp. Those return "product not found" with close matches, and you're not charged.
No reviewer job titles. The source doesn't publish them; you get industry and company size.
Reviewer names are first name + last initial, the way Capterra shows them.
A few older reviews come from GetApp or Software Advice rather than Capterra; reviewSource tells you which.

Issues tab on the actor page for bugs.

Capterra Reviews Scraper on Apify

This article was drafted with AI and published by me.

iOS vs Android: compare an app's recent reviews against its lifetime rating in Python

Gio Rich — Wed, 14 Oct 2026 15:03:28 +0000

The star rating on a store page is a lifetime average over tens of millions of ratings. It barely moves, so it can't tell you whether this week's release went badly, or whether Android users are having a worse time than iPhone users right now. The newest reviews can.

This post pulls the newest reviews for the same app from both stores with my App Store & Google Play Reviews Scraper, then puts three numbers side by side per platform: lifetime rating, recent average, and share of 1-2 star reviews. My earlier post built a Slack alert for negative reviews; this one is a comparison report.

What a real run showed

On 23 September 2026 I pulled the 30 newest US reviews per app. Thirty is a small sample, so read these as a demo of the method, not a verdict on the apps:

App	Store	Lifetime rating	Newest 30, average	1-2 star share
Uber	App Store	4.90 (16.9M ratings)	3.77	30%
Uber	Google Play	4.74 (19.9M ratings)	3.00	50%
Spotify	App Store	4.77 (42.2M ratings)	3.93	20%
Spotify	Google Play	4.35 (36.4M ratings)	4.03	20%

Two things stand out. First, recent written reviews run well below the lifetime number everywhere. That's normal: most lifetime ratings are a quick star tap with no text, and people who bother to write are more often unhappy. So compare recent to recent, not recent to lifetime. Second, on that basis Spotify looked the same on both platforms, while Uber's Android reviews were clearly worse than its iOS ones that day.

Google Play rows also carry the app version the reviewer had installed. In the Uber sample, 21 of the 27 reviews with a version were on 4.648.10003. For Spotify, 13 of 22 were on 9.1.84.2231. When low ratings pile onto one version that isn't simply the most installed one, you've found your regression.

The code

pip install apify-client

from collections import Counter, defaultdict
from apify_client import ApifyClient

# label -> (App Store numeric ID, Google Play package name)
APPS = {
    "Uber": ("368677368", "com.ubercab"),
    "Spotify": ("324684580", "com.spotify.music"),
}
PER_APP = 500

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/app-reviews-scraper").call(run_input={
    "apps": [app_id for pair in APPS.values() for app_id in pair],
    "country": "us",
    "sort": "newest",
    "maxReviewsPerApp": PER_APP,
})

label_of = {app_id: label for label, pair in APPS.items() for app_id in pair}
groups = defaultdict(list)
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    groups[(label_of.get(r["appId"], r["appName"]), r["store"])].append(r)

print(f'{"app":<10}{"store":<12}{"lifetime":>9}{"recent":>8}{"1-2 star":>10}{"n":>6}')
for (label, store), reviews in sorted(groups.items()):
    n = len(reviews)
    recent = sum(r["rating"] for r in reviews) / n
    low = sum(1 for r in reviews if r["rating"] <= 2) / n
    print(f'{label:<10}{store:<12}{reviews[0]["appRating"]:>9.2f}{recent:>8.2f}{low:>10.0%}{n:>6}')

    # Google Play only: which versions are the low ratings on?
    if store == "googlePlay":
        all_versions = Counter(r["version"] for r in reviews if r.get("version"))
        low_versions = Counter(r["version"] for r in reviews if r.get("version") and r["rating"] <= 2)
        for version, bad in low_versions.most_common(3):
            print(f'    v{version}: {bad} of {all_versions[version]} reviews are 1-2 star')

appId is the numeric ID on App Store rows and the package name on Play rows, which is why one lookup table handles both. The store names differ (Uber - Rides, Eats, Hotels on iOS, Uber: Rides, eats, hotels on Android), so don't group by appName.

Reading the output

Recent vs recent across platforms is the comparison that means something. A 0.3+ gap on a few hundred reviews is worth a look at the release notes for the worse platform.
Per-version share, not per-version count. The newest version always has the most reviews. What matters is the bad of total ratio per version, which the last loop prints.
Run it on a competitor. The same script with a rival's two IDs tells you which platform they're weakest on.

Things to keep in mind

Use a few hundred reviews per app. My table above used 30 per app to keep the example readable; at that size one angry afternoon moves the average.
App Store reviews have no app version. Apple doesn't publish it, so the version breakdown is Android only.
Reviews are per country. us and gb are different sets. Run each country separately if you care about more than one.
Google Play returns one language per run (language, default en).
Reviewer names are personal data. The script doesn't need them; drop the column if you store the rows.

Cost

$0.25 per 1,000 reviews, no start fee. The script above (2 apps, 2 stores, 500 each = 2,000 reviews) costs about $0.50 per run. Apify's free plan includes $5 of monthly credit.

App Store & Google Play Reviews Scraper on Apify · Python examples: app-reviews-scraper-python on GitHub

This article was drafted with AI and published by me.

Audit SSL certificate expiry and email providers for a list of domains in Python

Gio Rich — Tue, 13 Oct 2026 15:00:57 +0000

Expired certificates still take sites down, and the usual reason is boring: the domain was on nobody's list. Agencies, IT teams and anyone who inherited a pile of client or company domains have the same problem. You don't need a monitoring product to get a first answer. You need one table: domain, certificate issuer, expiry date, days left.

My Website Tech Stack Detector does a TLS handshake and DNS lookups for every site it checks, so that table falls out of it for free. My earlier post used it to find Shopify stores in a domain list. This one ignores the web technologies and uses only the certificate and DNS fields.

The fields

Every result row carries these, alongside the tech stack:

{
  "domain": "deathwishcoffee.com",
  "tlsIssuer": "Let's Encrypt / YE1",
  "tlsExpires": "2026-10-24T15:59:32.000Z",
  "emailProvider": "Barracuda",
  "emailSenders": ["Microsoft 365", "Zendesk", "HubSpot", "SendGrid"],
  "dnsProvider": "GoDaddy",
  "mxRecords": ["d344701a.ess.barracudanetworks.com", "d344701b.ess.barracudanetworks.com"]
}

That's a real row from my run on 24 September 2026. emailProvider comes from the MX records, emailSenders from the SPF record (the services the domain allows to send mail as it), and dnsProvider from the nameservers.

What 46 real domains looked like

I ran a mixed list of 46 domains: stores, blogs, news sites, framework homepages. 44 came back with a certificate (one domain doesn't exist, and neverssl.com is plain HTTP on purpose).

Who issued the certificates:

Issuer	Sites
Let's Encrypt	21
Google Trust Services	8
GlobalSign	6
DigiCert	4
Amazon	3
Sectigo	1
Microsoft	1

Days left on the day of the run:

	Sites
Under 35 days	6
Under 60 days	20
Over 90 days	12

The median was 66 days. The shortest were kyliecosmetics.com and deathwishcoffee.com at 30 days each, the longest cnn.com at 178. None of that is alarming by itself: all 21 Let's Encrypt certificates had between 30 and 89 days left, which is what a healthy 90-day certificate with auto-renewal looks like. The row to worry about is a short remaining life on an issuer that sells one-year certificates, because that one is probably renewed by hand.

Email was less varied than I expected: of the 45 sites that loaded, 24 use Google Workspace, 6 Microsoft 365, and 3 sit behind Proofpoint.

The code

pip install apify-client

import csv
from datetime import datetime, timezone
from apify_client import ApifyClient

WARN_DAYS = 21   # flag anything closer than this

with open("domains.txt", encoding="utf-8") as f:
    domains = [line.strip() for line in f if line.strip()]

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/website-tech-stack-detector").call(run_input={
    "urls": domains,
    "includeDns": True,          # TLS, MX, SPF and nameserver fields
    "includeEvidence": False,    # smaller output, we only want the headline fields
})

now = datetime.now(timezone.utc)
rows = []
for s in client.dataset(run["defaultDatasetId"]).iterate_items():
    if s.get("error"):
        rows.append((s["domain"], "ERROR", s["error"], "", "", ""))
        continue
    expires = s.get("tlsExpires")
    days = (datetime.fromisoformat(expires.replace("Z", "+00:00")) - now).days if expires else None
    rows.append((
        s["domain"],
        "NO TLS" if days is None else days,
        s.get("tlsIssuer") or "",
        (expires or "")[:10],
        s.get("emailProvider") or "",
        s.get("dnsProvider") or "",
    ))

# errors and missing certificates first, then soonest expiry
rows.sort(key=lambda r: r[1] if isinstance(r[1], int) else -1)

with open("cert-audit.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["domain", "days_left", "issuer", "expires", "email_provider", "dns_provider"])
    w.writerows(rows)

for r in rows:
    if not isinstance(r[1], int) or r[1] < WARN_DAYS:
        print("CHECK", *r)

Schedule it weekly on Apify or from cron, and send the CHECK lines wherever your team reads alerts. Dead domains come back as an error row instead of vanishing, which is useful in itself: a domain that stopped resolving is something you want to hear about.

Why keep the email columns

Offboarding and migrations. If you're moving a company from one mail host to another, emailProvider across every domain it owns shows which ones were missed.
Spoofing surface. emailSenders lists every service the SPF record trusts. A long list with tools nobody uses any more is worth trimming.
Inherited domains. dnsProvider tells you where to log in to fix any of the above.

Things to keep in mind

It reads the certificate served on the homepage host. Other subdomains (api., shop., mail.) have their own certificates; add them to the list as separate entries.
Short life isn't bad news on its own. Let's Encrypt certificates last 90 days and Google Trust Services ones about the same, so 30 days left is normal there. Flag by issuer, or use a threshold under 30 days like the code above.
Behind a CDN you see the CDN's certificate. For Cloudflare-proxied sites that's the edge certificate, which is the one visitors get, but it says nothing about the origin.
emailProvider is the inbound gateway. A domain filtered by Proofpoint, Mimecast or Barracuda shows that name, not the mailbox host behind it. emailSenders often fills in the rest.

Cost

$3.00 per 1,000 results, one result = one website. Domains that fail to load are saved as an error row but not charged. A weekly check of 200 domains is about $0.60 a week, and Apify's free plan includes $5 of monthly credit.

Website Tech Stack Detector on Apify · Python examples: website-tech-stack-detector-python on GitHub

This article was drafted with AI and published by me.

Send new 1-star App Store and Google Play reviews to Slack with Python

Gio Rich — Mon, 12 Oct 2026 15:01:59 +0000

App Store and Google Play reviews are where users tell you a release broke something, usually before your crash reporter makes it obvious. But the two stores have different formats, the App Store's public feed stops early, and nobody wants to check both consoles every morning.

I'm an 18-year-old engineering student, and I built App Store & Google Play Reviews Scraper to pull both stores into one table with the same field names. This post covers the Python and Node calls, the output, and a daily Slack alert for new 1- and 2-star reviews.

How it works

It reads the stores' public web data over plain HTTP, with no headless browser and no login. App metadata comes from Apple's public iTunes lookup and from the data embedded in the Play app page. Reviews come from the public endpoints each store's website uses, 200 per request. In testing, 20,000 reviews (Spotify and Uber, both stores) took under 30 seconds, with 0 failed requests and 0 duplicates.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/app-reviews-scraper").call(run_input={
    "apps": [
        "https://apps.apple.com/us/app/spotify-music-and-podcasts/id324684580",
        "com.spotify.music",
    ],
    "country": "us",
    "sort": "newest",
    "maxReviewsPerApp": 200,
})

for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(r["store"], r["rating"], r["reviewDate"][:10], (r["text"] or "")[:60])

You can mix App Store URLs or numeric IDs (324684580) with Play URLs or package names (com.spotify.music) in one list.

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/app-reviews-scraper").call({
  apps: ["com.spotify.music"],
  country: "gb",
  language: "en",
  sort: "mostHelpful",
  maxReviewsPerApp: 1000,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const replied = items.filter((r) => r.developerReply).length;
console.log(`${replied} of ${items.length} reviews have a developer reply`);

What comes back

One row per review; both stores share the same fields, and anything a store doesn't have is null. A real Google Play row, trimmed:

{
  "store": "googlePlay",
  "appId": "com.spotify.music",
  "appName": "Spotify: Music and Podcasts",
  "country": "us",
  "reviewId": "1e96015e-4577-4274-b4de-f71e0a4eae8e",
  "rating": 5,
  "title": null,
  "text": "The app has has gotten better with music suggestions, but recently podcast video episodes have trouble playing.",
  "reviewDate": "2026-09-23T05:14:13.000Z",
  "reviewerName": "Hennessy",
  "version": "9.1.82.2161",
  "thumbsUp": 4,
  "developerReply": "Hi there! Have you tried a quick reinstall? ...",
  "developerReplyDate": "2024-01-03T12:49:34.000Z",
  "appRating": 4.35,
  "appRatingCount": 36360224,
  "appInstalls": "1,000,000,000+",
  "appUpdated": "2026-09-18T07:51:07.000Z"
}

App Store rows have a title and isEdited but no per-review version (Apple doesn't publish it). Play rows have version for about 84% of reviews, which is what lets you tie complaints to a specific release.

Use case: a daily Slack alert for new negative reviews

The idea: every morning, fetch only reviews since yesterday, only 1 and 2 stars, and post them to a Slack channel through an incoming webhook.

pip install apify-client requests

import datetime
import requests
from apify_client import ApifyClient

SLACK_WEBHOOK = "<YOUR_SLACK_INCOMING_WEBHOOK_URL>"
APPS = ["324684580", "com.spotify.music"]  # replace with your own app IDs

since = (datetime.date.today() - datetime.timedelta(days=1)).isoformat()

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/app-reviews-scraper").call(run_input={
    "apps": APPS,
    "country": "us",
    "sort": "newest",
    "stars": [1, 2],
    "reviewsSince": since,
    "maxReviewsPerApp": 0,
})

lines = []
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    store = "iOS" if r["store"] == "appStore" else "Android"
    version = f' v{r["version"]}' if r.get("version") else ""
    text = (r.get("text") or "").replace("\n", " ")[:300]
    lines.append(f'*{r["rating"]}★ {store}{version}* {r.get("title") or ""}\n>{text}')

if lines:
    requests.post(SLACK_WEBHOOK, json={"text": f"{len(lines)} new 1-2★ reviews since {since}\n\n" + "\n\n".join(lines)})

With sort: "newest" and reviewsSince, the run stops as soon as it reaches older reviews, so a daily run only pays for the new ones. If the Android rows cluster on one version, you've probably found a regression.

What it costs

$0.25 per 1,000 reviews. Reviews filtered out by stars or reviewsSince are free, and so are apps that turn out not to exist. No start fee.

The daily alert above for a mid-size app: often under a hundred reviews a day, a few cents a month
100,000 reviews for a research dataset = $25
Apify's free plan gives $5 of monthly credit, about 20,000 reviews

Limits

App Store reviews have no app version. Apple's public data doesn't include it.
Google Play returns one language per run (language). Run twice for two languages.
Reviews are per country. us and gb have different reviews; run each country separately.
Star filters on Play go one star at a time: with [1, 2] you get newest 1-star first, then 2-star. Play has no "most critical" sort, so the actor walks star by star.
Reviewer names and avatars are personal data under GDPR/CCPA. Have a lawful reason before storing them.

Issues tab on the actor page if something's wrong.

App Store & Google Play Reviews Scraper on Apify

This article was drafted with AI and published by me.

Benchmark a product category on Alibaba before you source it (Python, real numbers)

Gio Rich — Sun, 11 Oct 2026 15:01:24 +0000

Before contacting a single supplier, it helps to know what "normal" looks like for a category: the usual starting price, the usual minimum order, how long production takes, and whether certified products cost more. Otherwise the first quote you get becomes your anchor.

My earlier post ranked individual suppliers by unit price at a given order size. This one zooms out: it scrapes 75 listings per category with my Alibaba Product Search Scraper and summarises the whole page of results.

The numbers from a real run

I ran four searches (75 listings each) and took medians:

Search	Median lowest price	Median MOQ	Median lead time	Verified suppliers	Sponsored results
bluetooth earbuds	$5.98	10	10 days	32 / 75	1
yoga mat	$9.00	20	31 days	60 / 75	23
custom tote bags	$0.45	100	10 days	61 / 75	25
led strip lights	$1.50	10	10 days	62 / 75	37

A few things stand out straight away:

Certification premium is real for electronics. Earbuds listing any certification (FCC, CE, RoHS were the most common) had a median low price of $6.51 versus $5.53 without one. LED strips: $2.00 certified versus $1.20 without. If you're selling into the EU or US you need those certificates anyway, so $5.53 is not your real benchmark.
Yoga mats take 3x longer to make. A 31-day median lead time versus 10 for the others changes your reorder planning.
Half the LED strip results were ads (37 of 75). Sorting by position alone mostly shows you who paid.

The script

pip install apify-client

from statistics import median
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/alibaba-product-search-scraper").call(run_input={
    "searchTerms": ["bluetooth earbuds", "led strip lights"],
    "maxResultsPerSearch": 75,
    "includeAds": True,   # keep sponsored listings so we can count them
})

by_search = {}
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    by_search.setdefault(item["searchTerm"], []).append(item)


def med(values):
    values = [v for v in values if v is not None]
    return median(values) if values else None


for term, items in by_search.items():
    certified = [i for i in items if i.get("productCertifications")]
    plain = [i for i in items if not i.get("productCertifications")]
    print(f"\n{term} ({len(items)} listings)")
    print(f"  median low price   ${med([i['priceMin'] for i in items])}")
    print(f"  certified / not    ${med([i['priceMin'] for i in certified])} ({len(certified)}) "
          f"vs ${med([i['priceMin'] for i in plain])} ({len(plain)})")
    print(f"  median MOQ         {med([i['moq'] for i in items])}")
    print(f"  median lead time   {med([i['leadTimeDays'] for i in items])} days")
    print(f"  verified suppliers {sum(1 for i in items if i.get('verifiedSupplier'))}")
    print(f"  sponsored (ads)    {sum(1 for i in items if i.get('isAd'))}")

The fields doing the work

Every listing carries the fields this needs, for example (real row, trimmed):

{
  "searchTerm": "bluetooth earbuds",
  "priceMin": 6.5, "priceMax": 9.9, "priceText": "US $6.50-$9.90",
  "priceTiers": [
    { "minQuantity": 10, "maxQuantity": 89, "price": 9.9 },
    { "minQuantity": 5000, "maxQuantity": null, "price": 6.5 }
  ],
  "moq": 10, "leadTimeDays": 14,
  "productCertifications": ["FCC", "UKCA", "CE", "ROHS"],
  "isAd": false,
  "supplierYears": 10, "verifiedSupplier": true, "tradeAssurance": true,
  "supplierRating": 4.5, "responseRate": "98.5%"
}

priceMin is the price at the biggest quantity tier, so it's the best case. For a benchmark at your own order size, read priceTiers the way the earlier post did.

Limits

Medians from 75 listings describe what Alibaba shows for that search, not the whole market. Run a couple of related terms ("wireless earbuds", "tws earbuds") before you trust a number.
leadTimeDays and certifications are what suppliers list; they're claims until you see paperwork.
Prices exclude shipping and duties.

Cost

$1.50 per 1,000 results (one result = one listing). The four-category snapshot above is 300 results, about $0.45. Apify's free plan includes $5 of monthly credit.

Alibaba Product Search Scraper on Apify · Python examples: alibaba-product-scraper-python on GitHub

This article was drafted with AI and published by me.

Which US states should your ads target? Google Trends interest by region in Python

Gio Rich — Fri, 09 Oct 2026 15:06:04 +0000

Most Google Trends tutorials stop at the interest-over-time line. The part I find more useful for actual decisions is interest by region: where in the country people search for one thing more than another. If you're splitting an ad budget across states, or picking which city gets a launch first, that's the table you want.

This post pulls that data with my Google Trends Scraper from Python and turns it into a ranked list of states. My earlier post covered the basics and a weekly keyword tracker; this one is only about regions.

Two ways to read regional data

One keyword: each region gets a 0-100 score, where 100 is the region with the highest share of searches for that term. For mortgage rates over the past 12 months (US), Wyoming scored 100, Kansas 49 and Colorado 48.
A comparison (up to 5 comma-separated keywords on one line): each region's values split that region's searches between your keywords and add up to 100. This is the one for "where do people want A rather than B".

A real comparison: iPhone vs Samsung Galaxy

I ran iphone, samsung galaxy for the US, past 12 months, with region resolution set to states. Nationally the gap is huge (average 61.4 vs 4.3 on the timeline), but the state split shows where Samsung does relatively best:

State	iphone	samsung galaxy
Wyoming	82	18
Kansas	92	8
New York	92	8
District of Columbia	92	8
Florida	93	7
...
Vermont	96	4
Louisiana	97	3

So a Samsung accessory seller would get roughly 6x the relative interest in Wyoming compared to Louisiana. Those are real values from the run, all 51 regions came back in one result.

The code

pip install apify-client

import csv
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/google-trends-scraper").call(run_input={
    "searchTerms": ["iphone, samsung galaxy"],
    "geo": "US",
    "timeRange": "today 12-m",
    "regionResolution": "REGION",       # states; "DMA" = US metro areas, "CITY" = cities
    "includeRelatedQueries": False,     # only regions needed: faster run
    "includeRelatedTopics": False,
})

TARGET = "samsung galaxy"

for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    rows = []
    for reg in r.get("interestByRegion", []):
        values = reg["values"]
        total = sum(values.values()) or 1
        rows.append((reg["geoName"], reg["geoCode"], values[TARGET], round(values[TARGET] / total, 3)))

    rows.sort(key=lambda x: x[3], reverse=True)
    for name, code, value, share in rows[:10]:
        print(f"{name:<22} {code:<6} {value:>3}  share {share:.0%}")

    with open("target-states.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        w.writerow(["state", "code", TARGET, "share"])
        w.writerows(rows)

Each item in interestByRegion looks like this:

{ "geoCode": "US-WY", "geoName": "Wyoming", "values": { "iphone": 82, "samsung galaxy": 18 } }

The geoCode values (US-WY, US-KS, ...) match the region codes most ad platforms accept in bulk location uploads, so the CSV is close to ready for a location bid adjustment sheet.

Going finer: metro areas

Set regionResolution to "DMA" to get US designated market areas instead of states. That's the same geography TV and many local ad buys use. "CITY" works too, but Google only returns cities with enough volume, so small keywords come back sparse.

Things to keep in mind

Shares, not volumes. Wyoming at 18 doesn't mean Wyoming has many Samsung searches; it means that of the searches for these two terms in Wyoming, a bigger slice went to Samsung. Combine it with population or your own conversion data before moving real budget.
Keep the comparison on one line. Values from separate searches are on different scales and can't be compared to each other.
Small regions are noisy. Google Trends samples its data, and low-population states can swing a few points between runs. Run it twice, or use 12 months instead of 30 days, before you act on a small difference.
hasData: false means the term doesn't have enough volume for Google to show anything.

Cost

$2.50 per 1,000 results, and one comparison (up to 5 keywords, all 51 states included) is one result. Checking 20 product pairs every month comes to about $0.05. Apify's free plan includes $5 of monthly credit.

Google Trends Scraper on Apify · Python examples: google-trends-api-python on GitHub

This article was drafted with AI and published by me.

Tracking Google Flights prices with Python (a fare watcher in 40 lines)

Gio Rich — Mon, 05 Oct 2026 03:32:16 +0000

Google shut down its QPX Express flight API in 2018, and there's no public Google Flights API to replace it. If you want to watch a fare for a trip, compare dates, or feed real itineraries into a travel app, you either click through the site every day or scrape it.

I'm an 18-year-old engineering student, and I built Google Flights Scraper for that. This post covers the Python and Node calls, the output, and a small fare watcher for a trip from Charlotte to Tokyo.

How it works

The actor makes plain HTTP requests (no headless browser) and reads the same results the Google Flights site shows. A one-way search is one request, takes about a second and usually returns 10-40 itineraries. A round trip lists outbound flights, then fetches the returns for the top few (3 by default) and outputs one row per complete outbound + return pair. It runs fine at 256 MB of memory.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/google-flights-scraper").call(run_input={
    "origin": "JFK",
    "destination": "LAX",
    "departureDate": "2026-11-12",
    "adults": 1,
    "cabinClass": "economy",
    "currency": "USD",
    "maxResults": 10,
})

for f in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f["price"], f["airlines"], f["stops"], f["totalDuration"], f["departureDateTime"])

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/google-flights-scraper").call({
  origin: "SFO",
  destination: "ORD",
  departureDate: "2026-10-20",
  returnDate: "2026-10-23",
  maxStops: "0",
  returnFlightsForTopOutbound: 3,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const t of items) {
  console.log(t.price, t.flightNumbers.join("+"), "back:", t.returnFlight?.flightNumbers.join("+"));
}

origin and destination also take several airports (JFK,EWR,LGA) or a Google city ID like /m/02_286 for all New York airports.

What comes back

One row per itinerary. A real row from a test run, Charlotte to Tokyo Narita one-way, trimmed:

{
  "searchId": "CLT-NRT 2026-12-03",
  "tripType": "one_way",
  "currency": "USD",
  "category": "best",
  "price": 1014,
  "priceLevel": "typical",
  "airlines": ["Air Canada"],
  "flightNumbers": ["AC 8746", "AC 9"],
  "stops": 1,
  "totalDuration": "17 hr 35 min",
  "departureDateTime": "2026-12-03T08:55:00-05:00",
  "arrivalDateTime": "2026-12-04T16:30:00+09:00",
  "arrivalDayOffset": 1,
  "layovers": [
    { "airport": "YYZ", "city": "Toronto", "durationMinutes": 98, "durationText": "1 hr 38 min" }
  ],
  "segments": [
    { "flightNumber": "AC 8746", "operatedBy": "Air Canada Express - Jazz", "aircraft": "Embraer 175", "legroom": "31 in", "emissionsKg": 142 },
    { "flightNumber": "AC 9", "aircraft": "Boeing 777", "legroom": "31 in", "emissionsKg": 582 }
  ],
  "emissionsKg": 724,
  "typicalEmissionsKg": 817,
  "emissionsDifferencePercent": -11,
  "googleFlightsUrl": "https://www.google.com/travel/flights/booking?tfs=..."
}

Times are local airport times with the UTC offset, so you can compute real elapsed time across time zones. priceLevel is Google's own low / typical / high verdict for that route and date, which is handy for alerts.

Use case: watch a trip across several dates

Say you're flexible on when you fly Charlotte to Tokyo in December. The searches input takes many routes and dates in one run, so you can check a week of departure dates at once and log the cheapest option for each.

import csv
import datetime
import os
from apify_client import ApifyClient

DATES = [f"2026-12-{d:02d}" for d in range(1, 8)]

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/google-flights-scraper").call(run_input={
    "searches": [{"origin": "CLT", "destination": "NRT", "departureDate": d} for d in DATES],
    "maxStops": "1",
    "currency": "USD",
})

cheapest = {}
for f in client.dataset(run["defaultDatasetId"]).iterate_items():
    day = f["searchId"]
    if f["price"] is not None and (day not in cheapest or f["price"] < cheapest[day]["price"]):
        cheapest[day] = f

today = datetime.date.today().isoformat()
new_file = not os.path.exists("clt-nrt.csv")
with open("clt-nrt.csv", "a", newline="", encoding="utf-8") as out:
    w = csv.writer(out)
    if new_file:
        w.writerow(["checked", "search", "price", "priceLevel", "airlines", "stops", "duration"])
    for day, f in sorted(cheapest.items()):
        w.writerow([today, day, f["price"], f["priceLevel"], "+".join(f["airlines"]), f["stops"], f["totalDuration"]])
        if f["priceLevel"] == "low":
            print(f'LOW FARE: {day} ${f["price"]} {f["googleFlightsUrl"]}')

Run it once a day. The CSV gives you a price history per departure date, and the priceLevel == "low" check prints a booking link when Google itself says the fare is below normal.

What it costs

$0.50 per 1,000 results, one result = one itinerary row. A one-way search usually returns 10-40 itineraries, so a single route/date costs about $0.005-$0.02.

The 7-date watcher above, daily for a month: roughly 7 × 25 × 30 ≈ 5,000 rows ≈ $2.60/month
Set maxResults to 5 and it drops to about $0.50/month
Apify's free plan gives $5 of monthly credit, about 10,000 itineraries

Limits

One-way and round trip only. No multi-city (3+ legs) yet.
No per-agency booking prices (Expedia vs the airline) and no baggage fees. googleFlightsUrl opens the itinerary on Google Flights, where those options are listed.
Initial results page only. Google sometimes loads extra "other flights" later; in tests I got 36 JFK-LAX options, 18 Madrid-Barcelona and 7 Charlotte-Tokyo.
Up to 9 passengers, about 11 months ahead (Google's own limits).
Fares change constantly. scrapedAt tells you when each row was captured.

If a round-trip return lookup fails after retries you still get the outbound row with returnFlight: null, not a crashed run. Bugs go to the Issues tab on the actor page.

Google Flights Scraper on Apify

This article was drafted with AI and published by me.

Getting Google Trends data in Python without fighting 429 errors

Gio Rich — Sat, 03 Oct 2026 15:07:00 +0000

If you've used pytrends, you've probably seen 429 Too Many Requests. Google Trends throttles anything that looks automated, and a script that worked yesterday can fail today. There's no official public API for it either.

I'm an 18-year-old engineering student, and I built Google Trends Scraper mostly around that one problem. This post covers calling it from Python and Node, what the output looks like, and a small weekly keyword tracker.

How it deals with rate limits

Everything runs over plain HTTP, no headless browser. Each session first gets its own Google cookie, then makes one small flow per search (explore, then the data widgets). When Google returns a 429, that session is retired and the search retries on a fresh session and proxy IP with backoff and jitter. Results are saved per search as soon as they're ready, so one stubborn search never blocks the rest. A RUN_SUMMARY record lists anything that failed and why, and failed searches aren't charged.

In my tests, 9 searches plus 2 trending-now countries (29 results) took about 20 seconds.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/google-trends-scraper").call(run_input={
    "searchTerms": ["air fryer", "chatgpt, gemini, claude"],
    "geo": "US",
    "timeRange": "today 12-m",
})

for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    for s in r.get("keywordStats", []):
        print(s["keyword"], s["average"], s["changePercent"])

One line in searchTerms is one search. Put up to 5 comma-separated terms on a line to compare them on the same 0-100 scale.

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/google-trends-scraper").call({
  trendingNowGeos: ["US", "GB"],
  includeInterestOverTime: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const t of items.filter((i) => i.type === "trending")) {
  console.log(t.geo, t.rank, t.title, t.approxTraffic);
}

What comes back

One result per search, with every data type nested inside. A real result (US, past 12 months), arrays shortened:

{
  "type": "search",
  "searchTerm": "chatgpt, gemini, claude",
  "keywords": ["chatgpt", "gemini", "claude"],
  "geo": "US",
  "hasData": true,
  "keywordStats": [
    { "keyword": "chatgpt", "average": 77.6, "peakValue": 100, "peakDate": "2025-10-19T00:00:00.000Z", "latestValue": 76, "changePercent": -20.2 },
    { "keyword": "gemini", "average": 26.9, "peakValue": 35, "latestValue": 33, "changePercent": 20.9 },
    { "keyword": "claude", "average": 17.9, "peakValue": 34, "latestValue": 19, "changePercent": 263.6 }
  ],
  "interestOverTime": [
    { "date": "2026-09-13T00:00:00.000Z", "formattedTime": "Sep 13 – 19, 2026", "values": { "chatgpt": 76, "gemini": 33, "claude": 19 }, "isPartial": false }
  ],
  "interestByRegion": [
    { "geoCode": "US-CA", "geoName": "California", "values": { "chatgpt": 57, "gemini": 26, "claude": 17 } }
  ],
  "relatedQueriesRising": [
    { "query": "how to use chatgpt effectively", "keyword": "chatgpt", "formattedValue": "+2,050%", "isBreakout": false }
  ]
}

changePercent compares the average of the last quarter of the period to the first quarter, with partial (still-running) weeks left out. Trending-now rows look like this:

{ "type": "trending", "geo": "US", "rank": 1, "title": "dodgers schedule", "approxTraffic": "1000+", "news": [{ "title": "The teams no one wants to play in October", "source": "MLB.com" }] }

Use case: a weekly keyword tracker for content planning

Say you write about kitchen gear and want to know every week which products are gaining interest and which rising queries you should write about.

import csv
import datetime
import os
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/google-trends-scraper").call(run_input={
    "searchTerms": ["air fryer, instant pot, slow cooker", "espresso machine"],
    "geo": "US",
    "timeRange": "today 3-m",
    "includeInterestByRegion": False,
    "includeRelatedTopics": False,
})

today = datetime.date.today().isoformat()
new_file = not os.path.exists("trends-log.csv")
with open("trends-log.csv", "a", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    if new_file:
        w.writerow(["date", "search", "keyword", "average", "latest", "changePercent"])
    for r in client.dataset(run["defaultDatasetId"]).iterate_items():
        for s in r.get("keywordStats", []):
            w.writerow([today, r["searchTerm"], s["keyword"], s["average"], s["latestValue"], s["changePercent"]])
        for q in r.get("relatedQueriesRising", [])[:5]:
            print(f'rising for {q["keyword"]}: {q["query"]} ({q["formattedValue"]})')

Run it weekly. The CSV builds a history you can chart, and the printed rising queries are a ready-made list of article ideas. Remember that values are only comparable inside one search, which is why the three cookers share one line.

What it costs

$2.50 per 1,000 results. One result is one search (a comparison of up to 5 keywords counts as one, with timeline, regions and related queries included) or one trending-now search. Failed searches aren't charged.

The weekly tracker above: 2 results a week ≈ $0.02/month
1,000 searches = $2.50
Apify's free plan gives $5 of monthly credit, about 2,000 results

Limits

Up to 5 keywords per comparison (Google's own limit).
Values are Google's relative 0-100 index, not search volumes. 100 is the peak for that search.
Numbers can move a few points between requests, even in the browser. Google Trends works from a sample.
Related queries and topics only appear with enough volume. Related topics have become rare in Google's responses lately; you get an empty array, not an error. hasData: false means too little volume.
Trending now returns what Google's feed lists, typically about 10 searches per country.

If a run shows lots of "rate-limited, rotating session" warnings, that's handled; if searches still fail, switch the proxy to RESIDENTIAL or lower maxConcurrency. Anything else, use the Issues tab on the actor page.

Google Trends Scraper on Apify

This article was drafted with AI and published by me.

Build an Alibaba supplier shortlist with Python (prices, MOQ, verified badges)

Gio Rich — Fri, 02 Oct 2026 15:05:29 +0000

Sourcing on Alibaba means opening dozens of tabs, squinting at price ranges like "US $8.00-$11.00", and trying to remember which supplier had the lower MOQ and which one was actually verified. The price you care about depends on how many units you're ordering, and that's buried in each listing's quantity tiers.

I'm an 18-year-old engineering student, and I built Alibaba Product Search Scraper to get all of that into one table. This post covers calling it from Python and Node, the output, and a small script that ranks suppliers by unit price at a given order quantity.

How it works

Alibaba's search page loads its product grid as JSON from an internal endpoint. The actor calls that endpoint directly over plain HTTP (no headless browser), 48 products per request, using the same filter parameters the site uses for price, MOQ, supplier country and verified status. A search of 1,000 products takes about 20 seconds.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/alibaba-product-search-scraper").call(run_input={
    "searchTerms": ["yoga mat"],
    "maxResultsPerSearch": 100,
    "maxMoq": 100,
    "verifiedSuppliersOnly": True,
})

for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(p["priceText"], p["moqText"], p["supplierName"], p["supplierYears"])

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/alibaba-product-search-scraper").call({
  searchTerms: ["led strip lights"],
  maxResultsPerSearch: 200,
  supplierCountry: "CN",
  includeAds: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, "products");

What comes back

One item per product, 55 fields. A real row for "yoga mat", trimmed:

{
  "searchTerm": "yoga mat",
  "productId": "1601583212388",
  "title": "High Quality Eco-Friendly Premium Extra Large Yoga Gym Mat Custom Logo ...",
  "url": "https://www.alibaba.com/product-detail/High-Quality-Eco-Friendly-Premium-Extra_1601583212388.html",
  "priceMin": 8,
  "priceMax": 11,
  "priceText": "US $8.00-$11.00",
  "priceTiers": [
    { "minQuantity": 50, "maxQuantity": 999, "price": 11, "unit": "pieces" },
    { "minQuantity": 1000, "maxQuantity": 4999, "price": 10, "unit": "pieces" },
    { "minQuantity": 5000, "maxQuantity": 9999, "price": 9, "unit": "pieces" },
    { "minQuantity": 10000, "maxQuantity": null, "price": 8, "unit": "pieces" }
  ],
  "moq": 50,
  "moqUnit": "pieces",
  "soldCount": 378,
  "leadTimeDays": 35,
  "isAd": false,
  "supplierName": "DONGGUAN TONGTIANXIA RUBBER CO., LTD.",
  "supplierCountryCode": "CN",
  "supplierYears": 19,
  "verifiedSupplier": true,
  "goldSupplier": true,
  "tradeAssurance": true,
  "supplierRating": 4.8,
  "supplierReviewCount": 25,
  "responseRate": "99.9%",
  "supplierEmployees": 130,
  "supplierCertifications": ["ISO", "FSC"],
  "supplierVerifiedBy": "SGS"
}

priceTiers is the part that makes this useful. It's the full quantity ladder, so you can work out what you'd actually pay per unit at your order size.

Use case: rank suppliers by real unit price at your order quantity

You want 1,000 yoga mats from a verified supplier with at least a few years on Alibaba. Here's a script that reads each product's tiers, finds the price at 1,000 units, and writes a sorted shortlist.

import csv
from apify_client import ApifyClient

ORDER_QTY = 1000

def price_at(product, qty):
    tiers = product.get("priceTiers") or []
    for t in tiers:
        top = t["maxQuantity"] if t["maxQuantity"] is not None else float("inf")
        if t["minQuantity"] <= qty <= top:
            return t["price"]
    # one flat price and no tiers: fall back to the listed max
    return product.get("priceMax")

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/alibaba-product-search-scraper").call(run_input={
    "searchTerms": ["yoga mat tpe", "yoga mat cork"],
    "maxResultsPerSearch": 300,
    "maxMoq": ORDER_QTY,
    "verifiedSuppliersOnly": True,
    "includeAds": False,
})

rows = []
for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    unit = price_at(p, ORDER_QTY)
    if unit is None or (p.get("supplierYears") or 0) < 3:
        continue
    rows.append([unit, unit * ORDER_QTY, p["supplierName"], p["supplierYears"],
                 p.get("supplierRating"), p.get("responseRate"), p["moq"], p["url"]])

rows.sort(key=lambda r: r[0])
with open("yoga-mat-shortlist.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["unitPrice", "orderTotal", "supplier", "years", "rating", "responseRate", "moq", "url"])
    w.writerows(rows)
print(f"{len(rows)} candidates written")

This is a starting list, not a decision. Search prices are list prices from the card; shipping, options and your final quote will move them. But it gets you from hundreds of tabs to 20 suppliers worth messaging.

Tip: Alibaba serves at most about 960 products per keyword, so split broad terms into specific ones ("yoga mat tpe", "yoga mat cork") the way the script does.

What it costs

$1.50 per 1,000 results, one result = one product saved. Duplicates and failed requests aren't charged.

The shortlist job above (2 keywords × 300) = 600 products ≈ $0.90
10,000 products = $15
Apify's free plan gives $5 of monthly credit, about 3,300 products

Limits

About 960 unique products per keyword. Alibaba serves 20 pages of 48, whatever match count it reports.
Alibaba matches loosely. Narrow or misspelled keywords still return related products. Check titles if precision matters.
USD list prices from the search card. Final price depends on quantity, options and shipping.
Country-level supplier location only. Province and city are on the profile page, which the actor doesn't open.
Search results only. No full product specs, SKU/variant lists or review texts in this version.
minPrice / maxPrice take whole dollars.

Issues tab on the actor page if anything's off.

Alibaba Product Search Scraper on Apify

This article was drafted with AI and published by me.

Scrape public Telegram channels with Python (no login, no API ID)

Gio Rich — Tue, 29 Sep 2026 15:06:06 +0000

If you want posts from a public Telegram channel, the usual route is Telethon or Pyrogram. They work, but you need a Telegram account, a phone number, an API ID and a session file, and you're logging in as a user to do it. For reading public channels that's a lot of setup.

Every public channel with a web preview is also readable at t.me/s/<channel> in a normal browser. I'm an 18-year-old engineering student, and I built Telegram Channel Scraper on top of that page. This post shows how to call it, what the output looks like, and a small daily digest project.

How it works

t.me/s/<channel> returns server-rendered HTML with up to 20 posts per page and a ?before=N cursor to the older ones. The actor fetches those pages over plain HTTP (no headless browser), parses each message block with cheerio, and walks back through the history. In my tests, 250 posts took 7 seconds and 480 posts took 12 seconds.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/telegram-channel-scraper").call(run_input={
    "channels": ["durov", "bloomberg"],
    "maxPostsPerChannel": 100,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["type"] == "post":
        print(item["date"], item["views"], item["text"][:80] if item["text"] else "")

The dataset mixes two kinds of rows: type: "post" for each post and type: "channel" for one info row per channel. Filter on type.

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/telegram-channel-scraper").call({
  channels: ["https://t.me/telegram"],
  onlyPostsNewerThan: "30 days",
  searchQuery: "update",
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const posts = items.filter((i) => i.type === "post");
console.log(posts.map((p) => `${p.url} (${p.views} views)`));

What comes back

A real post row from a run on 24 Sep 2026, trimmed (the CDN URL is shortened and I kept 2 of 8 reactions):

{
  "type": "post",
  "channel": "bloomberg",
  "channelId": "1488156064",
  "postId": 3225,
  "url": "https://t.me/bloomberg/3225",
  "date": "2026-09-22T03:26:17.000Z",
  "text": "Alibaba unveils new AI chip it says is China's most powerful and targets 20 gigawatts of data center capacity by 2032. https://bloom.bg/4ydroys ...",
  "views": 13400,
  "viewsText": "13.4K",
  "reactionsTotal": 173,
  "reactions": [
    { "emoji": "🔥", "isPaid": false, "count": 62 },
    { "emoji": "❤", "isPaid": false, "count": 33 }
  ],
  "mediaType": "photo",
  "photos": [{ "url": "https://cdn5.telesco.pe/file/...jpg" }],
  "links": ["https://bloom.bg/4ydroys"],
  "hashtags": [],
  "forwardedFrom": null,
  "edited": false
}

And the channel row:

{
  "type": "channel",
  "username": "bloomberg",
  "title": "Bloomberg",
  "subscribers": 175000,
  "verified": true,
  "previewAvailable": true
}

Posts also carry videos (with duration and thumbnail), files, polls with option percentages, forwarded-from and reply-to info when present.

Use case: a daily digest of new posts

Say you follow a handful of news or project channels and want one daily summary with the most-viewed posts, stored so you never see the same post twice.

import sqlite3
from apify_client import ApifyClient

CHANNELS = ["bloomberg", "durov", "telegram"]

db = sqlite3.connect("telegram.db")
db.execute("""CREATE TABLE IF NOT EXISTS posts (
    url TEXT PRIMARY KEY, channel TEXT, date TEXT, views INTEGER, text TEXT)""")

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/telegram-channel-scraper").call(run_input={
    "channels": CHANNELS,
    "onlyPostsNewerThan": "1 day",
    "includeChannelInfo": False,
})

new = []
for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    if p["type"] != "post":
        continue
    cur = db.execute("INSERT OR IGNORE INTO posts VALUES (?,?,?,?,?)",
                     (p["url"], p["channel"], p["date"], p["views"], p.get("text") or ""))
    if cur.rowcount:
        new.append(p)
db.commit()

for p in sorted(new, key=lambda x: x["views"] or 0, reverse=True)[:10]:
    first_line = (p.get("text") or "").split("\n")[0][:100]
    print(f'{p["views"]:>8}  {p["channel"]:<10} {first_line}  {p["url"]}')

Run it once a day. onlyPostsNewerThan: "1 day" makes the scraper stop as soon as posts get older than a day, so you only pay for new posts, and the url primary key handles any overlap between runs.

What it costs

$1.50 per 1,000 results. A result is a post or a channel-info row. Failed requests and duplicates aren't charged.

The digest above: if the three channels post ~30 times a day combined, that's ~900 posts a month ≈ $1.35/month
Archiving a 10,000-post channel history = $15
Apify's free plan gives $5 of monthly credit, about 3,300 posts

Limits

Public channels with a web preview only. No private channels, invite links (t.me/+...), groups or bots. Those return just a channel row with previewAvailable: false.
No comments. Discussion-group replies aren't in the web preview.
Counts are rounded the way Telegram shows them. 13.4K becomes 13400; the raw text is kept in viewsText.
Media URLs are signed CDN links that expire. Download files soon after the run if you need them.
Very large videos have no direct file URL (tooLargeForPreview: true), just the thumbnail, duration and post link.
Custom premium emoji reactions come back as a customEmojiId without the emoji character. The counts are still right.

If a channel returns 0 posts, the RUN_SUMMARY record in the key-value store says why. Anything else, the Issues tab on the actor page reaches me.

Telegram Channel Scraper on Apify

This article was drafted with AI and published by me.

Detect a website's tech stack in bulk with Python (a Wappalyzer-style lookup)

Gio Rich — Mon, 28 Sep 2026 15:05:12 +0000

"What is this site built with?" is easy to answer for one site with a browser extension. It gets annoying when you have 500 domains in a spreadsheet and want to know which ones run Shopify, which use HubSpot, or who's still on an old WordPress install.

I'm an 18-year-old engineering student, and I built Website Tech Stack Detector for bulk lookups like that. This post shows how to call it from Python and Node, what you get back, and how I'd use it to pull Shopify stores out of a domain list.

How it works

Each site gets one HTTP request with browser-like headers, plus DNS lookups (MX, TXT, NS, SOA, CNAME, A, PTR) and one TLS handshake. Headers, cookies, meta tags, script URLs, inline scripts and the HTML are matched against the open Wappalyzer rule set (7,600+ technology fingerprints). DOM rules are checked with cheerio instead of a browser, so analysis takes about 0.3 seconds per page.

The part I care about most: every detected technology comes with a confidence score and the evidence that matched (a header, a script URL, a DNS record...). You can check any result yourself instead of trusting a black box.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/website-tech-stack-detector").call(run_input={
    "urls": ["allbirds.com", "minimalistbaker.com", "nextjs.org"],
    "minConfidence": 50,
})

for site in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(site["domain"], site["cms"], site["ecommerce"], site["emailProvider"])

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/website-tech-stack-detector").call({
  urls: ["allbirds.com", "minimalistbaker.com"],
  includeEvidence: false, // smaller output
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const s of items) console.log(s.domain, s.technologyNames.join(", "));

What comes back

One item per website. A real row for minimalistbaker.com, trimmed (the technologies array had 26 entries; I kept one):

{
  "domain": "minimalistbaker.com",
  "statusCode": 200,
  "blocked": false,
  "technologyCount": 26,
  "cms": ["WordPress"],
  "ecommerce": ["WooCommerce"],
  "analytics": ["Google Analytics"],
  "tagManagers": ["Google Tag Manager"],
  "advertising": ["Amazon Advertising"],
  "cdn": ["Cloudflare"],
  "programmingLanguages": ["PHP"],
  "server": "cloudflare",
  "tlsIssuer": "Google Trust Services / WE1",
  "tlsExpires": "2026-12-04T21:16:49.000Z",
  "emailProvider": "Google Workspace",
  "dnsTechnologies": ["Google Workspace"],
  "technologies": [
    {
      "name": "Cloudflare",
      "confidence": 100,
      "categories": ["CDN"],
      "source": "website",
      "evidence": [
        { "type": "headers", "key": "server", "match": "cloudflare" },
        { "type": "dns", "key": "soa", "match": ".cloudflare.com" }
      ]
    }
  ]
}

The headline columns (cms, ecommerce, analytics, cdn, paymentProcessors, liveChat and so on) are flat arrays, so the dataset drops straight into a spreadsheet. dnsTechnologies is kept separate on purpose: a TXT verification record for Atlassian means the company uses Atlassian, not that Atlassian runs on the website.

Use case: find the Shopify stores in a domain list

Say you build Shopify apps or run a Shopify agency and you have a CSV of 1,000 ecommerce domains. You want only the Shopify ones, plus what analytics and email tools they use, so your outreach is relevant.

import csv
from apify_client import ApifyClient

with open("domains.csv", encoding="utf-8") as f:
    domains = [row[0].strip() for row in csv.reader(f) if row]

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/website-tech-stack-detector").call(run_input={
    "urls": domains,
    "minConfidence": 50,
    "includeEvidence": False,
})

with open("shopify-stores.csv", "w", newline="", encoding="utf-8") as out:
    w = csv.writer(out)
    w.writerow(["domain", "analytics", "marketingAutomation", "paymentProcessors", "emailSenders"])
    for s in client.dataset(run["defaultDatasetId"]).iterate_items():
        if "Shopify" in (s.get("ecommerce") or []):
            w.writerow([
                s["domain"],
                "; ".join(s.get("analytics") or []),
                "; ".join(s.get("marketingAutomation") or []),
                "; ".join(s.get("paymentProcessors") or []),
                "; ".join(s.get("emailSenders") or []),
            ])

emailSenders comes from the SPF record, so it shows services the domain authorised to send mail (Klaviyo, Mailchimp, SendGrid...). That's often a better signal of the marketing stack than what's visible on the homepage.

In my test run of 46 mixed domains, all 9 Shopify stores and all 9 WordPress sites were identified correctly, and 45 of 46 were analysed in 25 seconds (the one failure was a domain that doesn't exist).

What it costs

$3.00 per 1,000 results, one result = one website analysed. Sites that fail to load are saved as an error row for your records but never charged.

The 1,000-domain job above ≈ $3.00
10,000 domains = $30
Apify's free plan gives $5 of monthly credit, about 1,600 sites

Limits

Homepage only (or the exact URL you pass). A tool that only loads on checkout won't be seen unless you give that URL.
No JavaScript execution. The big one: analytics tags loaded through Google Tag Manager can be missed. The actor sees GTM, but not always what's inside the container. Tags pasted directly into the page are detected.
Bot walls. Some big sites (G2, Indeed, some banks) return a challenge page. You still get headers, DNS and TLS with blocked: true.
Hosting hints are hints. Behind Cloudflare, the origin host is hidden and you'll see the CDN.
Fingerprints are signatures, not certainty. Use minConfidence: 50 for the strictest list.

Bug reports go to the Issues tab on the actor page, which I read.

Website Tech Stack Detector on Apify

This article was drafted with AI and published by me.

Scraping Coles supermarket prices and weekly specials with Python

Gio Rich — Mon, 28 Sep 2026 01:57:39 +0000

Coles is one of Australia's two big supermarkets, and it sits behind Imperva bot protection. That's why most quick scripts people write against coles.com.au die after a few requests. If you want grocery price history, a half-price alert, or unit-price comparisons across a category, you need something that holds up.

I'm an 18-year-old engineering student, and I built Coles Scraper for this. This post covers calling it from Python and Node, what the output looks like, and a small weekly specials tracker.

How it works

coles.com.au is a Next.js site. Every search, category and product page has a matching /_next/data/<buildId>/...json route that returns the page data as JSON. The actor calls those routes directly over plain HTTP (no headless browser), picks up the new build ID whenever Coles deploys, and uses Australian residential proxies. It rotates to a fresh session when Imperva pushes back. In my test runs, 2,100 products (the full half-price list plus the whole bakery department) took 53 seconds with 0 failed requests.

Python example

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/coles-scraper").call(run_input={
    "searchQueries": ["tim tam", "milk"],
    "maxItems": 50,
})

for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(p["name"], p["price"], p["unitPriceText"], p.get("specialType"))

Node example

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_APIFY_TOKEN>" });

const run = await client.actor("rel8ble/coles-scraper").call({
  specials: "halfprice",
  maxItems: 0, // 0 = everything Coles lists
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`${items.length} half-price products`);

What comes back

A real row from the half-price specials list (no details):

{
  "productId": "2993706",
  "fullName": "COCA-COLA ZERO SUGAR SOFT DRINK BOTTLE 1.25L",
  "price": 2.25,
  "wasPrice": 4.5,
  "saveAmount": 2.25,
  "discountPercent": 50,
  "unitPriceText": "$1.80/ 1L",
  "specialType": "half_price",
  "offerDescription": "save $2.25",
  "available": true,
  "availableQuantity": 4824,
  "promotionalLimit": 12,
  "department": "Drinks",
  "category": "Soft Drinks",
  "aisle": "Soft Drink Bottles",
  "source": "specials",
  "specialsFilter": "halfprice"
}

With includeDetails: true you also get the barcode (GTIN), ingredients, allergens, country of origin and the full nutrition panel. A trimmed real example from a "chocolate" search:

{
  "productId": "2351720",
  "name": "Dairy Milk Hazelnut Chocolate Block",
  "brand": "Cadbury",
  "price": 5.5,
  "wasPrice": 8,
  "specialType": "price_drop",
  "unitPriceText": "$3.06/ 100g",
  "barcode": "9300617064923",
  "allergens": "Contains Hazelnut, Milk, Soy\nMay Contain Peanut, Wheat",
  "countryOfOrigin": "Australia",
  "nutrition": {
    "servingSize": "25g",
    "per100": { "Energy": "2340 kJ", "Protein": "9.0 g", "Sugars - Total": "46.4 g" }
  }
}

specialType is one of half_price, price_drop, multibuy, percent_off or special, so filtering for one kind of deal is a one-liner.

Use case: a weekly half-price tracker

Coles specials start on Wednesday. The idea: every Wednesday morning (AEST), pull the half-price list, keep it as a CSV with the date, and print anything on your personal watchlist.

import csv
import datetime
from apify_client import ApifyClient

WATCHLIST = ["tim tam", "coffee", "dishwasher", "olive oil"]

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("rel8ble/coles-scraper").call(run_input={
    "specials": "halfprice",
    "maxItems": 0,
})
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())

week = datetime.date.today().isoformat()
with open(f"coles-halfprice-{week}.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["productId", "name", "brand", "size", "price", "wasPrice", "unitPriceText"])
    for p in items:
        w.writerow([p["productId"], p["name"], p.get("brand"), p.get("size"),
                    p["price"], p.get("wasPrice"), p.get("unitPriceText")])

for p in items:
    name = f'{p.get("brand", "")} {p["name"]}'.lower()
    if any(word in name for word in WATCHLIST):
        print(f'{p["name"]} {p.get("size", "")}: ${p["price"]} (was ${p.get("wasPrice")})')

Schedule it for Wednesday morning with cron or an Apify Schedule. After a few weeks, productId lets you see how often a product cycles back to half price, which is the actual useful part: you stop buying things at full price that go half price every month.

If you care about one store's prices, set storeId (the number at the end of a store page URL, e.g. .../ardeer-7799 is 7799). Otherwise you get the online store's prices.

What it costs

$1.50 per 1,000 results, and one result is one product saved. Product details (barcode, nutrition, etc.) cost nothing extra per result, and failed or blocked requests are never charged.

A full half-price list in my test was 1,549 products ≈ $2.32 per week
A 20-item keyword watchlist instead of the full list is a few cents a week
Apify's free plan gives $5 of monthly credit, about 3,300 products

Limits

Coles' own totals are approximate. A category that says 550 products can return 549 or 551, because Coles re-ranks items between pages. Duplicates are removed.
Online prices by default. In-store prices can differ; set storeId for a specific store.
includeDetails doubles the request count, so leave it off when you only need prices.
No customer reviews in this version.
Sponsored products are skipped unless you turn on includeSponsored.

Questions or broken fields: the Issues tab on the actor page comes straight to me.

Coles Scraper on Apify

This article was drafted with AI and published by me.