DEV Community: Siftwright

Fix youtube-transcript-api RequestBlocked and IpBlocked on servers

Siftwright — Tue, 20 Oct 2026 14:47:40 +0000

The script worked yesterday. It fetched a transcript in half a second on your laptop, you wired it into a worker, deployed it, and now every call ends with this:

RequestBlocked:
Could not retrieve a transcript for the video https://www.youtube.com/watch?v=jNQXAC9IVRw!
This is most likely caused by:

YouTube is blocking requests from your IP. This usually is due to one of the following reasons:
- You have done too many requests and your IP has been blocked by YouTube
- You are doing requests from an IP belonging to a cloud provider (like AWS, Google Cloud Platform, Azure, etc.). ...

That is the real output of youtube-transcript-api 1.2.4, produced on 20 October 2026 from a cloud container, for the first YouTube video ever uploaded. Nothing is wrong with your code. This post explains what the error means, which fixes work, which ones to avoid, and shows a small, tested fallback you can drop into an existing pipeline.

What RequestBlocked and IpBlocked mean

The library is open source and does not use an official YouTube API. It makes the same requests a browser makes when it plays a video: it asks YouTube for the player data, finds the caption tracks in the response, and downloads the one you asked for. YouTube answers those requests differently depending on where they come from.

Both exceptions live in youtube_transcript_api._errors, and they are related:

RequestBlocked is the general case: YouTube refused to serve the player data to your address.
IpBlocked is a subclass of RequestBlocked. You can catch RequestBlocked and cover both.

The library's own error text names the two usual causes. You made a lot of requests from one address, or your address belongs to a cloud provider. Home and office connections are rarely blocked. Datacenter ranges (AWS, GCP, Azure and most VPS hosts) frequently are. That is why the same code passes on a laptop and fails the moment it is deployed.

Two practical consequences:

Retrying does not help. We ran three attempts, two seconds apart, from the blocked container. All three raised RequestBlocked. A block is a property of the address, not a transient glitch, so a retry loop only adds latency.
A block hides the real answer. When YouTube refuses you, the library cannot tell you whether the video has captions at all. A video that does not exist also raises RequestBlocked from a blocked address. You learn about the video only after you get past the block.

Not every failure is a block. The same module defines TranscriptsDisabled, NoTranscriptFound, VideoUnavailable, AgeRestricted, PoTokenRequired and others. Those describe the video or the request, and switching IPs will not change them. Your error handling should treat the two groups differently, which the fallback code below does.

Your options, honestly compared

1. Run it somewhere that is not a datacenter

If this is a personal script, run it on your own machine or a home server. It is free and it works. It stops being an answer when the job needs to run on a schedule in the cloud, or when more than one person's work depends on it.

2. Route requests through a proxy

The library has first-class proxy support. WebshareProxyConfig takes a username and password for residential proxies, and GenericProxyConfig takes any HTTP or HTTPS proxy URL:

from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api.proxies import GenericProxyConfig

api = YouTubeTranscriptApi(
    proxy_config=GenericProxyConfig(
        http_url="http://user:pass@proxy.example.com:8000",
        https_url="http://user:pass@proxy.example.com:8000",
    )
)
snippets = api.fetch("jNQXAC9IVRw").snippets

We did not run this one, because we have no proxy to point it at. The signature above is from the installed library. The caveats are worth knowing before you buy anything. The proxy has to be residential or otherwise not on a blocked range, since a datacenter proxy has the same problem as your server. You pay for it by traffic or by IP. And you take over the monitoring: when the pool gets flagged, your jobs fail until you notice.

3. Authenticate with cookies

The library's own error message mentions that authenticating requests with cookies can keep things working for a while, and in the same breath warns that YouTube may eventually permanently ban the account used. Do not tie an account you care about to a scraper.

4. Call an API that does the proxying for you

This moves the problem to a service whose whole job is to keep requests working. You send a video URL and get the transcript back as JSON, from any server. The tradeoff is cost: you pay per result instead of paying for proxy traffic. Whether that is cheaper depends on your volume. A few hundred transcripts a month is usually cheaper through an API than through a proxy plan you have to configure and watch. Millions a month is a different calculation, and at that scale you should price both.

We run one of these: the Siftwright YouTube Transcript API, and a free transcript tool in the browser for one-off videos. We are biased, so here is the useful part regardless of whose API you pick: the pattern below works with any provider.

The pattern: use the library first, fall back only on blocks

You do not have to rip out a working library to fix a blocked deployment. A good compromise is to keep the free path as the default and call the paid path only when YouTube blocks you. That keeps the cost proportional to the problem.

The rules for the wrapper:

Fall back only on RequestBlocked (which includes IpBlocked).
Do not fall back on TranscriptsDisabled, NoTranscriptFound or VideoUnavailable. The answer is "no", and a second service will say the same thing and may charge you to say it.
Return which path produced the text, so you can see in your logs how often you are blocked.

import os

import requests
from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api._errors import (
    IpBlocked,
    NoTranscriptFound,
    RequestBlocked,
    TranscriptsDisabled,
    VideoUnavailable,
)

API = "https://siftwright.com/v1/youtube-transcript"
KEY = os.environ["SIFTWRIGHT_API_KEY"]


def via_library(video_id):
    snippets = YouTubeTranscriptApi().fetch(video_id, languages=["en"]).snippets
    return " ".join(s.text for s in snippets)


def via_api(video_id):
    r = requests.post(
        API,
        headers={"Authorization": f"Bearer {KEY}"},
        json={
            "urls": [f"https://www.youtube.com/watch?v={video_id}"],
            "outputFormats": ["text"],
        },
        timeout=300,
    )
    r.raise_for_status()
    item = r.json()["results"][0]
    if item["status"] != "ok":
        raise LookupError(item.get("error", item["status"]))
    return item["text"]


def get_transcript(video_id):
    """Return (source, text). Only network blocks fall back."""
    try:
        return "library", via_library(video_id)
    except (RequestBlocked, IpBlocked):
        return "api", via_api(video_id)
    except (TranscriptsDisabled, NoTranscriptFound, VideoUnavailable) as err:
        raise LookupError(type(err).__name__) from err


for vid in ["jNQXAC9IVRw", "dQw4w9WgXcQ", "aaaaaaaaaaa"]:
    try:
        source, text = get_transcript(vid)
        print(vid, source, len(text.split()), "words:", text[:50])
    except LookupError as err:
        print(vid, "no transcript:", err)

Install with pip install youtube-transcript-api requests. We ran exactly this file from the same blocked container with a real key:

jNQXAC9IVRw api 39 words: All right, so here we are, in front of the elephan
dQw4w9WgXcQ api 487 words: [♪♪♪] ♪ We're no strangers to love ♪ ♪ You know th
aaaaaaaaaaa no transcript: This video is unavailable

Read the output carefully, because it shows the pattern working as designed. The first two videos were blocked by the library, so both went through the API (api). On a laptop they would print library and cost nothing. The third ID does not exist. The library was blocked on it, so it fell back, and the API gave the real answer: This video is unavailable.

That call is also a small proof of the billing rule. The request contained two URLs in an earlier test, one real and one dead, and the response reported "count": 2, "billed": 1. Videos that return no transcript are not billed.

Doing it in bulk

If you are fetching many videos, do not call the fallback once per video. The API accepts a list of URLs in urls and returns one result per video, so you can send the blocked ones in a single request. A simple structure:

Loop over IDs with the library, collecting the ones that raise RequestBlocked.
Stop trying the library once the first few calls in a run are all blocked. If three in a row fail, the address is blocked and the rest will fail too. Send everything remaining straight to the API.
Send the collected IDs in batches to the API with outputFormats set to ["text"] (or ["text", "segments"] if you need timestamps).

Step 2 matters more than it looks. In our test a blocked attempt took under a second, which sounds harmless until you multiply it: 1,000 videos against a blocked address is more than ten minutes of guaranteed failures before the real path even starts.

For the full response shape, languages, SRT and VTT output, and playlist handling, see our YouTube transcript API guide for Python and Node.js and the API docs.

Other things worth checking before you blame the IP

A few errors look similar but have a different cause:

TranscriptsDisabled or NoTranscriptFound: the creator turned captions off, or no track exists in the language you requested. Check the languages list, and consider asking for a fallback language. No proxy fixes this.
AgeRestricted: the video is age-gated, so anonymous requests cannot read it. Skip it or handle it manually.
PoTokenRequired and YouTubeDataUnparsable: YouTube changed what its player returns, and the library needs an update. Run pip install -U youtube-transcript-api before anything else, and pin a version in production so an upgrade is a choice you make.
Works for some videos only: that is a hint that the failing ones have a video-level restriction rather than an address problem.

Choosing, in one paragraph

If you run it yourself, on your own connection, at low volume: keep using the library and do nothing else. If it must run in the cloud and you have a few hundred transcripts a month, use the fallback pattern above so you only pay when you are blocked. If YouTube transcripts are central to your product and your volume is large, price a managed proxy pool against a per-result API with your real numbers before deciding. And in every case, handle the "genuinely no captions" errors separately, because they are answers and not failures.

If you want to try the API path without writing code first, the free transcript tool runs the same extraction in your browser. For code, plans start at $29/month with 10,000 results included.

Collect company social links, emails and phones from a list of websites in Python

Siftwright — Mon, 19 Oct 2026 12:54:26 +0000

Company websites already publish their social profiles and contact details; the annoying part is collecting them for a whole list of domains. Here is a small Python script that does it through an Apify Actor billed per website ($2 per 1,000 sites, and sites that fail to load are not charged).

The script

import os, requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = ("https://api.apify.com/v2/acts/siftwright~contact-details-extractor/"
       "run-sync-get-dataset-items")

sites = ["https://www.python.org", "https://github.com", "https://www.mozilla.org"]
r = requests.post(URL, params={"token": TOKEN, "timeout": 90},
                  json={"startUrls": sites, "maxPagesPerDomain": 3}, timeout=120)
r.raise_for_status()

for row in r.json():
    print(row["domain"], row["status"], row["emails"], row["phones"],
          row["linkedin"], row["twitter"])

Real output

I ran exactly this on 2026-10-19. Each site returned status: "ok" with 3 pages scanned:

domain	linkedin	twitter / X	instagram	emails	phones
www.python.org	python-software-foundation	ThePSF	none	none	none
www.mozilla.org	mozilla-corporation	none	mozilla	none	none
github.com	github	github	github	none	none

Note the empty email and phone columns: these three sites simply don't publish an address on the pages scanned, and the Actor does not guess. Smaller company sites with a visible contact page are where emails and phones show up. Raise maxPagesPerDomain if the contact page is deeper in the site.

Pricing and limits

One charge per website that loads, $0.002 each; dead domains are free.
It reads public pages only: no logins, no proxies by default, so heavily protected sites may return a failed status.
Fields per site: emails, phones, facebook, twitter, linkedin, instagram, youtube, pagesScanned.

Try it: Contact Details Extractor on Apify. It also works from AI agents through Apify's MCP server. More tools: siftwright.com.

Build a daily news brief in Python with Google News search, billed per article

Siftwright — Sun, 18 Oct 2026 19:09:08 +0000

Need fresh news headlines in a script, a RAG pipeline or a media-monitoring job, without scraping Google or wiring up a paid news API? Here is a small Python brief generator built on the Google News Scraper Apify Actor. It is billed per article returned ($1.50 per 1,000), and an empty or failed search costs nothing.

Input

The Actor takes query, maxItems (1 to 200), language and country. Each result has title, link, source, publishedAt (ISO 8601), snippet and status.

The script

import os
from collections import Counter
import requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = "https://api.apify.com/v2/acts/siftwright~google-news-scraper/run-sync-get-dataset-items"

def news(query, n=10, language="en", country="US"):
    r = requests.post(URL, params={"token": TOKEN},
                      json={"query": query, "maxItems": n, "language": language, "country": country},
                      timeout=180)
    r.raise_for_status()
    return [a for a in r.json() if a.get("status") == "ok"]

watchlist = ["open source LLM", "vector database"]
for q in watchlist:
    items = sorted(news(q), key=lambda a: a["publishedAt"], reverse=True)
    print(f"\n## {q} ({len(items)} articles)")
    for a in items[:3]:
        print(f"- {a['publishedAt'][:10]} {a['source']}: {a['title']}\n  {a['link']}")
    print("top sources:", Counter(a["source"] for a in items).most_common(3))

run-sync-get-dataset-items starts the run and returns the dataset in one HTTP call, which suits short jobs like this one.

Real output

I ran it on 2026-10-18 with a watchlist of two queries and 10 articles each. Sorted newest first, the top hit for "open source LLM" was a HackerNoon piece dated 2026-10-18 on GPU memory for running open LLMs locally, and for "vector database" an article dated 2026-10-13 about Elastic's serverless vector database. Your results will differ as the news changes.

Two notes from testing:

link values are Google News redirect URLs. They open the article in a browser, but if you need the publisher URL you have to resolve the redirect yourself.
snippet is often just the title plus the outlet name, so don't count on it for body text.

Variations

Change language and country (for example de and DE) to read another Google News edition.
Schedule the same Actor on Apify and send the dataset to a webhook for daily monitoring.
It also works with AI agents through Apify's MCP server, so an assistant can run the search itself.

More details and a curl example: siftwright.com/google-news-api. Questions: support@siftwright.com.

Full-page website screenshots and PDFs from Python or curl, billed per capture

Siftwright — Mon, 12 Oct 2026 20:54:31 +0000

Screenshots and PDFs of web pages are a common chore: link previews, visual regression checks, archiving a page as evidence, feeding page images to a vision model. Running your own headless browser works, but you end up maintaining Chromium, fonts and timeouts. Here is a small pay-per-result alternative, with the real output of a test run.

I am building Pageframe, an Apify Actor that captures full-page PNG/JPEG screenshots or print-ready PDFs. It costs $0.0015 per successful capture, and failed URLs are not charged.

The input

{
  "urls": ["https://example.com"],
  "format": "png",
  "fullPage": true,
  "viewportWidth": 1280,
  "viewportHeight": 800,
  "deviceScaleFactor": 1,
  "blockCookieBanners": true,
  "delaySec": 0
}

Other options: format can be png, jpeg or pdf (with pdfPageFormat A4, A3, Letter or Legal), deviceScaleFactor 2 or 3 gives retina captures, waitForSelector waits for an element before capturing, and blockCookieBanners makes a best-effort click on common consent buttons.

From Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("siftwright/pageframe-screenshots").call(run_input={
    "urls": ["https://example.com"],
    "format": "png",
    "fullPage": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["status"], item["fileUrl"])

From curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~pageframe-screenshots/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"format":"pdf","pdfPageFormat":"A4"}'

Real output

This is the dataset item from an actual run on Apify, capturing https://example.com:

{
  "url": "https://example.com",
  "status": "ok",
  "format": "png",
  "fileUrl": "https://api.apify.com/v2/key-value-stores/iSfPTldM2Pvavg8Ac/records/capture-1791797595722-zokzr1.png",
  "sizeBytes": 71152,
  "durationMs": 289
}

The run billed one capture event (plus Apify's tiny per-run start fee). The image itself is stored in the run's key-value store at fileUrl.

Honest limits

Pages behind a login are not supported.
Cookie banner dismissal is best-effort; unusual consent widgets may still show.
Heavy pages that load content late may need waitForSelector or delaySec.

Use it from an AI agent

Every Siftwright Actor works through Apify's MCP server, so an agent can call Pageframe as a tool. There is also a free tool page at siftwright.com if you just want to try one capture first.

Feedback or a missing option? Write to support@siftwright.com.

Get YouTube transcripts in bulk: text, SRT and VTT from Python, pay only per result

Siftwright — Wed, 07 Oct 2026 12:55:17 +0000

Getting a YouTube transcript sounds trivial until you need it for 500 videos: the official Data API doesn't return caption text for videos you don't own, libraries break when YouTube changes a page, and a plain requests loop gets rate-limited fast.

This post shows a small, honest workflow: send video URLs or IDs to an Apify Actor, get back clean text, timestamped segments and ready-to-use SRT/VTT subtitles, and only pay for transcripts that were actually returned.

Try it without writing code

Paste a link into the free tools page at https://siftwright.com/tools/ or open the Actor directly: https://apify.com/siftwright/youtube-transcript-extractor

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/youtube-transcript-extractor").call(run_input={
    "urls": ["jNQXAC9IVRw"],          # URL, Shorts link, playlist, channel or bare ID
    "language": "en",
    "outputFormats": ["text", "segments", "srt"],
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["status"], item["wordCount"])
    print(item["text"])

Real output

I ran it on the first video ever uploaded to YouTube ("Me at the zoo", 19 seconds). The result, trimmed:

{
  "videoId": "jNQXAC9IVRw",
  "status": "ok",
  "title": "Me at the zoo",
  "channelName": "jawed",
  "durationSeconds": 19,
  "language": "en",
  "isAutoGenerated": false,
  "availableLanguages": [{"code": "en"}, {"code": "de"}],
  "wordCount": 39,
  "text": "All right, so here we are, in front of the elephants the cool thing about these guys is that they have really... really really long trunks and that's cool (baaaaaaaaaaahhh!!) and that's pretty much all there is to say"
}

The srt field comes back as a ready-to-save subtitle file:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

2
00:00:05,318 --> 00:00:07,974
the cool thing about these guys is that they have really...

curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~youtube-transcript-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"urls":["jNQXAC9IVRw"],"outputFormats":["text","srt"]}'

Useful options

Playlists and channels: pass a playlist or @handle URL and set maxVideosPerSource.
Translation: translateTo: "de" uses YouTube's built-in machine translation.
Prefer human captions: on by default; falls back to auto-generated.
Metadata (title, channel, views, keywords) is included at no extra charge.

Pricing and limits

Pricing is pay per event: $0.003 per transcript returned ($3 per 1,000). Videos with no captions, private videos and failures are reported in the dataset with a status and are not charged. Limits: a video needs captions (manual or auto-generated) to have a transcript, and some fields such as publish date can be empty depending on what YouTube exposes.

Using it from an AI agent

The same Actor works through Apify's MCP server, and Siftwright also runs a hosted MCP endpoint at https://siftwright.com/mcp so an agent can fetch transcripts as a tool call.

If something breaks on a video you care about, tell us at support@siftwright.com and include the video ID.

Bulk domain age, expiry and DNS audit in Python: RDAP with WHOIS fallback

Siftwright — Mon, 05 Oct 2026 19:10:36 +0000

A domain list is a surprisingly useful dataset. Which of your 500 customer domains expire this quarter? Which lead domains were registered last month (often a trust signal)? Which of your own domains have no DMARC record? All of it is public data, but getting it for a whole list means juggling RDAP, legacy WHOIS servers and DNS lookups.

This tutorial uses one Apify Actor, Domain WHOIS & DNS Lookup, to turn a list of domains into clean JSON rows: registrar, creation and expiry dates, age, nameservers, plus MX, SPF and DMARC. I'm the maker, so the pricing is stated up front: $1 per 1,000 domains, and invalid inputs, duplicates and failed lookups are free.

The code

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("siftwright/domain-whois-dns-lookup").call(run_input={
    "domains": ["stripe.com", "github.io", "https://www.bbc.co.uk/news", "spiegel.de", "this-name-should-not-exist-siftwright-test.com"],
})
for d in client.dataset(run.default_dataset_id).iterate_items():
    if not d["registered"]:
        print(f'{d["domain"]:<14} not registered (likelyAvailable={d["likelyAvailable"]})'); continue
    dns = d.get("dns") or {}
    soon = d["daysUntilExpiry"] is not None and d["daysUntilExpiry"] < 90
    print(f'{d["domain"]:<14} {d["registrar"] or "?":<30} age {d["ageDays"]} days  src={d["source"]}'
          f'{"  EXPIRES<90d" if soon else ""}{"  NO-DMARC" if not dns.get("dmarc") else ""}{"  NO-SPF" if not dns.get("spf") else ""}')

Install with pip install apify-client (3.x; on 2.x use run["defaultDatasetId"]) and set APIFY_TOKEN.

Real output

This is the output of that exact script, run on 2026-10-05 (the last row is a made-up name, so it comes back unregistered):

this-name-should-not-exist-siftwright-test.com not registered (likelyAvailable=True)
stripe.com     SafeNames Ltd.                 age 11346 days  src=rdap
github.io      MarkMonitor Inc.               age 4958 days  src=whois  NO-DMARC
spiegel.de     ?                              age None days  src=whois
bbc.co.uk      British Broadcasting Corporation age 11619 days  src=rdap

Things worth noticing:

https://www.bbc.co.uk/news was reduced to the registrable domain bbc.co.uk automatically.
.io-style and .de domains have no usable RDAP, so the Actor falls back to classic WHOIS (src=whois).
.de registries publish very little. spiegel.de has no registrar or creation date in public data, so those fields are null rather than guessed. Check for None before doing arithmetic.
Registrant names and emails are usually redacted because of GDPR. Don't build on them.

Ideas to build on it

Expiry watchlist: run weekly on your portfolio and alert when daysUntilExpiry < 60.
Lead scoring: low ageDays plus no MX can flag throwaway domains.
Email deliverability audit: list every domain missing SPF or DMARC.

Pricing and limits

Pay-per-event at $0.001 per domain looked up (registered or not). Apify's free plan includes monthly credits, so you can try it at no cost. AI agents can also call it through Apify's MCP server: https://mcp.apify.com?tools=siftwright/domain-whois-dns-lookup.

Full details and the product page: siftwright.com/domain-whois-lookup/. Questions: support@siftwright.com.

Clean a signup list in bulk: MX checks, disposable domains and typo suggestions in Python

Siftwright — Sat, 03 Oct 2026 04:53:49 +0000

Bad addresses in a signup list cost you bounces, sender reputation and wasted sends. Most checks you can run without sending a single email: syntax, whether the domain has MX records, whether it is a disposable provider, and whether it looks like a typo of a common domain. This tutorial runs those checks in bulk with the Bulk Email Validator Actor on Apify.

What it checks

Syntax and domain split
MX lookup (does the domain accept mail at all)
Disposable-provider detection
Role addresses (info@, admin@), free providers, plus-tags
Typo suggestions (gmial.com -> gmail.com)

It does not do SMTP mailbox probing, so a "deliverable" verdict means the domain can receive mail, not that the specific mailbox exists. Pricing is pay-per-event, $0.0008 per email, and it only charges for emails it actually returns.

Run it from Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/email-validator-mx-disposable").call(run_input={
    "emails": ["jane@github.com", "test@mailinator.com", "someone@gmial.com"],
    "checkMx": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["email"], item["verdict"], item["reason"], item["suggestion"])

Real output

This is the actual result of a test run with those three addresses (3 emails returned, 3 charged):

email	verdict	reason	note
jane@github.com	deliverable	-	MX: github-com.mail.protection.outlook.com
test@mailinator.com	risky	disposable	has MX, but a throwaway provider
someone@gmial.com	risky	possible_typo	suggestion: someone@gmail.com, no MX

Cleaning a list

Keep deliverable, drop or review risky, and discard anything invalid:

clean = [i["email"] for i in items if i["verdict"] == "deliverable"]
review = [(i["email"], i["reason"], i["suggestion"]) for i in items if i["verdict"] == "risky"]

The typo suggestions are useful for signup forms: show "Did you mean someone@gmail.com?" instead of silently accepting a bad address.

Limits

No SMTP mailbox verification, so catch-all domains and non-existent mailboxes on valid domains pass the MX check. Use it as a cheap first filter before sending, not as a guarantee.

Actor: https://apify.com/siftwright/email-validator-mx-disposable

Turn a list of company websites into contact details: emails, phones and socials in Python

Siftwright — Fri, 02 Oct 2026 14:51:21 +0000

You have a list of company websites and nothing else: no email, no phone, no LinkedIn page. Maybe it came from a conference exhibitor page, a directory export or a CRM with half the columns empty. Every prospecting workflow starts the same way: open the site, hunt for the contact page, copy an address into a spreadsheet.

That is mechanical work, so it can be scripted. But there is a trap on the way: scraping every address a website mentions gives you a column full of legal@, privacy@ and copyright@ mailboxes, plus the occasional named employee. Extraction is the easy half. The useful half is deciding which of the addresses you found is one you should actually write to.

This guide does both. We take six websites, extract public contact details from each, and produce an enriched CSV with one best email per company. Everything below was run on 2 October 2026 against real sites, and the output shown is the real output, including the cases where the honest answer is "nothing usable".

What you need

Python 3.9 or newer (the code uses str.removeprefix, which arrived in 3.9).
An Apify account and API token. The free plan includes monthly credits, far more than this tutorial uses.
The official client: pip install apify-client. The code uses version 3.x, where runs come back as objects (run.default_dataset_id). On the 2.x client, use run["defaultDatasetId"].

We use the Contact Details Extractor, which we publish on the Apify Store. Give it homepage URLs and it checks the homepage plus pages that look like contact, about or support pages, up to a page limit you set. It reads emails from visible text and mailto: links, phone numbers from text and tel: links, and social profiles from structured data (sameAs) and the site's header, footer and navigation. It costs $0.002 per website ($2 per 1,000), and sites that fail to load are free. This whole tutorial cost about one cent.

export APIFY_TOKEN="apify_api_..."

The lead list

leads.csv is deliberately small and mixed: a few software companies and one domain that does not exist, so we can see how failures look.

company,website
Basecamp,basecamp.com
Mozilla,mozilla.org
Ghost,ghost.org
Buffer,buffer.com
Plausible,plausible.io
Example (dead),this-domain-does-not-exist-sw123.com

Step 1: one run for the whole list

The Actor takes all the websites in one run and scans them in parallel, so you do not loop over rows:

run = client.actor("siftwright/contact-details-extractor").call(
    run_input={"startUrls": [row["website"] for row in leads], "maxPagesPerDomain": 6}
)
results = {
    item["domain"].removeprefix("www."): item
    for item in client.dataset(run.default_dataset_id).iterate_items()
}

maxPagesPerDomain is the crawl depth per website (default 6, maximum 15). It does not change the price, because you are charged once per website, so raising it costs only time.

Notice that we key the results by domain, not by the string we sent. The Actor normalises each input to a full URL (basecamp.com comes back as https://basecamp.com), so a lookup on the original CSV value would miss. We normalise our own rows the same way with a small helper:

def domain_of(site):
    host = urlparse(site if "//" in site else "https://" + site).hostname or ""
    return host.removeprefix("www.")

What comes back

One item per website. Here is the real item for plausible.io:

{
  "domain": "plausible.io",
  "url": "https://plausible.io",
  "status": "ok",
  "emails": ["hello@plausible.io"],
  "phones": [],
  "facebook": null,
  "twitter": "https://twitter.com/PlausibleHQ",
  "linkedin": "https://www.linkedin.com/company/plausible-analytics/",
  "instagram": null,
  "youtube": null,
  "pagesScanned": 6
}

The dead domain comes back as a row too, with a reason and no charge:

{
  "domain": "this-domain-does-not-exist-sw123.com",
  "status": "error",
  "error": "fetch failed",
  "pagesScanned": 0
}

Across the six sites, five loaded and the Actor's log reported "5 website(s) scanned, 1 failed to load (not charged)".

What the raw data looks like

This is where honest tooling matters, so here is every result for the five sites that loaded:

Site	Emails found	Phones	Socials
basecamp.com	one address belonging to a named person	none	none
ghost.org	`support@ghost.org`	none	none
plausible.io	`hello@plausible.io`	none	X, LinkedIn
mozilla.org	`trademark-permissions@mozilla.com`	none	LinkedIn, Instagram
buffer.com	`hello@`, `legal@`, `copyright@`, `privacy@`, `security@`	two	Facebook, X, LinkedIn, Instagram

Three things stand out, and none of them is a bug:

Most sites publish very little. Companies deliberately keep their public contact surface small. Two of five sites gave us no phone number at all, because they do not publish one. An extractor can only report what a site shows.
The addresses are not all equal. Buffer publishes five. Only one is a way to start a business conversation. Mozilla's only public address on the pages checked is a trademark-permissions mailbox, which is a department for one specific purpose.
The data is as the website wrote it. Buffer's second phone number came back as +1-800)-952-5210, with a stray bracket exactly as the page formatted it. We clean that on our side.

Raw extraction output is a list of candidates, not a finished lead list. Hence step 2.

Step 2: rank the emails

We sort addresses into three buckets by their local part, the bit before the @:

# Role mailboxes you can usually write to about business, best first.
PREFERRED = ["sales", "hello", "contact", "info", "team", "partnerships", "support"]
# Mailboxes that exist for something else: never use them for outreach.
AVOID = ("legal", "privacy", "security", "copyright", "abuse", "dmca", "press",
         "trademark", "noreply", "no-reply", "careers", "jobs", "billing")


def rank_email(email):
    local = email.split("@")[0].lower()
    if any(word in local for word in AVOID):
        return 99
    for i, word in enumerate(PREFERRED):
        if local == word:
            return i
    return 50  # looks like a named person: keep it out of automated outreach


def best_email(emails):
    ranked = sorted(emails, key=rank_email)
    if ranked and rank_email(ranked[0]) < 50:
        return ranked[0]
    return ""

The rule is conservative on purpose. A role address like hello@ is a mailbox the company set up to receive messages from strangers. A legal@ or privacy@ mailbox is for formal requests, and a cold pitch there is both ineffective and annoying. An address that looks like one person's name, such as the one we found on Basecamp's site, is a personal address. Even when it is published, it is better left out of automated mailings, and it is personal data in the legal sense, which matters if you or your recipients are in the EU or UK. best_email returns an empty string in that case, and we keep the original in an all_emails column so a human can decide.

Phones need tidying because they arrive as they appeared on the page:

def clean_phone(phone):
    # The Actor returns phones as found on the page, so tidy them here.
    phone = re.sub(r"[^\d+]", "", phone)
    return phone if 8 <= len(phone) <= 15 else ""

Stripping everything except digits and + turns +1-800)-952-5210 into +18009525210, and the length check (E.164 numbers have at most 15 digits) drops fragments that are not phone numbers.

Step 3: write the enriched CSV

rows = []
for lead in leads:
    item = results.get(domain_of(lead["website"]), {})
    if item.get("status") != "ok":
        rows.append({**lead, "status": item.get("error", "not scanned")})
        continue
    emails = item["emails"]
    rows.append({
        **lead,
        "status": "ok",
        "best_email": best_email(emails),
        "all_emails": " ".join(emails),
        "phone": next((p for p in map(clean_phone, item["phones"]) if p), ""),
        "linkedin": item["linkedin"] or "",
        "twitter": item["twitter"] or "",
    })

Failed sites stay in the file with their error instead of vanishing, so nobody wonders why a company is missing and you can fix the typo and re-run only those rows. The full script, which also writes leads_enriched.csv, is at the end. Its real output:

Basecamp        ok           -                    -
Mozilla         ok           -                    -
Ghost           ok           support@ghost.org    -
Buffer          ok           hello@buffer.com     +18007787879
Plausible       ok           hello@plausible.io   -
Example (dead)  fetch failed -                    -

Three of the five loaded sites yield a usable address (two hello@ and one support@), and two yield none, which is a realistic result for modern software companies that hide behind contact forms. Lists of local businesses, agencies and shops usually do better, since they tend to publish a plain info@ address and a phone number. Test on a sample of your own list first; at $2 per 1,000 websites that costs next to nothing.

What this does not do

It does not guess or verify addresses. It reports what the site publishes. It does not generate firstname@company.com patterns or check whether a mailbox exists.
It does not read contact forms, JavaScript-only pages or anything behind a login. If a site shows its address only after a script runs, it will be missed.
It checks a limited number of pages. The default is 6 pages per site; an address buried on page 40 is out of reach.
Social links come from structured data and header, footer and navigation only. That is deliberate: a page-wide scan picks up links to other people's profiles in blog posts and comments.

Use the data responsibly

Public does not mean unrestricted. Before you email anyone, check the rules that apply to you: CAN-SPAM in the US, and GDPR plus the ePrivacy rules in the EU and UK, which treat a named person's work address as personal data and set conditions for marketing to it. Role mailboxes at companies are the easiest case. Always identify yourself, say why you are writing, and honour opt-outs. This guide is not legal advice.

The full script

import csv
import os
import re
from urllib.parse import urlparse

from apify_client import ApifyClient


def domain_of(site):
    host = urlparse(site if "//" in site else "https://" + site).hostname or ""
    return host.removeprefix("www.")


client = ApifyClient(os.environ["APIFY_TOKEN"])

with open("leads.csv", newline="") as f:
    leads = list(csv.DictReader(f))

run = client.actor("siftwright/contact-details-extractor").call(
    run_input={"startUrls": [row["website"] for row in leads], "maxPagesPerDomain": 6}
)
results = {
    item["domain"].removeprefix("www."): item
    for item in client.dataset(run.default_dataset_id).iterate_items()
}

# Role mailboxes you can usually write to about business, best first.
PREFERRED = ["sales", "hello", "contact", "info", "team", "partnerships", "support"]
# Mailboxes that exist for something else: never use them for outreach.
AVOID = ("legal", "privacy", "security", "copyright", "abuse", "dmca", "press",
         "trademark", "noreply", "no-reply", "careers", "jobs", "billing")


def rank_email(email):
    local = email.split("@")[0].lower()
    if any(word in local for word in AVOID):
        return 99
    for i, word in enumerate(PREFERRED):
        if local == word:
            return i
    return 50  # looks like a named person: keep it out of automated outreach


def best_email(emails):
    ranked = sorted(emails, key=rank_email)
    if ranked and rank_email(ranked[0]) < 50:
        return ranked[0]
    return ""


def clean_phone(phone):
    # The Actor returns phones as found on the page, so tidy them here.
    phone = re.sub(r"[^\d+]", "", phone)
    return phone if 8 <= len(phone) <= 15 else ""


rows = []
for lead in leads:
    item = results.get(domain_of(lead["website"]), {})
    if item.get("status") != "ok":
        rows.append({**lead, "status": item.get("error", "not scanned")})
        continue
    emails = item["emails"]
    rows.append({
        **lead,
        "status": "ok",
        "best_email": best_email(emails),
        "all_emails": " ".join(emails),
        "phone": next((p for p in map(clean_phone, item["phones"]) if p), ""),
        "linkedin": item["linkedin"] or "",
        "twitter": item["twitter"] or "",
    })

fields = ["company", "website", "status", "best_email", "all_emails", "phone", "linkedin", "twitter"]
with open("leads_enriched.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(rows)

for row in rows:
    print(f"{row['company']:<15} {row['status']:<12} {row.get('best_email') or '-':<20} {row.get('phone') or '-'}")

Next steps

Combine this with the Tech Stack Detector to segment the list first, as in find every Shopify, WordPress and HubSpot site in a lead list, and only extract contacts for the segment you actually sell to.
If you prefer a plain HTTP API with an API key over the Apify client, the same extraction is available as POST /v1/contact-details in the Siftwright API. For the cost comparison, see pay-per-result vs subscription pricing.

Questions or a site that should work and does not? Write to support@siftwright.com.

One schema for Greenhouse, Lever, Ashby and more: pulling job postings without writing five parsers

Siftwright — Thu, 01 Oct 2026 08:54:11 +0000

Every company hosts its careers page on an applicant tracking system (ATS): Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee. Each has its own JSON shape, so tracking jobs across companies usually means writing one parser per platform. Here is how to get them all in one schema with a single call.

The idea

Give the ATS Jobs Scraper a list of board URLs, company websites or ats:board shorthands. It detects the platform, pulls the open jobs, applies your filters, and only then charges ($0.001 per job returned; boards it cannot read and filtered-out jobs are free).

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/ats-jobs-scraper").call(run_input={
    "companies": ["https://boards.greenhouse.io/stripe", "https://jobs.lever.co/palantir"],
    "titleKeywords": ["data engineer"],
    "maxJobsPerCompany": 3,
    "includeDescription": False,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["company"], job["ats"], job["title"], job["location"], job["url"])

What I got back (real run, 2026-10-01)

I ran exactly that input. Stripe (Greenhouse) returned one matching job and Palantir (Lever) returned none for that title filter, so I was charged for one job:

Stripe | Greenhouse | Staff Software Engineer, Risk Data Engineering | Toronto | posted 2026-08-06

Each item has the same fields whatever the ATS: company, ats, title, department, location, locations, remote, employmentType, postedAt, updatedAt, url, applyUrl, salary (when the board publishes it) and optional descriptionText.

Useful filters

titleKeywords / excludeTitleKeywords: e.g. include "engineer", exclude "intern"
locations, departments, remoteOnly
postedWithinDays: run it daily with 1 to get only new postings

Limits

It reads public job boards only. Companies on other platforms (Workday, custom sites) are skipped and not charged. Salary and remote fields are only present when the board provides them.

curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~ats-jobs-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"companies":["greenhouse:airbnb"],"remoteOnly":true,"maxJobsPerCompany":5}'

It also works through Apify's MCP server, so an AI agent can call it as a tool.

Find every Shopify, WordPress and HubSpot site in a lead list: tech stack lookup with evidence in Python

Siftwright — Thu, 01 Oct 2026 01:47:21 +0000

Originally published on siftwright.com.

If you sell a Shopify app, a WordPress plugin or a migration service, the first question about any lead is the same: what is their website actually built on? A lead list with that one extra column is worth far more than one without it, because you can stop pitching a Shopify app to companies that run Magento.

Tools like Wappalyzer and BuiltWith answer the question, but their APIs are priced for teams that look up thousands of sites a month. If you have a list of a few hundred companies from a conference, a CRM export or a directory, you want something closer to "pay a fraction of a cent per website and get the answer back as JSON".

This guide does exactly that. We take a small CSV of company websites, detect the technologies on each one, and write an enriched CSV with four new columns: platform, the evidence for that platform, email provider and hosting. Then we print the list grouped by platform, so you can see your segments at a glance.

Everything here was run on 1 October 2026 against real, public websites. The output shown is the real output.

What you need

Python 3.9 or newer.
An Apify account and API token. The free plan includes monthly credits, which covers this tutorial many times over.
The official client: pip install apify-client. The code uses version 3.x, where runs come back as objects (run.default_dataset_id). On the older 2.x client, use run["defaultDatasetId"] instead.

We use the Tech Stack Detector, which we built and publish on the Apify Store. It loads each website once over plain HTTPS, reads the response headers, cookies, meta tags, script and stylesheet URLs, and checks the domain's MX, SPF and NS records. It costs $0.002 per website scanned ($2 per 1,000), and sites that fail to load are free. The ten sites in this tutorial cost under two cents.

Set your token once:

export APIFY_TOKEN="apify_api_..."

The lead list

Here is the input, leads.csv. It is deliberately mixed: two direct-to-consumer brands, a news site, a magazine, a few software companies, a non-profit and one domain that does not exist, so we can see how failures are handled.

company,website
Allbirds,allbirds.com
Gymshark,gymshark.com
TechCrunch,techcrunch.com
The New Yorker,newyorker.com
Basecamp,basecamp.com
Notion,notion.so
HubSpot,hubspot.com
Mozilla,mozilla.org
Ghost,ghost.org
Example (dead),this-domain-does-not-exist-sw123.com

In real life this is your CRM export. The only column the code relies on is website, and it accepts bare domains or full URLs.

One call for the whole list

The detector takes a list of websites and scans them in parallel, so there is no need to loop and call it once per row:

import csv
import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

with open("leads.csv", newline="") as f:
    leads = list(csv.DictReader(f))

run = client.actor("siftwright/tech-stack-detector").call(
    run_input={"urls": [row["website"] for row in leads], "includeDns": True}
)
# apify-client 3.x returns a Run object; on 2.x use run["defaultDatasetId"]
results = {item["input"]: item for item in client.dataset(run.default_dataset_id).iterate_items()}

We key the results by input, the exact string we sent, because the site you ask for is often not the site you land on. notion.so redirects to www.notion.com, and gymshark.com ended up on a regional checkout domain. Matching on the input keeps every result attached to the right row of your CSV.

includeDns turns on the MX, SPF and NS checks. They are what tell you a company uses Google Workspace or Microsoft 365 for email, which is often as useful for targeting as the website platform itself. It costs nothing extra.

What comes back

Each website produces one item. Here is the one for ghost.org, shortened:

{
  "input": "ghost.org",
  "finalUrl": "https://ghost.org/",
  "status": "ok",
  "httpStatus": 200,
  "generator": "Hugo 0.119.0",
  "technologyCount": 11,
  "byCategory": {
    "DNS provider": ["Cloudflare DNS"],
    "Email provider": ["Google Workspace"],
    "Hosting": ["Netlify"],
    "Static site generator": ["Hugo"],
    "Transactional email": ["Mandrill"],
    "UI framework": ["Tailwind CSS"]
  },
  "technologies": [
    { "name": "Netlify", "categories": ["Hosting"],
      "evidence": ["header server: Netlify", "header x-nf-request-id: 01M3TJ4..."] },
    { "name": "Google Workspace", "categories": ["Email provider"],
      "evidence": ["dns mx: alt1.aspmx.l.google.com"] }
  ]
}

Two fields do most of the work:

byCategory groups the technology names by what they are. It is the quickest way to ask "which CMS?" or "which email provider?".
technologies carries the evidence for every detection: the header, meta tag, asset URL or DNS record that matched. This matters more than it looks. When a salesperson asks "are we sure they're on Shopify?", you can answer with header powered-by: Shopify instead of "the tool said so".

A failed site comes back as a row too, with status: "error" and a reason, and it is not charged:

{
  "input": "this-domain-does-not-exist-sw123.com",
  "status": "error",
  "error": "Domain not found (DNS lookup failed)"
}

Turning detections into segments

A lead list needs one platform per company, not fifteen technology names. So we pick the first match from a short list of categories, in order of how specific they are: an ecommerce platform beats a CMS, and a CMS beats a static site generator.

from collections import defaultdict

PLATFORM_CATEGORIES = ["Ecommerce", "CMS", "Headless CMS", "Static site generator"]

def first_in(item, categories):
    for category in categories:
        names = item.get("byCategory", {}).get(category)
        if names:
            return names[0]
    return ""

def evidence_for(item, name):
    for tech in item.get("technologies", []):
        if tech["name"] == name:
            return tech["evidence"][0]
    return ""

Then we walk the original leads, in their original order, and build one output row per company:

segments = defaultdict(list)
rows = []
for lead in leads:
    item = results.get(lead["website"], {})
    if item.get("status") != "ok":
        rows.append({**lead, "status": item.get("error", "not scanned")})
        continue
    platform = first_in(item, PLATFORM_CATEGORIES)
    email = first_in(item, ["Email provider"])
    segments[platform or "Custom / unknown"].append(lead["company"])
    rows.append({
        **lead,
        "status": "ok",
        "platform": platform,
        "platform_evidence": evidence_for(item, platform) if platform else "",
        "email_provider": email,
        "hosting": first_in(item, ["Hosting", "CDN"]),
        "analytics": ", ".join(item.get("byCategory", {}).get("Analytics", [])
                               + item.get("byCategory", {}).get("Tag manager", [])),
        "tech_count": item["technologyCount"],
    })

Failed sites stay in the file with their error message instead of silently disappearing. That way nobody wonders why a company is missing, and you can fix typos in the domain column and run just those rows again.

Writing the enriched CSV

with open("leads_enriched.csv", "w", newline="") as f:
    fields = ["company", "website", "status", "platform", "platform_evidence",
              "email_provider", "hosting", "analytics", "tech_count"]
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(rows)

for platform, companies in sorted(segments.items(), key=lambda kv: -len(kv[1])):
    print(f"{platform:<18} {len(companies)}  {', '.join(companies)}")

Running the whole script printed this:

Custom / unknown   3  The New Yorker, Basecamp, Mozilla
Shopify            2  Allbirds, Gymshark
WordPress          1  TechCrunch
Contentful         1  Notion
HubSpot CMS        1  HubSpot
Hugo               1  Ghost

And leads_enriched.csv came out like this:

company	platform	platform_evidence	email_provider	hosting
Allbirds	Shopify	header powered-by: Shopify	Microsoft 365	Cloudflare
Gymshark	Shopify	header powered-by: Shopify		Cloudflare
TechCrunch	WordPress	meta generator: WordPress 6.9.9
The New Yorker			Google Workspace	Amazon CloudFront
Basecamp				Cloudflare
Notion	Contentful	asset https://images.ctfassets.net		Vercel
HubSpot	HubSpot CMS	header x-hs-hub-id: 53	Google Workspace	Cloudflare
Mozilla			Google Workspace	Google Cloud
Ghost	Hugo	meta generator: Hugo 0.119.0	Google Workspace	Netlify
Example (dead)

The run reported nine websites scanned and one failed, and Apify's billing showed nine charged events: nine websites at $0.002 is $0.018. The dead domain cost nothing.

Reading the results honestly

A few things in that table are worth understanding before you hand it to a sales team.

"Custom / unknown" usually means custom. Basecamp and The New Yorker run their own application stacks, so no off-the-shelf platform shows up. That is a real answer, and for a Shopify app vendor it is the right one: not a fit.

Blank email provider does not mean "no email". TechCrunch and Gymshark route mail through security gateways (Mimecast and Proofpoint showed up under "Email security"), which hide the mailbox provider behind them. If email matters to your pitch, add the Email security category to that column.

Redirects are followed. Notion's notion.so is now notion.com, and the result describes the site people actually see. Check finalUrl if a result looks surprising.

One page, no browser. The detector reads the HTML, headers and DNS of the page you give it and does not execute JavaScript. Tools that only appear after client-side scripts run can be missed, and the catalogue (around 200 technologies) is smaller than Wappalyzer's or BuiltWith's. For "which CMS, which shop platform, which email provider, which host" on a list of companies, that has been plenty in our tests. For a full inventory of every marketing pixel on a single site, use a browser-based tool.

Making it repeatable

Once the script works, a few small changes make it useful week after week:

Run only new leads. Keep the enriched CSV, and before calling the Actor, drop rows whose website already has a status of ok.
Cap the spend. Pass maxTotalChargeUsd in the run options, for example client.actor(...).call(run_input=..., max_total_charge_usd=1), and the run stops at that budget. At $0.002 per website, a dollar covers 500 sites.
Schedule it. Apify can run the Actor on a schedule with a saved input, and you can pull the latest dataset from a cron job or an n8n, Make or Zapier flow.
Use it from an agent. The same Actor is available through Apify's MCP server, so an AI agent can look up a prospect's stack mid-conversation: https://mcp.apify.com?tools=siftwright/tech-stack-detector.

The full script

Here is everything in one file, exactly as we ran it. Save it as segment_leads.py next to your leads.csv and run python segment_leads.py.

import csv
import os
from collections import defaultdict

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

with open("leads.csv", newline="") as f:
    leads = list(csv.DictReader(f))

run = client.actor("siftwright/tech-stack-detector").call(
    run_input={"urls": [row["website"] for row in leads], "includeDns": True}
)
# apify-client 3.x returns a Run object; on 2.x use run["defaultDatasetId"]
results = {item["input"]: item for item in client.dataset(run.default_dataset_id).iterate_items()}

PLATFORM_CATEGORIES = ["Ecommerce", "CMS", "Headless CMS", "Static site generator"]

def first_in(item, categories):
    for category in categories:
        names = item.get("byCategory", {}).get(category)
        if names:
            return names[0]
    return ""

def evidence_for(item, name):
    for tech in item.get("technologies", []):
        if tech["name"] == name:
            return tech["evidence"][0]
    return ""

segments = defaultdict(list)
rows = []
for lead in leads:
    item = results.get(lead["website"], {})
    if item.get("status") != "ok":
        rows.append({**lead, "status": item.get("error", "not scanned")})
        continue
    platform = first_in(item, PLATFORM_CATEGORIES)
    email = first_in(item, ["Email provider"])
    segments[platform or "Custom / unknown"].append(lead["company"])
    rows.append({
        **lead,
        "status": "ok",
        "platform": platform,
        "platform_evidence": evidence_for(item, platform) if platform else "",
        "email_provider": email,
        "hosting": first_in(item, ["Hosting", "CDN"]),
        "analytics": ", ".join(item.get("byCategory", {}).get("Analytics", []) + item.get("byCategory", {}).get("Tag manager", [])),
        "tech_count": item["technologyCount"],
    })

with open("leads_enriched.csv", "w", newline="") as f:
    fields = ["company", "website", "status", "platform", "platform_evidence", "email_provider", "hosting", "analytics", "tech_count"]
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(rows)

for platform, companies in sorted(segments.items(), key=lambda kv: -len(kv[1])):
    print(f"{platform:<18} {len(companies)}  {', '.join(companies)}")

If your list is large or you need the data refreshed on a schedule without running anything yourself, the Tech Stack Detector page has the pricing and limits, and our custom data feeds cover scheduled deliveries.

Find the bugs your users are complaining about: mining App Store reviews with Python

Siftwright — Wed, 30 Sep 2026 23:40:52 +0000

Your one-star reviews are the cheapest bug reports you will ever get. People take the time to write down what broke, on which device, after which update, and they do it in public. The trouble is reading them: the App Store shows one country at a time, mixes praise with complaints, and has no export button.

This guide pulls the low-star reviews for an app into Python, then answers three questions a product or QA team actually asks:

Where are the complaints coming from?
Which app versions collect the most of them?
What are people complaining about?

Everything below was run on 30 September 2026 against a real app (Notion's iOS app, picked because it is popular and has plenty of reviews). The output you see is the real output, not a mock-up.

What you need

Python 3.9 or newer.
An Apify account and API token. The free plan includes monthly credits, which is far more than this tutorial uses.
The official client: pip install apify-client. The code uses version 3.x, where runs come back as objects (run.default_dataset_id). On the older 2.x client, use run["defaultDatasetId"] instead.

We use the App Store Reviews Scraper, which we built and publish on the Apify Store. It reads Apple's public customer-review feed, so you get the same reviews anyone can read on the App Store, as JSON. You can search by app name, pass several countries in one run, and filter by stars before anything is charged. The price is $0.0001 per review returned ($0.10 per 1,000), so the 300 reviews in this tutorial cost three cents.

Set your token once:

export APIFY_TOKEN="apify_api_..."
pip install apify-client

Step 1: fetch only the reviews you care about

The key idea is to filter at the source. You don't want 5-star "love it!" reviews for a bug hunt, so ask for 1 and 2 stars only with maxRating: 2. Filtered-out reviews are never returned or billed.

run_input = {
    "apps": ["Notion"],            # a name, an App Store link or a numeric id
    "countries": ["us", "gb", "ca", "au"],
    "maxReviewsPerApp": 300,       # across all countries
    "maxRating": 2,                # only 1- and 2-star reviews are returned (and billed)
}
run = client.actor("siftwright/app-store-reviews-scraper").call(run_input=run_input)
reviews = list(client.dataset(run.default_dataset_id).iterate_items())

A few details worth knowing:

App names work, but links are exact. The name is resolved with Apple's own App Store search and the top match is used. "Notion" matched Notion: Notes, Tasks, AI by Notion Labs. If a name is ambiguous, paste the App Store link or the numeric id instead.
Every country is a separate review pool. A review written in the UK store does not show up in the US store. Listing several countries is how you get a fuller picture; the scraper de-duplicates across them.
maxReviewsPerApp is a total, not per country. Countries are read in order until the cap is reached. In our run the cap of 300 was hit after the US and UK, so Canada and Australia were never fetched and never charged. Raise the cap if you want every listed country.
Apple's feed has a ceiling. It returns up to 500 reviews per app, per country, per sort order (most recent or most helpful). For a big app that is the most recent few weeks or months, not its whole history.

Each item looks like this (a real row, shortened):

{
  "appName": "Notion: Notes, Tasks, AI",
  "country": "us",
  "rating": 1,
  "title": "Keep flickering",
  "text": "It just keep pulling everything up. Unusable on Mac.",
  "appVersionReviewed": "1.7.340",
  "date": "2026-09-23T...Z"
}

The appVersionReviewed field is the one that makes this useful for engineering teams: it tells you which build the reviewer was on.

Step 2: check what you paid for

Pay-per-result pricing is only fair if you can check it. Charges are recorded on the run itself, and the counters settle a moment after the run finishes, so re-read the run before printing them:

run = client.run(run.id).get()
print(f"{len(reviews)} low-star reviews, charged: {run.charged_event_counts}")

Our run printed 300 low-star reviews, charged: {'apify-actor-start': 1, 'review': 300}: one charge per review in the dataset, plus Apify's standard start fee of $0.00005.

Step 3: where are complaints coming from?

print(Counter(r["country"] for r in reviews).most_common())

Our output was [('us', 207), ('gb', 93)]. With a higher cap you would see all four countries. For apps with localised content, pricing or payment options, a single country standing out is often the first clue that a problem is regional.

Step 4: which versions collect the most low-star reviews?

App versions are strings like 1.7.341, so sort them numerically rather than alphabetically (otherwise 1.7.99 sorts after 1.7.341):

by_version = Counter(r["appVersionReviewed"] for r in reviews)
for version, n in sorted(by_version.items(), key=lambda kv: [int(x) for x in re.findall(r"\d+", kv[0])], reverse=True)[:6]:
    print(f"{version:>10}  {n:3d}  {'#' * n}")

For a fast-shipping app like Notion, which releases every few days, the counts per version are small (2 to 5 for the most recent builds in our run). The signal gets much stronger when you run this on a schedule and compare one release with the previous one: a version that suddenly collects three times the usual low-star reviews is worth a look before the next release goes out.

Step 5: tag reviews with explainable themes

You could send every review to an LLM, and for deep analysis you probably should. But a first pass with plain keyword themes is instant, free, and, most importantly, explainable: anyone on the team can see why a review was tagged.

THEMES = {
    "crash / bug": r"crash\w*|bugs?|buggy|glitch\w*|broken|flicker\w*|not working|doesn.t work",
    "slow / lag": r"slow|lag\w*|loading|takes forever|freez\w*",
    "sync / offline": r"sync\w*|offline|no internet|connection",
    "login / account": r"log ?in|sign ?in|account|password|locked out",
    "ai features": r"ai|artificial intelligence",
    "pricing": r"price|pricing|subscription|paywall|expensive",
    "mobile / ipad": r"mobile|iphone|ipad|phone app",
}
PATTERNS = {theme: re.compile(rf"\b(?:{p})\b") for theme, p in THEMES.items()}

Two lessons from getting this right on real data:

Wrap every alternative in word boundaries. Our first draft had \b only in front of the first word, so lag also matched "flag" and ai could match inside other words. \b(?:...)\b around the whole group fixes that.
Then allow the endings people actually type. With strict boundaries, bug no longer matched "bugs" or "buggy", and the crash theme dropped from 110 reviews to 45. bugs?|buggy|crash\w* brought it back. Always spot-check a few matches per theme before trusting the numbers.

Counting is then a simple loop, keeping one example title per theme so the output is self-explaining (see the full script below). Our run printed:

crash / bug      110 (37%)  e.g. [us 1.7.341] what's happening?
ai features       90 (30%)  e.g. [us 1.7.340] AI is terrible
mobile / ipad     89 (30%)  e.g. [us 1.7.341] Can’t make checklists in Mobil app
slow / lag        40 (13%)  e.g. [us 1.7.339] Too slow and takes up too much data storage
login / account   15 (5%)  e.g. [us 1.7.328] Great before the most recent update (2026)
pricing            8 (3%)  e.g. [us 1.7.333] Why so bloated?
sync / offline     6 (2%)  e.g. [us 1.7.327] Ready to switch to an offline note app

A review can carry several themes, so the percentages add up to more than 100%. Read the result as "what share of unhappy reviewers mention this", not as a pie chart. For this app, stability and the mobile experience dominate, and AI features come up in almost a third of low-star reviews, which is exactly the kind of thing a product team wants to know about a recent feature push.

Step 6: save it for a spreadsheet or an LLM

with open("low_star_reviews.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["date", "country", "rating", "appVersionReviewed", "title", "text"], extrasaction="ignore")
    w.writeheader()
    w.writerows(reviews)

The CSV opens in any spreadsheet. It is also a good input for an LLM prompt such as "group these reviews into the five most common problems, quote one review for each, and name the app version where each problem first appears". Keeping the version and country columns in the file lets the model answer that second part.

We left author names out of the CSV on purpose. You rarely need them for product analysis, and not storing personal data you don't need keeps you on the right side of GDPR and similar laws.

The full script

This is the exact script that produced the output above:

import csv
import os
import re
from collections import Counter, defaultdict

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "apps": ["Notion"],            # a name, an App Store link or a numeric id
    "countries": ["us", "gb", "ca", "au"],
    "maxReviewsPerApp": 300,       # across all countries
    "maxRating": 2,                # only 1- and 2-star reviews are returned (and billed)
}
run = client.actor("siftwright/app-store-reviews-scraper").call(run_input=run_input)
reviews = list(client.dataset(run.default_dataset_id).iterate_items())
run = client.run(run.id).get()  # re-read the run: charge counters settle a moment after it finishes
print(f"{len(reviews)} low-star reviews, charged: {run.charged_event_counts}")

# 1. Where do the complaints come from?
print(Counter(r["country"] for r in reviews).most_common())

# 2. Which versions collect the most low-star reviews?
by_version = Counter(r["appVersionReviewed"] for r in reviews)
for version, n in sorted(by_version.items(), key=lambda kv: [int(x) for x in re.findall(r"\d+", kv[0])], reverse=True)[:6]:
    print(f"{version:>10}  {n:3d}  {'#' * n}")

# 3. Tag each review with simple, explainable themes.
THEMES = {
    "crash / bug": r"crash\w*|bugs?|buggy|glitch\w*|broken|flicker\w*|not working|doesn.t work",
    "slow / lag": r"slow|lag\w*|loading|takes forever|freez\w*",
    "sync / offline": r"sync\w*|offline|no internet|connection",
    "login / account": r"log ?in|sign ?in|account|password|locked out",
    "ai features": r"ai|artificial intelligence",
    "pricing": r"price|pricing|subscription|paywall|expensive",
    "mobile / ipad": r"mobile|iphone|ipad|phone app",
}
PATTERNS = {theme: re.compile(rf"\b(?:{p})\b") for theme, p in THEMES.items()}
theme_counts = Counter()
examples = defaultdict(list)
for r in reviews:
    text = f"{r['title']} {r['text']}".lower()
    for theme, pattern in PATTERNS.items():
        if pattern.search(text):
            theme_counts[theme] += 1
            if len(examples[theme]) < 2:
                examples[theme].append(f"[{r['country']} {r['appVersionReviewed']}] {r['title']}")

for theme, n in theme_counts.most_common():
    print(f"{theme:<16} {n:3d} ({n / len(reviews):.0%})  e.g. {examples[theme][0]}")

# 4. Save everything for a spreadsheet or an LLM pass later.
with open("low_star_reviews.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["date", "country", "rating", "appVersionReviewed", "title", "text"], extrasaction="ignore")
    w.writeheader()
    w.writerows(reviews)
print("saved low_star_reviews.csv")

And its complete output:

300 low-star reviews, charged: {'apify-actor-start': 1, 'review': 300}
[('us', 207), ('gb', 93)]
   1.7.341    2  ##
   1.7.340    5  #####
   1.7.339    5  #####
   1.7.338    2  ##
   1.7.337    2  ##
   1.7.336    3  ###
crash / bug      110 (37%)  e.g. [us 1.7.341] what's happening?
ai features       90 (30%)  e.g. [us 1.7.340] AI is terrible
mobile / ipad     89 (30%)  e.g. [us 1.7.341] Can’t make checklists in Mobil app
slow / lag        40 (13%)  e.g. [us 1.7.339] Too slow and takes up too much data storage
login / account   15 (5%)  e.g. [us 1.7.328] Great before the most recent update (2026)
pricing            8 (3%)  e.g. [us 1.7.333] Why so bloated?
sync / offline     6 (2%)  e.g. [us 1.7.327] Ready to switch to an offline note app
saved low_star_reviews.csv

Making it a weekly habit

A one-off report is interesting; a weekly one is useful. A few ways to run it on a schedule:

Apify schedules: save the input as a task and schedule it; the dataset of every run stays available through the API.
Filter by date: pass since (for example the date of your last run) so each run only returns, and only charges for, new reviews.
Watch competitors too: add competing apps to apps. What users hate about a rival is a feature list for your roadmap.
Alert on spikes: compare this week's crash-theme count with last week's and post to Slack when it doubles.

Limits to keep in mind

Apple's feed gives the most recent (or most helpful) 500 reviews per app, country and sort order. It is not a full archive.
Developer replies are not in the feed.
Star ratings without text (ratings-only) are not reviews and are not included.
Keyword themes are a first pass. They miss sarcasm, other languages and unusual wording. Use them to decide where to look, then read the reviews.

Wrapping up

In about 60 lines you went from "we should read our reviews" to a per-version, per-theme picture of what unhappy users are saying, for three cents. The same script works for any app on the App Store and, with a list of competitors, for your whole market.

If you want to try it, the App Store Reviews Scraper runs on Apify's free plan credits. Questions or ideas are welcome at support@siftwright.com.

Originally published on the Siftwright blog.

Scraping Google News search results with the public RSS feed (no API key, no browser)

Siftwright — Mon, 28 Sep 2026 11:59:11 +0000

Google News doesn't have a public search API. If you want structured articles for a keyword — title, source, link, publish date — the usual options are a paid SERP API or scraping the HTML search results page (fragile, breaks often, and against Google's ToS for the main search UI).

There's a third option that's easy to miss: Google News publishes an unauthenticated RSS feed for any search query, at news.google.com/rss/search?q=.... It's meant for feed readers, but it's public, stable, and returns clean structured data. This tutorial builds a small scraper around it.

The endpoint

GET https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en

No API key, no auth, no proxy needed — it's a plain public RSS/XML endpoint. Each <item> in the feed has a title, link, source, and publish date, which is really all "Google News search results" means in structured form.

Parsing it

import { parseStringPromise } from 'xml2js';

async function searchGoogleNews(query) {
    const url = `https://news.google.com/rss/search?q=${encodeURIComponent(query)}&hl=en-US&gl=US&ceid=US:en`;
    const res = await fetch(url);
    const xml = await res.text();
    const parsed = await parseStringPromise(xml, { explicitArray: false });

    const items = parsed?.rss?.channel?.item;
    if (!items) return [];

    const list = Array.isArray(items) ? items : [items];
    return list.map((item) => ({
        title: item.title,
        link: item.link,
        source: item.source?._ ?? item.source,
        publishedAt: item.pubDate,
        snippet: item.title, // Google's feed doesn't include a separate summary
    }));
}

That's genuinely the whole scraper. No headless browser, no proxy rotation, no CAPTCHA handling — it's a public feed, so it behaves like one.

Real output

Running it for "artificial intelligence regulation" today returns real, current articles like:

{
    "title": "Bill Gates says unchecked AI could 'cause a billion deaths' in call for regulation",
    "link": "https://news.google.com/rss/articles/CBMi...",
    "source": "The Guardian",
    "publishedAt": "2026-09-27T09:00:00.000Z",
    "snippet": "Bill Gates says unchecked AI could 'cause a billion deaths' in call for regulation The Guardian"
}

{
    "title": "Trump rejects AI regulation, citing parallels with climate change, in U.N. address",
    "link": "https://news.google.com/rss/articles/CBMi...",
    "source": "Scientific American",
    "publishedAt": "2026-09-22T16:00:00.000Z",
    "snippet": "Trump rejects AI regulation, citing parallels with climate change, in U.N. address Scientific American"
}

(Links are truncated here for length — the real output includes the full Google News redirect URL, which resolves to the original article.)

What to watch for

No summary field. Google's RSS feed repeats the title as the description, so don't expect a real snippet/abstract — if you need one, you'd have to fetch the linked article separately.
Links are Google redirect URLs, not the publisher's direct URL. They resolve fine in a browser or with a follow-redirects fetch, but store them as-is if you just need a working link.
Rate limits are unpublished. It's a public feed, not a documented API — keep request volume reasonable and add backoff if you see errors.
Empty results are normal, not an error. An obscure or misspelled query can legitimately return zero items — don't treat that as a failure.

Try it hosted

I turned this exact pattern into a small pay-per-event Apify Actor — Google News Scraper — so you don't have to run and maintain the scraper yourself. It charges $0.0015 per article returned and nothing for empty or failed searches, and it's callable directly by AI agents through Apify's MCP server if you're wiring this into an agent pipeline instead of a script.

Source and more real examples: github.com/siftwright/siftwright-tools