DEV Community: Siftwright

Visual regression testing with a screenshot API and pixel diffs in Python

Siftwright — Tue, 10 Nov 2026 15:51:20 +0000

Functional tests tell you the checkout button still works. They do not tell you it turned white-on-white after a CSS refactor, or that a footer slid under a banner on the pricing page. Visual regression testing catches that class of bug: take a screenshot of a page, compare it with an approved one, and fail when too much has changed.

You do not need a heavy framework for this. Many teams already have a staging or production URL they want to keep an eye on, and the two hard parts, rendering the page and comparing two images, are small. This post builds a complete check in about 70 lines of Python: capture with a screenshot API, diff with Pillow, ignore areas that change on purpose, and exit non-zero on a regression so any scheduler or deploy script can act on it.

We ran the code on 10 November 2026 against a real page, and we found a flaw in our first threshold. That is covered below because it is the most useful thing in the post.

A note on what was tested: the captures came from the public Pageframe Screenshots Actor, the engine behind our screenshot API. The /v1/screenshot call in the script uses the same options, but we did not re-call that endpoint with a key for this post. The difference is only how the image is downloaded (the API's fileUrl is a signed link, the Actor's store needs your token).

The approach

Visual regression has four steps:

Capture the page at a fixed viewport so runs are comparable.
Compare the new capture with a stored baseline, pixel by pixel.
Decide whether the difference is noise or a regression.
Show where it changed, so a human can approve or fix it quickly.

Steps 1 and 4 matter more than people expect. A capture that is not repeatable (different viewport, a cookie banner that appears on some runs) produces false failures, and a bare "FAIL" with no picture trains people to ignore the check.

Capture settings that keep runs repeatable

Use the same options on every run. The ones that matter:

curl https://siftwright.com/v1/screenshot \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing/"],
    "format": "png",
    "fullPage": true,
    "viewportWidth": 1280,
    "viewportHeight": 800,
    "blockCookieBanners": true
  }'

PNG, not JPEG. JPEG compression adds noise that a pixel diff reads as change. Use lossless output for comparisons.
A fixed viewport. Pick one width and keep it. If you care about mobile, run a second named check at 390 wide.
Full page. Layout bugs often live below the fold, in the footer or a long table.
Cookie banner handling. Dismissal is best effort. If a banner still appears, it shows up as a diff, which is the correct behaviour: you want to know.

A failed capture returns status: "error" and does not count against your plan, so a flaky target does not cost you money. The script treats that as an error, not a pass.

The script

Install Pillow (pip install pillow) and set SIFTWRIGHT_API_KEY:

import io, json, os, re, sys, urllib.request
from pathlib import Path
from PIL import Image, ImageChops

API = "https://siftwright.com/v1/screenshot"
BASE = Path("baselines")
OUT = Path("vrt-out")
TOLERANCE = 16      # per-channel difference (0-255) ignored as antialiasing noise
MAX_CHANGED = 0.002 # fail if more than 0.2% of pixels changed
TILE = 80           # region report granularity in px

def capture(url, **opts):
    body = {"urls": [url], "format": "png", "fullPage": True,
            "viewportWidth": 1280, "viewportHeight": 800, **opts}
    req = urllib.request.Request(API, json.dumps(body).encode(),
        {"Authorization": f"Bearer {os.environ['SIFTWRIGHT_API_KEY']}",
         "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        res = json.load(r)["results"][0]
    if res["status"] != "ok":
        raise RuntimeError(res.get("error", res["status"]))
    with urllib.request.urlopen(res["fileUrl"], timeout=60) as r:   # signed link, no key needed
        return Image.open(io.BytesIO(r.read())).convert("RGB")

def slug(url):
    return re.sub(r"[^a-z0-9]+", "-", url.lower().split("://", 1)[-1]).strip("-")

def compare(base, cur, ignore=()):
    """Return (changed_ratio, bbox, hot_tiles, diff_image, size_note)."""
    note = None
    if base.size != cur.size:
        note = f"size changed {base.size} -> {cur.size}"
        w, h = min(base.width, cur.width), min(base.height, cur.height)
        base, cur = base.crop((0, 0, w, h)), cur.crop((0, 0, w, h))
    d = ImageChops.difference(base, cur)
    r, g, b = d.split()
    mask = ImageChops.lighter(ImageChops.lighter(r, g), b).point(
        lambda v: 255 if v > TOLERANCE else 0)
    for x0, y0, x1, y1 in ignore:                      # blank out known-dynamic areas
        mask.paste(0, (x0, y0, x1, y1))
    changed = mask.histogram()[255]
    ratio = changed / (mask.width * mask.height)
    tiles = []
    if changed:
        for ty in range(0, mask.height, TILE):
            for tx in range(0, mask.width, TILE):
                t = mask.crop((tx, ty, tx + TILE, ty + TILE)).histogram()[255]
                if t > TILE * TILE * 0.02:             # tile is >2% changed
                    tiles.append((tx, ty))
    overlay = Image.new("RGB", cur.size, (255, 0, 0))
    dim = Image.blend(cur, Image.new("RGB", cur.size, (255, 255, 255)), 0.6)
    return ratio, mask.getbbox(), tiles, Image.composite(overlay, dim, mask), note

def check(url, update=False, ignore=(), **opts):
    BASE.mkdir(exist_ok=True); OUT.mkdir(exist_ok=True)
    name, path = slug(url), BASE / f"{slug(url)}.png"
    cur = capture(url, **opts)
    if update or not path.exists():
        cur.save(path)
        return {"url": url, "status": "baseline-saved"}
    ratio, bbox, tiles, diff, note = compare(Image.open(path).convert("RGB"), cur, ignore)
    ok = ratio <= MAX_CHANGED and not tiles and note is None
    if not ok:
        cur.save(OUT / f"{name}-current.png"); diff.save(OUT / f"{name}-diff.png")
    return {"url": url, "status": "pass" if ok else "FAIL", "changed": round(ratio, 5),
            "bbox": bbox, "hot_tiles": len(tiles), "note": note}

if __name__ == "__main__":
    update = "--update" in sys.argv
    urls = [a for a in sys.argv[1:] if not a.startswith("--")]
    results = [check(u, update=update) for u in urls]
    for r in results:
        print(json.dumps(r))
    sys.exit(1 if any(r["status"] == "FAIL" for r in results) else 0)

Run it once to save baselines, then again whenever you want to check:

python vrt.py https://example.com/pricing/          # first run: saves the baseline
python vrt.py https://example.com/pricing/          # later runs: compare
python vrt.py --update https://example.com/pricing/ # accept the new look as the baseline

On a failure you get vrt-out/<name>-diff.png: the current page washed out, with changed pixels in solid red. Open it and you see at once whether the change is a moved button or a rewritten paragraph.

Design decisions, and why

A per-channel tolerance of 16. Antialiasing and font hinting produce tiny colour differences that are not bugs. Ignoring differences of 16 or less on each channel removes that noise and still catches any real colour change, which is far larger.

A tile check on top of the ratio. This is the mistake we made. Our first version failed only when more than 0.2% of all pixels changed. We tested it on the full-page capture of siftwright.com/pricing/, which is 1280 by 5785 pixels, then painted a button-sized red rectangle (200 by 48) onto a copy to simulate a regression. The result:

Check	Value
Changed pixels	0.133% of the page
Threshold	0.2%
First version verdict	pass, wrong

On a long page, a clearly visible broken button is a small share of the pixels. A global percentage dilutes real bugs the longer the page gets. The fix is the tile report: split the diff into 80-pixel tiles and fail if any tile is more than 2% changed. With that rule the same test produced six hot tiles and a FAIL, with a bounding box of (100, 300, 301, 349) pointing at the rectangle. Keep the global ratio too, because it catches a sitewide change (a new font, say) that no single tile flags.

Size changes fail on their own. If the page height changes, everything below the change shifts and the comparison turns into red noise. The script reports size changed (1280, 5785) -> (1280, 5825) and fails without pretending to diff the shifted content. That is usually what you want: someone added or removed content, so look.

Ignore regions for things that change on purpose. Dates, ad slots, "last updated" labels, live counters. Pass rectangles as (x0, y0, x1, y1) and the script blanks them out of the mask. Coordinates are exclusive on the right and bottom, so add one pixel if you measured from a drawing tool. In our test, ignoring the exact rectangle brought the changed ratio back to 0.0 and the check passed.

What we measured

We ran the code against https://siftwright.com/pricing/, captured by the Actor with the settings above:

Test	Result
Capture the page, save a baseline	1280x5785 PNG saved
Capture again and compare	0.0 changed, pass
Add a 200x48 red box	0.133% changed, 6 hot tiles, FAIL, bbox on the box
Same box, listed in `ignore`	0.0 changed, pass
Page 40 px taller	FAIL, size change reported
One black pixel changed	pass, below the noise floor on purpose

Two captures of the same page seconds apart were pixel-identical, and the same held for two captures of the Hacker News front page. That is a favourable case: the pages are static. A page with a carousel, a video or a rotating testimonial will not be, and you should expect to add ignore regions or wait for a selector before capturing. Behaviour on your own pages depends on what they render.

The synthetic regression was painted onto a real capture rather than produced by changing the live page, because we cannot edit a page we do not control. The diff logic is what we tested there; the capture was real and repeatable.

Practical advice

Store baselines in version control or object storage, keyed by URL and viewport. Review the diff image when you update them, like any other code review.
Compare like with like. Staging versus a production baseline will differ in data. Capture a baseline from the same environment you test.
Wait for the page to settle. If content loads late, pass a waitForSelector option (the Actor supports it) so you do not capture a loading state.
Keep the page list short. Ten to twenty pages that matter (home, pricing, checkout, a long article, a form) give most of the benefit. Every extra page is a capture to pay for and a diff to review.
Do not run it on every commit by default. Scheduled runs against staging, or one run per release, are usually enough and cost far less.

Limits

Pixel diffing is blunt. It cannot tell a deliberate redesign from a bug, so someone has to approve baselines. It is sensitive to fonts and rendering engine differences, which is why you capture with the same service every time instead of comparing a local browser capture with a cloud one. And it does not test behaviour: a page that looks right can still be broken.

For pure design-system work on components in isolation, a dedicated tool such as Storybook-based visual testing may suit you better. For watching real pages at real URLs, with nothing to install beyond Pillow, the approach above is enough to catch the regressions that embarrass you.

Try it

The screenshot API takes a URL and returns a PNG, JPEG or PDF, with full-page capture, a fixed viewport and cookie-banner handling. Plans start at $29 per month for 10,000 successful results, and failed captures are not counted. You can try a capture for free with the website screenshot tool, then read the screenshot API page for options and pricing. If you want to see how to batch captures and cache them, our post on website thumbnails and link previews covers it, and the screenshot API tutorial has the basics in Python and Node.

Website thumbnails and link previews from code: capture, crop, cache

Siftwright — Fri, 06 Nov 2026 15:48:45 +0000

Directories, bookmark managers, link-in-bio pages, "sites using X" galleries and internal dashboards all need the same small thing: a picture of a web page, at a size that fits a card. Writing that yourself looks like a one-line Playwright call until you have a few thousand URLs, a few of which are dead, slow or behind a cookie banner.

This post builds a small thumbnail service in Python: one API call per five URLs, three cropped sizes per page (including a 1200x630 image for link previews), a cache so you do not re-capture pages needlessly, and sane handling for URLs that fail. We ran the code on 6 November 2026 against real sites. One caveat up front, repeated at the end: the tests called the public Pageframe Screenshots Actor, which is the engine behind our screenshot API, not the /v1/screenshot endpoint itself. The only difference in the script is where the image is downloaded from, and we mark it below.

Thumbnail versus link preview: pick the right shot

Two different jobs get called "thumbnail":

A card thumbnail is a small picture inside your own UI, usually 16:9, 320 to 640 pixels wide.
A link preview is the og:image other sites show when someone pastes your URL into Slack, X or LinkedIn. The common recommendation is 1200x630 (roughly 1.91:1) at a modest file size.

Both are best taken above the fold: a viewport-sized capture, not a full-page one. A full-page capture of a long landing page is a 1280x9000 image that becomes unreadable lint when scaled to 320 pixels wide. So the plan is to capture one 1280x720 viewport, then derive every size from that single image. One capture, three crops, one bill.

The capture call

The screenshot endpoint takes up to five URLs per request and returns one result per URL, each with a status. The options that matter here:

curl https://siftwright.com/v1/screenshot \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com", "https://news.ycombinator.com"],
    "format": "jpeg",
    "fullPage": false,
    "viewportWidth": 1280,
    "viewportHeight": 720,
    "blockCookieBanners": true
  }'

fullPage: false is the important one. format: "jpeg" keeps the source image small; for thumbnails you will recompress anyway. blockCookieBanners (on by default) makes a best-effort attempt to dismiss consent dialogs, so your card shows the page and not a "We value your privacy" overlay. It is best effort: unusual banners can survive.

For a successful URL the result has status: "ok" and a signed fileUrl. The link downloads without your key and stays valid for 7 days, so download and store the image rather than hot-linking it. For a failure you get status: "error" and a message, and the capture is not counted against your plan.

The script

Install Pillow (pip install pillow) and set SIFTWRIGHT_API_KEY. The whole thing:

import hashlib, io, json, os, time, urllib.request
from pathlib import Path
from PIL import Image

API = "https://siftwright.com/v1/screenshot"
SIZES = {"sm": (320, 180), "md": (640, 360), "og": (1200, 630)}
CACHE = Path("thumb-cache")
TTL = 24 * 3600            # re-capture after a day
RETRY_AFTER = 3600         # don't hammer a URL that just failed

def capture(urls, headers=None, **opts):
    """One API call for up to 5 URLs. Returns the results list."""
    body = {"urls": urls, "format": "jpeg", "fullPage": False,
            "viewportWidth": 1280, "viewportHeight": 720, **opts}
    req = urllib.request.Request(API, json.dumps(body).encode(),
        {"Authorization": f"Bearer {os.environ['SIFTWRIGHT_API_KEY']}",
         "Content-Type": "application/json", **(headers or {})})
    with urllib.request.urlopen(req, timeout=300) as r:
        return json.load(r)["results"]

def download(url, headers=None):
    req = urllib.request.Request(url, headers=headers or {})
    with urllib.request.urlopen(req, timeout=60) as r:
        return r.read()

def cover(img, size):
    """Scale to fill the box, then crop from the top: page headers carry the brand."""
    w, h = size
    s = max(w / img.width, h / img.height)
    img = img.resize((round(img.width * s), round(img.height * s)), Image.LANCZOS)
    return img.crop((0, 0, w, h))

def key(url):
    return hashlib.sha256(url.encode()).hexdigest()[:16]

def fresh(url):
    meta = CACHE / f"{key(url)}.json"
    if not meta.exists():
        return False
    m = json.loads(meta.read_text())
    age = time.time() - m["at"]
    return age < (TTL if m["ok"] else RETRY_AFTER)

def thumbnails(urls, capture_fn=capture, download_fn=lambda res: download(res["fileUrl"])):
    CACHE.mkdir(exist_ok=True)
    todo = [u for u in dict.fromkeys(urls) if not fresh(u)]   # dedupe, skip cache hits
    for i in range(0, len(todo), 5):                          # the API takes 5 URLs per call
        for res in capture_fn(todo[i:i + 5]):
            k = key(res["url"])
            meta = {"url": res["url"], "at": time.time(), "ok": res["status"] == "ok"}
            if meta["ok"]:
                img = Image.open(io.BytesIO(download_fn(res))).convert("RGB")
                for name, size in SIZES.items():
                    cover(img, size).save(CACHE / f"{k}-{name}.jpg", quality=80, optimize=True)
            else:
                meta["error"] = res.get("error", res["status"])[:120]
            (CACHE / f"{k}.json").write_text(json.dumps(meta))
    return {u: (CACHE / f"{key(u)}-md.jpg") if (CACHE / f"{key(u)}-md.jpg").exists() else None
            for u in urls}

if __name__ == "__main__":
    import sys
    for url, path in thumbnails(sys.argv[1:]).items():
        print(url, "->", path or "failed")

A few design choices worth explaining:

Crop from the top, not the centre. A site's logo and headline sit at the top. Centre-cropping a 16:9 capture into a different aspect ratio chops the brand off.
Deduplicate before calling. dict.fromkeys removes repeats while preserving order. If your input is scraped from many pages, repeats are common, and every duplicate would be a paid capture.
Cache successes and failures differently. A good thumbnail is reused for a day. A failure is remembered for an hour, so a dead domain in a 10,000-row list does not trigger a retry on every run.
The key is a hash of the URL. Normalise URLs first (lowercase host, strip tracking parameters) if you want example.com/?utm_source=x and example.com/ to share a thumbnail.

What we measured

We ran the script on eight inputs: seven distinct sites, one repeated, and one deliberately dead domain (no-such-domain-siftwright-test.invalid). Here is what happened on the first run:

Measure	Result
API calls for 8 inputs	2 (one batch of 5, one of 2: 7 unique URLs after dedupe, dead domain included)
Thumbnails written	6 sites, 3 sizes each
Dead domain	`status: "error"`, `ERR_NAME_NOT_RESOLVED`, no files, no crash
Wall time, first run	9.8 s end to end, including downloads and resizing
Wall time, second run	0.0 s, no API calls (everything fresh in the cache)

File sizes for the Hacker News front page, at JPEG quality 80: 8 KB at 320x180, 29 KB at 640x360 and 92 KB at 1200x630. The other sites were between 8 KB and 32 KB at the middle size; sparse pages compress much better than dense ones. We also confirmed a plain HTTP 404 (httpbin.org/status/404) is reported as an error ("Page responded with HTTP 404") and not captured, which is what you want for a gallery: a 404 page makes a bad thumbnail.

In the same test, github.com came back as an 8 KB image at the middle size. That was a mostly plain page at the capture time, which is a reminder that thumbnails show whatever the page looks like at that moment, including interstitials, empty states and A/B variants. Spot-check a sample of your output before shipping a gallery of 10,000.

Using the image as a link preview

Once you have the 1200x630 file on your own storage or CDN, the meta tags are short:

<meta property="og:image" content="https://cdn.example.com/thumbs/3f2a9c1d4b7e8a60-og.jpg">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="630">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="https://cdn.example.com/thumbs/3f2a9c1d4b7e8a60-og.jpg">

Use an absolute HTTPS URL. If your pages are rendered per URL (a directory entry per site, say), generate the tag from the same key(url) the script uses. Platforms cache previews aggressively, so changing the image later may not update an already shared link.

A thumbnail of someone else's site as a preview for your directory page is common practice, but check the usage terms of what you display, and do not capture pages behind logins or pages you are not entitled to republish.

Run it as a nightly job

For a directory that adds a few hundred sites a day, a cron entry is enough. Pipe URLs from a file, and the cache makes reruns cheap:

xargs -n 5000 python thumbs.py < new-urls.txt

Cache-friendly habits that keep the bill predictable:

Capture on first sight, refresh on a schedule. Weekly is plenty for most directories. Do not recapture on every page view.
Queue captures; never capture in a request handler. A capture takes from a fraction of a second to a few seconds. Show a placeholder and fill the thumbnail in when the job finishes.
Store the failure reason. The cache metadata above keeps error, so you can show "site unreachable" in an admin view and drop persistent failures from the list.
Respect rate limits. The API allows 60 requests per minute per key. At five URLs per call that is up to 300 pages a minute, far above what a nightly job needs.

Cost

On the API, plans start at $29 per month for 10,000 successful results, so a thumbnail costs about $0.003 on a full Starter plan, and less on larger tiers. Failed captures do not count. If your volume is occasional, the Pageframe Actor is pay-per-result at $1.50 per 1,000 captures with no subscription. Our pricing page has a calculator, and the earlier post on pay-per-result versus subscription explains when each one wins.

If you only need a handful of images, skip the code: the free website screenshot tool does single captures in the browser.

Limits to know about

Public pages only. There is no login or interaction step. For pages that need clicks or cookies, run Playwright yourself. Our screenshot API guide covers when that is the better choice.
Some sites block automated browsers. Expect a small share of URLs to return errors or challenge pages. The script's failure cache keeps that from becoming a retry storm.
Thumbnails go stale. A redesign is not reflected until the cache expires.
What we did not test. The /v1/screenshot call with a key was not re-run for this post. We ran the same engine through the public Actor, whose results use the same fields (url, status, fileUrl, error) and whose image links need your Apify token for the download instead of being pre-signed. That one difference is why thumbnails() takes capture_fn and download_fn parameters. Everything else, including batching, cropping, caching and failure handling, ran as shown.

Get a key and try it

The screenshot endpoint, the Actor and the free tool all use the same renderer. If you want thumbnails for a list of URLs today, start from the Siftwright API plans and the docs, paste the script above and point it at your list. Questions or edge cases we should cover next: support@siftwright.com.

Extract emails, phones and social links from a list of websites with Python

Siftwright — Thu, 05 Nov 2026 21:53:23 +0000

Sales and research scripts often start with "get me a contact address for each company on this list." Here is a small Python script that does it with the Contact Details Extractor Actor on Apify, plus the real output from a test run today.

What it returns

For each website you give it, the Actor scans the home page and a few likely contact pages and returns emails, phones and social profile links. Only addresses actually found on the pages are returned (no guessing, no pattern-generated emails).

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/contact-details-extractor").call(run_input={
    "startUrls": ["https://stripe.com", "https://www.python.org", "https://apify.com"],
    "maxPagesPerDomain": 3,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["domain"], item["emails"], item["phones"], item["linkedin"])

Real output (test run, 3 sites, 3 pages each)

domain	emails	phones	linkedin
stripe.com	sales@stripe.com	+1 888 926 2289	yes
apify.com	hello@apify.com	none found	yes
www.python.org	none found	none found	yes

Note the honest part: python.org's scanned pages publish no email, so the result is an empty list. A status field tells you whether the site loaded, and you can see how many pages were scanned.

Pricing

Pay per event: about $0.002 per website. The three-site test above cost well under a cent. New Apify accounts get free monthly credit, so you can try it without paying.

curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~contact-details-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://apify.com"],"maxPagesPerDomain":3}'

Try it: https://apify.com/siftwright/contact-details-extractor . It also works from AI agents through Apify's MCP server.

Brand monitoring on a budget: a Python script that alerts you to new press coverage

Siftwright — Tue, 03 Nov 2026 15:47:48 +0000

Most brand monitoring tools charge for dashboards you open twice a month. If what you need is "tell me when a new article mentions us, or mentions us next to the word breach", a short script and a news search API cover it.

This post builds that script in about 60 lines of Python. It runs a few Google News queries, filters out noise, remembers what it has already shown you and prints only what is new. We ran every step on 3 November 2026 using the public Siftwright Google News Scraper Actor on Apify, which is the same engine behind our Google News API. We use "Notion" as the example brand because it is a common word, which makes it a good test of noise filtering.

What a budget monitor needs (and what it does not)

A useful press-coverage alert has four parts:

Queries that match your brand, plus a second set for risk words such as outage, breach or lawsuit.
Noise filtering, because brand names double as ordinary words and companies publish about themselves.
De-duplication across runs, so you see each article once.
A delivery step: stdout, email, Slack, whatever you already read.

What it does not do: sentiment scoring, social media, podcasts, paywalled archives or guaranteed completeness. Google News is an index of publishers it chooses to include. If you need legal-grade coverage tracking, a paid media-monitoring service is the right tool. For early warning on a startup or a side project, this is enough.

Step 1: Fetch results for a query

Google News search accepts operators such as quoted phrases, OR and when: for recency. Putting them in the query string works through the Actor too. Here is the fetch function:

import json, os, urllib.request

ACTOR = "https://api.apify.com/v2/acts/siftwright~google-news-scraper/run-sync-get-dataset-items"

def fetch(query, max_items=50):
    body = json.dumps({"query": query, "maxItems": max_items}).encode()
    req = urllib.request.Request(f"{ACTOR}?token={os.environ['APIFY_TOKEN']}", body,
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=150) as r:
        return json.load(r)

For the query "Notion" when:7d with maxItems 20, the call returned HTTP 201 and 20 records. Every record has the same seven fields:

{
  "query": "\"Notion\" when:7d",
  "title": "ZoomInfo Launches MCP Integration With Notion Through GTM.AI",
  "link": "https://news.google.com/rss/articles/CBMi...?oc=5",
  "source": "Yahoo Finance",
  "publishedAt": "2026-11-03T13:30:00.000Z",
  "snippet": "ZoomInfo Launches MCP Integration With Notion Through GTM.AI Yahoo Finance",
  "status": "ok"
}

Two details matter for monitoring. First, link is a Google redirect URL, not the publisher's URL. It is still unique per article, which is all we need for de-duplication. Second, snippet is mostly the title plus the source name, so do not expect an article summary. If you need the full text, you fetch the publisher page yourself afterwards.

Step 2: Choose queries that find coverage, not noise

Two query sets work well together:

QUERIES = [
    '"Notion" when:2d',                                  # brand name, last 2 days
    '"Notion" (outage OR breach OR lawsuit) when:7d',    # risk terms, last week
]

Quote the brand name so Google matches the phrase. Use a longer when: window for rare risk terms, because a breach story from five days ago is still news to you if your script did not run last week.

If your brand name is a common word, add context words: "Acme" (software OR startup OR funding). If it is unique, the bare quoted name is fine. Expect to tune this list after the first run; the first results show you which kinds of noise your name attracts.

Step 3: Filter noise

Two filters removed obvious junk in our run, and you will probably add your own:

IGNORE_SOURCES = {"Notion"}                  # the brand's own posts are not coverage
IGNORE_WORDS = ("horoscope", "giveaway")     # recurring noise for your brand name

The first one matters more than it looks. In our test, a result with source "Notion" was the company's own page, not press coverage. Dropping your own publications and your PR wire pages keeps the digest honest.

Also skip records where status is not ok. The Actor reports empty searches and failures as records with another status rather than raising an error, so a quiet day produces a record you can ignore instead of a crash.

Step 4: Remember what you have seen

Because the same article stays in results for days, you need state. A JSON file is enough for one machine:

import hashlib

def item_id(r):
    # the link path is stable per article; drop the tracking query string
    return hashlib.sha1(r["link"].split("?")[0].encode()).hexdigest()[:16]

The script stores each ID with a timestamp. On the first run it records everything as a baseline and prints no alerts, otherwise your first notification is a wall of old articles. If you move this to a server, replace the file with a small database table or a key-value store.

The complete script

import hashlib, json, os, sys, urllib.request
from datetime import datetime, timezone

ACTOR = "https://api.apify.com/v2/acts/siftwright~google-news-scraper/run-sync-get-dataset-items"
STATE = "seen.json"

QUERIES = [
    '"Notion" when:2d',                       # brand name, last 2 days
    '"Notion" (outage OR breach OR lawsuit) when:7d',  # risk terms
]
IGNORE_SOURCES = {"Notion"}                   # the brand's own posts are not coverage
IGNORE_WORDS = ("horoscope", "giveaway")      # recurring noise for your brand name

def fetch(query, max_items=50):
    body = json.dumps({"query": query, "maxItems": max_items}).encode()
    req = urllib.request.Request(f"{ACTOR}?token={os.environ['APIFY_TOKEN']}", body,
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=150) as r:
        return json.load(r)

def item_id(r):
    # links are Google redirect URLs; the path segment is stable per article
    return hashlib.sha1(r["link"].split("?")[0].encode()).hexdigest()[:16]

def load_seen():
    try:
        return json.load(open(STATE))
    except FileNotFoundError:
        return {}

def main():
    seen = load_seen()
    fresh, first_run = {}, not seen
    for q in QUERIES:
        for r in fetch(q):
            if r.get("status") != "ok":
                continue
            if r["source"] in IGNORE_SOURCES or any(w in r["title"].lower() for w in IGNORE_WORDS):
                continue
            i = item_id(r)
            if i not in seen and i not in fresh:
                fresh[i] = r
    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
    for i in fresh:
        seen[i] = now
    json.dump(seen, open(STATE, "w"))
    print(f"{len(fresh)} new mentions" + (" (first run: baseline)" if first_run else ""))
    if not first_run:
        for r in sorted(fresh.values(), key=lambda r: r["publishedAt"], reverse=True):
            print(f"- {r['publishedAt'][:16]} | {r['source']} | {r['title'][:80]}")

if __name__ == "__main__":
    main()

Run it by hand first, then schedule it.

Real output

We ran the script three times against the live Actor.

Run 1 built the baseline:

35 new mentions (first run: baseline)

The two queries together produced 35 unique articles after filtering and de-duplication.

Run 2, immediately afterwards, printed:

0 new mentions

Run 3 followed after we deleted two stored IDs from seen.json to simulate fresh articles. The script printed:

3 new mentions
- 2026-11-03T13:33 | StreetInsider | ZoomInfo launches MCP integration with Notion for B2B data access
- 2026-11-03T13:30 | Yahoo Finance | ZoomInfo Launches MCP Integration With Notion Through GTM.AI
- 2026-10-29T07:00 | The Media Online | In the move-fast-and-break-things age of AI, some rules on reputation remain

We removed two IDs and got three alerts. We did not trace the extra one; the most likely explanation is that the result set shifted slightly between runs, which is normal for a live news index. The simulation is crude: it proves the dedupe and the output, not how many fresh stories a real day brings. We did not run the script over several days, so we have no measured alert rate to quote. The risk-word query returned only 5 articles over seven days, all of which were already in the baseline. For a brand in that position, silence is the expected result, and that is what you want from an alarm.

Scheduling and delivery

Run the script from cron once or twice a day:

0 8,16 * * * cd /opt/brandwatch && APIFY_TOKEN=... python3 brandwatch.py >> brandwatch.log

To deliver alerts somewhere you will see them, send the printed lines to a Slack incoming webhook, an email, or a ticket. Post one message per run, not one per article, so a busy day does not flood a channel. Keep the token in an environment variable or a secrets manager and never commit it.

Cost and limits

Pay only for results. The Actor charges per result, so a narrow query with when:2d is cheap and a broad one with maxItems: 100 is not. Keep maxItems near the number of articles you expect per window.
Result cap. In our earlier RSS versus API comparison a single search topped out at 100 records, so split broad brands into several narrower queries instead of asking for more.
Coverage. Google News does not index every publisher, and links are redirects. Verify any story that matters at the source.
Language and region. Add language and country to the Actor input if you monitor a non-US market, one run per market.
Duplicates across outlets. The same wire story appears under many publishers. The example script treats each as separate. Group by normalised title if that bothers you.

If you run this in production, move from the Actor to the API subscription once your volume is steady: plans start at $29/month and cover the same Google News data as JSON. For a wider look at assembling feeds, see the news monitoring pipeline guide, and if you only want to try a query once, the free Google News tool needs no code.

Summary

A budget brand monitor is a loop: query, filter, de-duplicate, deliver. The script above does that with one HTTP call per query, tested against live data for the fetch, baseline and de-duplication paths. Start with two queries, read the first week of output, and tighten the noise filters before you add anything else.

Google News RSS vs a Google News API: what you actually get (with working Python)

Siftwright — Fri, 30 Oct 2026 14:48:25 +0000

Google does not offer an official Google News API. If you need news results in code, you end up with one of two options: the undocumented Google News RSS feed, or a third-party API that returns the same kind of data as JSON. Both work. They differ in ways that only show up once you run them for a while.

This post compares them using real runs from 30 October 2026, with Python for each. We sell one of the options (the Siftwright Google News API), so we say plainly where the free RSS feed is the better choice.

The Google News RSS feed

Google News search is available as an RSS feed at a predictable URL:

https://news.google.com/rss/search?q=electric+vehicles&hl=en-US&gl=US&ceid=US:en

q is the query, hl the interface language, gl the country and ceid the country-and-language edition. A plain request returned HTTP 200 and 100 <item> elements in our run. Each item has a title, a link, a GUID, a publication date, an HTML description and a <source> element with the publisher.

Here is a minimal client using only the standard library:

import urllib.parse, urllib.request
import xml.etree.ElementTree as ET
from email.utils import parsedate_to_datetime

def google_news_rss(query, hl="en-US", gl="US", ceid="US:en"):
    url = "https://news.google.com/rss/search?" + urllib.parse.urlencode(
        {"q": query, "hl": hl, "gl": gl, "ceid": ceid})
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=30) as r:
        root = ET.fromstring(r.read())
    items = []
    for it in root.findall("./channel/item"):
        src = it.find("source")
        items.append({
            "title": it.findtext("title"),
            "link": it.findtext("link"),
            "source": src.text if src is not None else None,
            "published": parsedate_to_datetime(it.findtext("pubDate")).isoformat(),
        })
    return items

for i in google_news_rss("electric vehicles")[:3]:
    print(i["published"], "|", i["source"], "|", i["title"])

Real output:

2026-10-29T21:02:00+00:00 | Electrek | Toyota’s EV sales more than doubled in a year, but gas cars are getting crushed in China - Electrek
2026-10-29T22:00:00+00:00 | The Washington Post | Why Americans can’t buy these world-class EVs - The Washington Post
2026-10-27T16:00:00+00:00 | The Washington Post | Column | I went to Mexico to test-drive the cars Americans can’t buy - The Washington Post

For a hobby project, a personal dashboard or a quick prototype, this is hard to beat: it is free and needs no signup. Other languages work by changing the edition. A search for Elektroautos with hl=de&gl=DE&ceid=DE:de returned 100 German-language items, the first from Spiegel.

What the RSS feed leaves for you to clean up

Running the client above and inspecting the 100 items exposed the following.

The publisher name is glued onto the title. In our run, all 100 titles ended with - <source>, as in ... getting crushed in China - Electrek. If you show or deduplicate titles, you have to strip that suffix yourself.

Every link is a Google redirect. All 100 links were news.google.com/rss/articles/... URLs, not the publisher's address. They open the article in a browser, but if you want the publisher's domain for analysis you need to follow the redirect, or use the <source url> attribute (which gives the publisher's homepage, not the article).

The description is HTML. It contains a link to the article plus related-coverage links, not a text snippet. Parsing it for a summary is fragile.

A hard cap of 100 items. Our test query returned exactly 100, and so did the same query with when:1d and when:7d added. For broad queries, you see the newest slice, not everything. Narrow the query to go deeper.

Parameters are required. Requests without the hl/gl/ceid parameters were answered with a 302 redirect instead of a feed (we did not follow it). Build the URL with all three.

Terms. The feed carries a copyright notice stating it is made available "solely for the purpose of rendering Google News results within a personal feed reader for personal, non-commercial use". Read that before you build a commercial product on it. We are not lawyers and this is not legal advice, but it is the reason many teams look at paid APIs.

No guarantees. It is an undocumented endpoint. Nothing promises it will keep its format, stay unthrottled at volume or stay available.

The same search through a Google News API

A Google News API does the extraction and clean-up for you and returns JSON. The call below uses the public Siftwright Google News Scraper Actor on Apify, which is the same engine behind our API; pay-per-event pricing means you can try it on Apify's free plan credit:

import os, json, urllib.request

def google_news_actor(query, max_items=100, language="en", country="US"):
    url = ("https://api.apify.com/v2/acts/siftwright~google-news-scraper/"
           "run-sync-get-dataset-items?token=" + os.environ["APIFY_TOKEN"])
    body = json.dumps({"query": query, "maxItems": max_items,
                       "language": language, "country": country}).encode()
    req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)

for r in google_news_actor("electric vehicles", 5):
    print(r["publishedAt"], "|", r["source"], "|", r["title"][:60], "|", r["status"])

Real output:

2026-10-29T21:02:00.000Z | Electrek | Toyota’s EV sales more than doubled in a year, but gas cars  | ok
2026-10-29T22:00:00.000Z | The Washington Post | Why Americans can’t buy these world-class EVs | ok
2026-10-27T16:00:00.000Z | The Washington Post | Column | I went to Mexico to test-drive the cars Americans c | ok
2026-10-30T06:00:00.000Z | Barron's | ‘This Is a Knife Fight:’ Inside China’s Ruthless EV Blitz Up | ok
2026-10-29T23:02:00.000Z | Reuters | Chinese EVs win over some in South Korea, but Japan is provi | ok

Each record has query, title, link, source, publishedAt (ISO 8601 in UTC), snippet and status. The title comes without the source suffix, and the publisher is its own field. The same query matched the same articles as the RSS feed, which is what you would expect.

To be exact about what is and is not different:

Links are still Google redirects. The link values in our run were news.google.com/rss/articles/... URLs, same as the feed. The API does not magically resolve them. If you need publisher URLs, you still follow the redirect yourself.
The cap is the same. Asking for maxItems: 150 returned 100 records.
The sources are the same. Both read Google News. The API does not give you a different or bigger index.

So the value of an API is not extra data. It is the clean schema, the language and country parameters, an explicit status per item, and a service you call with a key and a price rather than an endpoint you hope keeps working.

Side by side

	Google News RSS	Siftwright Google News API
Cost	Free	From $29/month (10,000 results), or $1.50 per 1,000 articles on Apify
Signup	None	API key, or Apify account
Max items per query	100	100
Output	XML	JSON (or CSV on Apify)
Publisher name	Glued to the title, plus a `<source>` element	Separate `source` field
Date format	RFC 822 string	ISO 8601 UTC
Snippet	HTML with related links	Plain `snippet` field
Article links	Google redirect URLs	Google redirect URLs
Failed or empty search	Empty feed or a redirect	Empty searches are not billed
Commercial use	Feed notice says personal, non-commercial	Governed by our terms; you are responsible for lawful use
Stability promise	None	A service we operate

Which one should you use?

Use the RSS feed when:

you are prototyping, learning or building something for yourself;
you run a handful of queries and do not mind writing the clean-up code above;
you are comfortable with the feed's notice and its lack of guarantees.

Use an API when:

the data feeds something a customer pays for, and you need a clear, supported source;
you run many queries and want a consistent schema without maintaining parsing code;
you want to know what each item cost and why an item failed.

In either case, store the results and deduplicate them rather than re-reading the same feed. The pipeline in our news monitoring guide shows queries, dedupe, SQLite storage and a daily digest, and the same code works with either source if you map the fields to title, link, source and published.

Adapting the RSS client to the same schema

If you start with RSS and might switch later, normalise now. A small adapter makes the switch a one-line change:

import re

def normalise_rss(item):
    title = item["title"]
    if item["source"] and title.endswith(" - " + item["source"]):
        title = title[: -len(item["source"]) - 3]
    return {"title": title, "link": item["link"],
            "source": item["source"], "published": item["published"]}

def normalise_api(row):
    return {"title": row["title"], "link": row["link"],
            "source": row["source"], "published": row["publishedAt"]}

Every downstream step (dedupe, storage, alerts) then works on one shape. For deduplication, key on the normalised title plus source rather than the link, since a story syndicated to several outlets has several links but the same headline.

What it costs

On our plans, a result is one article that came back with status ok, and empty searches are free. Plans start at $29 per month with 10,000 successful results (see pricing). If you prefer no subscription, the Apify Actor charges $1.50 per 1,000 articles. For the 100-article test above, that is $0.15. A quick comparison for your own volume is on the pricing calculator, and our Google News API page has a live demo with no signup.

A note on testing: the RSS code and the Actor call above were run on 30 October 2026 exactly as shown. We did not re-run the /v1/google-news REST endpoint with a fresh key for this post, since the Actor returns the same record shape, and the fields in the API's response are listed in the docs.

Summary

Google has no official News API. The RSS feed is free and decent for personal projects, but it returns XML with the publisher glued to the title, HTML descriptions and Google redirect links, and its own notice limits it to personal, non-commercial use.
A News API returns the same articles with a clean schema, ISO dates, a separate source field and per-item status. It does not give you a larger index or publisher links.
Whichever you pick, normalise to one schema early, deduplicate on title plus source, and keep your queries narrow, because both stop at 100 items.

Questions or a case this post missed? Email support@siftwright.com.

YouTube subtitles to SRT and VTT from code: language tracks, cue timing and a fallback

Siftwright — Tue, 27 Oct 2026 14:51:32 +0000

You have a list of YouTube videos and you need subtitle files for them: .srt for an editor or a video player, .vtt for an HTML <track> element, maybe a second language for the same video. Doing it by hand in the YouTube UI does not scale past a handful, and the usual libraries hand you raw caption segments that you still have to turn into valid files.

This post covers the whole path: fetching captions, choosing a language track, building SRT and VTT correctly (including the one timing quirk that breaks players), and what to do when a translation request fails. The sample output below comes from real runs on 27 October 2026 against the first YouTube video ever uploaded, jNQXAC9IVRw (19 seconds, so the files stay short enough to read).

What an SRT and a VTT file actually are

Both formats are plain text lists of cues. A cue is a time range and the text to show.

SRT numbers every cue and uses a comma before the milliseconds:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

VTT starts with a WEBVTT header, drops the numbers (they are optional) and uses a dot:

WEBVTT

00:00:01.200 --> 00:00:03.360
All right, so here we are, in front of the elephants

That is the entire difference for simple captions. Which means the interesting work is not the syntax. It is getting correct timings and the right language in the first place.

Step 1: fetch the captions

The Siftwright YouTube transcript API can return srt and vtt strings directly through its outputFormats option, next to the raw segments. Requesting segments as well is worth it, because they let you build your own files when you need a different line length or style.

curl https://siftwright.com/v1/youtube-transcript \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.youtube.com/watch?v=jNQXAC9IVRw",
       "language": "en",
       "outputFormats": ["segments", "srt", "vtt"]}'

In Python, saving one pair of files per video:

import os
import requests

API = "https://siftwright.com/v1/youtube-transcript"
KEY = os.environ["SIFTWRIGHT_API_KEY"]

def fetch(url, language="en"):
    r = requests.post(
        API,
        headers={"Authorization": f"Bearer {KEY}"},
        json={"url": url, "language": language,
              "outputFormats": ["segments", "srt", "vtt"]},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["results"][0]

item = fetch("https://www.youtube.com/watch?v=jNQXAC9IVRw")
if item["status"] == "ok":
    open(f'{item["videoId"]}.{item["language"]}.srt', "w", encoding="utf-8").write(item["srt"])
    open(f'{item["videoId"]}.{item["language"]}.vtt', "w", encoding="utf-8").write(item["vtt"])

A note on what was and was not tested. For this post we ran the same request, with the same input fields, against the public YouTube Transcript Extractor Actor, the engine behind the endpoint, and checked every output below against it. The response envelope of the endpoint (a results array, as shown in our earlier tutorial) is taken from our docs rather than from a fresh call with a new key.

The Actor returned this for the zoo video:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

2
00:00:05,318 --> 00:00:07,974
the cool thing about these guys is that they have really...

3
00:00:07,974 --> 00:00:12,616
really really long trunks

Notice cue 2 ends at 07,974, exactly where cue 3 begins. That is deliberate, and it is the quirk the next section is about.

Step 2: the overlap problem

Raw YouTube caption segments come as start and duration. For auto-generated captions in particular, a segment's start + duration often runs past the start of the next segment, because the text is being rolled up on screen while the next words arrive.

If you naively compute end = start + duration, you get cues that overlap. Overlapping cues are legal in VTT but they stack on screen, and strict SRT consumers either reject them or show two cues at once. The fix is simple: never let a cue end after the next one starts.

Two consecutive segments from the zoo video show the fields involved:

{"start": 5.318, "duration": 2.656, "text": "the cool thing about these guys is that they have really..."}
{"start": 7.974, "duration": 4.642, "text": "really really long trunks"}

Here 5.318 + 2.656 is exactly 7.974, so nothing overlaps. In auto-generated captions the rolled-up text can make a segment's computed end land after the next segment's start, so the clamp matters; it costs nothing when nothing overlaps.

Step 3: build your own files from segments

You need your own builder when you want different line lengths (broadcast guidelines often use 42 characters per line and two lines per cue), when you merge segments, or when you want to post-process text. This is the whole thing in standard-library Python:

import re

def ts(sec, sep):
    ms = round(sec * 1000)
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02}:{m:02}:{s:02}{sep}{ms:03}"

def wrap(text, width=42):
    lines, cur = [], ""
    for w in text.split():
        if cur and len(cur) + 1 + len(w) > width:
            lines.append(cur)
            cur = w
        else:
            cur = f"{cur} {w}".strip()
    lines.append(cur)
    return "\n".join(lines)

def cues(segments):
    out = []
    for i, seg in enumerate(segments):
        start = seg["start"]
        end = start + seg["duration"]
        if i + 1 < len(segments):          # never run past the next cue
            end = min(end, segments[i + 1]["start"])
        text = " ".join(seg["text"].split())
        if text and end > start:
            out.append((start, end, wrap(text)))
    return out

def to_srt(segments):
    return "\n".join(
        f"{n}\n{ts(a, ',')} --> {ts(b, ',')}\n{t}\n"
        for n, (a, b, t) in enumerate(cues(segments), 1))

def to_vtt(segments):
    body = "\n".join(
        f"{ts(a, '.')} --> {ts(b, '.')}\n{t}\n" for a, b, t in cues(segments))
    return "WEBVTT\n\n" + body

We checked this builder against the API's own srt output for the zoo video. The timings match cue for cue; the only difference is that the builder wraps long lines at 42 characters, so the first cue becomes two lines:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the
elephants

Whether that is better depends on where the subtitles are shown. For a narrow mobile player, wrapping helps. For a wide web player it is just noise, which is why we return the unwrapped version by default and give you the segments to wrap yourself.

If you already have an SRT string and just need VTT, converting it is three lines, because VTT is nearly a subset:

def srt_to_vtt(srt):
    srt = srt.replace("\r\n", "\n").strip() + "\n"
    srt = re.sub(r"(\d\d:\d\d:\d\d),(\d{3})", r"\1.\2", srt)
    srt = re.sub(r"^\d+\n(?=\d\d:)", "", srt, flags=re.M)   # drop cue numbers
    return "WEBVTT\n\n" + srt

Run on the API's SRT for the zoo video, this produced a file identical to the vtt field the API returned.

Step 4: choose the language track deliberately

A video can carry several caption tracks, and a track can be manual (written by the uploader) or auto-generated. Two things follow:

Ask for preferManual: true (the default) so a human-written track wins over a machine one in the same language.
Read availableLanguages in the response to see what actually exists instead of guessing.

For the zoo video the response lists two tracks, English and German, both manual. Asking for German with "language": "de" returned a real German track, not a machine translation:

1
00:00:00,000 --> 00:00:05,067
Also hier sind wir vor den Elefanten.

2
00:00:05,067 --> 00:00:12,467
Das Coole an den Typen ist dass sie sehr, sehr, sehr, lange Rüssel haben.

Note that the German cues have different timings from the English ones. Tracks are separate files written separately, so you cannot reuse one track's timings for another. If you ship both, ship both files.

Step 5: translation, and what happens when it fails

For languages the video does not have a track for, you can ask for a translation with translateTo (for example "translateTo": "fr"). It uses YouTube's own caption translation, so quality is machine quality, and it is a service we depend on rather than control.

We tested this, and it is worth being honest about the result. Two translation requests from our test environment, translateTo: "es" and translateTo: "fr", both came back with status: "translation_failed". The first said Translation failed (fetch failed) and the second Translation failed (HTTP 429), which is YouTube rate-limiting the request. In both cases the original-language transcript was returned instead, and the error message states that it is free of charge.

That behaviour is the useful part, and your code should expect it:

def subtitles(url, want="es"):
    r = requests.post(API, headers={"Authorization": f"Bearer {KEY}"},
        json={"url": url, "translateTo": want,
              "outputFormats": ["segments", "srt", "vtt"]}, timeout=120)
    r.raise_for_status()
    item = r.json()["results"][0]

    if item["status"] == "ok":
        return item, item["language"]            # translated (or native) track
    if item["status"] == "translation_failed":
        # original language came back, unbilled: keep it, retry the translation later
        return item, item["sourceLanguage"]
    raise RuntimeError(f'{item["status"]}: {item.get("error")}')

A failed translation is not a reason to drop the video. Store the original-language file, record that the translation is pending, and retry later with a delay. A 429 is the textbook case for backing off rather than hammering.

If you need translations you can depend on at volume, a better pipeline is to fetch the source-language segments once and send the cue texts to a translation service you control, keeping the original start and end values. That keeps timings stable and moves the failure risk to a service you can pick and pay for.

Practical checks before you ship the files

A few small checks catch most subtitle bugs:

Save as UTF-8. German, French and every non-Latin script break on a default Windows code page. Always pass encoding="utf-8".
Validate monotonic cues. Assert that each cue's start is at or after the previous cue's end. The builder above guarantees it; third-party SRTs often do not.
Name files by language. videoId.en.srt and videoId.de.srt. Players and HTML <track srclang> both want the language code.
Skip videos without captions. The API marks them with an error status and does not bill them, so a batch job should log and move on.
Respect the source. Captions belong to the video's uploader. Fine for accessibility, search and personal study; check the licence before republishing them.

What it costs

The Siftwright API plans start at $29 per month and count only results with status ok, so videos without captions and failed translations do not use your quota. If you only need a few thousand files, the pay-per-result Apify Actor is simpler. The pricing page has a calculator, and the post on pay-per-result vs subscription pricing explains where the break-even sits.

To check what a video offers before writing any code, the free YouTube transcript tool downloads TXT, SRT and VTT in the browser, no key needed.

Originally published on siftwright.com.

Track hiring signals across Greenhouse, Lever and Ashby with one Python script

Siftwright — Mon, 26 Oct 2026 19:11:02 +0000

Hiring is one of the clearest public buying signals: a company opening "Data Engineer" roles is about to spend on data tooling. The catch is that every company uses a different applicant tracking system (ATS), and each has its own API shape.

This tutorial uses the ATS Jobs Scraper Actor, which reads the public job-board feeds of Greenhouse, Lever, Ashby, SmartRecruiters and Recruitee and returns one schema. It costs $1 per 1,000 jobs returned after your filters; companies without a supported board and filtered-out jobs are free.

The script

import csv, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("siftwright/ats-jobs-scraper").call(
    run_input={
        "companies": ["greenhouse:airbnb", "lever:palantir", "ramp.com"],
        "titleKeywords": ["data engineer", "analytics"],
        "excludeTitleKeywords": ["senior", "staff", "manager", "director"],
        "includeDescription": False,
        "maxJobsPerCompany": 20,
    },
    max_total_charge_usd=0.05,  # hard spend cap for the run
)

jobs = list(client.dataset(run.default_dataset_id).iterate_items())
with open("jobs.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["company", "ats", "title", "location", "url"])
    for j in jobs:
        w.writerow([j["company"], j["ats"], j["title"], j["location"], j["url"]])

summary = client.key_value_store(run.default_key_value_store_id).get_record("SUMMARY")
for c in summary["value"]["companies"]:
    print(f'{c["input"]:20} {c["status"]:10} {c["ats"] or "-":12} open={c["jobsOnBoard"]} matched={c["jobsMatched"]} returned={c["jobsReturned"]}')
print(len(jobs), "jobs written to jobs.csv")

Install and run:

pip install apify-client
export APIFY_TOKEN=your_token
python hiring.py

max_total_charge_usd caps what the run can spend, so a typo in a filter cannot surprise you.

What I got

I ran this on 2026-10-26 against three companies. The output:

greenhouse:airbnb    ok         Greenhouse   open=187 matched=7 returned=7
lever:palantir       ok         Lever        open=307 matched=1 returned=1
ramp.com             ok         Ashby        open=173 matched=0 returned=0
8 jobs written to jobs.csv

Eight jobs, for example "Data Engineer - Talent Acquisition Operations" (Palantir, New York) and "Software Engineer, Analytics Infrastructure" (Airbnb, Brazil). Ramp had 173 open roles and none matched these filters, so it returned nothing and cost nothing. Notice ramp.com was just a website: the Actor found the Ashby board from the careers page link. The SUMMARY record reports how each board was detected (detectedVia), so you can check it.

Limits worth knowing

Only five ATS platforms. Workday, iCIMS, Taleo and custom career sites come back as "not found" (free).
If you give a name instead of a board URL, the board slug is guessed; pass ats:board such as greenhouse:airbnb when you need certainty.
Salary is only present when the company publishes it. None of the jobs above had one.
Filters are substring matches on the title, so "analytics" also matched finance roles. Tighten titleKeywords for cleaner output.

Use it from an AI agent

The same Actor works through Apify's MCP server, so an agent can ask "which of these companies are hiring data engineers?" and pay per result.

Store page: https://apify.com/siftwright/ats-jobs-scraper. More tools: https://siftwright.com

Summarise any YouTube video with an LLM: transcript chunking that keeps timestamps

Siftwright — Fri, 23 Oct 2026 14:48:20 +0000

Pasting a YouTube transcript into a chat window works for a five-minute video. It stops working at about the time the video gets interesting. A two-hour lecture is roughly 20,000 words, which is more than you want to stuff into one prompt, and even when it fits, a single giant prompt tends to produce a summary that favours the start and the end and skips the middle.

The other thing a plain-text transcript loses is where things were said. A summary that says "the speaker explains attention" is much less useful than one that says "attention is explained at 25:10" with a link that opens the video there. This post builds that: a small Python pipeline that fetches a transcript with timestamps, splits it into chunks that respect caption boundaries, summarises each chunk, combines the notes, and rewrites every cited time into a clickable link.

We ran the transcript and chunking parts against a real video on 23 October 2026: Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out", 6,980 seconds long. The LLM calls are the one piece we could not run for you, and we say exactly where below.

Step 1: get segments, not just text

Most "transcript to text" tools return one big string. For summarisation you want the timestamped segments, because the timestamps are what let you cite. The Siftwright YouTube Transcript API returns them when you ask for segments:

curl -s -X POST https://siftwright.com/v1/youtube-transcript \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.youtube.com/watch?v=jNQXAC9IVRw"], "outputFormats": ["segments"]}'

Each result carries the video's metadata plus a segments array. These are the first three segments of the first YouTube video ever uploaded, exactly as returned:

[
  {"start": 1.2,   "duration": 2.16,  "text": "All right, so here we are, in front of the elephants"},
  {"start": 5.318, "duration": 2.656, "text": "the cool thing about these guys is that they have really..."},
  {"start": 7.974, "duration": 4.642, "text": "really really long trunks"}
]

start and duration are in seconds. Two properties of caption data matter for what follows:

Segments are short. The 6,980-second lecture came back as 2,955 segments, about 7 words each. A segment is a caption line, not a sentence.
Segments can overlap, or leave gaps. In the lecture, the first segment starts at 0.199 s and lasts 4.7 s, but the second starts at 2.52 s, before the first has finished. In the zoo video above, there is a gap instead: the first segment ends at 3.4 s and the next starts at 5.3 s. Auto-generated captions are drawn with a rolling window, so never assume start + duration\ equals the next start\.

If a video has no captions, the result comes back with a non-ok status and is not billed. Check status before you use segments; the code below does.

Step 2: chunk by segment, keep the time range

The naive way to chunk is to split the text every N characters. That cuts sentences in half and throws away the timestamps. Chunking by segments keeps both: you add whole segments to a chunk until it reaches a word budget, then start the next one. The start of the first segment and the end of the last give you the chunk's time range for free.

Two small details improve the result. First, a few segments of overlap between chunks, so a thought that straddles a boundary appears in both summaries instead of being cut. Second, a time marker like [7:49] inserted roughly every 30 seconds inside the chunk text. Without markers the model has no idea when anything happened; with a marker on every segment you waste tokens. Every 30 seconds is a reasonable middle.

import json, sys

def fmt(t):
    t = int(t)
    h, m, s = t // 3600, t % 3600 // 60, t % 60
    return f"{h}:{m:02d}:{s:02d}" if h else f"{m}:{s:02d}"

def chunk_segments(segments, max_words=1500, overlap_segments=5):
    """Group caption segments into chunks of ~max_words, never splitting a segment.
    Each chunk keeps its start/end time so a summary can point back to the video."""
    chunks, cur, words = [], [], 0
    for seg in segments:
        n = len(seg["text"].split())
        if cur and words + n > max_words:
            chunks.append(cur)
            cur = cur[-overlap_segments:]          # small overlap so a sentence cut at the edge appears twice
            words = sum(len(s["text"].split()) for s in cur)
        cur.append(seg)
        words += n
    if cur:
        chunks.append(cur)
    out = []
    for c in chunks:
        out.append({
            "start": c[0]["start"],
            "end": c[-1]["start"] + c[-1]["duration"],
            # one [m:ss] marker per ~30 s keeps the prompt small but lets the model cite times
            "text": render(c),
        })
    return out

def render(seg_list, every=30):
    lines, last = [], -every
    for s in seg_list:
        if s["start"] - last >= every:
            lines.append(f"\n[{fmt(s['start'])}] ")
            last = s["start"]
        lines.append(s["text"] + " ")
    return "".join(lines).strip()

We ran this on the lecture. 2,955 segments became 15 chunks of about 1,500 words each. The time ranges are what you would hope for: the first chunk covers 0:00 to 8:01, the second 7:49 to 15:52 (the overlap is the 12-second overlap between them), the third 15:38 to 24:07, and the last 1:53:07 to 1:56:21 with 598 words. The text of a chunk starts like this:

[7:49] we're going to focus on the Transformer neural network itself okay so I created a new Google collab uh jup notebook here and this will allow me to later easily share this code ...

Note the uh and the missing punctuation. That is what auto-generated captions look like. Models cope with it fine, and the summaries they write are cleaner than the input, but do not be surprised when the raw text looks rough.

Choosing the chunk size

1,500 words is a default, not a law. Smaller chunks (500 to 800 words) give more detailed notes and more calls. Larger chunks (3,000+) are cheaper but start losing the middle of each chunk. For lectures and tutorials, 1,000 to 2,000 words is a sensible range. For a fast podcast with lots of topic changes, go smaller.

Step 3: map, then reduce

This is the classic map-reduce summarisation pattern:

Map: ask the model for short notes on each chunk, keeping the [m:ss] markers of the moments it mentions.
Reduce: give the model all the notes (not the full transcript) and ask for the final summary, again citing times.

The reduce step only sees the notes, which are a small fraction of the transcript, so it fits in a single prompt even for long videos. Here is the whole pipeline, including the fetch and the time-to-link rewrite:

import json, os, re, requests
from chunker import chunk_segments, fmt   # the chunker above, saved as chunker.py

API = "https://siftwright.com/v1/youtube-transcript"

def get_video(url):
    r = requests.post(API, headers={"Authorization": f"Bearer {os.environ['SIFTWRIGHT_API_KEY']}"},
                      json={"urls": [url], "outputFormats": ["segments"]}, timeout=300)
    r.raise_for_status()
    item = r.json()["results"][0]
    if item["status"] != "ok":
        raise LookupError(item.get("error", item["status"]))
    return item

def summarize_video(url, llm):
    """llm(prompt: str) -> str is any chat-model call you already have."""
    v = get_video(url)
    chunks = chunk_segments(v["segments"])
    notes = []
    for c in chunks:                                   # map: one call per chunk
        notes.append(llm(
            "Summarize this part of a video transcript in 3-5 bullets. "
            "Keep the [m:ss] markers of the moments you mention.\n\n" + c["text"]))
    joined = "\n\n".join(f"## {fmt(c['start'])}-{fmt(c['end'])}\n{n}" for c, n in zip(chunks, notes))
    final = llm("Combine these section notes into a summary with a short overview and "
                "5-8 key points. Cite times like [12:34].\n\n" + joined)       # reduce
    # turn [h:mm:ss] / [m:ss] into links that open the video at that moment
    def link(m):
        parts = [int(p) for p in m.group(1).split(":")]
        secs = sum(p * 60 ** i for i, p in enumerate(reversed(parts)))
        return f"[{m.group(1)}](https://youtu.be/{v['videoId']}?t={secs})"
    return v["title"], re.sub(r"\[(\d+(?::\d{2}){1,2})\]", link, final)

llm is deliberately a plain function that takes a prompt string and returns a string, so you can plug in whichever model and SDK you already use.

The last step is the one readers will actually notice. The regular expression finds every [12:34] or [1:02:03] in the final summary, converts it to seconds, and rewrites it as a youtu.be/<id>?t=<seconds> link. YouTube opens the video at that second when it sees the t parameter.

What we tested, and what we did not

We ran the full pipeline against the real lecture with a stand-in llm function that returns a canned bullet quoting a timestamp from its input. That tests everything except the model: the API call, the 15 chunks, the 16 total calls (15 map plus one reduce), the largest prompt at 1,535 words, and the link rewrite. The output was:

Let's build GPT: from scratch, in code, spelled out.
16 llm calls; max prompt words 1535
- discusses a topic around [1:00:41](https://youtu.be/kCc8FmEb1nY?t=3641)

1:00:41 becomes t=3641, which is 3,600 + 41. The rewrite works for both m:ss and h:mm:ss.

We did not run it with a real language model, so we cannot show you a real summary or tell you what it costs on your provider. That is a function of the model and the length of the video: 15 map calls of about 1,500 words in, a few bullets out, plus one reduce call. You can estimate it before you start by counting the words in the chunks, which chunk_segments already gives you.

Edge cases worth handling

Very long videos. The reduce prompt grows with the number of chunks. A two-hour lecture produced 15 chunks, which is fine. A ten-hour stream would produce about 75, and 75 sets of notes may be too long for one call. The fix is another round: reduce groups of ten notes into section summaries, then reduce those. The same function works at every level.

Videos without captions. Some videos have none, and some have captions in a different language. The API accepts a language parameter and a translateTo option, and it reports isAutoGenerated and availableLanguages in the result, so you can tell the user why a video could not be summarised instead of failing silently.

Playlists and channels. The same endpoint accepts a playlist or channel URL and returns one result per video (see maxVideosPerSource). Running the loop over a playlist gives you a summary of each lecture in a course. Billing is per video that returns a transcript, so check the video count before you run a large playlist.

Prompt injection. A transcript is untrusted text. A video can literally say "ignore your instructions and..." out loud. Keep your instructions in the system prompt, treat the transcript as data, and do not give the summarising model tools that can act on anything. A summary pipeline needs none.

Copyright and terms. Summaries for your own research are one thing; republishing someone's content is another. Check the terms that apply to you before publishing summaries at scale.

Making it cheaper

Cache by video ID. Transcripts of a published video almost never change. Store the segments and the summary keyed by videoId and a version of your prompt.
Summarise once, ask many times. If users will ask questions about the video later, keep the chunk notes. Retrieval over 15 sets of notes with timestamps is far cheaper than re-reading the whole transcript on every question.
Skip the map step for short videos. Under about 3,000 words, a single call with the markers inserted is simpler and usually just as good.

Summary

Timestamped segments are the key ingredient. With them, chunking keeps boundaries intact, the model can cite times, and a three-line regular expression turns those citations into links. Without them you get a block of text and a summary nobody can verify.

If you do not need code for one-off videos, the free transcript tool gives you the text, timestamped lines, SRT and VTT in the browser. For the pipeline above, the API is 10,000 results for $29 a month; our Python and Node.js guide covers the rest of the request options, and if your own server gets blocked when it talks to YouTube directly, see the RequestBlocked fix.

Give your AI agent web data tools over MCP: transcripts, contacts and news, billed per result

Siftwright — Wed, 21 Oct 2026 12:56:27 +0000

Agents are good at deciding what data they need and bad at fetching it. Instead of writing glue code, you can give an MCP client (Claude Desktop, Cursor, VS Code and others) a few Apify Actors as tools and let it call them directly. Here is a minimal setup using three small Actors, with the real prices and the limits.

The config

Apify hosts an MCP server at mcp.apify.com. The tools parameter limits it to the Actors you name, and your Apify API token goes in an Authorization header:

{
  "mcpServers": {
    "siftwright-data": {
      "url": "https://mcp.apify.com?tools=siftwright/youtube-transcript-extractor,siftwright/contact-details-extractor,siftwright/google-news-scraper",
      "headers": {
        "Authorization": "Bearer <APIFY_TOKEN>"
      }
    }
  }
}

Apify adds run and dataset helper tools (such as fetching dataset items) automatically, so the agent can read results. Check Apify's MCP docs for the current format; it has changed before.

The three tools

Actor	What it returns	Price
YouTube Transcript Extractor	Transcript of a video as text, SRT or VTT	$0.003 per transcript
Contact Details Extractor	Emails, phones and social links from a website	$0.002 per website
Google News Scraper	News articles for a search topic	$0.0015 per article

These are pay-per-event Actors: you are charged for results that were produced, and failed lookups are not charged. Apify also charges a tiny per-run start fee ($0.00005).

Three prompts to try

"Get the transcript of this video and give me five bullet points: "
"Find a contact email and LinkedIn page for example.com, acme.io and foo.dev."
"What are the 10 latest news stories about ? Group them by theme."

The agent picks the Actor, fills the input, runs it and reads the dataset. A single call costs a fraction of a cent.

Honest limits

Running Actors needs an Apify account and token. Without one, the server only offers search and docs tools.
YouTube blocks some server IPs, so a small share of videos can fail. Those are not charged.
Contact extraction only finds what is published on the site's own pages. It does not guess addresses.
Agents can loop. Set a spending limit on your Apify account and keep the tools list short.
Siftwright also runs its own hosted MCP server for its API plans, which needs a paid key. The Apify route above does not.

Try it without an agent first

The free browser tools at siftwright.com/tools run the same lookups with no signup, so you can see the output shape before wiring anything up.

Fix youtube-transcript-api RequestBlocked and IpBlocked on servers

Siftwright — Tue, 20 Oct 2026 14:47:40 +0000

The script worked yesterday. It fetched a transcript in half a second on your laptop, you wired it into a worker, deployed it, and now every call ends with this:

RequestBlocked:
Could not retrieve a transcript for the video https://www.youtube.com/watch?v=jNQXAC9IVRw!
This is most likely caused by:

YouTube is blocking requests from your IP. This usually is due to one of the following reasons:
- You have done too many requests and your IP has been blocked by YouTube
- You are doing requests from an IP belonging to a cloud provider (like AWS, Google Cloud Platform, Azure, etc.). ...

That is the real output of youtube-transcript-api 1.2.4, produced on 20 October 2026 from a cloud container, for the first YouTube video ever uploaded. Nothing is wrong with your code. This post explains what the error means, which fixes work, which ones to avoid, and shows a small, tested fallback you can drop into an existing pipeline.

What RequestBlocked and IpBlocked mean

The library is open source and does not use an official YouTube API. It makes the same requests a browser makes when it plays a video: it asks YouTube for the player data, finds the caption tracks in the response, and downloads the one you asked for. YouTube answers those requests differently depending on where they come from.

Both exceptions live in youtube_transcript_api._errors, and they are related:

RequestBlocked is the general case: YouTube refused to serve the player data to your address.
IpBlocked is a subclass of RequestBlocked. You can catch RequestBlocked and cover both.

The library's own error text names the two usual causes. You made a lot of requests from one address, or your address belongs to a cloud provider. Home and office connections are rarely blocked. Datacenter ranges (AWS, GCP, Azure and most VPS hosts) frequently are. That is why the same code passes on a laptop and fails the moment it is deployed.

Two practical consequences:

Retrying does not help. We ran three attempts, two seconds apart, from the blocked container. All three raised RequestBlocked. A block is a property of the address, not a transient glitch, so a retry loop only adds latency.
A block hides the real answer. When YouTube refuses you, the library cannot tell you whether the video has captions at all. A video that does not exist also raises RequestBlocked from a blocked address. You learn about the video only after you get past the block.

Not every failure is a block. The same module defines TranscriptsDisabled, NoTranscriptFound, VideoUnavailable, AgeRestricted, PoTokenRequired and others. Those describe the video or the request, and switching IPs will not change them. Your error handling should treat the two groups differently, which the fallback code below does.

Your options, honestly compared

1. Run it somewhere that is not a datacenter

If this is a personal script, run it on your own machine or a home server. It is free and it works. It stops being an answer when the job needs to run on a schedule in the cloud, or when more than one person's work depends on it.

2. Route requests through a proxy

The library has first-class proxy support. WebshareProxyConfig takes a username and password for residential proxies, and GenericProxyConfig takes any HTTP or HTTPS proxy URL:

from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api.proxies import GenericProxyConfig

api = YouTubeTranscriptApi(
    proxy_config=GenericProxyConfig(
        http_url="http://user:pass@proxy.example.com:8000",
        https_url="http://user:pass@proxy.example.com:8000",
    )
)
snippets = api.fetch("jNQXAC9IVRw").snippets

We did not run this one, because we have no proxy to point it at. The signature above is from the installed library. The caveats are worth knowing before you buy anything. The proxy has to be residential or otherwise not on a blocked range, since a datacenter proxy has the same problem as your server. You pay for it by traffic or by IP. And you take over the monitoring: when the pool gets flagged, your jobs fail until you notice.

3. Authenticate with cookies

The library's own error message mentions that authenticating requests with cookies can keep things working for a while, and in the same breath warns that YouTube may eventually permanently ban the account used. Do not tie an account you care about to a scraper.

4. Call an API that does the proxying for you

This moves the problem to a service whose whole job is to keep requests working. You send a video URL and get the transcript back as JSON, from any server. The tradeoff is cost: you pay per result instead of paying for proxy traffic. Whether that is cheaper depends on your volume. A few hundred transcripts a month is usually cheaper through an API than through a proxy plan you have to configure and watch. Millions a month is a different calculation, and at that scale you should price both.

We run one of these: the Siftwright YouTube Transcript API, and a free transcript tool in the browser for one-off videos. We are biased, so here is the useful part regardless of whose API you pick: the pattern below works with any provider.

The pattern: use the library first, fall back only on blocks

You do not have to rip out a working library to fix a blocked deployment. A good compromise is to keep the free path as the default and call the paid path only when YouTube blocks you. That keeps the cost proportional to the problem.

The rules for the wrapper:

Fall back only on RequestBlocked (which includes IpBlocked).
Do not fall back on TranscriptsDisabled, NoTranscriptFound or VideoUnavailable. The answer is "no", and a second service will say the same thing and may charge you to say it.
Return which path produced the text, so you can see in your logs how often you are blocked.

import os

import requests
from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api._errors import (
    IpBlocked,
    NoTranscriptFound,
    RequestBlocked,
    TranscriptsDisabled,
    VideoUnavailable,
)

API = "https://siftwright.com/v1/youtube-transcript"
KEY = os.environ["SIFTWRIGHT_API_KEY"]


def via_library(video_id):
    snippets = YouTubeTranscriptApi().fetch(video_id, languages=["en"]).snippets
    return " ".join(s.text for s in snippets)


def via_api(video_id):
    r = requests.post(
        API,
        headers={"Authorization": f"Bearer {KEY}"},
        json={
            "urls": [f"https://www.youtube.com/watch?v={video_id}"],
            "outputFormats": ["text"],
        },
        timeout=300,
    )
    r.raise_for_status()
    item = r.json()["results"][0]
    if item["status"] != "ok":
        raise LookupError(item.get("error", item["status"]))
    return item["text"]


def get_transcript(video_id):
    """Return (source, text). Only network blocks fall back."""
    try:
        return "library", via_library(video_id)
    except (RequestBlocked, IpBlocked):
        return "api", via_api(video_id)
    except (TranscriptsDisabled, NoTranscriptFound, VideoUnavailable) as err:
        raise LookupError(type(err).__name__) from err


for vid in ["jNQXAC9IVRw", "dQw4w9WgXcQ", "aaaaaaaaaaa"]:
    try:
        source, text = get_transcript(vid)
        print(vid, source, len(text.split()), "words:", text[:50])
    except LookupError as err:
        print(vid, "no transcript:", err)

Install with pip install youtube-transcript-api requests. We ran exactly this file from the same blocked container with a real key:

jNQXAC9IVRw api 39 words: All right, so here we are, in front of the elephan
dQw4w9WgXcQ api 487 words: [♪♪♪] ♪ We're no strangers to love ♪ ♪ You know th
aaaaaaaaaaa no transcript: This video is unavailable

Read the output carefully, because it shows the pattern working as designed. The first two videos were blocked by the library, so both went through the API (api). On a laptop they would print library and cost nothing. The third ID does not exist. The library was blocked on it, so it fell back, and the API gave the real answer: This video is unavailable.

That call is also a small proof of the billing rule. The request contained two URLs in an earlier test, one real and one dead, and the response reported "count": 2, "billed": 1. Videos that return no transcript are not billed.

Doing it in bulk

If you are fetching many videos, do not call the fallback once per video. The API accepts a list of URLs in urls and returns one result per video, so you can send the blocked ones in a single request. A simple structure:

Loop over IDs with the library, collecting the ones that raise RequestBlocked.
Stop trying the library once the first few calls in a run are all blocked. If three in a row fail, the address is blocked and the rest will fail too. Send everything remaining straight to the API.
Send the collected IDs in batches to the API with outputFormats set to ["text"] (or ["text", "segments"] if you need timestamps).

Step 2 matters more than it looks. In our test a blocked attempt took under a second, which sounds harmless until you multiply it: 1,000 videos against a blocked address is more than ten minutes of guaranteed failures before the real path even starts.

For the full response shape, languages, SRT and VTT output, and playlist handling, see our YouTube transcript API guide for Python and Node.js and the API docs.

Other things worth checking before you blame the IP

A few errors look similar but have a different cause:

TranscriptsDisabled or NoTranscriptFound: the creator turned captions off, or no track exists in the language you requested. Check the languages list, and consider asking for a fallback language. No proxy fixes this.
AgeRestricted: the video is age-gated, so anonymous requests cannot read it. Skip it or handle it manually.
PoTokenRequired and YouTubeDataUnparsable: YouTube changed what its player returns, and the library needs an update. Run pip install -U youtube-transcript-api before anything else, and pin a version in production so an upgrade is a choice you make.
Works for some videos only: that is a hint that the failing ones have a video-level restriction rather than an address problem.

Choosing, in one paragraph

If you run it yourself, on your own connection, at low volume: keep using the library and do nothing else. If it must run in the cloud and you have a few hundred transcripts a month, use the fallback pattern above so you only pay when you are blocked. If YouTube transcripts are central to your product and your volume is large, price a managed proxy pool against a per-result API with your real numbers before deciding. And in every case, handle the "genuinely no captions" errors separately, because they are answers and not failures.

If you want to try the API path without writing code first, the free transcript tool runs the same extraction in your browser. For code, plans start at $29/month with 10,000 results included.

Collect company social links, emails and phones from a list of websites in Python

Siftwright — Mon, 19 Oct 2026 12:54:26 +0000

Company websites already publish their social profiles and contact details; the annoying part is collecting them for a whole list of domains. Here is a small Python script that does it through an Apify Actor billed per website ($2 per 1,000 sites, and sites that fail to load are not charged).

The script

import os, requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = ("https://api.apify.com/v2/acts/siftwright~contact-details-extractor/"
       "run-sync-get-dataset-items")

sites = ["https://www.python.org", "https://github.com", "https://www.mozilla.org"]
r = requests.post(URL, params={"token": TOKEN, "timeout": 90},
                  json={"startUrls": sites, "maxPagesPerDomain": 3}, timeout=120)
r.raise_for_status()

for row in r.json():
    print(row["domain"], row["status"], row["emails"], row["phones"],
          row["linkedin"], row["twitter"])

Real output

I ran exactly this on 2026-10-19. Each site returned status: "ok" with 3 pages scanned:

domain	linkedin	twitter / X	instagram	emails	phones
www.python.org	python-software-foundation	ThePSF	none	none	none
www.mozilla.org	mozilla-corporation	none	mozilla	none	none
github.com	github	github	github	none	none

Note the empty email and phone columns: these three sites simply don't publish an address on the pages scanned, and the Actor does not guess. Smaller company sites with a visible contact page are where emails and phones show up. Raise maxPagesPerDomain if the contact page is deeper in the site.

Pricing and limits

One charge per website that loads, $0.002 each; dead domains are free.
It reads public pages only: no logins, no proxies by default, so heavily protected sites may return a failed status.
Fields per site: emails, phones, facebook, twitter, linkedin, instagram, youtube, pagesScanned.

Try it: Contact Details Extractor on Apify. It also works from AI agents through Apify's MCP server. More tools: siftwright.com.

Build a daily news brief in Python with Google News search, billed per article

Siftwright — Sun, 18 Oct 2026 19:09:08 +0000

Need fresh news headlines in a script, a RAG pipeline or a media-monitoring job, without scraping Google or wiring up a paid news API? Here is a small Python brief generator built on the Google News Scraper Apify Actor. It is billed per article returned ($1.50 per 1,000), and an empty or failed search costs nothing.

Input

The Actor takes query, maxItems (1 to 200), language and country. Each result has title, link, source, publishedAt (ISO 8601), snippet and status.

The script

import os
from collections import Counter
import requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = "https://api.apify.com/v2/acts/siftwright~google-news-scraper/run-sync-get-dataset-items"

def news(query, n=10, language="en", country="US"):
    r = requests.post(URL, params={"token": TOKEN},
                      json={"query": query, "maxItems": n, "language": language, "country": country},
                      timeout=180)
    r.raise_for_status()
    return [a for a in r.json() if a.get("status") == "ok"]

watchlist = ["open source LLM", "vector database"]
for q in watchlist:
    items = sorted(news(q), key=lambda a: a["publishedAt"], reverse=True)
    print(f"\n## {q} ({len(items)} articles)")
    for a in items[:3]:
        print(f"- {a['publishedAt'][:10]} {a['source']}: {a['title']}\n  {a['link']}")
    print("top sources:", Counter(a["source"] for a in items).most_common(3))

run-sync-get-dataset-items starts the run and returns the dataset in one HTTP call, which suits short jobs like this one.

Real output

I ran it on 2026-10-18 with a watchlist of two queries and 10 articles each. Sorted newest first, the top hit for "open source LLM" was a HackerNoon piece dated 2026-10-18 on GPU memory for running open LLMs locally, and for "vector database" an article dated 2026-10-13 about Elastic's serverless vector database. Your results will differ as the news changes.

Two notes from testing:

link values are Google News redirect URLs. They open the article in a browser, but if you need the publisher URL you have to resolve the redirect yourself.
snippet is often just the title plus the outlet name, so don't count on it for body text.

Variations

Change language and country (for example de and DE) to read another Google News edition.
Schedule the same Actor on Apify and send the dataset to a webhook for daily monitoring.
It also works with AI agents through Apify's MCP server, so an assistant can run the search itself.

More details and a curl example: siftwright.com/google-news-api. Questions: support@siftwright.com.