DEV Community: Joshua Smith

Turn a list of prospect websites into honest page-speed reports for outreach

Joshua Smith — Wed, 14 Oct 2026 13:10:13 +0000

A cold email saying "your website is slow" gets deleted. One saying "your services page takes N seconds to show its main content on a phone, and two fixes would shorten that" gets read, because the owner can check it. Below: pages to pick, four numbers, a ranking script and the etiquette. Google's PageSpeed Insights API does the measuring; PageSpeed Insights Audit is an Apify Actor that runs it over a list of URLs.

Pick the pages that matter

Audit three or four pages per prospect, chosen as a customer would use the site:

The homepage, because the owner has seen it.
The page that earns money: the main service, product or booking page.
One page that attracts visitors, such as a popular blog post.
The contact or checkout page.

Use mobile, since Google Search ranks with mobile data. Open each URL in a browser: Google must fetch it publicly, so logins and staging fail.

Four numbers an owner can follow

Performance score (scores.performance), 0 to 100. 90 or more is "good", 50 to 89 "needs improvement".
Time until the main content shows (labMetrics.lcpMs, milliseconds, so divide by 1,000). Describe it as when the biggest thing on screen, usually a hero image or headline, has appeared.
What real visitors get (fieldData.overallCategory: FAST, AVERAGE or SLOW). It is Chrome user data, stronger evidence than the lab score. It is null for low-traffic pages, and originFallback: true means whole-site data. Say which you have.
The biggest fix (the first entry of opportunities, with its title and savingsMs).

If they disagree (a poor score but FAST visitor data), say so or drop the prospect.

Turn opportunities into plain English

Google words findings for developers ("Eliminate render-blocking resources"). Write one sentence per common id, about what a visitor notices:

render-blocking-resources: "Visitors wait on a blank screen while files load."
unused-javascript: "The page downloads code it never runs."

Present savings as estimates, show at most three, and do not add them up, because they overlap. For an id without a sentence, use Google's title rather than guess.

Run the audits

Input for two prospects. categories is narrowed because a speed report needs nothing else; includeAuditDetails adds opportunities:

{
    "urls": [
        "https://www.prospect-one.example/",
        "https://www.prospect-one.example/services",
        "https://www.prospect-two.example/",
        "https://www.prospect-two.example/book"
    ],
    "strategy": "mobile",
    "categories": ["performance"],
    "includeAuditDetails": true
}

Save it as input.json and make one call:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~pagespeed-insights-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" -d @input.json > results.json

The call waits for the run to finish, so keep it to a short list; for a longer one, start the run in the Console and download the Output tab as JSON. Allow about 10 minutes per 100 URLs at the default concurrency.

Rank prospects and write one summary each

Run python rank_prospects.py results.json (Python 3.9+). The script groups pages by hostname, keeps successful mobile records, ranks sites by mean performance score (lowest first) and writes one report-01-<site>.md-style file per site.

import json, sys
from collections import defaultdict
from urllib.parse import urlparse

PLAIN = {
    "render-blocking-resources": "Visitors wait on a blank screen while files load.",
    "unused-javascript": "The page downloads code it never runs.",
}
host = lambda u: urlparse("//" + u.split("//", 1)[-1]).hostname.removeprefix("www.")
score = lambda p: p["scores"]["performance"]

ok, bad = defaultdict(list), defaultdict(list)
for r in json.load(open(sys.argv[1], encoding="utf-8")):
    if r.get("strategy") == "mobile":
        usable = r.get("success") and None not in (score(r), r["labMetrics"]["lcpMs"])
        (ok if usable else bad)[host(r["url"])].append(r)

avg = lambda h: sum(map(score, ok[h])) / len(ok[h])
ranked = sorted((h for h in ok if min(map(score, ok[h])) < 90), key=avg)

for rank, h in enumerate(ranked, 1):
    pages = sorted(ok[h], key=score)
    out = [f"# {h}: where the site loses points", "",
           f"Google Lighthouse, mobile, {pages[0]['fetchedAt'][:10]}. Scores move a few points between runs.", "",
           "| Page | Score | Main content visible | Real visitors |", "|---|---|---|---|"]
    fixes = []
    for p in pages:
        field = p.get("fieldData") or {}
        real = field.get("overallCategory", "NONE").lower().replace("none", "not enough data")
        if field.get("originFallback"):
            real += " (whole site)"
        out.append(f"| {p['url']} | {score(p)} | {p['labMetrics']['lcpMs'] / 1000:.1f} s | {real} |")
        for o in (p.get("opportunities") or [])[:2]:
            ms = o.get("savingsMs") or 0
            note = f" Estimated saving: {ms / 1000:.1f} s." if ms else ""
            fixes.append((ms, f"- {p['url']}: {PLAIN.get(o['id'], o['title'])}{note}"))
    if fixes:
        out += ["", "## Biggest fixes", "", "Estimates overlap; do not add them up.", ""]
        out += [text for _, text in sorted(fixes, reverse=True)[:3]]
    open(f"report-{rank:02d}-{h}.md", "w", encoding="utf-8").write("\n".join(out) + "\n")
    print(rank, h, f"avg {avg(h):.0f}", f"failed {len(bad[h])}")

for h in sorted(set(ok) | set(bad)):
    if h not in ranked:
        print("not reported:", h, "(all 90+)" if h in ok else "(failed)")

Sites where every page scores 90 or more are skipped: there is no honest story. Failed pages are counted, not reported, so an unloadable page never reads as a bad score. Good pages stay in the table. Read each file before sending.

Be accurate, be kind

Public pages only. Never audit staging sites or anything behind a login.
Be accurate. Give the URL, date, device, and that it is a lab result; re-run borderline pages.
Do not exaggerate. Quote what you measured, and never turn seconds into money: you cannot see their analytics.
Do not scare. A score of 62 is "needs improvement", not "failing". Say what is fine.
Send once and accept a no. Check outreach rules where you and the prospect are based.

Cost and limits

$0.004 per audited page and device strategy. Worked example: 40 prospects with 3 pages each on mobile is 120 audits, or $0.48. Adding desktop makes 240 audits, $0.96. If 10 of the 120 pages cannot be loaded, 110 audits are billed: $0.44. No start-up charge, and a run stops at the maximum cost you set.

Variance. Lighthouse scores differ between runs by a few points.
Lab versus field. scores.performance is a lab estimate; real-user data is stronger but needs enough traffic.
Failed pages. Pages Google cannot load return free success: false records with an errorType such as dns or timeout.
API key. By default the Actor uses a built-in key shared by all its users: 25,000 requests a day and a few hundred a minute between them. When that runs out you get free rate-limited records; retry later, or paste your own free Google key into apiKey for a dedicated quota.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/pagespeed-insights-audit. PageSpeed Insights and Lighthouse are Google trademarks; the Actor is not affiliated with Google.

Track competitor YouTube uploads every week and send them to Slack or Google Sheets

Joshua Smith — Sat, 10 Oct 2026 13:09:17 +0000

Competitor tracking is usually a habit you skip: open a dozen channel pages, scroll to find what is new, guess how it did. A weekly job fixes that: once a week, fetch only the videos each channel published in the last seven days, store them, and compare with last week. Below: what to measure, how a date filter keeps cost low, a comparison script, and the limits.

What to track each week

Each video record from YouTube's official Data API carries enough for four signals:

Upload cadence. New videos per channel per week, from channelTitle and publishedAt.
Title patterns. Length, numbers, question marks and "how to" openers from title. One week proves little; a few months of rows show habits.
Length. durationSeconds. Shorts appear in channel uploads like any other video, so split formats before taking a median.
Views after N days. viewCount next to publishedAt and scrapedAt, so every number comes with the video's age when it was read.

Why the date filter keeps a weekly job cheap

A channel's uploads can be read newest first, so "what is new this week" means reading until you reach a video older than your cutoff, then stopping. Reading uploads costs about 1 quota unit per 50 videos. A search page costs 100, so do not build this on search.

Best Damn YouTube Scraper does that through its publishedAfter input. Paging stops as soon as older videos appear, and billing is per delivered video, so the back catalogue is never delivered and never billed. Without it, a weekly run at the default maxResults of 50 re-delivers each channel's latest 50 videos, mostly ones you already have, and bills them again.

The weekly input

Use this input and swap in your competitors' @handles, channel URLs or channel IDs:

{
    "startUrls": [
        "https://www.youtube.com/@YouTube",
        "https://www.youtube.com/@jawed"
    ],
    "publishedAfter": "7 days",
    "maxResults": 0
}

maxResults: 0 means no cap, which is safe because the date filter does the limiting. publishedAfter also takes a date such as 2026-01-31 or a period such as 2 weeks. Save the input as a task, put it on a weekly schedule in Apify, and add your own YouTube API key in apiKey (see limits).

From code

For your own cron, one call returns the dataset:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~youtube-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": ["https://www.youtube.com/@YouTube", "https://www.youtube.com/@jawed"], "publishedAfter": "7 days", "maxResults": 0 }' \
  > "week-$(date +%F).json"

A small week-over-week comparison

From week two, compare the two latest files with python compare.py this-week.json last-week.json:

import json, sys
from collections import defaultdict
from datetime import datetime
from statistics import mean, median

def ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

def load(path):
    rows = json.load(open(path, encoding="utf-8"))
    good = [r for r in rows if r.get("success") and r.get("liveStatus") != "upcoming"]
    return good, {r.get("errorType") for r in rows if not r.get("success")}

def summarise(rows):
    by_channel = defaultdict(list)
    for r in rows:
        by_channel[r["channelTitle"]].append(r)
    return {ch: {
        "uploads": len(vs),
        "minutes": round(median(v["durationSeconds"] for v in vs) / 60, 1),
        "title_chars": round(mean(len(v["title"]) for v in vs)),
        "age_days": round(median((ts(v["scrapedAt"]) - ts(v["publishedAt"])).days for v in vs)),
        "views": median(v["viewCount"] or 0 for v in vs),
    } for ch, vs in by_channel.items()}

this_rows, errors = load(sys.argv[1])
last_rows, _ = load(sys.argv[2])
if errors:
    print("WARNING: failed records", errors, "- this week may be incomplete")

now, before = summarise(this_rows), summarise(last_rows)
for ch in sorted(set(now) | set(before)):
    n, b = now.get(ch, {}), before.get(ch, {})
    print(ch, "uploads", b.get("uploads", 0), "->", n.get("uploads", 0), n)

It keeps only success: true records and skips liveStatus: "upcoming" streams, which have no views yet. It warns when any record failed: a quota error produces free rate-limited records instead of a failed run, so an empty week would otherwise look like a quiet one. Median age sits beside median views so you can see what the views belong to.

Compare like with like. A video one day old has fewer views than one six days old, so a change in median views mostly reflects when the channel posted. For a fairer number, take a second reading: bare video IDs are valid startUrls entries, so a week later you can send last week's IDs and read the same videos again. Each re-read is billed at the usual per-video price.

Send it to Slack or a sheet

Apify integrations move finished results onward without a download. The Actor's README lists Make, Zapier, n8n, Slack, Google Drive and Airbyte, with Slack posts and Google Sheets rows as example uses.

Slack: the Slack integration guide covers sending a task's results to a channel. Good for noticing new videos, poor for history.
Google Sheets: append rows every week so the history builds. Treat videoId as the key and drop duplicates, because a delayed run can overlap the previous window.
Your own endpoint: webhooks fire when a run succeeds, fails or times out. Point one at a small service that runs the comparison and posts the summary.

Cost and limits

$0.001 per video delivered, so 1,000 videos cost $1.00. Private, deleted and unavailable videos, invalid inputs and quota errors cost nothing, there is no start-up charge, and a run stops at the maximum cost you set.

Worked example, with assumed numbers: 20 channels publish 60 videos between them in a week. At 60 x $0.001, a weekly run costs $0.06, or $3.12 over 52 weeks. Without the filter, 20 channels at 50 videos each return up to 1,000 videos per run: $1.00 a week, $52.00 a year, nearly all of it videos you already have. Re-reading last week's 60 videos for the second reading adds another $0.06 a run.

Limits:

Quota. By default the Actor uses a built-in key whose 10,000 daily units are shared by all its users. For a scheduled workload, create your own free key in the Google Cloud Console with the YouTube Data API v3 enabled, and paste it into apiKey. When a quota runs out, the run stops cleanly and returns free rate-limited records; quotas reset at midnight Pacific Time. The SUMMARY record in the key-value store lists failures and quota units used.
Young videos. Early view counts are small and noisy. Compare videos of similar age.
Public data only. No login is used, so nothing from a channel owner's private analytics is available, and private, unlisted and members-only videos are not included. likeCount and commentCount are null when hidden, and YouTube rounds subscriber counts.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-scraper. YouTube is a trademark of Google, and the Actor is not affiliated with Google.

Find bug reports and feature requests in YouTube review-video comments

Joshua Smith — Fri, 09 Oct 2026 13:10:13 +0000

People who watch a long review or a setup tutorial are informed critics, and they leave their complaints in the comments. This post covers choosing the videos, filtering by keyword and date, de-duplicating, tagging comments as bug, feature request, question or praise, and ranking the result for a product or support team.

Where the feedback is

In rough order of usefulness:

Setup and tutorial videos. Viewers get stuck here: questions and bug reports.
Independent reviews and unboxings of your product. Feature gaps, price, build quality.
The same videos for a competitor. Their users' complaints are your opening; their praise is the bar.
Your own channel.

Search YouTube for the product name plus "review", "unboxing", "tutorial", "problem" and "vs" your main rival. Keep public videos with comments on, published after the product version you care about. Private and members-only videos give you nothing, and videos made for kids never have comments.

Filter before you collect

Keyword. The Actor's keyword option passes your words to YouTube's own comment search, so you pay only for matching comments. It applies to top-level comments, so a complaint inside a reply thread is not searched. It is also YouTube's search, not a regular expression, so do not assume exact substring matching; check a sample. Run one theme at a time (battery, then crash, then refund). A filtered run shows what people say, not what share of viewers say it; for shares, also pull an unfiltered sample.

Date. Set "Only comments after" to the release date of the version you are studying, as a date (2026-03-01) or a relative period such as 3 months. With "Newest first" the Actor stops paging once it reaches older comments. Use "Top comments" instead for long-standing complaints other viewers have voted up.

Collecting the comments

Best Damn YouTube Comments Scraper is an Apify Actor that reads comments through the official YouTube Data API v3: no browser, no login. Give it video URLs (or a channel, and it reads that channel's most recent videos):

{
    "startUrls": [
        "https://www.youtube.com/watch?v=VIDEO_ID_1",
        "https://www.youtube.com/watch?v=VIDEO_ID_2"
    ],
    "searchTerms": "battery",
    "sortBy": "time",
    "publishedAfter": "3 months",
    "maxComments": 200,
    "authorData": "hash"
}

maxComments is per video. authorData: "hash" swaps the commenter's channel ID for a stable hash and drops name and avatar. Save the block as input.json and call the API:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~youtube-comments-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" -d @input.json

Each record is one comment. Fields used below: success, commentId, commentUrl, text, likeCount, isByVideoOwner, authorChannelId. Unreadable videos come back as free records with success: false and an errorType such as comments-disabled.

De-duplicate, tag and group

De-duplicate because the same video can arrive twice (as a URL and through a channel), merged keyword runs overlap, and copy-pasted comments inflate a theme. Match on commentId and on normalised text. Skip comments where isByVideoOwner is true; the creator is not the audience.

Tag with rules where the first match wins (bugs, then requests, questions, praise) and group by theme. Counting distinct authorChannelId values works with hashed IDs, so you count people without knowing who they are.

import json, re
from collections import defaultdict
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("josh99smith/youtube-comments-scraper").call(
    run_input=json.load(open("input.json")))

seen, groups = set(), defaultdict(lambda: {"people": set(), "likes": 0, "links": []})
TAGS = [  # first match wins
    ("bug", r"crash|broke|stopped working|doesn'?t work|won'?t (start|connect)"),
    ("request", r"i wish|should (have|add)|please add|would be (nice|great)"),
    ("question", r"\?|how (do|can) i"),
    ("praise", r"love|great|works well"),
]
THEMES = {"battery": r"batter|charg", "app": r"\bapp\b|firmware", "price": r"price|expensive"}

for c in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not c["success"] or c["isByVideoOwner"]:
        continue
    t = c["text"].lower()
    key = re.sub(r"\W+", " ", t).strip()
    if c["commentId"] in seen or key in seen:
        continue
    seen.update([c["commentId"], key])
    tag = next((n for n, p in TAGS if re.search(p, t)), "other")
    for theme, p in THEMES.items():
        if re.search(p, t):
            g = groups[(tag, theme)]
            g["people"].add(c["authorChannelId"])
            g["likes"] += c["likeCount"]
            g["links"].append(c["commentUrl"])

for (tag, theme), g in sorted(groups.items(), key=lambda kv: (-len(kv[1]["people"]), -kv[1]["likes"])):
    print(tag, theme, len(g["people"]), g["likes"], g["links"][:2])

Replace THEMES with your product's vocabulary. When keyword themes stop being enough, embed each text with any sentence-embedding model, cluster the vectors (k-means is fine) and name each cluster by reading its top comments.

Turn it into a prioritised list

Rank each tag-and-theme group by distinct commenters, then total likeCount; a like is another viewer saying "me too". Each row needs the theme, tag, commenter count, likes, commentUrl links for context, and an owner: bugs to engineering or support, requests to product, repeated questions to whoever writes docs or tutorials, praise to marketing as things not to break.

Read a random sample from every tag before sharing. Keyword rules misread sarcasm: "love how it never connects" lands under praise here. Commenters are self-selected, not your customer base, so treat the list as a starting point.

Cost and limits

Price. $0.0004 per delivered comment or reply ($0.40 per 1,000). Ten videos capped at 200 comments each is at most 2,000 comments, so at most $0.80. Videos with comments off, private or deleted videos, invalid URLs and quota errors are free; there is no start fee and a run stops at the maximum cost you set.
Quota. By default the Actor uses a built-in API key shared by all its users, with a daily quota of 10,000 units and one unit per page of up to 100 comments. If it runs out, the run stops cleanly and the remaining videos come back as free rate-limited records; the quota resets at midnight Pacific Time. Your own free key from the Google Cloud Console, pasted into apiKey, gives you roughly a million comments a day. The SUMMARY record in the key-value store lists comments per video, failures and quota units used.
Fewer comments than shown. YouTube's count includes replies and held or removed comments; the API returns what is currently public.
Public data only. No login, so nothing private or members-only.

Treat comments as aggregate research

Comments are personal data under GDPR and similar laws, so you need a legitimate purpose. Work with themes, not people. Use hash or omit for authorData, do not contact commenters, and if you quote a comment in a report, leave the name out.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-comments-scraper. YouTube is a trademark of Google; the Actor is not affiliated with Google.

Build a YouTube influencer shortlist from subscriber counts and recent views

Joshua Smith — Thu, 08 Oct 2026 13:10:09 +0000

Most sponsorship shortlists start the same way: open a channel, glance at the subscriber count, move on. That works for two creators. For forty, you want a sortable table that answers a better question than "who is biggest?"

Why subscriber count alone misleads

A subscriber total is a running tally of everyone who ever clicked subscribe. It says nothing about how many of them see the next video. A big, quiet channel can reach fewer people per upload than a smaller one that publishes steadily.

These are the numbers that tend to matter for a sponsorship:

Typical views per recent video. This is the reach you are buying. Use the median of the last ten or so uploads, not the average: one viral video drags an average far above a normal upload.
Views per video as a share of subscribers. If recent videos get views close to the subscriber count, the audience still shows up. A low ratio is a flag, not a verdict: some channels grow through search rather than subscribers, so compare creators in the same niche.
Upload consistency. The gap between uploads tells you whether there will be a fresh video to carry your sponsor next month.
Engagement per view. Likes and comments divided by views is noisy, so use it as a tie-breaker, not a ranking.

The vanity numbers are the subscriber total on its own, lifetime views (old hits dominate), and a creator's single best video. I am not quoting benchmark percentages: they differ by niche, and nothing in the data below supports one. Compare the creators on your list against each other. Running everyone through the same formulas makes the odd ones out obvious, and gives you something to show whoever approves the budget.

Step 1: channel stats for the whole list

The YouTube Data API publishes these numbers, and the Best Damn YouTube Scraper Actor calls it for a whole list at once. Set Output to Channels only and you get one record per channel: title, handle, url, subscriberCount, viewCount, videoCount, country, publishedAt, description, keywords, thumbnailUrl and uploadsPlaylistId.

This input pulls one row per channel. @handles, channel URLs and channel IDs all work.

{
    "startUrls": [
        "https://www.youtube.com/@YouTube",
        "https://www.youtube.com/@jawed",
        "https://www.youtube.com/channel/UCBR8-60-B28hp2BmDPdntcQ"
    ],
    "outputMode": "channels"
}

The same run over HTTP:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~youtube-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": ["https://www.youtube.com/@YouTube", "https://www.youtube.com/@jawed"], "outputMode": "channels" }'

Download the dataset as CSV or Excel, or send it to Google Sheets.

Two things to expect. YouTube rounds public subscriber counts to three significant figures (for example 46,300,000), so rank with them, do not measure with them. And channels can hide the count entirely, which comes back as null. Do not read that as zero: leave the cell blank and rank that creator on views instead. A handle that cannot be found comes back as a free record, such as {"success": false, "input": "@thisHandleDoesNotExist", "errorType": "not-found"}, so a typo never silently vanishes.

Step 2: add recent videos

The channel record only has lifetime totals. For recent reach and upload frequency, run the same list in the default Videos mode with maxResults set to 10. Channels are read newest first, so you get each creator's latest ten uploads, with viewCount, likeCount, commentCount, publishedAt, durationSeconds, channelUrl and channelSubscribers.

Clean up before comparing:

likeCount and commentCount are null when the owner hides likes or turns comments off. Skip those rows rather than counting zero.
Scheduled streams appear with liveStatus: "upcoming". Drop them.
Shorts are included, and durationSeconds lets you separate them from long-form. Mixing them blurs the median.
Very recent uploads are still collecting views, so ignore the last few days.

Step 3: compute the metrics

On the channels sheet, with subscriberCount in column B, viewCount in C and videoCount in D:

E2: =IFERROR(C2/D2, "")        lifetime views per video
F2: =IFERROR(E2/B2, "n/a")     that, as a share of subscribers ("n/a" when hidden)

That is a rough first pass, since lifetime numbers lean on old hits. For recent-video metrics, a short script beats spreadsheet formulas:

from collections import defaultdict
from datetime import datetime
from statistics import median
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("josh99smith/youtube-scraper").call(run_input={
    "startUrls": ["https://www.youtube.com/@YouTube", "https://www.youtube.com/@jawed"],
    "maxResults": 10,
})

by_channel = defaultdict(list)
for v in client.dataset(run["defaultDatasetId"]).iterate_items():
    if v["success"] and v["liveStatus"] != "upcoming":
        by_channel[v["channelUrl"]].append(v)

rows = []
for channel, vids in by_channel.items():
    views = [v["viewCount"] for v in vids]
    dates = sorted(datetime.fromisoformat(v["publishedAt"].replace("Z", "+00:00")) for v in vids)
    gap_days = (dates[-1] - dates[0]).days / max(len(dates) - 1, 1)
    subs = vids[0]["channelSubscribers"]
    like_rates = [v["likeCount"] / v["viewCount"] for v in vids if v["likeCount"] is not None and v["viewCount"]]
    rows.append((channel, median(views), median(views) / subs if subs else None,
                 round(gap_days, 1), median(like_rates) if like_rates else None))

# channel, median views, views / subscribers, days between uploads, like rate
for row in sorted(rows, key=lambda r: r[1], reverse=True):
    print(row)

Sort by median views, then read the ratio and the upload gap beside it. The top of that list is your shortlist.

Cost and limits

Channels only: $0.001 per channel delivered, so 100 channels cost $0.10.
Videos mode: $0.001 per video, 1,000 for $1.00. A hundred channels at ten videos each is 1,000 videos, or $1.00.
Private, deleted and unavailable items, invalid inputs and quota errors cost nothing. There is no start fee, and a run stops at the maximum cost you set.
It uses the official YouTube Data API v3, not page scraping, on a shared API key with a daily quota of 10,000 units. Reading channels and videos costs about 1 unit per 50 videos, so lists like this barely register. Searches cost 100 units per page, so the shared key allows up to 5 search terms and 100 results per term per run. For heavy use, create a free key in the Google Cloud Console and paste it into YouTube API key. When a quota runs out, the run stops cleanly with free rate-limited records; quotas reset at midnight Pacific Time.
It returns public channel and video data only. It does not look for emails or other contact details and collects nothing about viewers. The description text comes back as the creator published it.
Audience location, brand fit and whether views are genuine are not in this data. The metrics narrow a list; they do not replace looking at the content.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-scraper. YouTube is a trademark of Google, and the Actor is not affiliated with Google.

Sentiment analysis on YouTube comments with Python: handling sarcasm, emoji and spam

Joshua Smith — Wed, 07 Oct 2026 13:09:50 +0000

Comments are the cheapest audience research you have, and the hardest to read. A video with a few thousand of them is too many to skim, and a single sentiment score from a library will mislead you. This article covers why the scores go wrong, what to do about it, and then a short pipeline: collect comments from the official API, classify them in Python, summarise per video.

Why comment sentiment is noisy

Most sentiment tools were built on reviews or tweets and then pointed at YouTube. Four things break them:

Sarcasm. "Great, another ten-minute sponsor read" contains a positive word and a negative meaning. Lexicon tools score the words, not the intent.
Emoji and slang. "This is sick" and a row of skull emoji are praise. Some tools cover part of this, many ignore it.
Short text. "first" and "lol" carry almost no sentiment, but they still count in your percentages.
Spam. Link drops, "check my channel" and copy-pasted replies are not audience opinion.

There is a fifth, quieter problem: context. A reply of "same" means nothing without the comment above it.

What to do about it

Clean before you score. Drop the creator's own comments, anything with a link, very short comments and exact duplicates.
Compare, do not read absolutes. A tool that is biased in a consistent way still shows whether this video is better or worse than your last five.
Weight by likes. A negative comment with many likes speaks for more viewers than one with none.
Check against a hand-labelled sample. Label 50 comments yourself and see how often the tool agrees. If it is poor, swap in a better model or an LLM with a fixed three-label prompt and run the same check.
Read what is behind the number. Always look at the most-liked comments in each bucket.

Getting the comments

The official YouTube Data API v3 returns comments, but you handle paging, turning a channel into its recent videos, replies and quota yourself. Best Damn YouTube Comments Scraper is an Apify Actor that wraps the API and returns one flat record per comment. This input collects comments for sentiment work, from one video plus a channel's three most recent uploads:

{
    "startUrls": ["https://www.youtube.com/watch?v=jNQXAC9IVRw", "https://www.youtube.com/@YouTube"],
    "maxVideosPerChannel": 3,
    "maxComments": 500,
    "authorData": "hash"
}

Replace the channel with your own @handle. maxComments applies per video, maxVideosPerChannel sets how many recent uploads are read, and the default sort is top comments, which the Actor's docs recommend for sentiment work. authorData: "hash" replaces each commenter's channel ID with a stable hash and drops name and avatar, so you can still see when one account posts again and again without storing who it is.

Save that as input.json and call the API:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~youtube-comments-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json -o comments.json

Each delivered record has text, likeCount, replyCount, isReply, parentCommentId, isByVideoOwner, publishedAt, videoId, videoTitle and commentUrl, plus success: true. isByVideoOwner still works with pseudonymised authors.

Classify and summarise

This uses VADER (pip install vaderSentiment), a lexicon tool that handles some slang and emoji. It is generic: swap sia.polarity_scores for any model.

import json
from collections import Counter, defaultdict
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

rows = [r for r in json.load(open("comments.json")) if r["success"]]

seen, comments = set(), []
for r in rows:  # clean
    text = r["text"].strip()
    if r["isByVideoOwner"] or "http" in text or len(text) < 4 or text.lower() in seen:
        continue
    seen.add(text.lower())
    comments.append(r)

sia = SentimentIntensityAnalyzer()
for r in comments:  # score: +/-0.05 is VADER's usual cut-off
    c = sia.polarity_scores(r["text"])["compound"]
    r["label"] = "positive" if c >= 0.05 else "negative" if c <= -0.05 else "neutral"

by_video = defaultdict(list)
for r in comments:
    by_video[r["videoTitle"]].append(r)

for title, items in by_video.items():  # summarise
    count, weight = Counter(), Counter()
    for r in items:
        count[r["label"]] += 1
        weight[r["label"]] += 1 + r["likeCount"]
    print(title, len(items), "comments")
    for lab in ("positive", "neutral", "negative"):
        print(f"  {lab:9}{count[lab] / len(items):4.0%} of comments {weight[lab] / sum(weight.values()):4.0%} like-weighted")

negative = sorted((r for r in comments if r["label"] == "negative"), key=lambda r: r["likeCount"], reverse=True)
for r in negative[:5]:  # read the evidence
    print(r["likeCount"], r["text"][:100], r["commentUrl"])

The output is a three-line table per video and the five most-liked negative comments with links back to YouTube. Run it before and after a change, such as a new format or sponsor, and compare. I am not showing results here because they depend entirely on your channel.

Cost and limits

Price: $0.40 per 1,000 comments, charged per delivered comment or reply. The input above reads at most 4 videos (one video plus 3 from the channel) at 500 comments each, so up to 2,000 comments, or $0.80. You can set a maximum cost per run.
Free: videos with comments off, private or deleted videos, invalid URLs and quota errors are not charged. A video with comments off comes back as {"success": false, "errorType": "comments-disabled", ...}. Made-for-kids videos never have comments.
Quota: by default the Actor uses a shared API key. Each key has a daily quota of 10,000 units, and one page of up to 100 comments costs one unit, shared with every other user. For regular or large runs, create your own free key in Google Cloud Console and paste it into the apiKey input, which the docs put at roughly a million comments a day. When a quota runs out, the run stops cleanly and the remaining videos are returned as free rate-limited records. It resets at midnight Pacific Time.
Replies: off by default. Set includeReplies to true to get them; they carry isReply and parentCommentId, count toward maxComments and are billed like any comment.
Coverage: public videos only, no login, so no members-only content. You may get fewer comments than the count YouTube shows, because that count includes replies and comments held for review or removed as spam.
Privacy: comments are personal data under GDPR and similar laws. Use hash or omit for authorData if you do not need to know who wrote what.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-comments-scraper. YouTube is a trademark of Google; the Actor is not affiliated with Google.

Export every video from a YouTube channel, with views and subscribers, to a spreadsheet

Joshua Smith — Tue, 06 Oct 2026 20:25:47 +0000

To see what a competitor's channel really publishes, you want every video in one table: upload date, length, views, likes, comments, and the channel's subscriber count. YouTube Studio covers your own channel, not someone else's, and copying titles from a channel page by hand stops being fun around video 40. The official YouTube Data API v3 returns all of it for any public channel. Here is how it works, where it bites, and a no-code way to run it.

What the YouTube Data API gives you

The YouTube Data API v3 is free and returns public data as JSON. Four resources matter for this job:

channels: title, subscriber count, total views, video count, and the ID of the channel's uploads playlist.
playlistItems: the videos in a playlist, 50 per page.
videos: details and statistics for up to 50 video IDs per call.
search: keyword search. It can list a channel's videos too, but it is the wrong tool for that, as the next section explains.

Every API key has a daily quota of 10,000 units. Reading channels, playlists and videos is cheap, about 1 unit per 50 videos. Search is not: each search page costs 100 units.

Why you list uploads through the uploads playlist

The obvious approach is search filtered by channel. It costs 100 units a page, and YouTube caps search results at 500, so you cannot even reach the end of a large channel.

Every channel instead has an automatically maintained uploads playlist. Its ID is in the channels response under contentDetails.relatedPlaylists.uploads (it is the channel ID with the UC prefix swapped for UU). playlistItems pages through it newest first, with no 500-result cap, at a small fraction of search's cost.

The catch is that playlistItems returns IDs and titles, not statistics. So the loop is:

Call channels with the handle or channel ID to get the uploads playlist ID and the subscriber count.
Call playlistItems page by page until there is no nextPageToken.
Send the video IDs to videos in batches of 50 for views, likes, comments, duration and tags.
Add the channel's stats to every row.

Expect rough edges in the raw responses. Counts arrive as strings, duration is ISO 8601 (PT1H2M3S), likes and comments are missing when the owner hides them, and private or deleted videos show up in the playlist but not in videos.

Which columns are worth keeping

For competitor research a handful of fields does most of the work:

publishedAt and durationSeconds: upload cadence, and short versus long formats. Shorts are ordinary uploads, so length is how you separate them.
viewCount, likeCount, commentCount: raw performance. Compare channels per subscriber, not in absolute numbers.
channelSubscribers: the denominator for views per subscriber.
title and tags: the topics and phrasing the channel bets on.
category and liveStatus: keep live streams from skewing your averages.

These are today's numbers. The API does not return history, so to track growth, rerun the export on a schedule and keep every run with its scrapedAt timestamp.

The no-code version

Best Damn YouTube Scraper is an Apify Actor that runs the loop above against the official API (channels, playlistItems, search and videos). No browser runs and no login is used, so nothing breaks when YouTube changes its layout.

Paste channel URLs, @handles or channel IDs into YouTube URLs.
Set Max results per channel, playlist or search to 0, which means all videos, newest first.
Click Start, then download the Output tab as CSV or Excel, or connect an integration to push it to Google Sheets.

Even channels with thousands of uploads finish in a few seconds. Example input for two channels:

{
    "startUrls": ["https://www.youtube.com/@YouTube", "https://www.youtube.com/@jawed"],
    "maxResults": 0,
    "maxDescriptionLength": 200
}

maxDescriptionLength trims descriptions so spreadsheet cells stay readable (0 keeps the full text). Add "publishedAfter": "90 days" to limit the export to recent uploads. If a video appears under more than one input, it is returned and billed once.

One record per video, trimmed from the real example in the Actor's docs:

{
    "success": true,
    "videoId": "jNQXAC9IVRw",
    "title": "Me at the zoo",
    "publishedAt": "2005-04-24T03:31:52Z",
    "durationSeconds": 19,
    "viewCount": 435887912,
    "likeCount": 19914774,
    "commentCount": 10626831,
    "channelTitle": "jawed",
    "channelSubscribers": 6610000
}

If you only need one row per channel, set "outputMode": "channels" and you get subscriberCount, viewCount, videoCount and country per channel instead of a list of videos.

From code

One HTTP call runs the Actor and writes the CSV:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~youtube-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN&format=csv" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": ["https://www.youtube.com/@YouTube"], "maxResults": 0 }' \
  -o channel-videos.csv

The same input works with the apify-client packages for Python and JavaScript.

Cost and limits

$0.001 per video delivered, so 1,000 videos cost $1.00 and a 500-video channel costs $0.50. In Channels only mode the price is the same per channel. There is no start-up charge, and private, deleted and unavailable videos, invalid inputs and quota errors cost nothing. Each run stops at the maximum cost you set.

Quota is the limit to plan around. By default the Actor uses a built-in API key shared by all its users, with the usual 10,000 units a day. Whole-channel exports are cheap, but search is expensive, so the shared key allows up to 5 search terms and 100 results per term per run. For heavy or scheduled work, create your own free key in the Google Cloud Console, enable "YouTube Data API v3", and paste the key into YouTube API key. When a quota runs out, the run stops cleanly and returns free rate-limited records. Quotas reset at midnight Pacific Time.

Other limits to know:

Only public data. Private, unlisted, members-only and removed videos are not included, and nothing from a logged-in account is available.
likeCount and commentCount are null when the owner hides likes or turns comments off.
Subscriber counts are rounded by YouTube to three significant figures, and channels that hide the count return null.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/youtube-scraper. YouTube is a trademark of Google, and the Actor is not affiliated with Google.

RSS, Atom and JSON Feed to clean JSON, with feed discovery and only-new-items mode

Joshua Smith — Wed, 30 Sep 2026 13:10:02 +0000

Feeds are the most reliable structured data on the web, and every parser you write handles the first three feeds fine and breaks on the fourth. Here is what breaks and a way to stop caring.

Why feed parsing is messier than it looks

Four formats: RSS 2.0, RSS 1.0 (RDF), Atom, and JSON Feed.
Content lives in description, content:encoded, Atom content, or content_html, often wrapped in CDATA with HTML entities escaped once or twice.
Authors are author, dc:creator, or Atom author/name, sometimes only at feed level.
Dates come in RFC 822, ISO 8601, and things that are neither.
Images hide in media:content, itunes:image, an enclosure, or the first <img> in the body.
Many people give you a website URL, not the feed URL, so you need <link rel="alternate"> discovery and a fallback probe of /feed, /rss.xml, /atom.xml.

The two-minute version

RSS Feed to JSON takes feed URLs or plain website URLs, discovers the feed, and returns one normalised record per item with HTML and plain-text content, categories, enclosures and a lead image.

Paste feed or website URLs into Feed or website URLs.
Optionally set Max items per feed and Published after.
Click Start. Items appear within seconds; send them to Google Sheets, Slack, Make, Zapier or a webhook from the Integrations tab.

For a scheduled digest, add a Schedule and switch on Only new items since the last run: the Actor remembers delivered items in a state store and marks each record isNew, and skipped items are not billed.

From code

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "<YOUR_API_TOKEN>" });
const run = await client.actor("josh99smith/rss-feed-to-json").call({
    feedUrls: ["https://blog.apify.com/rss/", "https://hnrss.org/frontpage", "https://www.theverge.com"],
    maxItemsPerFeed: 50,
    plainText: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => `${i.feedTitle}: ${i.title}`));

What you get back

{
    "feedUrl": "https://blog.apify.com/rss/",
    "feedTitle": "Apify Blog",
    "feedType": "rss",
    "id": "64884d212b8e8a0001df7576",
    "title": "12 LangChain alternatives for building AI agents in 2026",
    "url": "https://blog.apify.com/langchain-alternatives/",
    "author": "Theo Vasilis",
    "publishedAt": "2026-09-15T12:00:00.000Z",
    "summary": "LangChain is a powerful framework for developing LLM apps, but ...",
    "contentText": "LangChain is still the most widely used framework ...",
    "categories": ["AI", "Tool comparisons"],
    "imageUrl": "https://storage.ghost.io/.../langchain-alternatives-1.png",
    "isNew": true
}

A page with no discoverable feed returns one free { "success": false, "errorType": "not-found" } record.

Cost and limits

$0.0005 per delivered item (2,000 items = $1), no start fee, failed feeds and monitor-mode skips free, cost cap per run. Feeds behind logins or bot protection are reported as blocked, not bypassed.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/rss-feed-to-json to your MCP client and ask "what are the latest 10 posts on blog.apify.com?".

Disclosure: I built this Actor. Source: github.com/josh99smith/rss-feed-to-json.

Get every URL of a website from its sitemaps, without crawling it

Joshua Smith — Tue, 29 Sep 2026 13:10:44 +0000

Before a migration, a content audit, or a crawl budget review you want the list of pages a site itself says it has. Crawling gets you there slowly and noisily. The sitemap gets you there in seconds, if you handle the format properly.

What a sitemap reader has to handle

Discovery: robots.txt may list several Sitemap: lines; if not, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt.
Indexes: large sites publish a sitemap index pointing at dozens of child sitemaps, sometimes nested.
Compression: .xml.gz is common on big sites.
Extensions: xhtml:link rel="alternate" hreflang="..." for language versions, image:image entries, lastmod, changefreq, priority.
Size: a single file can hold 50,000 URLs; an ecommerce site can have millions across files, so you need a cap and streaming parsing or you run out of memory.

The two-minute version

Sitemap URL Extractor does discovery, follows indexes recursively, decompresses gzip, parses the extensions, and returns one record per page URL.

Paste website URLs or sitemap URLs into Website or sitemap URLs.
Set Max URLs per site (default 5,000) and optionally Include / Exclude URL patterns such as **/blog/** or *.pdf.
Click Start; export CSV, Excel or JSON, or feed the URL list into another Actor.

Switch on List sitemap files only to get one row per sitemap file (URL count, depth, lastmod) instead, which is the quickest way to map a large site's sitemap structure.

From code

curl -X POST "https://api.apify.com/v2/acts/josh99smith~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://www.apify.com", "https://blog.apify.com/sitemap.xml"], "maxUrlsPerSite": 5000, "excludePatterns": ["*.pdf"] }'

What you get back

{
    "site": "https://www.apify.com",
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "url": "https://apify.com/store",
    "lastmod": "2026-09-17T06:12:41.000Z",
    "changefreq": "daily",
    "priority": 0.8,
    "alternates": [{ "hreflang": "en", "href": "https://apify.com/store" }],
    "imageCount": 0
}

A site with no sitemap, or one hidden behind bot protection, comes back as a single free { "success": false, "errorType": "not-found" | "blocked" | "dns" } record.

Cost and limits

$0.0002 per extracted URL: a 5,000-URL site costs $1. Failed sites and unloadable sitemap files are free, there is no start fee, and both the per-site cap and the run's cost cap protect you from a surprise on a million-URL site. The Actor reads sitemap files only and never fetches the pages themselves.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/sitemap-url-extractor to your MCP client and ask "list every URL under /blog/ on blog.apify.com".

Disclosure: I built this Actor. Source: github.com/josh99smith/sitemap-url-extractor.

Long-tail keyword research from Google Autocomplete, hundreds of seeds at once

Joshua Smith — Mon, 28 Sep 2026 13:11:24 +0000

Type best crm for into Google and the drop-down shows ten to fifteen phrases real people search for. Keyword tools charge a subscription to collect those at scale. The endpoint behind the box is public and free; the work is in the expansion and de-duplication.

How the suggest endpoint works

https://suggestqueries.google.com/complete/search?client=chrome&q=best+crm+for&gl=us&hl=en returns JSON with the suggestions, their type and a relevance score. Tools get more out of it by expanding each seed:

Alphabet: best crm for a, best crm for b, ... best crm for z
Questions: how best crm for, what best crm for, why ...
Prepositions: best crm for vs, best crm for with, ... without
Numbers: best crm for 2, ... 10
Depth 2: re-query every suggestion found at depth 1

One seed with alphabet, question and preposition expansions at depth 2 is roughly 190 requests and can yield around 1,000 unique phrases. Send those too fast from one IP and you get 429s, so you need small delays, low concurrency and retries, plus per-seed de-duplication.

The two-minute version

Google Autocomplete Scraper does the expansion, throttling and de-duplication for any country and language.

Enter your seed keywords in Seed keywords, one per line. Partial phrases work best.
Set Country and Language for the market.
Tick the expansions you want and optionally set Depth to 2. Click Start, then export CSV/Excel or send to Google Sheets.

From code

curl -X POST "https://api.apify.com/v2/acts/josh99smith~google-autocomplete-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "keywords": ["best crm for", "how to start a podcast"], "country": "us", "language": "en", "expandQuestions": true, "maxSuggestionsPerKeyword": 50, "depth": 1 }'

What you get back

One record per unique suggestion:

{
    "seedKeyword": "best crm for",
    "query": "what best crm for",
    "suggestion": "what's the best crm for small business",
    "position": 1,
    "relevance": 601,
    "suggestType": "QUERY",
    "modifierType": "question",
    "modifier": "what",
    "country": "us",
    "language": "en"
}

relevance is Google's own score, useful for ranking suggestions within a seed. modifierType tells you which expansion produced the phrase, so question-style queries are one filter away.

Cost and limits

$0.003 per seed keyword, flat, whatever expansions and depth you enable and however many suggestions come back; 15 or 1,000 cost the same. A keyword whose requests fail on rate limiting or a network error is not charged and comes back with an errorType. Per-suggestion pricing elsewhere runs $0.0005 to $0.001 each, which for a 1,000-suggestion seed is $0.50 to $1.00 instead of $0.003.

Suggestions change by country, language and time, so two runs are not identical, and rare phrases may return nothing. No search result pages are scraped and no personal data is collected.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/google-autocomplete-scraper to your MCP client and ask for suggestions around any phrase; the agent can group them into topics for you.

Disclosure: I built this Actor. Source: github.com/josh99smith/google-autocomplete-scraper.

Full-page website screenshots and PDFs from a URL list, with cookie banners hidden

Joshua Smith — Sun, 27 Sep 2026 13:10:13 +0000

page.screenshot({ fullPage: true }) is one line in Playwright. Getting a screenshot that looks like what a visitor sees takes about two hundred more. Here is what goes wrong and a way to skip it.

Why naive full-page screenshots look broken

Lazy-loaded images never load because nothing scrolled. You get grey boxes below the fold.
Sticky headers and chat widgets are position: fixed, so in a stitched full-page capture they repeat on every screen or cover the content.
Cookie banners sit on top of everything. Clicking "accept" on someone else's site on a user's behalf is not something you want a bot doing; hiding the overlay with CSS is.
Emoji and CJK text render as boxes on a bare Linux container without the right fonts.
Network idle never arrives on pages with analytics beacons, so waitUntil: "networkidle" hangs until the timeout.

Each of these is a small fix. Together, plus device presets, retries, file storage and a sane error record for pages that fail, they are a project.

The two-minute version

Website Screenshot API runs headless Chromium on Apify with all of the above handled: it scrolls to trigger lazy loading, pins fixed elements so they appear once, hides consent pop-ups with CSS (nothing is clicked), ships fonts for emoji and CJK, and reports failed pages as free records.

Paste your URLs into Website URLs.
Pick a Device preset (desktop, laptop, tablet, mobile or a custom viewport), PNG or JPEG, and Full page or first screen only.
Optionally tick Also render a PDF, add Hide elements selectors for chat widgets, set Dark mode, or under Advanced set Wait for element, cookies and extra headers. Click Start.

Images land in the run's key-value store; the dataset record links to each file.

From code

curl -X POST "https://api.apify.com/v2/acts/josh99smith~website-screenshot-api/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://apify.com", "https://www.wikipedia.org"], "device": "laptop", "format": "jpeg", "quality": 80, "fullPage": true }'

What you get back

{
    "url": "https://www.wikipedia.org",
    "success": true,
    "statusCode": 200,
    "title": "Wikipedia",
    "screenshotUrl": "https://api.apify.com/v2/key-value-stores/AbCdEf123/records/screenshot-002-www-wikipedia-org.jpg",
    "format": "jpeg",
    "width": 1366,
    "height": 1101,
    "fullPage": true,
    "device": "laptop",
    "sizeBytes": 172806,
    "renderTimeMs": 1139,
    "warnings": []
}

screenshotUrl is a direct link you can drop into an <img> tag, a Slack message or a report. Pages that fail come back as { "success": false, "errorType": "dns" | "timeout" | "blocked" | ... }.

Cost and limits

$0.005 per screenshot and $0.004 per PDF when enabled. Failed pages are free, there is no start fee, and each run stops at your cost cap. Pages behind a login need you to supply your own cookies (an input field exists for that); the Actor does not log in anywhere and does not bypass bot protection, it reports it as blocked.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/website-screenshot-api to your MCP client and ask "take a full-page mobile screenshot of apify.com". The agent gets the image URL back.

Disclosure: I built this Actor. Source: github.com/josh99smith/website-screenshot-api.

A remote jobs API from five public job boards, normalised into one schema

Joshua Smith — Sat, 26 Sep 2026 13:21:34 +0000

If you are building a job board, a Slack alert, or a dataset of remote salaries, the good news is that several remote job boards publish public JSON APIs or RSS feeds and explicitly allow reuse with attribution. The bad news is that each one has its own shape.

The public sources

RemoteOK: https://remoteok.com/api (JSON, attribution required)
Remotive: https://remotive.com/api/remote-jobs (JSON)
Himalayas: https://himalayas.app/jobs/api (JSON)
Jobicy: https://jobicy.com/api/v2/remote-jobs (JSON)
We Work Remotely: RSS, one main feed plus a feed per category

Salary appears as "$170k - $200k" in one, as two integers in another, and not at all in a third. Location is free text: "Northern America, LATAM, Europe". Employment type is full_time, Full-Time or missing. Descriptions are HTML. And the same listing shows up on three boards.

The two-minute version

Remote Jobs Scraper & Aggregator API calls the five sources, parses salaries into numbers, maps locations to a compact remoteRegion, unifies employment types, adds a plain-text description, and keeps each title + company pair once.

Keep all five Job boards selected, or untick some.
Add Keywords (matched on title, company and tags) and Categories, set Posted within (days).
Click Start. A default run (five boards, 200 jobs each, last 30 days) yields 500 to 700 unique listings in under a minute.

For alerts: add a Schedule (every 3 to 6 hours is plenty) and switch on Only new items since the last run. The Actor remembers what it has delivered in a named state store and marks each record isNew; skipped listings are never billed.

From code

curl -X POST "https://api.apify.com/v2/acts/josh99smith~remote-jobs-aggregator/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "keywords": ["python", "typescript"], "categories": ["programming"], "postedWithinDays": 7, "maxJobsPerSource": 200 }'

What you get back

{
    "id": "jobicy:153610",
    "source": "Jobicy",
    "title": "Manager, Clinical Review and Quality Assurance",
    "company": "SmithRx",
    "location": "USA",
    "remoteRegion": "US",
    "category": "Healthcare & Medical",
    "tags": ["Healthcare & Medical", "Senior"],
    "employmentType": "Full-Time",
    "salaryMin": 136000,
    "salaryMax": 192000,
    "salaryCurrency": "USD",
    "salaryPeriod": "yearly",
    "descriptionText": "Who We Are:\n\nSmithRx is a rapidly growing ...",
    "url": "https://jobicy.com/jobs/153610-manager-clinical-review-and-quality-assurance-2",
    "publishedAt": "2026-09-18T14:43:04.000Z",
    "isNew": true
}

A board that is down or rate-limiting produces one free { "success": false, "source": "Remotive", "errorType": "rate-limited" } record and the other boards continue.

Cost and limits

$0.001 per job record (1,000 jobs = $1). Filtering, de-duplication, monitor-mode skips and failed boards are free; each run stops at your cost cap. The Actor uses only official public APIs and feeds; it is not a LinkedIn or Indeed scraper and will not become one. Respect each board's attribution terms if you republish listings.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/remote-jobs-aggregator to your MCP client and ask "find remote TypeScript jobs posted in the last 7 days".

Disclosure: I built this Actor. Source: github.com/josh99smith/remote-jobs-aggregator.

PDF URLs to clean text and RAG chunks with one API call

Joshua Smith — Fri, 25 Sep 2026 13:10:21 +0000

Every RAG pipeline starts with "get the text out of these PDFs". pdf.js or pypdf will do it in five lines for a clean file on disk. The mess starts with the inputs people actually have: Google Drive share links, files with a password, 400-page manuals where you need pages 12 to 40, a login page that comes back with a 200 and a .pdf extension, and paragraphs broken into one line per visual row with hyphen-
ated words.

What "extract text" really involves

Rewrite cloud share links (Google Drive, Dropbox, OneDrive, SharePoint) to direct-download URLs.
Stream the download with a size cap and check the %PDF signature before parsing, so HTML error pages are reported, not parsed.
Read the text layer page by page, reassemble lines into paragraphs, repair hyphenation.
Pull metadata (title, author, dates, producer) from the info dictionary and XMP, and links from annotations.
Detect scanned, image-only files and say so instead of returning empty strings.
Chunk with overlap for embeddings, keeping the page number on each chunk.

The two-minute version

PDF Text Extractor does the list above with Mozilla's pdf.js, from a list of URLs, and returns JSON or Markdown.

Paste PDF links into PDF URLs; share links work as they are.
Pick an Output format (text, Markdown or both), optionally a Page range like 1-5, 8, 12-, a Chunk size for RAG, and a PDF password under Advanced.
Click Start. Each file appears in the Output tab as it finishes.

From code

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("josh99smith/pdf-text-extractor").call(run_input={
    "urls": ["https://arxiv.org/pdf/1706.03762", "https://bitcoin.org/bitcoin.pdf"],
    "outputFormat": "markdown",
    "chunkSize": 1500,
    "chunkOverlap": 200,
})
for doc in client.dataset(run["defaultDatasetId"]).iterate_items():
    if doc["success"]:
        for chunk in doc["chunks"]:
            embed(chunk["text"], metadata={"url": doc["url"], "page": chunk["page"]})

What you get back

{
    "url": "https://bitcoin.org/bitcoin.pdf",
    "success": true,
    "status": "ok",
    "fileName": "bitcoin.pdf",
    "fileSizeBytes": 184292,
    "pageCount": 9,
    "wordCount": 1263,
    "hasText": true,
    "metadata": { "title": null, "creator": "Writer", "producer": "OpenOffice.org 2.4", "creationDate": "2009-03-24T17:33:15.000Z", "pdfVersion": "1.4", "encrypted": false },
    "links": ["www.bitcoin.org"],
    "text": "Bitcoin: A Peer-to-Peer Electronic Cash System\n\nSatoshi Nakamoto ...",
    "pages": [{ "page": 1, "text": "Bitcoin: A Peer-to-Peer ...", "charCount": 3049 }],
    "chunks": [{ "index": 0, "page": 1, "text": "Bitcoin: A Peer-to-Peer ...", "charCount": 1246 }]
}

A scanned file returns status: "no-text-layer", a wrong password errorType: "encrypted", an HTML page errorType: "not-a-pdf". All of those are free records.

Cost and limits

$0.004 per successfully processed PDF, no per-page fee: a one-page flyer and a 400-page manual cost the same, and page ranges, passwords, share-link resolution and chunking are included. Scanned PDFs, invalid URLs, non-PDF responses, oversized and unreachable files cost nothing. There is no OCR in this version; if your corpus is scans, this is not the tool yet.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/pdf-text-extractor to your MCP client and ask "extract the text of https://arxiv.org/pdf/1706.03762 and summarise section 3".

Disclosure: I built this Actor. Source: github.com/josh99smith/pdf-text-extractor. Parsing is powered by pdf.js (Apache-2.0).