DEV Community: Harsh Maur

YouTube Transcripts for RAG - Ask a Whole Channel, Get Answers With Timestamps

Harsh Maur — Tue, 20 Oct 2026 13:56:07 +0000

I wanted to ask Andrej Karpathy's YouTube channel questions and get back the exact moment in the video where he answers them. Not a summary. A link that starts the video at the right second.

His three most recent videos together are almost 10 hours long. This post shows how I turned them into a searchable index in about 60 lines of Python: 15,285 caption segments, 598 one-minute chunks, and a search that answers "what is flash attention" with &t=7241s.

Full disclosure: I built the transcript scraper used here and sell it on the Apify Store. Everything below is a real run from 20 October 2026, with the real output.

Why YouTube transcripts are awkward to get in bulk

For one video, you can open the transcript panel and copy it. For a whole channel, the usual paths are:

The YouTube Data API. It lists videos, but it only downloads captions for videos you own.
An open-source transcript library to fetch captions in your own code.
A hosted scraper that takes a channel, playlist or list of videos and returns the transcripts as JSON.

I use option 3, YouTube Transcript Scraper on Apify, because it returns transcripts ready for indexing. You need no YouTube account, no Google API key and no cookies.

What you need

A free Apify account. The free plan includes $5 of credit each month.
Your Apify API token (Console → Settings → API & Integrations).
Python 3 with requests. Nothing else: the search below is plain Python, so you can run it before you pick a vector database.

Step 1: Get every transcript from the channel

The input takes a channel handle. maxVideos limits how many of its newest videos you get:

{
  "channelUrls": ["@AndrejKarpathy"],
  "maxVideos": 3,
  "transcriptFormat": "both"
}

transcriptFormat: "both" returns the timestamped segments and the joined plain text. For RAG you want the segments, because they carry the start time of each line. You can also pass videoUrls, playlistUrls, a dateFrom and dateTo window, and a language code. If a video has no captions in that language, YouTube translates them, and wasTranslated tells you so.

Here is one real row from my run, with the long fields cut:

{
  "videoId": "l8pRSuU81PU",
  "url": "https://www.youtube.com/watch?v=l8pRSuU81PU",
  "title": "Let's reproduce GPT-2 (124M)",
  "channelName": "Andrej Karpathy",
  "publishedAt": "2024-06-09T16:31:35-07:00",
  "durationSeconds": 14486,
  "viewCount": 1212773,
  "language": "en",
  "availableLanguages": ["en"],
  "isAutoGenerated": true,
  "wasTranslated": false,
  "status": "success",
  "transcript": [
    { "start": 0.16, "duration": 4.52, "end": 4.68, "text": "hi everyone so today we are going to be" },
    { "start": 2.36, "duration": 3.64, "end": 6, "text": "continuing our Zero to Hero series and" }
  ],
  "transcriptText": "hi everyone so today we are going to be continuing our Zero to Hero series and ..."
}

That one video has 6,049 segments and 223,000 characters of text. Each row also includes the description (with the chapter list, when the creator wrote one), likeCount, commentCount, keywords and the thumbnail.

Notice isAutoGenerated: true. All three videos use YouTube's automatic captions, so the text has no punctuation and has some misheard words ("chpd" for ChatGPT). Embeddings handle that well. A keyword search like the one below is more sensitive to it.

Step 2: Chunk by time, not by tokens

Most RAG tutorials split text every N tokens. For video, split by time instead. Each chunk then keeps the start time of its first segment, and that start time becomes the citation:

def chunk(video, window=60):
    chunks, words, start = [], [], None
    for seg in video["transcript"]:
        if start is None:
            start = seg["start"]
        words.append(seg["text"])
        if seg["end"] - start >= window:
            chunks.append((video, int(start), " ".join(words)))
            words, start = [], None
    if words:
        chunks.append((video, int(start), " ".join(words)))
    return chunks

60 seconds is about 150 to 200 spoken words, which is a good size for both embeddings and an LLM context window. A link to the chunk is video["url"] + f"&t={start}s".

Step 3: Search the chunks

Here is the full script. It runs the scraper, chunks every video, and ranks the chunks with BM25, the classic keyword ranking that Lucene and Elasticsearch use by default:

import math, os, re, sys, requests
from collections import Counter

ACTOR = "harshmaur~youtube-transcript-scraper"
TOKEN = os.environ["APIFY_TOKEN"]

videos = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json={"channelUrls": ["@AndrejKarpathy"], "maxVideos": 3, "transcriptFormat": "both"},
    timeout=300,
).json()
videos = [v for v in videos if v["status"] == "success"]

def chunk(video, window=60):
    chunks, words, start = [], [], None
    for seg in video["transcript"]:
        if start is None:
            start = seg["start"]
        words.append(seg["text"])
        if seg["end"] - start >= window:
            chunks.append((video, int(start), " ".join(words)))
            words, start = [], None
    if words:
        chunks.append((video, int(start), " ".join(words)))
    return chunks

chunks = [c for v in videos for c in chunk(v)]
print(f"{len(videos)} videos, {sum(len(v['transcript']) for v in videos)} segments, {len(chunks)} chunks")

tokenize = lambda s: re.findall(r"[a-z0-9]+", s.lower())
docs = [Counter(tokenize(text)) for _, _, text in chunks]
avg_len = sum(sum(d.values()) for d in docs) / len(docs)
df = Counter(term for d in docs for term in d)

def bm25(query, doc, k1=1.5, b=0.75):
    length = sum(doc.values())
    score = 0.0
    for term in tokenize(query):
        if term not in doc:
            continue
        idf = math.log(1 + (len(docs) - df[term] + 0.5) / (df[term] + 0.5))
        tf = doc[term]
        score += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / avg_len))
    return score

def ask(question, k=3):
    ranked = sorted(range(len(chunks)), key=lambda i: bm25(question, docs[i]), reverse=True)
    print(f"\nQ: {question}")
    for i in ranked[:k]:
        video, start, text = chunks[i]
        print(f"- {video['title']} @ {start // 3600}:{start // 60 % 60:02d}:{start % 60:02d}  {video['url']}&t={start}s")
        print(f"  \"{text[:160]}...\"")

for q in sys.argv[1:]:
    ask(q)

Run it with your questions as arguments:

APIFY_TOKEN=... python3 ask_channel.py "when should I use a thinking model" "why are LLMs bad at spelling" "how does gradient accumulation work" "what is flash attention"

The real output

3 videos, 15285 segments, 598 chunks

Q: when should I use a thinking model
- Deep Dive into LLMs like ChatGPT @ 2:39:55  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=9595s
  "on many math uh and kind of code problems that have verifiable Solutions so there's like an answer three Etc now these thinking models you can access in for exa..."
- How I use LLMs @ 0:30:24  https://www.youtube.com/watch?v=EWvNQjAaOHw&t=1824s
  "you now when you go to Gro for example when I start a new conversation with grock um when you put the question here like hello you should put something importan..."
- Deep Dive into LLMs like ChatGPT @ 3:27:03  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=12423s
  "labeling instructions at open a so that's what we're getting back now I do think that the uh things change a little bit when you actually go and reach for one o..."

Q: why are LLMs bad at spelling
- Deep Dive into LLMs like ChatGPT @ 1:47:56  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=6476s
  "was Data labeler and I was creating one of these one of these would be uh a really terrible answer for the assistant and the other would be okay and so I'd like..."
- Deep Dive into LLMs like ChatGPT @ 2:03:32  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=7412s
  "third character and the actual truth is u2s uqs uh which looks correct to me so um again an example of spelling related tasks not working very well a very famou..."
- Deep Dive into LLMs like ChatGPT @ 2:00:37  https://www.youtube.com/watch?v=7xTGNNLPyMI&t=7237s
  "the model to copy paste those token IDs and um kind of unpack them into Dots here and so it creates this string and then it calls python routine. count and then..."

Q: how does gradient accumulation work
- Let's reproduce GPT-2 (124M) @ 2:45:50  https://www.youtube.com/watch?v=l8pRSuU81PU&t=9950s
  "processed per step is B * T * gradient accumulation so long story short here we have the optimization it looks uh reasonable right we're starting at a good spot..."
- Let's reproduce GPT-2 (124M) @ 2:43:54  https://www.youtube.com/watch?v=l8pRSuU81PU&t=9834s
  "well uh so where's the loss after that cross entropy and by default the reduction uh here in Cross entropy is also I don't know why they don't show it but it's ..."
- Let's reproduce GPT-2 (124M) @ 2:40:59  https://www.youtube.com/watch?v=l8pRSuU81PU&t=9659s
  "loss objective and this is the average the one over four because there are four independent examples here and then we have the four examples and their mean squa..."

Q: what is flash attention
- Let's reproduce GPT-2 (124M) @ 2:00:41  https://www.youtube.com/watch?v=l8pRSuU81PU&t=7241s
  "attention will come here and we will take out these four lines and Flash attention implements these four lines really really quickly and how does it do that wel..."
- Let's reproduce GPT-2 (124M) @ 2:04:40  https://www.youtube.com/watch?v=l8pRSuU81PU&t=7480s
  "single fused kernel flash attention and that's what we are about to use so great example I think of being aware of um memory hierarchy the fact that flops don't..."
- Let's reproduce GPT-2 (124M) @ 2:02:40  https://www.youtube.com/watch?v=l8pRSuU81PU&t=7360s
  "we're getting a t BYT Matrix of attention which is a Million numbers even for a single head at a single batch index at like so so basically this is a ton of mem..."

The run took 35 seconds and returned all three transcripts.

What worked and what did not:

Specific technical terms work very well. All three "flash attention" hits are inside the chapter "flash attention, 96ms" (2:00:18 in the GPT-2 video). The three "gradient accumulation" hits are all inside the "gradient accumulation" chapter, which starts at 2:34:09.
The spelling question hit the right place. The chapter "tokenization revisited: models struggle with spelling" starts at 2:01:11. Two of the three hits overlap it: the chunk at 2:00:37 runs into it, and the one at 2:03:32 is in the middle of it.
Even the loosely worded question found real answers. The top hit, 2:39:55 in "Deep Dive", is where he says to use thinking models for prompts that need advanced reasoning, and GPT-4o for simpler questions. The second hit is the last minute of the chapter "Thinking models and when to use them" (0:22:54 in "How I use LLMs"). BM25 still only matches words, so a question phrased differently from what he said can miss. That is the point where you swap BM25 for embeddings: embed each chunk's text once, embed the question, and rank by cosine similarity. The chunks and the &t= links stay the same.

Once you have the top chunks, give their text to an LLM with the instruction "answer only from these excerpts and cite the links", and you have a channel Q&A bot with clickable sources.

Tips for a real index

Use the chapter list. When the description contains lines like 00:22:54 Thinking models and when to use them, parse them and store the chapter title with each chunk. It is a strong extra signal for search.
Overlap the chunks. Start a new chunk every 30 seconds with a 60-second window, so an answer that crosses a border is not split in two.
Keep a stable key. Use videoId + start as the chunk ID, so you can run the scraper again on a schedule and add only new videos. dateFrom limits each run to recent uploads.
Check isAutoGenerated. If a creator uploaded their own captions, the text has punctuation and correct names, and keyword search works better.

Use it from an AI agent

If you work in Claude, Cursor or another MCP client, you can give the agent the scraper as a tool and let it fetch transcripts itself:

{
  "mcpServers": {
    "youtube-transcripts": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.apify.com/?tools=harshmaur/youtube-transcript-scraper"]
    }
  }
}

The first time your client starts the server, a browser window opens to connect your Apify account.

What this costs

$5 per 1,000 transcripts on the free plan, and less on paid plans. You pay only for transcripts that come back. A private, deleted or caption-less video returns a row with status: "failed", and that row is free. There is no start fee.

My three-video run cost 1.4 cents. Indexing a channel with 200 videos costs about $1, once. For more than about 20 videos, do not use the run-and-wait endpoint, which waits at most 5 minutes. Start the run with POST /v2/acts/{ACTOR}/runs, wait for it to finish, then read /v2/datasets/{defaultDatasetId}/items. After that, a weekly run with dateFrom costs a few cents.

FAQ

Can I download all transcripts from a YouTube channel at once?
Yes. Put the channel handle in channelUrls and set maxVideos to the number of videos you want. Playlists work the same way with playlistUrls.

Do I need a YouTube API key?
No. You need only an Apify API token, which comes free with an Apify account.

What happens when a video has no transcript?
You get a row with status: "failed" and no charge. The row has only videoId, url, errorCode and error, with no transcript fields, so filter on status == "success" as the script does.

Can I get transcripts in another language?
Set language to a code like es or de. If the video has captions in that language, you get them. If not, you get a machine translation and wasTranslated is true. If no translation is available, you get the original captions, and language shows which language you got.

Do I get SRT or VTT files?
Yes. Set transcriptFormat to srt or vtt, and each row has an srt or vtt field that holds the subtitle file as text. If you already have an SRT or VTT file from somewhere else and only want the words, this free SRT and VTT to text converter removes the timestamps and the repeated lines that YouTube's automatic captions often have.

Try it

YouTube Transcript Scraper on Apify. Paste a channel handle into the input form, press Start, and copy the script above to search what comes back.

If you build a channel Q&A bot with it, tell me in the comments what you indexed and which retrieval worked best for you.

The Apify links in this post include my referral code.

How I Find Winning Facebook Ads (Days Running Is Not Enough)

Harsh Maur — Tue, 20 Oct 2026 13:55:31 +0000

The usual advice for finding winning Facebook ads is simple: open the Meta Ad Library, find ads that have run for a long time, and copy what they do. An advertiser does not keep paying for an ad that loses money.

I tested that advice on 60 active US ads for "magnesium glycinate" on 20 October 2026. The median ad had already run for 129 days. 52 of the 60 were past the usual 30-day "winner" mark. When almost every ad is a "winner", the rule tells you nothing.

What did work: looking for the same hook launched more than once. When a brand keeps putting the same opening line into new ads, that hook is working for them. This post shows how to pull the ads, rank them, and find those repeated hooks with a short Python script.

I built the scraper used here and sell it on the Apify Store, so keep that in mind. All numbers below come from real runs.

Why "days running" breaks in a busy niche

The Ad Library shows ads that are active now. In a mature niche, most active ads are old, because the new ones that failed were already switched off. That is survivorship: the losers left the list before you looked. So:

In a new or small niche, "running 30+ days" is a useful filter.
In a busy niche like supplements, compare each ad against the niche median instead. Here the top quarter of ads had run 168 days or more.

What the Ad Library does not tell you

For normal commercial ads in the US, Meta shows no spend, no impressions and no reach. In my data, spend, impressions_lower and reach_lower were null for every ad. Meta shows spend and impressions only for political and issue ads. Ads shown in the EU and UK also show reach.

So run time and repetition are the best public signals of what is working. Neither is proof, but together they are much stronger than either alone.

Step 1: Pull the ads

Build your search on facebook.com/ads/library: choose a country, "All ads", and type a keyword. Copy the URL from the address bar. (Or build it with a free Ad Library URL builder, which also sets filters like media type and date.) Then run Facebook Ads Library Scraper with it:

{
  "urls": [{ "url": "https://www.facebook.com/ads/library/?active_status=active&ad_type=all&country=US&q=magnesium%20glycinate&search_type=keyword_unordered&media_type=all" }],
  "count": 60
}

Facebook matches keywords loosely. In my run, 50 of the 60 ads mentioned magnesium somewhere in the ad (text, headline or cards). The other 10 were for other products, from general vitamin brands to pharmacy deals at Walgreens and CVS. If you need exact matches, filter on the ad text after the run. For a clean list of one competitor's ads, use their Page instead.

You can paste a brand's Facebook Page URL to get all ads from that one advertiser. You need no Facebook login and no Meta API token.

The run took 34 seconds. Each ad comes back as one flat row. These are the fields I used, from a real ad:

{
  "page_name": "Allison Arnold - Migraine Coach",
  "start_date_iso": "2026-06-07T07:00:00.000Z",
  "is_active": true,
  "ad_duration_days": 135,
  "is_long_running": true,
  "headline": "I Charge $189 for a Migraine IV Drip. My Patients Keep Asking About a $29 Roll-On I Keep at the Front Desk.",
  "cta_text": "Learn more",
  "display_format": "IMAGE",
  "link_url": "https://florusrelief.com/blogs/news/every-migraine-tool-you-have-works-downstream-of-the-source-here-is-what-works-at-it",
  "publisher_platform": ["FACEBOOK", "INSTAGRAM", "AUDIENCE_NETWORK", "MESSENGER", "THREADS"],
  "ad_library_url": "https://www.facebook.com/ads/library/?id=987150940600917",
  "spend": null
}

ad_duration_days is the number of days between the start date and today (or the end date, for ads that stopped). is_long_running is true from 30 days on. The ad text, headline and media links sit at the top level, so the rows drop straight into a spreadsheet.

Step 2: Rank by days live and find repeated hooks

import os, re, statistics, requests
from collections import defaultdict

ACTOR = "harshmaur~facebook-ads-library-scraper"
TOKEN = os.environ["APIFY_TOKEN"]
SEARCH = ("https://www.facebook.com/ads/library/?active_status=active&ad_type=all"
          "&country=US&q=magnesium%20glycinate&search_type=keyword_unordered&media_type=all")

ads = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json={"urls": [{"url": SEARCH}], "count": 60},
    timeout=300,
).json()

# Catalog ads show a template placeholder (product.name in double braces), so their copy says nothing.
PLACEHOLDER = "{" * 2 + "product"
ads = [a for a in ads if PLACEHOLDER not in (a["headline"] or "") + (a["ad_text"] or "")]

days = [a["ad_duration_days"] for a in ads]
print(f"{len(ads)} ads, median {statistics.median(days):.0f} days live, "
      f"{sum(a['is_long_running'] for a in ads)} flagged long-running")

# Group the same opening line from the same page: one hook, many launches.
hooks = defaultdict(list)
for a in ads:
    opening = re.sub(r"\s+", " ", a["ad_text"] or a["headline"] or "")[:60]
    hooks[(a["page_name"], opening)].append(a)

def score(group):
    return (len(group), max(a["ad_duration_days"] for a in group))

for (page, opening), group in sorted(hooks.items(), key=lambda kv: score(kv[1]), reverse=True)[:6]:
    runs = sorted((a["ad_duration_days"] for a in group), reverse=True)
    print(f"\n{page}: {len(group)} ad(s), days live {runs}")
    print(f"  \"{opening}...\"")
    print(f"  {group[0]['display_format']} -> {group[0]['link_url']}")

Two details matter:

Drop catalog ads first. 8 of my 60 ads were catalog (dynamic product) ads. Their headline is literally the placeholder product.name in double curly braces, because Meta fills it in per viewer. They run for months, but their copy teaches you nothing.
Group by page and opening line. The first 60 characters of the ad text are the hook. The same page reusing the same hook in a new ad is the signal.

The real output

52 ads, median 121 days live, 45 flagged long-running

Restelle: 4 ad(s), days live [210, 69, 52, 39]
  "Tired of restless nights? 😴 Introducing our NEW Sleep Patche..."
  IMAGE -> https://myrestelle.com/pages/lp-sleep-2

Allison Arnold - Migraine Coach: 2 ad(s), days live [213, 151]
  "I tracked my migraines for 2,190 days. It took me six years ..."
  IMAGE -> https://florusrelief.com/blogs/news/every-migraine-tool-you-have-works-downstream-of-the-source-here-is-what-works-at-it

We Heart Nutrition: 2 ad(s), days live [211, 187]
  "Why our magnesium glycinate isn't like the others... Researc..."
  IMAGE -> https://www.weheartnutrition.com/

Top 5 Choices: 2 ad(s), days live [155, 117]
  "We've tested them all you don't have to...."
  VIDEO -> https://top5-choices.com/magnesiumreview

Best Supplement Finds: 2 ad(s), days live [152, 133]
  "We tested 47 magnesium supplements for brain fog, cortisol, ..."
  VIDEO -> https://goldsealsupplements.com/pages/magnesium

(The script printed one more group. I cut it here to keep the list short.)

What the winners have in common

1. Restelle ran one hook in four ads. The first ad has run 210 days. Then they added the same opening line in new ads at 69, 52 and 39 days. A brand does that when one message keeps converting and they want more creative variety around it. That is the strongest "winner" signal in the whole dataset.

2. The landing page is often not the store. Look at the links. "Allison Arnold - Migraine Coach" sends people to a blog post on florusrelief.com. "Top 5 Choices" and "Best Supplement Finds" send people to "we tested them all" review pages. These are advertorials: a story or a comparison page that sells before the product page. Several of the longest-running ads in this niche use this pattern.

3. Several hooks are first-person stories with exact numbers. "I tracked my migraines for 2,190 days." "We tested 47 magnesium supplements." "I charge $189 for a migraine IV drip." None of the long-running hooks open with the product name.

4. Video and image both win. Of the 60 ads, 27 were video and 18 were image. The repeated hooks above are split between the two, so format alone was not the deciding factor here.

Turn it into a weekly swipe file

Save the input as a task in Apify Console and schedule it weekly.
Keep ad_archive_id from each run. An ad that shows up week after week, and a hook that gets new ad IDs, are the ones to study.
To follow one competitor, paste their Page URL instead of a keyword search.
For EU and UK ads, turn on scrapeAdDetails to also get reach, age and gender breakdowns and target countries.

What it costs

$0.50 per 1,000 ads, with no start fee. My 60-ad run cost 3 cents. A weekly run of 500 ads is about $1 a month.

FAQ

Can I see how much a competitor spends on Facebook ads?
Not for normal commercial ads in the US. Meta publishes spend only for political and issue ads. Ads shown in the EU and UK come with reach numbers, which are the closest public proxy.

What counts as a long-running Facebook ad?
There is no official definition. 30 days is a common rule of thumb, and it is what is_long_running uses. In a busy niche, compare against the niche median instead, as shown above.

Does a long-running ad always mean it is profitable?
No. Some brands leave ads on by mistake, and catalog ads run for months by design. Repetition (the same hook in several new ads) plus run time is a much stronger signal than run time alone.

Do I need a Facebook account or Meta API access?
No. The scraper reads the public Ad Library, the same one anyone can open without logging in.

Try it

Facebook Ads Library Scraper on Apify. Paste your niche's Ad Library URL, set count to 60, and sort the results by ad_duration_days.

Want an exact-phrase search, a date window or one platform only? Build the URL with this free Facebook Ad Library search URL builder, open it to check the ads, then paste it into urls.

What signal do you use besides run time? Tell me in the comments. I would like to add it to the script.

The links to my Actor include my Apify referral code.

Clockworks TikTok Scraper Alternative: Same Input JSON, 6x Cheaper

Harsh Maur — Tue, 20 Oct 2026 13:54:54 +0000

I pasted one TikTok scraping input into two Apify Actors on 20 October 2026. Same JSON, same account, under a minute apart. One run cost $0.071. The other cost $0.011, and its date filter also worked on the hashtag, which the first run's did not.

The first Actor is clockworks/tiktok-scraper, the most used TikTok scraper on Apify with about 29,000 monthly users. The second is TikTok Scraper. I built the second one, so read this with that in mind. Input fields match clockworks', so moving over takes one or two lines of code.

This post shows the exact test, the price math at 1,000 videos, which fields differ, and what the cheaper one does not do.

The test input

This is a normal "watch a hashtag and a brand account" input in clockworks' format:

{
  "hashtags": ["booktok"],
  "profiles": ["duolingo"],
  "resultsPerPage": 10,
  "oldestPostDateUnified": "2026-10-06",
  "shouldDownloadVideos": false,
  "shouldDownloadCovers": false,
  "shouldDownloadSubtitles": false,
  "shouldDownloadSlideshowImages": false,
  "proxyCountryCode": "None"
}

Ten videos from #booktok, ten from @duolingo, only videos posted since 6 October. I ran it on both Actors from the same Apify account on the Starter plan.

The results

	clockworks/tiktok-scraper	harshmaur/tiktok-scraper
Videos returned	20	20
Extra rows	none	1 profile row for @duolingo
#booktok videos inside the date window	0 of 10 (all from February to April 2023)	10 of 10 (all from 9 to 19 October 2026)
@duolingo videos inside the date window	10 of 10	10 of 10
Run time	18 s	31 s
Cost on the Starter plan	$0.071	$0.011

Three things stand out.

The price. Clockworks charged 20 results at $0.003, plus 10 "date filter" add-on events at $0.001, plus a $0.001 start fee. Mine charged 20 videos at $0.00035, plus one profile row at $0.004.

The date filter on the hashtag. Clockworks applied the date filter to the profile only (that is where the 10 add-on charges came from). The #booktok videos it returned were up to three years and eight months old. Check this in your own runs if you watch hashtags with a date filter.

Speed. Clockworks was faster: 18 seconds against 31. For a scheduled job this rarely matters. For a chat agent waiting on a reply, it might.

The price at 1,000 videos

These are the per-event prices from each Actor's pricing table on 20 October 2026. Clockworks has tiered prices that fall on bigger Apify plans. Mine is one flat price on every plan.

1,000 videos	clockworks	harshmaur
Free plan, no filters	$3.70	$0.35
Free plan, with a date filter	$5.00	$0.35
Starter plan, with a date filter	$4.00	$0.35
Business plan, with a date filter	$2.30	$0.35
1,000 comments, free plan	$1.25	$0.20

The gap closes on clockworks' top tiers. On clockworks' Diamond tier, plain results are $0.50 per 1,000.

Where mine costs more: video downloads are $10 per 1,000 against clockworks' $1.30 or less, and each scraped profile adds a profile row at $4 per 1,000 profiles. If you mainly download MP4s, or scrape thousands of profiles for their bio only, do the math for your own job before you switch.

Moving over: the two lines you change

If you call the Actor from code, you change the Actor ID and keep your input:

import os, requests

# ACTOR = "clockworks~tiktok-scraper"
ACTOR = "harshmaur~tiktok-scraper"
TOKEN = os.environ["APIFY_TOKEN"]

run_input = {
    "hashtags": ["booktok"],
    "profiles": ["duolingo"],
    "resultsPerPage": 3,
    "oldestPostDateUnified": "2026-10-06",
    "shouldDownloadVideos": False,
    "shouldDownloadCovers": False,
}

rows = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json=run_input,
    timeout=300,
).json()

videos = [r for r in rows if r.get("dataType", "video") == "video"]
for v in videos:
    print(v["createTimeISO"][:10], f"{v['playCount']:>10,} plays", v["authorMeta"]["name"], v["webVideoUrl"])

The real output of that script:

2026-10-13      4,703 plays laurinelexis https://www.tiktok.com/@laurinelexis/video/7695950683073187086
2026-10-19     59,400 plays probablyoffreading https://www.tiktok.com/@probablyoffreading/video/7698425693255798030
2026-10-11    370,200 plays caroline.montague https://www.tiktok.com/@caroline.montague/video/7695446883426110742
2026-10-19     85,100 plays duolingo https://www.tiktok.com/@duolingo/video/7698408739077901598
2026-10-12    665,400 plays duolingo https://www.tiktok.com/@duolingo/video/7695843389211708702
2026-10-13  2,300,000 plays duolingo https://www.tiktok.com/@duolingo/video/7696211706317409550

Besides the Actor ID, the dataType filter is the only other change most scripts need. Clockworks puts only videos in the main dataset. Mine puts videos, comments, followers and profiles in one dataset and marks each row with dataType. The .get("dataType", "video") default keeps the same script working on both Actors.

In the Apify Console it is even simpler: open your clockworks task, copy the input JSON, paste it into the new Actor's JSON tab, and start.

Which output fields differ

I compared every video row from both runs, including nested keys. 76 fields have the same name in both. Video fields such as createTimeISO, webVideoUrl, authorMeta.fans and the counts held the same values, apart from a few likes, comments and saves of drift, because the runs were seconds apart and people kept interacting. Some author fields differ: mine leaves authorMeta.bioLink and authorMeta.createTime empty and reports authorMeta.friends as 0, and in text it replaces line breaks with spaces.

Only in clockworks' output:

locationMeta.city, locationMeta.address and similar place details on videos tagged with a location.
authorMeta.commerceUserInfo, the business-account category.

Only in mine:

dataType, as above.
isAiGenerated, locationCreated, duetEnabled, stitchEnabled, suggestedWords.
authorMeta.region and authorMeta.secUid.
videoMeta.transcript, filled when you ask for TikTok's own subtitles.
aiAnalysis, filled when you turn on the paid AI analysis option.

If your spreadsheet uses place details or the business category, mine leaves those columns empty.

Input fields that do not carry over

Almost every clockworks field works as it is. These do not:

scrapeRelatedSearchWords and scrapeAdditionalAuthorMeta have no effect.
videoKvStoreIdOrName is ignored. Downloads always go to the run's own key-value store, and the links are in mediaUrls.
profileScrapeSections: ["stories"] returns no stories, because TikTok shows stories only to logged-in users.

Relative dates work in both: "30 days", "2 weeks", or just "30".

When to stay on clockworks

To be fair to a very good Actor:

You are on a Platinum or Diamond Apify plan, where their prices come close to mine.
You download a lot of videos.
You need place details or the business category on each video.
You want the Actor with a long track record: clockworks has over 140 million runs. Mine launched on 12 October 2026.

For everyone else, watching hashtags, brands or creators on a free or Starter plan, the same input now costs a sixth to a fourteenth as much.

Try it with your own input

Open TikTok Scraper on Apify, switch the input to JSON, and paste the input from your last clockworks run. Set resultsPerPage to 5 for a first test. For one hashtag and one profile, that costs less than a cent.

To check one video before you scrape a whole account, I use this free TikTok engagement rate calculator. Enter the playCount, diggCount, commentCount, shareCount and authorMeta.fans values from a row, and it gives the engagement rate by views and by followers.

If a field you rely on is missing or named differently, leave a comment with the field name. I would rather add it than have you write a mapping layer.

The links to my Actor include my Apify referral code.

Scrape LinkedIn Jobs Without Login or API Key (Half Were Recruiters)

Harsh Maur — Tue, 20 Oct 2026 13:54:42 +0000

This morning I pulled every "founding engineer" job posted in the US in the last 24 hours, straight into Python. No LinkedIn account, no cookies, no API key from LinkedIn. The run took 10 seconds and cost about 1.5 cents.

The more useful part came after that: half of those "founding engineer" listings were not posted by a startup at all. They came from recruiting agencies. If you are job hunting, building a job board, or looking for companies that are hiring, that difference matters. So this post also shows the 15-line filter I use to split them.

Full disclosure: I built the scraper used here and sell it on the Apify Store. Everything below is a real run from 20 October 2026, with the real output.

Why there is no official LinkedIn jobs API for you

LinkedIn has a Job Posting API, but it is for partners who post jobs (ATS vendors, job boards with a contract). It does not let you search or read other companies' listings. If you want job data, your options are:

Copy and paste from linkedin.com/jobs by hand.
Write your own scraper and maintain it every time LinkedIn changes its pages.
Use a hosted scraper that reads LinkedIn's public job pages, the ones anyone sees without logging in.

I went with option 3 and built LinkedIn Jobs Scraper on Apify. It never logs in, so there is no account to get banned and no cookie to refresh.

What you need

A free Apify account. The free plan includes $5 of credit each month.
Your Apify API token (Console → Settings → API & Integrations).
Python 3 with requests.

That is all. You do not need anything from LinkedIn.

Step 1: Describe the search

The input is plain JSON. These are the fields I used:

{
  "keywords": "founding engineer",
  "location": "United States",
  "datePosted": "1",
  "maxItems": 15,
  "scrapeCompany": true
}

datePosted: "1" means the past 24 hours. Use "7" for a week or "30" for a month.
maxItems caps the number of jobs, and you pay per job, so keep it low while you test.
scrapeCompany adds the employer's employee count, website and headquarters to each row. It costs nothing extra.

If you already built a search on linkedin.com/jobs, you can paste its URL instead. LinkedIn's own filter menu only offers 24 hours, a week or a month. I use this free LinkedIn job search URL builder when I want "posted in the last 2 hours" or another custom window:

{ "startUrls": [{ "url": "https://www.linkedin.com/jobs/search/?keywords=founding%20engineer&location=United%20States&f_TPR=r86400" }] }

Step 2: Run it from Python and get the jobs back

Apify has a "run and wait" endpoint that starts the scraper and returns the dataset in the same HTTP call. It is fine for a few hundred jobs.

import os, re, requests

ACTOR = "harshmaur~linkedin-jobs-scraper"
TOKEN = os.environ["APIFY_TOKEN"]

run_input = {
    "keywords": "founding engineer",
    "location": "United States",
    "datePosted": "1",        # past 24 hours
    "maxItems": 15,
    "scrapeCompany": True,    # adds employee count, website, HQ
}

jobs = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json=run_input,
    timeout=300,
).json()

Each item in jobs is one job. Here is a real one from my run, trimmed (the description fields are long):

{
  "title": "Founding Fullstack AI Engineer",
  "company": "AustinWorks",
  "companyUrl": "https://www.linkedin.com/company/austinworksrecruitment",
  "location": "San Francisco Bay Area",
  "postedTimeAgo": "9 hours ago",
  "employmentType": "Full-time",
  "seniorityLevel": "Entry level",
  "applicants": 80,
  "salary": "$150,000.00/yr - $210,000.00/yr",
  "salaryMin": 150000,
  "salaryMax": 210000,
  "salaryCurrency": "USD",
  "salaryPeriod": "YEAR",
  "jobPosterName": "Charlie Austin",
  "jobPosterTitle": "Connecting top talent across Software + FDE + AI",
  "jobPosterProfileUrl": "https://www.linkedin.com/in/charlie-austin-a4a87a170",
  "url": "https://www.linkedin.com/jobs/view/4478636704"
}

Every row also has descriptionText, descriptionMarkdown and descriptionHtml, plus industries, jobFunction, companyWebsite, companyEmployeesCount and companyHeadquarters. When LinkedIn does not show a value, the field is null instead of missing, so your code does not need try/except around every key.

Look at companyUrl in that example: austinworksrecruitment. That "founding engineer" job is a recruiter's listing.

Step 3: Filter out recruiting agencies

My first idea was to filter on industries == "Staffing and Recruiting". On real data, that caught only 2 of the agencies. Half of the agencies list "Software Development" or "Artificial Intelligence" as their industry, just like their clients.

What works better is to check the company name, its LinkedIn slug, its website and its industry together:

AGENCY = re.compile(r"recruit|staffing|talent|\bsearch", re.I)

def is_agency(job):
    text = " ".join(filter(None, [
        job["company"], job["companyUrl"], job.get("companyWebsite"), job.get("industries"),
    ]))
    return bool(AGENCY.search(text))

direct = [j for j in jobs if not is_agency(j)]
print(f"{len(jobs)} jobs, {len(jobs) - len(direct)} from agencies, {len(direct)} posted by the company")

for j in direct:
    pay = f"${j['salaryMin']:,}-${j['salaryMax']:,}" if j["salaryMin"] else "no pay listed"
    print(f"- {j['title']} @ {j['company']} ({j['companyEmployeesCount'] or '?'} staff), "
          f"{pay}, {j['applicants'] or '?'} applicants\n  {j['url']}")

The \b before search matters. Without it, "Research Services" counts as an agency.

The real output

15 jobs, 8 from agencies, 7 posted by the company
- Founding Aerospace Engineer @ Aether (1 staff), no pay listed, ? applicants
- Founding Engineer @ Bearcroft (4 staff), no pay listed, 61 applicants
- Founding Backend Engineer @ ALL KNOWN SERVICES (9 staff), no pay listed, 111 applicants
- Founding Full-Stack Engineer (up to $220k) @ Dex (40 staff), no pay listed, ? applicants
- Founding AI Engineer @ Convoy Health (4 staff), no pay listed, 62 applicants
- Founding Engineer @ Arcafeed (Acq. by Whop) (4 staff), no pay listed, 52 applicants
- Founding Product Engineer @ WhyHireWrong? (1 staff), no pay listed, ? applicants

(I removed the job URLs from this listing to keep it short. The script prints them.)

Three things surprised me:

Every salary in the sample came from an agency. Five of the 15 jobs had a parsed salary, and all five were recruiter listings. None of the direct startup posts filled in LinkedIn's pay field. Dex put "up to $220k" in the job title instead, so salary is null for that row. If pay matters to you, also search title and descriptionText for a dollar sign.
The filter is not perfect. ALL KNOWN SERVICES passed as "direct", but the person who posted it describes themselves as helping people land W2 contract roles. Adding jobPosterTitle to the text you check catches more agencies, with some false positives.
Applicant counts are mostly low. Most direct posts had under 65 applicants within a day. A null count (printed as ?) usually means LinkedIn showed "Be among the first 25 applicants". LinkedIn shows at most about 200, so 200 means "200 or more".

Two runs a few minutes apart did not return exactly the same 15 jobs. LinkedIn reorders its results, so for a daily feed, de-duplicate on jobId.

What this costs

You pay per job in the dataset. There is no start fee and no subscription. On Apify's free plan it is $1.00 per 1,000 jobs, and $0.40 to $0.47 per 1,000 on paid plans. Recruiter fields and company details are included in that price.

My 15-job test cost about 1.5 cents on the free plan. A daily run of 50 new jobs is roughly $0.70 a month on the Starter plan.

Get more than 1,000 jobs

LinkedIn stops every search near 1,000 results. To go past that, turn on splitByLocation. The scraper then runs your search in each large city of the country and removes duplicates:

{ "keywords": "software engineer", "splitByLocation": true, "splitCountry": "us", "maxItems": 5000 }

Make it a daily alert

In Apify Console, save the input above as a task.
Add a schedule that runs it every morning.
Pick Slack, Notion or Google Sheets in the scraper's Send results to an app option, or call the run-and-wait endpoint from your own cron job and keep the filter step.

With datePosted: "1" and a daily schedule, you see each new job once.

FAQ

Can I scrape LinkedIn jobs without logging in?
Yes. Job listings are public. Anyone can open linkedin.com/jobs in a private window and read them. The scraper reads those same public pages and never asks for your LinkedIn account.

Do I need a LinkedIn API key?
No. You need only an Apify API token, which comes free with an Apify account.

Can I get the recruiter's name for a job?
When the listing names the person who posted it, you get jobPosterName, jobPosterTitle and jobPosterProfileUrl. In my run, 7 of 15 jobs named someone.

Do LinkedIn's remote and experience filters still work?
LinkedIn moved to an AI job search in 2026 and dropped several classic filters. Date posted, company and Easy Apply still filter exactly. Results for remote, job type and experience level can be approximate. Filter on seniorityLevel or employmentType afterwards if you need an exact match.

Is it legal?
The scraper collects only public job listings that anyone can see without logging in. It does not touch profiles or anything behind a login. Check the rules for your own use case.

Try it

LinkedIn Jobs Scraper on Apify. Paste the JSON from step 1 into the input form, press Start, and you have a spreadsheet of jobs in about 10 seconds.

Want to test a search before you scrape it? Build the URL with the LinkedIn job search URL builder, open it in a private window to check the results, then paste it into startUrls.

If you build something with it, or the agency filter misses a pattern you see often, tell me in the comments.

The Apify links in this post include my referral code.

How to build a Reddit lead generation monitor with Apify MCP connectors

Harsh Maur — Tue, 01 Sep 2026 06:09:32 +0000

People do not always announce buying intent on a landing page. They ask for recommendations in a subreddit, complain about a workflow, or look for an alternative to a tool they already use.

I saw that pattern while building and testing my Reddit Scraper Actor. I kept repeating the same keywords and communities across runs. These were no longer one-off research jobs; they were monitors. The missing piece was delivery. A useful post could sit in an Apify dataset until someone remembered to check it.

I added optional Model Context Protocol (MCP) delivery so the Actor can send completed results to another service. This Reddit lead generation workflow uses Slack's official MCP server. For this guide, I configured the server, discovered its live tool schema, ran a one-result test, and verified the message in a private Slack channel. The first attempt failed for a non-obvious Slack setting, which is included below because it is exactly the kind of integration detail that polished examples tend to hide.

What we are building

Apify schedule -> Reddit Scraper -> structured Reddit posts and comments
  -> Apify MCP connector -> Slack MCP server -> private lead-alert channel -> human review

The Actor searches recent Reddit discussions, stores the complete results in its dataset, and sends selected posts through an Apify MCP connector. A Slack alert can contain the post title, subreddit, score, comment count, link, and selected comments.

A person still decides whether to respond. This workflow surfaces relevant public conversations; it does not automate unsolicited Reddit replies.

Two Apify features have similar names but opposite directions:

An Apify MCP connector lets an Actor call an external MCP server. That is what sends the alerts in this tutorial.
The Apify MCP server lets an external AI client call Apify Actors and tools. It is useful for Claude, ChatGPT, or Cursor, but it is not the delivery path here.

Prerequisites

You will need:

an Apify account;
Reddit Scraper;
permission to create a Slack app and a private destination channel;
permission to create Apify tasks and schedules; and
a small list of brand, competitor, and problem-related searches.

Slack exposes its official Streamable HTTP MCP endpoint at https://mcp.slack.com/mcp. It requires a registered internal or Marketplace Slack app with a confidential OAuth client. Slack does not support Dynamic Client Registration for this endpoint, so the one-click OAuth option is not enough; use Apify's Own OAuth client flow. Some Console versions label the same option Your own OAuth client.

1. Start with high-intent Reddit searches

A broad category keyword creates noise. A phrase describing a decision or painful workflow creates better alerts.

Broad query	Higher-intent query
scheduling software	alternative to calendly
appointment tools	keep missing client calls
booking app	double booked
CRM	looking for a simple CRM

A useful starter set contains your brand and common misspellings, two or three competitor names, a few problem phrases, and buying-intent language such as looking for, alternative to, or would pay for.

Reddit Scraper has four controls that matter for monitoring:

searchSort: "new" orders results chronologically;
searchTime: "day" limits the search window;
withinCommunity limits a run to one subreddit; and
maxPostsCount caps the posts collected.

Sorting by new does not remove old results, so use a time window as well.

Figure 1: Search for a problem or buying-intent phrase, sort by New, and use a recent time window.

Start with 10 to 20 posts. It is easier to improve five weak alerts than to clean up a Slack channel after sending hundreds.

2. Configure Slack for Reddit lead generation alerts

Create the private destination first: in Slack, select Add channels -> Create a new channel, give it a clear test name, and make it private. The Slack user who authorizes the app must be a member of that channel and able to post there.

Next open Slack app management, select Create New App -> From scratch, name the app, and choose the receiving workspace. In the new app, open OAuth & Permissions and add this redirect URL:

https://console.apify.com/oauth/mcp-proxy

Add chat:write as a User Token Scope. Then open the app's Agents page and enable Slack Model Context Protocol (MCP) Server. This toggle is separate from the app's OAuth scopes and is easy to miss. Open Basic Information -> App Credentials to find the client ID and client secret required by Apify.

In Apify Console, open Settings -> API & Integrations -> MCP connectors, select Add connector, and enter:

Name: Slack lead alerts
Server URL: https://mcp.slack.com/mcp
Authentication: Own OAuth client (or Your own OAuth client)
Authorization URL: https://slack.com/oauth/v2_user/authorize
Token URL: https://slack.com/api/oauth.v2.user.access

Supply the Slack app's client ID and client secret in Apify's secure fields. They should never appear in Actor input, datasets, source code, screenshots, or logs.

Review the Slack consent page instead of approving it mechanically. The workflow needs chat:write, not workspace-wide search, file, user, and channel-management permissions. In my August 2026 test, Apify generated the authorization popup with Slack's much broader advertised scope set. This was observed Apify behavior, not a Slack-supported scope picker. I completed the least-privilege flow as follows:

In the same OAuth popup, focus the address bar without closing or duplicating the window.
Change only the scope query parameter to chat:write and remove the empty user_scope parameter.
Preserve client_id, redirect_uri, state, code_challenge, code_challenge_method, and response_type exactly as Apify generated them.
Reload that edited URL in the same popup and verify that the consent screen requests only permission to send messages as the authorizing user.
Approve the request and return to Apify, where the connector should show the discovered tools.

That URL edit is a workaround for the current connector flow. Do not alter, share, or reuse the OAuth state or Proof Key for Code Exchange (PKCE) values. If your administrator does not approve this workaround, stop rather than granting the broader scopes and use an administrator-managed least-privilege integration instead. Re-check Apify's connector UI before following this step because native scope selection may replace the workaround.

After successful authorization, the live server exposed three tools with this scope:

slack_send_message;
slack_schedule_message; and
slack_send_message_draft.

The slack_send_message input schema required exactly two values:

{
  "channel_id": "string",
  "message": "string"
}

Optional fields included thread_ts, reply_broadcast, draft_id, and unfurl_app_links. I disabled Allow all in the connector and permitted only slack_send_message.

Figure 2: OAuth scope and connector tool allowlists are separate controls. This connector can call only the immediate-send operation.

Apify stores the Slack credential server-side. The Actor receives a connector ID and authenticates to Apify's MCP proxy with its run token; it never receives the Slack OAuth token.

3. Configure Reddit Scraper delivery

Open Reddit Scraper and expand Send to your apps (MCP integration, optional).

Copy the Slack channel ID before filling mcpTarget: open the destination channel, select its name to open channel details, and copy the Channel ID shown on the About tab. You can also copy the channel link and use the segment beginning with C. Do not use the human-readable #channel-name in mcpTarget.

Figure 3: The final configuration uses New, Last day, the authorized Slack connector, and one message per post.

Field	What it controls
`mcpConnector`	Managed connector selected in Apify Console
`mcpMode`	One message per post (`perPost`) or one digest (`summary`)
`mcpTarget`	Destination expected by the tool; for Slack, a channel ID
`mcpComments`	Ignore comments, bundle them, or send separate follow-ups
`mcpCommentsPerPost`	Highest-scoring comments included per post
`mcpMessage`	Deterministic `{{field}}` template
`mcpTool`	Exact tool-name override
`mcpArguments`	Extra rendered arguments merged over inferred values
`mcpMaxItems`	Maximum posts dispatched in `perPost` mode

For Slack lead alerts, use perPost. Each opportunity becomes a message that can be discussed or assigned independently.

A useful template is:

**{{title}}**
{{communityName}} | score {{score}} | {{commentsCount}} comments
{{postUrl}}

Templates are deterministic substitutions, not AI prompts. Missing values render as empty strings, and nested dataset paths are supported.

Figure 4: Control formatting, comment delivery, the exact MCP tool, and the dispatch cap.

Here is the schema-validated smoke-test input used for the final screenshots. Replace the two placeholders with values from your account:

{
  "searchTerms": ["AI"],
  "searchPosts": true,
  "searchComments": false,
  "searchCommunities": false,
  "searchSort": "new",
  "searchTime": "day",
  "crawlCommentsPerPost": false,
  "maxPostsCount": 1,
  "mcpConnector": "REPLACE_WITH_APIFY_CONNECTOR_ID",
  "mcpMode": "perPost",
  "mcpTarget": "REPLACE_WITH_SLACK_CHANNEL_ID",
  "mcpComments": "ignore",
  "mcpTool": "slack_send_message",
  "mcpMessage": "**{{title}}**\n{{communityName}} | score {{score}} | {{commentsCount}} comments\n{{postUrl}}",
  "mcpMaxItems": 1
}

The broad, harmless AI query makes a one-result transport test more reliable than a narrow lead phrase. If it returns no post for day, widen searchTime to week and rerun before changing any connector settings. Once delivery works, replace AI with the high-intent phrases from step 1, optionally set withinCommunity, and raise both caps gradually.

The live Slack schema uses channel_id and message. Reddit Scraper puts the mcpTarget value and the rendered message into the matching fields of the tool. You do not write a provider-specific request body.

4. Test the Reddit lead generation workflow

Run the Actor manually before creating a task or schedule. Use maxPostsCount: 1 and mcpMaxItems: 1 for the first end-to-end test.

My first run scraped one post, but the delivery to Slack failed with a connection error. The error message did not show the real cause.

The real cause was a Slack setting: the Slack app was not enabled for Slack MCP server access. After I enabled Agents -> Slack Model Context Protocol (MCP) Server, I recreated and reauthorized the Apify connector so it could rediscover tools. No change to the Actor input was needed.

The second run discovered the three Slack tools, called slack_send_message, and finished successfully. After restricting the connector to that one allowed tool, I repeated the one-item run; it dispatched one call and produced a second message in the private channel.

The first message showed the title in italics, not bold. The official Slack MCP tool uses standard Markdown, so write **{{title}}**, not *{{title}}*. The Actor default now uses the bold form, and the next test showed a bold title.

Figure 5: The test extracted one post, dispatched one MCP call through slack_send_message, and preserved a successful crawl result.

The private destination channel then contained the rendered Reddit alert and source link:

Figure 6: The external result confirms that connector authorization, schema mapping, and the Slack write all worked.

Check a manual run in this order:

Verify the expected Reddit rows in the dataset.
Confirm deliverable rows have dataType: "post".
Inspect the log for the selected MCP tool and dispatched-call count.
Open Slack and verify the channel, formatting, and links.
Compare the number of delivered posts with Slack messages.
Tighten searches before raising the caps.

If you set mcpTool, use the exact tool name from the connector. A wrong name does not fall back to automatic selection, and nothing is sent.

5. Turn the working input into a monitor

Once the manual run is clean:

Return to the Actor's Input tab, open Actions, and choose Save as a new task.
Name the task, save it, and open Schedules -> Create schedule in Apify Console.
Choose the saved task as the action, set an hourly or two-hour interval, and save the schedule.
Use Run now once and confirm the new run, dataset row, dispatch log, and Slack message before leaving the schedule enabled.
Review alert quality for several days and remove noisy searches before increasing volume.

Reddit Scraper does not currently keep a cross-run set of posts delivered through MCP. Overlapping search windows can therefore repeat a post. Reduce duplicates with a shorter searchTime, fewer overlapping queries, a slower schedule, or downstream deduplication on the Reddit post id.

How Reddit Scraper fills in Slack's fields

After the crawl, Reddit Scraper builds one message for each post, up to mcpMaxItems. In bundle mode, it adds the top comments to the same message. It then reads the tools of your connector and sends the messages one at a time.

For the Slack tool, the message goes to message and the channel ID goes to channel_id. Other MCP servers use other field names, such as text, content, rows, or properties. If a tool needs a field that the Actor does not fill, add it in mcpArguments.

Reddit lead generation failure modes

Scraping finishes before MCP delivery. A connector failure normally leaves the collected dataset intact, but partial delivery is possible.

Failure	What you see
Slack MCP Server toggle is disabled	The connection fails; enable the toggle, then reauthorize the connector
The connector is not available to the run	Nothing is sent, and the run log shows a delivery error
`mcpTool` names a tool that does not exist	Nothing is sent
The connector has no tool that can send a message	Nothing is sent
A tool call fails	Delivery stops at that message
Connector is empty	The Actor only scrapes
The run finds no posts	Nothing is sent

The Actor does not retry a failed message. It sends messages one at a time, and one failure stops the rest. Earlier Slack messages can already exist when the run log shows a delivery error. Check Slack before you run again, so you do not get duplicates.

mcpMaxItems caps posts, not total calls. In separate-comment mode, one post can generate one post message plus several comment messages.

Security, Reddit policies, and human review

Give the Slack app only the scopes required by the permitted MCP tool.
Disable Allow all and allow only slack_send_message for this connector.
Send test alerts to a designated private channel.
Keep mcpMaxItems low until the query quality is proven.
Prefer a managed connector over placing a bearer token in Actor input.
Never put OAuth credentials in datasets, templates, screenshots, or logs.
Treat public Reddit content as context, not consent for automated outreach.
Keep a person responsible for deciding whether and how to respond.

Use this workflow in accordance with Reddit's User Agreement, Public Content Policy, privacy requirements, and applicable law. Do not use it to profile sensitive traits, harass users, or automate spam.

Slack also warns about mixing data across several MCP servers. Review every endpoint that can receive workspace context, and avoid sending workspace data through an MCP service you do not trust.

After a temporary proof-of-concept, pause or delete the Apify schedule and task, remove the MCP connector from Apify, revoke the Slack app's OAuth grant, and archive the test channel. Delete the Slack app only if no other connector or workflow uses it.

What I would improve next

The useful first version is working: scheduled discovery, deterministic formatting, live tool discovery, least-privilege connector controls, comments, and safety caps.

The next improvements are no repeat alerts across runs, retries for short MCP errors, a count of delivered messages, and templates for each destination. The main lesson, however, is that query quality matters more than scrape volume. Start with three phrases indicating a real problem or decision, send a tiny sample to Slack, review it with a human, and improve the searches before scheduling more runs.

That turns Reddit Scraper from a dataset generator into a monitor a team can actually use.

DEV Community: Harsh Maur

YouTube Transcripts for RAG - Ask a Whole Channel, Get Answers With Timestamps

Why YouTube transcripts are awkward to get in bulk

What you need

Step 1: Get every transcript from the channel

Step 2: Chunk by time, not by tokens

Step 3: Search the chunks

The real output

Tips for a real index

Use it from an AI agent

What this costs

FAQ

Try it

How I Find Winning Facebook Ads (Days Running Is Not Enough)

Why "days running" breaks in a busy niche

What the Ad Library does not tell you

Step 1: Pull the ads

Step 2: Rank by days live and find repeated hooks

The real output

What the winners have in common

Turn it into a weekly swipe file

What it costs

FAQ

Try it

Clockworks TikTok Scraper Alternative: Same Input JSON, 6x Cheaper

The test input

The results

The price at 1,000 videos

Moving over: the two lines you change

Which output fields differ

Input fields that do not carry over

When to stay on clockworks

Try it with your own input

Scrape LinkedIn Jobs Without Login or API Key (Half Were Recruiters)

Why there is no official LinkedIn jobs API for you

What you need

Step 1: Describe the search

Step 2: Run it from Python and get the jobs back

Step 3: Filter out recruiting agencies

The real output

What this costs

Get more than 1,000 jobs

Make it a daily alert

FAQ

Try it

How to build a Reddit lead generation monitor with Apify MCP connectors

What we are building

Prerequisites

1. Start with high-intent Reddit searches

2. Configure Slack for Reddit lead generation alerts

3. Configure Reddit Scraper delivery

4. Test the Reddit lead generation workflow

5. Turn the working input into a monitor

How Reddit Scraper fills in Slack's fields

Reddit lead generation failure modes

Security, Reddit policies, and human review

What I would improve next

References