DEV Community: Emmanuel Uchenna

How to Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion

Emmanuel Uchenna — Wed, 09 Sep 2026 09:15:19 +0000

Generating an XML sitemap is only the first step in modern technical Search Engine Optimization (SEO). If you have ever published a new blog post, launched a product line on an e-commerce platform, or updated critical landing pages, you know the frustration of waiting days, weeks, or even months for Google to crawl and index your content.

Relying on search engine crawlers to discover changes organically is no longer viable for high-growth websites. If you run a dynamic content platform, an online store with rapidly changing inventory, or a news publication, delayed indexing directly equates to lost traffic, lower visibility, and missed revenue.

In this guide, we will walk through the process of building an automated, enterprise-grade SEO pipeline. We will start by covering automated URL discovery using the Fast Sitemap Generator Apify Actor. Then, we will look at programmatically pushing those URLs to Google using the Google Indexer & Instant SEO Submitter Actor. Finally, we will solve the critical tracking challenge by using Model Context Protocol (MCP) Connectors to stream real-time indexing logs directly into a Notion database.

The evolution of programmatic indexing

Historically, website owners managed SEO by submitting a static Extensible Markup Language (XML) sitemap through Google Search Console. While essential, submitting a sitemap is fundamentally a passive notification. You are effectively dropping a note in Google's mailbox, asking Googlebot to visit your site whenever it gets around to it.

For small, static websites, this passive approach works. However, for modern dynamic sites, it falls short due to several structural challenges:

Crawl budget constraints: Search engines assign a finite "crawl budget" to every domain. On large sites with thousands of pages, Googlebot may exhaust its budget on legacy pages, completely ignoring newly published or updated content.
Orphaned or deep pages: Content buried deep within subdirectories or pagination chains often fails to receive internal link equity, making it nearly invisible to organic crawling.
Time-sensitive content: E-commerce price updates, breaking news, real estate listings, and job postings require instant indexation. Waiting two weeks for organic discovery renders the content stale before it ever reaches search results.
Desktop crawler overhead: Traditional site auditors like Screaming Frog run locally, consuming heavy system resources, tying up developer machines, and requiring manual CSV exports to process updates.

Moving from passive crawling to active push pipelines

To solve these limitations, search engines introduced programmatic indexing APIs. Instead of waiting for Googlebot to discover changes, you can actively send a push notification to Google the instant a page is created or updated.

By combining cloud-based crawling with active API submissions and real-time database logging, you can replace manual SEO workflows with a fully automated "set and forget" pipeline.

Stage 1: Automated URL discovery with Fast Sitemap Generator

Before you can submit URLs to Google, you need an accurate, deduplicated list of active links from your website. Hand-coding XML files or relying on CMS plugins that crash on large databases creates maintenance bottlenecks.

The Fast Sitemap Generator Actor runs on the Apify platform as a serverless cloud program. It uses a Direct Connection to crawl websites at high speeds directly from data centers, without requiring proxy infrastructure.

Configuring the URL crawler

Setting up automated URL discovery requires configuring key parameters inside the Apify Console to balance thoroughness with execution speed.

Here is an example input configuration for a production crawl:

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 3,
  "maxPagesPerCrawl": 5000,
  "includePatterns": [
    ".*/blog/.*",
    ".*/products/.*"
  ],
  "excludePatterns": [
    ".*/admin/.*",
    ".*/login.*",
    ".*/cart.*",
    ".*/checkout/.*"
  ],
  "sitemapFormats": [
    "xml",
    "html",
    "txt"
  ],
  "respectRobotsTxt": true,
  "changefreq": "daily",
  "defaultPriority": 0.8,
  "includeImages": false
}

Key configuration settings explained

Start URLs (startUrls): The entry point for the crawler. Usually, this is your primary domain homepage.
Max Crawl Depth (maxCrawlDepth): Controls how deep the crawler traverses link structures. A depth of 3 to 5 is typically sufficient to discover all essential content while preventing infinite loops.
Regex Filtering (includePatterns and excludePatterns): Prevents cluttering your index with utility pages. You should explicitly exclude administrative paths like .*/admin/.*, checkout pages .*/cart.*, and query parameters that create duplicate content.
Robots.txt Respect (respectRobotsTxt): Ensures your pipeline adheres to site governance rules and crawler restrictions.

Output storage and Dataset generation

When execution completes, the Actor stores output artifacts in two locations:

Key-Value Store: Contains compiled sitemap.xml, sitemap.html, and sitemap.txt files ready for static hosting.
Apify Dataset: Stores structured JSON records for every discovered page. The Dataset ID produced during this run serves as the input payload for the next stage in our pipeline.

Stage 2: Programmatic submission via Google Cloud infrastructure

With your site URLs discovered and structured into an Apify Dataset, the next phase is notifying Google. This step relies on the official Google Indexing API, which allows site owners to submit batch update requests directly to Google's indexing pipeline.

Step 1: Setting up Google Cloud credentials

Before making programmatic requests, you must configure authentication through the Google Cloud Console.

Create a Google Cloud Project: Log in to Google Cloud Console, click the project dropdown, and select New Project. Name it seo-automation-pipeline.
Enable the Indexing API: Navigate to APIs & Services > Library, search for "Web Search Indexing API", and click Enable.
Create a Service Account: Go to IAM & Admin > Service Accounts, click Create Service Account, name it indexing-bot, and assign it the Project > Editor role.
Generate a JSON Key: Select your new service account, go to the Keys tab, click Add Key > Create new key, select JSON, and download the file. Keep this file secure, as it grants API permissions.

Service Account Email Example:
indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com

Step 2: Granting Google Search Console access

Enabling the API in Google Cloud is not enough on its own. Google requires verified domain ownership to prevent unauthorized indexing requests.

Open Google Search Console.
Select your verified domain property.
Navigate to Settings > Users and permissions.
Click Add user, paste your service account email address, set the role to Owner, and save.

IMPORTANT
Google Search Console requires Owner permissions for Service Accounts calling the Indexing API. Setting lower permission levels such as "Full" or "Restricted" will cause authentication errors during API calls.

Bypassing DIY limitations: Python vs. Apify Actors

Developers often attempt to write custom scripts to interact with the Google Indexing API. Below is a standard Python implementation using official Google authentication libraries:

import json
import requests
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession

SERVICE_ACCOUNT_FILE = "google-key.json"
INDEXING_API_ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish"

def get_authenticated_session(key_path):
 scopes = ["https://www.googleapis.com/auth/indexing"]
 credentials = service_account.Credentials.from_service_account_file(
 key_path, scopes=scopes
 )
    return AuthorizedSession(credentials)

def submit_url(session, url, action_type="URL_UPDATED"):
 payload = {
        "url": url,
        "type": action_type
 }
 response = session.post(INDEXING_API_ENDPOINT, json=payload)
    return response.status_code, response.json()

if __name__ == "__main__":
 session = get_authenticated_session(SERVICE_ACCOUNT_FILE)
 target_url = "https://example.com/blog/automated-seo-pipeline"
 status, result = submit_url(session, target_url)
    print(f"Status Code: {status}")
    print(f"Response: {json.dumps(result, indent=2)}")

Why custom scripts break at scale

While the DIY Python script works for single URLs, operating it in production exposes major infrastructure hurdles:

Rate limit handling: Google enforces daily quotas (typically 200 URL notifications per day for standard projects). Custom scripts will fail with HTTP 429 Too Many Requests unless you write exponential backoff algorithms.
State management: A basic script does not remember which URLs were submitted yesterday. Running it daily resubmits duplicate URLs, wasting your daily quota.
Execution server maintenance: You must deploy the script to a cloud server, manage cron schedules, and store credentials securely.

The solution: Google Indexer & Instant SEO Submitter

Instead of writing and hosting custom boilerplate code, we use the Google Indexer & Instant SEO Submitter Actor.

Feature	Custom Python Script	Google Indexer Actor
Quotas & Rate Limiting	Manual handling required	Built-in automatic rate limiting & backoff
Input Source	Hardcoded or manual files	Direct integration with Apify Datasets & Sitemaps
Pricing Model	Server hosting costs	Pay-Per-Event
Test Mode	Requires code flags	Built-in dry-run toggle
Monitoring	Custom logging setup	Real-time console logs & dataset outputs

To configure the Actor, pass the JSON credentials and your Dataset ID from Stage 1 directly into the input schema:

{
  "datasetId": "YOUR_STAGE_1_DATASET_ID",
  "serviceAccountJson": {
    "type": "service_account",
    "project_id": "seo-automation-pipeline",
    "private_key_id": "abcdef123456789",
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC...\n-----END PRIVATE KEY-----\n",
    "client_email": "indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com"
  },
  "action": "URL_UPDATED",
  "testMode": false
}

Stage 3: The missing link: Logging to Notion via MCP Connectors

Automating crawls and API submissions solves the execution side of SEO. However, engineering teams and SEO leads still face a visibility gap: How do you monitor pipeline results without manually checking console logs after each run?

Exporting CSVs after every run is tedious, and setting up custom webhooks requires maintaining dedicated receiver endpoints. This is where MCP connectors fit into the pipeline.

Understanding MCP connectors

The Model Context Protocol (MCP) is an open standard that lets Actors call third-party services like Notion, Slack, and GitHub on your behalf, using your own credentials, without the Actor ever holding your token directly.

MCP connectors on Apify work differently from the Apify MCP server. The Apify MCP server exposes Apify Actors as tools to external AI clients (such as Claude or Cursor). MCP connectors do the opposite: they let Actors running on the Apify platform call external MCP-compatible services. The two are independent and can be used together.

In this pipeline, we use the Notion Uploader Actor together with an Apify MCP connector to push the Google Indexer's results into a Notion database automatically after each run.

Setting up the Notion database

Create a dedicated database in your Notion workspace titled SEO Indexing Tracker.

Configure the database columns with the following properties:

Column Name	Property Type	Description
URL	Title	The target webpage URL submitted to Google
Submission Status	Select	`Indexed`, `Failed`, or `Quota Exceeded`
HTTP Code	Number	The status response code returned by Google (e.g., `200`, `429`, `403`)
Last Updated	Date	Timestamp of when the request occurred
Source Actor	Text	Name of the Apify Actor that executed the request

Authorizing the Notion MCP connector

MCP connectors are authorized once at the account level, not on individual Actor or task pages. To set one up:

Open Apify Console and go to Settings → API & Integrations → MCP connectors.
Click Add connector and select Notion.
Complete the OAuth flow — you will be redirected to Notion to authorize access to your workspace. The Apify platform never sees your token directly; authentication is handled server-side via the MCP proxy.
Once authorized, the connector appears as a selectable option when running any compatible Actor.

Connecting the Notion Uploader Actor

The Notion Uploader Actor reads any Apify dataset and uploads its rows into Notion. It accepts your MCP connector ID so you do not need to paste a Notion API key anywhere.

When the Notion Uploader runs as an Actor-to-Actor integration (triggered by the completion of the Google Indexer run), it automatically picks up the triggering run's output dataset — no explicit datasetId is needed in the payload template.

Here is an example input for the Notion Uploader when configured as a standalone run:

{
  "notionConnector": "<your-connector-id>",
  "notionDatabaseId": "https://www.notion.so/workspace/SEO-Indexing-Tracker-<db-id>",
  "datasetId": "<google-indexer-output-dataset-id>",
  "dedupeMode": "upsert",
  "keyProperty": "URL",
  "dryRun": false
}

Tip: Enable "dryRun": true on your first run. It resolves the field mapping, validates it against your Notion schema, and shows exactly what would be written — without creating any rows.

Orchestrating the full "set and forget" pipeline

Now that all three Actors are configured, we can chain them into an automated workflow: Fast Sitemap Generator discovers URLs, Google Indexer & Instant SEO Submitter submits them to Google, and Notion Uploader logs the results into Notion.

Step 1: Creating saved Actor tasks

An Actor task is a saved, reusable configuration of an Actor in Apify Console. Tasks let you pre-define all input parameters so they can be run on demand, via API, or on a schedule.

Crawler task: Open Fast Sitemap Generator in Apify Console, configure your startUrls, crawl depth, and regex filters, then click Save as a new task in the top-right corner. Name it sitemap-generator-task.
Indexer task: Open Google Indexer & Instant SEO Submitter, fill in your Google Cloud Service Account JSON key, then click Save as a new task. Name it google-indexer-task.
Uploader task: Open Notion Uploader, set your Notion MCP connector and target database URL, then click Save as a new task. Name it notion-uploader-task.

All saved tasks are listed under Saved tasks in the Apify Console left navigation.

Step 2: Chaining tasks using Actor-to-Actor integrations

The Integrations tab on each Actor or task page lets you chain runs together. When one task succeeds, Apify automatically triggers the next one and passes dynamic run data — such as the output dataset ID — via a payload template.

Chain 1: Sitemap Generator → Google Indexer

Open sitemap-generator-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose google-indexer-task as the target.
Set the trigger to Run succeeded.
Set the payload template so the indexer receives the crawler's output dataset:

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "action": "URL_UPDATED"
}

Chain 2: Google Indexer → Notion Uploader

Open google-indexer-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose notion-uploader-task as the target.
Set the trigger to Run succeeded.
Leave datasetId out of the payload template — the Notion Uploader automatically uses the triggering run's dataset when no explicit ID is provided:

{
  "dedupeMode": "upsert",
  "keyProperty": "URL"
}

Now the full chain fires end-to-end on its own. When sitemap-generator-task completes, Apify triggers google-indexer-task with the crawler's dataset. When the indexer finishes, Apify triggers notion-uploader-task, which reads the indexer's output dataset and writes each result row into your Notion database.

Step 3: Scheduling automated pipeline execution

Configure a recurring schedule so the entire chain runs automatically.

In the Apify Console left sidebar, click Schedules.
Click Create new schedule.
Select sitemap-generator-task as the target. The full chain — crawl, index, upload to Notion — fires automatically when this task runs.
Set the frequency using a Cron expression or one of the presets.

For high-frequency publishing (news, job boards): Schedule daily at 2:00 AM UTC.
For standard marketing blogs and commercial sites: Schedule weekly, every Monday morning.

Cron expression example (daily at 2:00 AM UTC):
0 2 * * *

Best practices for automated SEO indexing

To maintain a healthy indexing strategy while avoiding Google API flags, keep these best practices in mind:

1. Submit only modified or new content

Do not resubmit your entire site inventory every day. The Google Indexing API is intended for pages that have new content (URL_UPDATED) or pages that have been deleted (URL_DELETED). Resubmitting unchanged pages wastes daily quotas and can result in rate limiting.

2. Match canonical URLs strictly

Google requires strict URL matching. If your canonical structure uses https://example.com/blog/post-name/ with a trailing slash, ensure your crawler filters and API submissions match that format exactly. Submitting http:// or non-trailing-slash variations can trigger redirect loops and indexing failures.

3. Use Test Mode during initial setup

Before running full-scale API operations, turn on the Test Mode toggle in the Google Indexer & Instant SEO Submitter Actor. Test Mode executes the full crawl, verifies JSON authentication keys, and simulates Notion MCP logging without consuming your daily Google API quota.

4. Monitor quota usage in Google Cloud Console

Standard Google Cloud projects receive an initial quota of 200 Indexing API requests per day. If your site publishes hundreds of new pages daily, submit a quota increase request directly through the Quotas tab in Google Cloud Console.

Wrapping up

Building an automated technical SEO pipeline transforms indexing from a passive waiting game into a predictable, programmatic system.

Combining the cloud crawling capabilities of the Fast Sitemap Generator, the push notifications of the Google Indexer & Instant SEO Submitter, and real-time database logging through Notion MCP Connectors, you establish an automated SEO engine that operates reliably in the background.

Instead of waiting weeks for Google to discover your newest content organically, your site updates are crawled, verified, submitted, and logged in Notion within minutes.

Explore the Fast Sitemap Generator on the Apify Store.
Set up the Google Indexer & Instant SEO Submitter Actor with your Google Cloud Service Account.
Connect your Notion workspace via MCP Connectors to start tracking real-time indexing status today.

How to Automate Instagram Reels to Notion via MCP

Emmanuel Uchenna — Wed, 09 Sep 2026 09:14:53 +0000

Managing short-form video content across multiple campaigns often turns into an administrative nightmare. Your marketing team publishes dozens of public Reels, short video posts, and IGTV clips every month. Everything looks great on Instagram, but as soon as someone asks to archive these assets into a centralized brand library or content calendar, the entire operation grinds to a halt.

Suddenly, you are forced to rely on shady, ad-heavy download websites that fail half the time, paste individual links into manual file converters, or copy caption text into spreadsheets by hand. Even when you do manage to extract a direct video file URL from Instagram, that URL expires within hours due to Instagram's content delivery network (CDN) session restrictions.

What if you could automatically extract high-quality Instagram video files, store them permanently, and stream that structured data straight into Notion in real time?

With Instagram Video and Reel Downloader, the Notion Uploader Actor, and Apify Model Context Protocol (MCP) connectors, you can build a zero-maintenance, automated pipeline. This guide walks you through setting up an automated Instagram-to-Notion video content calendar that automatically syncs dataset outputs upon run completion without requiring manual CSV exports or fragile custom backend glue code.

The content archiving nightmare

Picture this scenario. You are a creative organization, and your creative team just finished a quarter-long influencer campaign. Thirty public Instagram Reels and video posts were published. Your client or department head wants every asset logged inside Notion with direct video links, preview thumbnails, shortcodes, captions, and publishing dates for future repurposing.

The traditional approach to this task usually follows one of two painful paths:

The manual downloader route: A team member spends hours copying Instagram links, pasting them into third-party downloader sites, closing popup ads, downloading .mp4 files locally, uploading those files into Google Drive or Notion, and typing metadata by hand. This process consumes dozens of work hours, produces inconsistent video resolutions, and risks downloading malware.
The legacy scraping route: A developer sets up a web scraper to pull Instagram data. The scraper runs for 10 minutes, completes its execution, and dumps a static JSON or CSV file to cloud storage. A developer then has to write custom webhooks or cron scripts to parse that output file, format the payload, and send REST API requests to Notion. If any record fails or if the script breaks mid-transit, debugging becomes a headache.

flowchart LR
    A["Scraper / Web Downloader"] --> B["CSV / JSON Export"]
    B --> C["Custom Glue Script / Manual Upload"]
    C --> D["Notion Database"]

Neither approach is ideal for growing teams or production workflows. Legacy scraping forces you to write custom integration pipelines to move data downstream. Meanwhile, manual downloading fails to scale beyond a handful of links.

Apify Actor integrations and Apify MCP connectors solve this problem seamlessly. Rather than manually writing custom export scripts or handling fragile API glue, you can attach the supported Notion Uploader Actor directly to your extraction workflow inside Apify Console.

flowchart LR
    A["Instagram Reels"] --> B["Instagram Video and Reel Downloader"]
    B -- On Run Succeeded --> C["Notion Uploader Actor Integration"]
    C -- Apify MCP Connector / API Key --> D["Notion Content Calendar"]

Once Instagram Video and Reel Downloader finishes extracting video assets and storing permanent .mp4 file links in Apify Key-Value Store, Apify automatically executes the Notion Uploader integration. Notion Uploader connects securely to your Notion workspace using an Apify MCP connector (configured in Apify Console under Settings → Integrations → MCP connectors) or a Notion API key, mapping fields and creating structured database rows automatically.

The pipeline architecture: how MCP connectors change the game

To understand why this architecture is so powerful, let's examine what happens under the hood during an automated run.

flowchart TD
    subgraph APIFY["APIFY PLATFORM"]
        A["Instagram /reel/ URLs"] --> B["Dual-Path Extraction Engine"]
        B --> C["Apify Key-Value Store<br/>(Permanent Video URL)"]
        B --> D["Apify Dataset<br/>(Structured Metadata)"]
    end

    D -- On Run Succeeded --> E["Notion Uploader Actor<br/>(filip_cicvarek/notion-uploader)"]
    E -- Apify MCP Connector / API Key --> F["Notion Content Calendar<br/>(Automated Database Rows)"]

1. Dual-path extraction engine

Instagram does not provide direct, permanent video download links through a simple public REST API. Furthermore, different post types behave differently:

Some standard post pages expose media tags within initial static HTML.
Complex Reel pages render content dynamically via JavaScript, obscuring video source URLs behind client-side network calls.

Instagram Video and Reel Downloader handles this seamlessly using a dual-path architecture. It first attempts a lightweight static HTTP request to extract the media link instantly. If the target page requires dynamic execution, the Actor automatically falls back to a browser session to capture media delivery requests.

2. Solving CDN URL expiration via Key-Value Store

Direct video URLs returned straight from Instagram's servers (*.cdninstagram.com or *.fbcdn.net) are session-bound and expire after a short period. If you paste a raw CDN video link into Notion, the video will stop playing within 24 to 48 hours.

To solve this, Instagram Video and Reel Downloader includes a storeVideoFiles configuration option in Apify Console. When set to true, the Actor streams the raw .mp4 file directly into Apify Key-Value Store during the run. This generates a permanent, publicly accessible URL (videoPublicUrl) that Notion can embed reliably forever.

3. Automated dataset push via Notion Uploader and MCP connectors

Model Context Protocol (MCP) is an open standard that allows tools and AI systems to exchange context and perform actions securely across application boundaries. In the Apify ecosystem, MCP connectors allow Actors like Notion Uploader to communicate directly with external services - such as Notion, HubSpot, Google Sheets, or Slack.

By configuring an Actor Integration on Instagram Video and Reel Downloader set to trigger on Run succeeded, Apify automatically hands off the generated dataset to Notion Uploader. Notion Uploader reads the structured items and uses the configured Notion MCP connector (or Notion API key) to map and upsert rows directly into your Notion database.

4. Zero-trust token security and governance

When connecting third-party services to web automations, storing API tokens in raw scripts or public Actor inputs poses severe security risks. Apify handles MCP authentication via OAuth 2.0 proxies managed centrally inside Apify Console under Settings → Integrations → MCP connectors.

Your Notion authorization token is never exposed in plain text, never saved in Actor dataset output, and never shared with the underlying script environment. The Apify platform handles identity verification and permission scope tokens securely behind the scenes.

Step 1: Setting up the Notion database

Before configuring the Actor, you need a designated Notion database structured to store video metadata and file references.

Open your Notion workspace and create a new database page (select Full page database or Inline database).
Name your database Instagram Content Calendar.
Configure the database properties to match the schema exported by Instagram Video and Reel Downloader:

Property Name	Notion Property Type	Purpose / Payload Source
Title	Title	Stores post caption snippet or title (`title` / `description`)
Instagram Link	URL	Direct link to original Instagram post (`canonicalUrl`)
Direct Video Link	URL	Permanent Apify storage link (`videoPublicUrl`)
Thumbnail	Files & media (or URL)	Preview thumbnail image (`thumbnailUrl`)
Shortcode	Text	Instagram unique ID (`shortcode`)
Media Type	Select	Post type (`reel`, `post`, or `tv`)
Fetch Date	Date	Timestamp of processing (`fetchedAt`)
Status	Status / Select	Processing result (`ready` or `error`)

erDiagram
    INSTAGRAM_CONTENT_CALENDAR {
        string Title "Summer Campaign Teaser"
        string Instagram_Link "instagram.com/reel/C123abc/"
        string Direct_Video_Link "api.apify.com/.../video-c123.mp4"
        string Media_Type "reel"
        string Status "ready"
    }

Ensure your Notion database view is set up to display URL and media properties clearly so your creative team can preview videos directly within the calendar workspace.

Step 2: Configuring Instagram Video and Reel Downloader

With your Notion database ready, navigate to Instagram Video and Reel Downloader on Apify Store.

Click Try for free or open the Actor inside Apify Console. Select the Input tab to configure your run.

Input JSON configuration

To ensure video URLs remain active inside Notion indefinitely, you must set storeVideoFiles to true. This instructs the Actor to download the binary media file into the Apify Key-Value Store and produce the videoPublicUrl parameter.

Here is an ideal production configuration snippet:

{
  "startUrls": [
    { "url": "https://www.instagram.com/reel/C8XyZ123abc/" },
    { "url": "https://www.instagram.com/p/C9AbC456def/" },
    { "url": "https://www.instagram.com/reel/C-Ghi789jkl/" }
  ],
  "storeVideoFiles": true,
  "storeThumbnailFiles": true,
  "includePageHtml": false,
  "filenamePrefix": "notion-archive-2026",
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Key configuration settings explained

startUrls: An array of target public Instagram URLs. Supports /reel/, /p/, and /tv/ paths.
storeVideoFiles: Set to true. This generates a permanent videoPublicUrl host link on api.apify.com, avoiding expired CDN link errors inside Notion.
storeThumbnailFiles: Set to true if you want stored key-value records for thumbnail images as well.
filenamePrefix: A custom string prepended to all stored files inside Apify storage (e.g., notion-archive-2026).
maxConcurrency: Set to 2 or 3. Instagram actively monitors aggressive parallel scraping. Keeping concurrency low ensures reliable extraction without triggering rate limits.
proxyConfiguration: Enabling Apify Proxy distributes requests across residential or datacenter IP pools, preventing unexpected block screens.

Step 3: Wiring the Notion Uploader integration in Apify Console

Once your input JSON is defined for Instagram Video and Reel Downloader, you attach the Notion Uploader Actor integration directly to the run configuration inside Apify Console.

1. Creating your Notion MCP connector (or API key)

Before attaching the integration, establish your connection to Notion inside Apify Console:

In Apify Console, open Settings → API & Integrations → MCP connectors.
Click Add connector and select Notion.
Complete the one-click OAuth authorization window to grant Apify access to your Notion workspace and database.
Apify Console generates a secure connector ID (e.g. mcp-connector-...) that allows Actors to interact with Notion with your permissions, without ever exposing raw private tokens.

(Alternative: You can also use a classic internal integration secret starting with ntn_ created at notion.so/profile/integrations. Be sure to invite your integration to the Notion database via *••• → Connections*).

flowchart LR
    A["Apify Console Settings"] --> B["MCP Connectors"]
    B --> C["Add Notion Connector (OAuth)"]
    C --> D["Secure Connector Session ID"]

2. Adding the Notion Uploader integration to your Actor

With your Notion MCP connector ready, wire the integration to Instagram Video and Reel Downloader:

Open Instagram Video and Reel Downloader in Apify Console.
Select the Integrations tab located in the navigation panel below the Actor title.
Click Add integration. Under Suggested for this Actor, click Notion Uploader (filip_cicvarek/notion-uploader). (Note: Apify Console also lists third-party integration routes like Make and Zapier under "Integrate with third-party tools", but Notion Uploader provides the direct, native Apify Actor integration path).
Set Start when to Run succeeded. This ensures Notion Uploader fires automatically the moment your Instagram extraction run completes cleanly.

3. Mapping fields and configuring Notion Uploader input

Under the Actor input section for the Notion Uploader integration, configure the target database and field mapping:

Notion connector: Select your authorized Notion MCP connector from the dropdown list (or paste your notionApiKey).
Notion database: Paste your Notion database URL or ID (notionDatabaseId).
Explicit field mapping (dataMapping): Map the dataset output fields from Instagram Video and Reel Downloader to your Notion database properties:

{
  "Title": "description",
  "Instagram Link": "canonicalUrl",
  "Direct Video Link": "videoPublicUrl",
  "Thumbnail": "thumbnailUrl",
  "Shortcode": "shortcode",
  "Media Type": "mediaType",
  "Fetch Date": "fetchedAt"
}

Duplicate handling (rerun safety): Set Dedupe mode (dedupeMode) to upsert and set Key property (keyProperty) to Instagram Link (or Shortcode). This guarantees that re-running the Actor with previously processed URLs will update existing database rows instead of creating duplicate entries.

Click Save integration. The integration is now active and bound to your Actor task.

Step 4: Executing the run and verifying the content calendar

With the input parameters set and the Notion Uploader integration saved, click the Start button in the top-right corner of Apify Console.

Watching automated workflow execution

As Instagram Video and Reel Downloader runs, you can monitor the progress through the Log tab:

The Actor extracts public Instagram video posts, Reels, and IGTV links specified in startUrls.
For each URL, it resolves media endpoints and streams binary .mp4 files into Apify Key-Value Store, generating permanent videoPublicUrl host links.
Integration handoff: When the extraction run finishes successfully (Run succeeded), Apify immediately launches Notion Uploader (filip_cicvarek/notion-uploader), passing the run's default dataset ID automatically.
Notion Database update: Notion Uploader uses your Notion MCP connector to read the dataset items, validate properties against your database schema, apply upsert deduplication, and populate your Instagram Content Calendar in Notion.

sequenceDiagram
    autonumber
    participant Downloader as Instagram Video Downloader
    participant KVS as Key-Value Store
    participant Integration as Notion Uploader Integration
    participant Notion as Notion Database

    Note over Downloader: Downloader starts execution
    Downloader->>KVS: Extracts Reel & uploads .mp4 file
    Note over Downloader: Run completes (Status: SUCCESS)
    Downloader->>Integration: Triggers Notion Uploader automatically
    Integration->>Notion: Reads dataset & upserts database rows
    Note over Notion: Content calendar fully updated!

If an invalid or private URL is encountered, the extraction Actor records a structured dataset item containing an errorCode (such as NO_VIDEO_URL) and errorMessage. Notion Uploader writes an error status row into Notion, maintaining a complete audit log without breaking the pipeline execution.

Extending the workflow with API triggers and scheduling

Building a one-off run inside Apify Console is great for testing, but real content operations require automated execution. You can automate this pipeline using code triggers or scheduled jobs.

Programmatic execution via JavaScript / TypeScript

You can trigger the integrated pipeline programmatically using the official apify-client SDK in Node.js or TypeScript by invoking your saved task (which includes the Notion Uploader integration):

import { ApifyClient } from 'apify-client';

// Initialize the client with your Apify API token
const client = new ApifyClient({
  token: 'YOUR_APIFY_API_TOKEN',
});

// Define input parameters with video storage enabled
const input = {
  startUrls: [
    { url: 'https://www.instagram.com/reel/C8XyZ123abc/' },
    { url: 'https://www.instagram.com/reel/C9AbC456def/' }
  ],
  storeVideoFiles: true,
  storeThumbnailFiles: true,
  filenamePrefix: 'weekly-content-sync',
  maxConcurrency: 2,
  proxyConfiguration: {
    useApifyProxy: true
  }
};

async function runInstagramSync() {
  console.log('Starting Instagram to Notion sync workflow...');

  // Call the saved task (or Actor). Configured integrations run automatically on completion.
  const run = await client.task('YOUR_SAVED_TASK_ID').call(input);

  console.log(`Run completed! View dataset at: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
  console.log('Notion Uploader integration triggered automatically.');
}

runInstagramSync();

Programmatic execution via Python

If your infrastructure runs on Python, use apify-client to trigger the sync:

from apify_client import ApifyClient

# Initialize client
client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Prepare input dictionary
run_input = {
    "startUrls": [
        {"url": "https://www.instagram.com/reel/C8XyZ123abc/"},
        {"url": "https://www.instagram.com/reel/C9AbC456def/"}
    ],
    "storeVideoFiles": True,
    "storeThumbnailFiles": True,
    "filenamePrefix": "python-content-sync",
    "maxConcurrency": 2,
    "proxyConfiguration": {
        "useApifyProxy": True
    }
}

# Run the saved task with Notion integration attached
print("Launching Instagram Video Downloader with Notion Uploader Integration...")
run = client.task("YOUR_SAVED_TASK_ID").call(task_input=run_input)

print(f"Workflow complete. Dataset ID: {run['defaultDatasetId']}")

Automated set-and-forget scheduling

To eliminate manual execution:

Save your input configuration and Notion Uploader integration inside Apify Console as a Saved Task (e.g., Instagram-to-Notion-Sync-Task).
Navigate to the Schedules tab in Apify Console and click Create schedule.
Link the schedule to your saved Actor task.
Set a cron cadence, such as every Friday at 5:00 PM (0 17 * * 5).

Every week, Apify will automatically run Instagram Video and Reel Downloader, extract new video links, upload permanent .mp4 binaries to Key-Value Store, and trigger Notion Uploader to push the assets into your Notion database.

Wrapping up

Relying on ad-ridden video download websites, manual file management, and fragile spreadsheet uploads is no longer necessary for modern media operations.

By combining the extraction capabilities of Instagram Video and Reel Downloader with the Notion Uploader Actor and Apify MCP connectors, you can build a seamless content archiving pipeline. Permanent video hosting via Apify Key-Value Store eliminates broken CDN links, while automated Actor integrations push structured data straight into Notion without requiring custom backend glue code.

Ready to automate your social media asset pipeline?

Try Instagram Video and Reel Downloader on Apify Store today, connect your Notion workspace via Notion Uploader, and build your automated content calendar in minutes.

Headless browsers vs. SERP APIs: Why your Playwright scraper fails in production

Emmanuel Uchenna — Wed, 09 Sep 2026 09:14:32 +0000

Every developer who builds web scrapers goes through the same initial rite of passage. You need search engine data for an SEO tracking tool, an AI retrieval pipeline, or a market research project. Your first instinct is to reach for a browser automation tool like Playwright, Puppeteer, or Selenium.

You write a quick script, launch a headless Chromium instance, navigate to Google, inspect the Document Object Model (DOM), and extract text from <h3> tags and organic result containers.

On your local development machine, it feels like magic. The script runs cleanly for your first five test queries.

Then, you deploy your script to a production server on AWS or DigitalOcean, and everything breaks down. Within minutes, your scraper encounters roadblock CAPTCHAs, gets flagged by automated anti-bot systems, or silently returns empty arrays because the search engine rolled out an unannounced layout tweak.

Environment	Execution Path	Production Outcome
Localhost (Prototyping)	Playwright ==> Residential Home IP ==> Google	🟢 200 OK (5 results parsed successfully)
Cloud Server (DIY)	Playwright ==> Datacenter Cloud IP ==> Google	🔴 429 Blocked / CAPTCHA Wall (Scraper fails)
Managed API (SearchApi)	App Worker ==> HTTP GET ==> SearchApi Infrastructure	⚡ 200 OK (Instant structured JSON payload)

While headless browsers are exceptional tools for end-to-end web testing and scraping low-security websites, using them to scrape Search Engine Results Pages (SERPs) is an expensive engineering trap.

In this guide, we break down the hidden operational costs of DIY browser automation, compare side-by-side Python implementations, and explore why switching to a dedicated SERP API like SearchApi.io saves your engineering team time, infrastructure budget, and maintenance headaches.

The real cost of doing it yourself

Developers naturally enjoy building systems from the ground up. However, building a custom search engine scraper means entering an ongoing cat-and-mouse game against some of the most sophisticated anti-bot security teams in the world.

When you scale a headless browser scraper beyond a handful of casual requests, you immediately run into four severe architectural bottlenecks:

flowchart LR
    Root["DIY Browser Scraping Bottlenecks"] --> B1["1. Proxy Infrastructure Tax<br/>High residential bandwidth costs"]
    Root --> B2["2. Fragile Selector Maintenance<br/>Random CSS class changes break code"]
    Root --> B3["3. Server Resource Drain<br/>150MB+ RAM per tab, CPU spikes"]
    Root --> B4["4. Anti-Bot Fingerprinting<br/>TLS, Canvas, WebGL, JA3 detection"]

1. The proxy infrastructure tax

Search engines can easily identify and block automated requests originating from data center IP ranges (such as AWS, Google Cloud Platform, or DigitalOcean). If you send five consecutive queries from an AWS EC2 instance without a proxy, your IP address is flagged almost immediately.

To bypass this restriction, you must purchase and configure a residential proxy network. Residential proxies route your requests through real residential consumer connections, making them appear legitimate. However, managing proxies introduces heavy technical complexity:

Bandwidth costs: Residential proxy providers charge based on data transfer, usually between $5 and $15 per gigabyte. When a headless browser loads a search results page, it downloads not just the text, but also JavaScript bundles, stylesheets, tracking pixels, and images. Loading these unnecessary assets inflates your proxy bandwidth bill significantly.
Pool rotation and stickiness: You must write custom pooling logic to rotate IP addresses across queries while maintaining sticky sessions when handling multi-page pagination.
Gateway latency: Routing traffic through multiple proxy hops adds anywhere from 1 to 4 seconds of network latency to every single request.

2. Fragile HTML selector maintenance

Search engines do not use semantic, human-readable HTML markup. Instead of providing clear class names like .search-result-item or .result-title, they rely on minified, randomized, or hashed CSS selectors that change frequently.

<!-- What you hope to see in the DOM -->
<div class="search-result">
  <h3 class="title">Web Scraping Guide</h3>
  <p class="snippet">Learn how to extract data...</p>
</div>

<!-- What Google actually renders -->
<div class="MjjYud">
  <div class="g Ww4FFb vt6AZc">
    <div class="kvH3df">
      <div class="VwiC3b yD755b xDu2fd">
        <span>Learn how to extract data...</span>
      </div>
    </div>
  </div>
</div>

If your code relies on fragile selectors like div.g or .VwiC3b, an unannounced A/B test or markup refresh by the search engine will cause your parser to fail silently. You will receive empty datasets, triggering alerts and requiring developers to drop their current sprint tasks to inspect DOM trees and update CSS queries.

3. Massive server resource drain

Headless browsers are full browser execution environments. Running an instance of headless Chromium or Firefox forces your host server to parse HTML, evaluate complex JavaScript scripts, construct DOM trees, and calculate CSS layout rules.

Memory consumption: A single headless browser tab typically consumes between 150MB and 300MB of RAM. If your application needs to handle 30 concurrent search queries, your server needs at least 8GB to 16GB of dedicated RAM just to keep the browser processes alive.
CPU spikes: Initializing browser contexts and rendering client-side JavaScript creates sharp CPU spikes, often requiring expensive multi-core cloud compute instances.
Process zombie leaks: Long-running browser processes frequently experience memory leaks. Without aggressive lifecycle management and process termination routines, orphaned Chromium processes will gradually consume all available server memory and crash your host container.

4. Advanced anti-bot fingerprinting and CAPTCHAs

Modern search engines employ sophisticated behavioral and cryptographic bot detection techniques. They do not just check your User-Agent header; they inspect deep browser characteristics:

TLS and JA3/JA4 fingerprinting: The way your networking stack negotiates SSL/TLS handshakes reveals whether your request comes from a genuine desktop browser or an automated runtime.
Navigator and JavaScript runtime properties: Automated environments often leak telltale attributes such as navigator.webdriver = true, missing system plugin arrays, or default WebGL vendor strings.
Canvas and audio fingerprinting: Scripts render hidden shapes or audio signals to generate a unique hardware signature.
Behavioral heuristics: Mouse movement velocity, keystroke timings, and scroll dynamics are analyzed in real time.

Even when using stealth plugins like puppeteer-extra-plugin-stealth or custom Playwright evasion flags, these countermeasures are brittle. Anti-bot vendors continuously update their detection models, leaving your custom scraper vulnerable to sudden blocks and CAPTCHA walls.

The abstracted alternative: Dedicated SERP APIs

A dedicated Search Engine Results Page (SERP) API transforms search engine data collection into a fully managed infrastructure service.

Instead of launching browsers, rotating proxies, handling CAPTCHAs, and maintaining brittle DOM selectors, you offload the entire operational pipeline to a specialized provider like SearchApi.io.

flowchart LR
    App["Your Application"] -->|1. Simple HTTP GET| API["SearchApi.io Engine"]

    subgraph ManagedInfra["Managed Infrastructure Layer"]
        API --> Proxies["Residential & Mobile Proxies"]
        API --> Browsers["Stealth Browser Fleet"]
        API --> Parsers["Auto-Updating DOM Parsers"]
    end

    ManagedInfra -->|2. Automated Request| Google["Search Engines (Google, Bing)"]
    Google -->|3. Live SERP Response| ManagedInfra
    API -->|4. Clean Structured JSON| App

SearchApi.io handles every layer of the extraction process behind a clean REST Application Programming Interface (API):

Intelligent proxy rotation: Automatically routes queries through optimized residential and mobile IP pools.
Fingerprint emulation: Emulates authentic browser signatures at both the network (TLS/HTTP2) and JavaScript runtime levels.
Automated parsing: Upstream parsers continuously monitor search engine layout updates. When Google changes a CSS class, SearchApi.io updates its internal parsers immediately, ensuring your application always receives structured, consistent JSON without code modifications.
Lightweight delivery: Your server sends a lightweight HTTP request and receives clean, parsed JSON in milliseconds, using minimal CPU and memory.

Side-by-side implementation: Playwright vs. SearchApi

To see the difference in code simplicity, maintainability, and resource footprint, let us compare two complete Python implementations designed to accomplish the exact same task: searching Google for "Top web scraping frameworks in 2026" and extracting the top organic results (position, title, URL, and snippet text).

Building a custom Playwright scraper in Python

The following script represents the DIY approach. It launches an automated Chromium browser, configures viewport parameters, attempts to bypass basic bot flags, waits for network idle states, and parses HTML using CSS selectors.

"""
DIY Google Search Scraper using Playwright
Requires: pip install playwright
Run: playwright install chromium
"""

import asyncio
from typing import Any, Dict, List
from playwright.async_api import async_playwright


async def scrape_google_diy(search_query: str) -> List[Dict[str, Any]]:
    print(f"Launching headless Chromium for query: '{search_query}'...")

    search_results: List[Dict[str, Any]] = []

    async with async_playwright() as p:
        # Launch Chromium with anti-detection flags
        browser = await p.chromium.launch(
            headless=True,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
                "--disable-setuid-sandbox",
                "--disable-infobars",
                "--window-size=1920,1080",
            ],
        )

        # Configure browser context with realistic desktop parameters
        context = await browser.new_context(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            viewport={"width": 1920, "height": 1080},
            locale="en-US",
            timezone_id="America/New_York",
        )

        page = await context.new_page()

        # Format Google search URL
        encoded_query = search_query.replace(" ", "+")
        url = f"https://www.google.com/search?q={encoded_query}&hl=en"

        try:
            # Navigate and wait for DOM network idle state
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)

            # Define fragile CSS selectors for organic search results
            organic_container_selector = "div.g, div.MjjYud"
            await page.wait_for_selector(
                organic_container_selector, timeout=7000
            )

            # Query all potential organic result cards
            containers = await page.query_selector_all(
                organic_container_selector
            )
            position = 1

            for container in containers:
                # Extract title from h3 tag
                title_elem = await container.query_selector("h3")
                # Extract link anchor tag
                link_elem = await container.query_selector("a")
                # Extract snippet text using known class heuristics
                snippet_elem = await container.query_selector(
                    "div.VwiC3b, div[style*='-webkit-line-clamp']"
                )

                title = (
                    await title_elem.inner_text() if title_elem else "N/A"
                )
                link = (
                    await link_elem.get_attribute("href")
                    if link_elem
                    else "N/A"
                )
                snippet = (
                    await snippet_elem.inner_text() if snippet_elem else "N/A"
                )

                # Filter out nested navigational links and empty blocks
                if (
                    title != "N/A"
                    and link.startswith("http")
                    and "google.com" not in link
                ):
                    search_results.append(
                        {
                            "position": position,
                            "title": title,
                            "link": link,
                            "snippet": snippet,
                        }
                    )
                    position += 1

                if len(search_results) >= 5:
                    break

        except Exception as error:
            print(f"Scraping failed due to selector timeout or block: {error}")
        finally:
            await context.close()
            await browser.close()

    return search_results


if __name__ == "__main__":
    query = "Top web scraping frameworks in 2026"
    results = asyncio.run(scrape_google_diy(query))

    print(f"\nRetrieved {len(results)} results via Playwright:")
    for item in results:
        print(f"[{item['position']}] {item['title']}\n    Link: {item['link']}")

The streamlined SearchApi approach

Now, let us examine the equivalent implementation using SearchApi.io. Notice how all browser lifecycle management, selector queries, and network evasions disappear.

"""
Production-ready Google Search Extraction using SearchApi
Requires: pip install requests python-dotenv
"""

import os
from typing import Any, Dict, List, Optional
from dotenv import load_dotenv
import requests

load_dotenv()


def scrape_google_with_searchapi(
    search_query: str,
    location: str = "United States",
    device: str = "desktop",
) -> List[Dict[str, Any]]:
    print(f"Requesting structured payload for: '{search_query}'...")

    api_key = os.getenv("SEARCHAPI_API_KEY")
    if not api_key:
        raise ValueError("Missing SEARCHAPI_API_KEY environment variable.")

    # SearchApi Google Search endpoint
    endpoint = "https://www.searchapi.io/api/v1/search"
    params = {
        "engine": "google",
        "q": search_query,
        "api_key": api_key,
        "location": location,
        "device": device,
        "hl": "en",
        "gl": "us",
    }

    try:
        response = requests.get(endpoint, params=params, timeout=15)
        response.raise_for_status()

        payload = response.json()
        organic_results = payload.get("organic_results", [])

        extracted_data = []
        for result in organic_results[:5]:
            extracted_data.append(
                {
                    "position": result.get("position"),
                    "title": result.get("title"),
                    "link": result.get("link"),
                    "displayed_link": result.get("displayed_link"),
                    "snippet": result.get("snippet"),
                }
            )

        return extracted_data

    except requests.exceptions.HTTPError as http_err:
        print(f"SearchApi HTTP error: {http_err.response.status_code} - {http_err.response.text}")
        return []
    except requests.exceptions.RequestException as error:
        print(f"Network request error: {error}")
        return []


if __name__ == "__main__":
    query = "Top web scraping frameworks in 2026"
    results = scrape_google_with_searchapi(query, location="New York, New York, United States")

    print(f"\nRetrieved {len(results)} results via SearchApi:")
    for item in results:
        print(f"[{item['position']}] {item['title']}\n    Link: {item['link']}\n    Display: {item['displayed_link']}")

Code walkthrough and key differences

Let us break down what happens behind the scenes in both codebases:

Metric	DIY Playwright Scraper	Managed SearchApi Solution
Code Complexity	~100 lines of boilerplate	~35 lines of standard Python
Dependencies	Playwright + Chromium binary (~300MB)	Standard HTTP client (`requests` < 5MB)
Memory Footprint (RAM)	150MB - 300MB per active tab	< 10MB per HTTP worker
Maintenance & Recovery	Manual DOM re-inspection on layout updates	Automatic upstream parser updates

Lines of code and complexity: The Playwright script requires over 90 lines of boilerplate, including browser process configuration, context options, selector query strings, element null-checks, and error catching. The SearchApi implementation requires under 35 lines of standard, clean Python.
Binary footprint: Running Playwright in a Docker container or serverless function requires downloading and bundling a full Chromium binary (~300MB). SearchApi runs with a lightweight HTTP client like requests or httpx (< 5MB total dependency size).
Data format integrity: In the Playwright snippet, if Google alters .VwiC3b to a new CSS class name, the script returns N/A for all snippets without throwing a visible network error. In contrast, SearchApi returns a normalized JSON object where fields like title, link, snippet, displayed_link, and sitelinks remain consistent regardless of Google's internal layout tests.

Technical performance and cost showdown

When deciding between building an internal headless browser cluster and integrating an API, you must evaluate total operational overhead across compute, proxies, reliability, and engineering salaries.

Detailed architectural comparison

The table below highlights the technical trade-offs between both approaches:

Feature / Metric	DIY Headless Browsers (Playwright / Puppeteer)	Dedicated SERP API (SearchApi)
Initial Setup Time	Days to weeks (proxy setup, evasion scripting)	5 minutes (API key integration)
Maintenance Overhead	High (ongoing selector patches and bot updates)	Zero (handled upstream by provider)
Memory Usage (RAM)	150MB - 300MB per concurrent tab	< 10MB per HTTP worker
CPU Utilization	High (JavaScript parsing and layout rendering)	Minimal (JSON deserialization)
Bandwidth per Query	1.5MB - 3.5MB (full HTML, CSS, JS assets)	15KB - 40KB (compressed JSON response)
Proxy Management	Manual (residential pools, rotation, sticky IPs)	Fully managed and automated
CAPTCHA Resolution	Requires third-party solvers or browser hooks	Handled automatically with 99.9% success
Data Format	Raw HTML / Unstructured DOM elements	Pre-parsed, structured JSON
Scalability	Complex (orchestrating browser clusters on Kubernetes)	Seamless (call API endpoints with high concurrency)
Availability / SLA	Unpredictable (prone to sudden bans)	Guaranteed 99.9% uptime SLA

Real-world cost breakdown at scale

Many engineering teams assume that DIY scraping is "free" because open-source tools like Playwright cost nothing to download. However, hosting and maintaining a high-volume scraper reveals significant hidden costs.

Let us model the estimated monthly expense of running 100,000 search queries per month across both architectures:

flowchart TD
    subgraph DIY["DIY Headless Browsers: ~$2,296 / month"]
        D_Proxies["Residential Proxies (200GB @ $8/GB)<br/>$1,600"]
        D_Eng["Engineering Maintenance (6 hrs @ $90/hr)<br/>$540"]
        D_Compute["Cloud Compute (2x 8GB Instances)<br/>$96"]
        D_Captcha["CAPTCHA Solver Service<br/>$60"]
    end

    subgraph API["Managed SERP API (SearchApi): $250 / month"]
        A_Tier["100k Searches (BigData Plan)<br/>$250"]
        A_Maint["Zero Infrastructure & Maintenance Overhead<br/>$0"]
    end

Proxy bandwidth amplification: Because headless browsers download client assets, 100k queries easily consume 200GB+ of residential proxy bandwidth. At $8 per GB, proxies alone cost $1,600 per month.
Server infrastructure: Running 20 to 30 concurrent Playwright instances requires dedicated virtual machines (e.g. AWS EC2 t4g.xlarge or c6i.xlarge), costing roughly $96 to $140 per month.
Engineering time: When selectors break twice a month, an engineer spends 3 to 4 hours diagnosing issues, testing selectors, and deploying hotfixes. At an average developer cost of $90 per hour, maintenance costs exceed $540 every month.
By contrast, using a managed SERP API (such as SearchApi's BigData Plan at $250 per month for 100,000 searches) consolidates your entire bill into a single, predictable subscription with a 99.9% SLA, saving over $2,000 every month while eliminating developer on-call fatigue.

Handling advanced search engine layouts and rich snippets

Modern search results are no longer simple lists of ten blue links. Search engines present dynamic, highly interactive features designed to answer user questions directly on the page:

People Also Ask (PAA): Accordion dropdowns that load dynamic content via asynchronous JavaScript calls upon being clicked.
Knowledge Graph panels: Rich sidebars containing entity facts, social profiles, founders, and related topics.
Google Maps and local packs: Interactive business listings with customer review ratings, operating hours, and geocoordinates.
AI Overviews and rich snippets: Dynamically streamed answers, product price tags, ratings, and video carousels.

flowchart TD
    SERP["Search Engine Results Page (SERP)"] --> AI["AI Overview / Rich Answer Box<br/>(Dynamic streaming content)"]
    SERP --> Main["Main Results Stream"]
    SERP --> Side["Entity Sidebar"]

    Main --> PAA["People Also Ask (PAA)<br/><i>(Interactive JavaScript accordions)</i>"]
    Main --> Maps["Local 3-Pack / Google Maps<br/>(Dynamic scroll & geocoordinates)"]
    Main --> Organic["Organic Search Results<br/>(Titles, URLs, sitelinks, snippets)"]

    Side --> KG["Knowledge Graph Panel<br/>(Entity attributes, social links, facts)"]

The challenge with DIY extraction of interactive elements

Extracting these advanced components with Playwright is remarkably complex:

To scrape People Also Ask questions, your script must locate each accordion element, issue click actions to trigger JavaScript expand events, wait for animation frames to finish, and parse the newly inserted DOM nodes.
To scrape Local Maps packs, your script must interact with map containers, handle scrolling viewports, and extract coordinates embedded inside nested JavaScript payloads.
To scrape Knowledge Graph boxes, your selectors must account for dozens of entity variations (companies, celebrities, movies, books, and locations), each of which renders with different HTML tags.

Writing and maintaining code for all these edge cases requires hundreds of lines of brittle automation logic.

Extracting structured SERP data effortlessly

With SearchApi, extracting interactive widgets requires zero browser actions. The API automatically parses every SERP feature into clean, dedicated JSON objects:

{
  "search_metadata": {
    "id": "search_65df89bc12e4",
    "status": "Success",
    "created_at": "2026-08-29T05:12:00.000Z",
    "request_time_taken": 0.62,
    "parsing_time_taken": 0.22,
    "total_time_taken": 0.84
  },
  "ai_overview": {
    "markdown": "Playwright is an open-source automation library and framework created by Microsoft for end-to-end testing across Chromium, Firefox, and WebKit.",
    "reference_links": [
      {
        "title": "Playwright: Fast and reliable end-to-end testing",
        "link": "https://playwright.dev",
        "source": "playwright.dev"
      }
    ]
  },
  "knowledge_graph": {
    "kgmid": "/g/11f555cn8l",
    "knowledge_graph_type": "Software library",
    "title": "Playwright",
    "type": "Software library",
    "description": "Playwright is an open-source automation library for browser testing and web scraping developed by Microsoft.",
    "source": {
      "name": "Wikipedia",
      "link": "https://en.wikipedia.org/wiki/Playwright_(software)"
    },
    "website": "https://playwright.dev"
  },
  "related_questions": [
    {
      "question": "Is Playwright better than Selenium?",
      "answer": "Playwright offers faster execution, native auto-waiting, built-in async support, and modern dev tools integration compared to Selenium.",
      "source": {
        "title": "Playwright vs Selenium: Modern Web Automation",
        "link": "https://example.com/playwright-vs-selenium",
        "displayed_link": "https://example.com > playwright-vs-selenium"
      }
    }
  ],
  "organic_results": [
    {
      "position": 1,
      "title": "Playwright: Fast and reliable end-to-end testing",
      "link": "https://playwright.dev",
      "displayed_link": "https://playwright.dev",
      "domain": "playwright.dev",
      "snippet": "Playwright enables reliable end-to-end testing for modern web apps across all modern rendering engines.",
      "sitelinks": {
        "inline": [
          {"title": "Docs", "link": "https://playwright.dev/docs/intro"},
          {"title": "Python API", "link": "https://playwright.dev/python/docs/intro"}
        ]
      }
    }
  ]
}

Your data pipeline can instantly access payload["ai_overview"]["markdown"], payload["knowledge_graph"], or payload["related_questions"] without having to simulate user clicks or manage animation delays.

Decision matrix: Choosing the right tool for your project

Headless browsers and SERP APIs both have distinct places in modern software engineering. The key is applying the right tool to the right problem.

flowchart TD
    Start{"What are you scraping?"}

    Start -->|Search Engine Results<br/>Google, Bing, Yahoo, Baidu| SERPBranch["Use a Dedicated SERP API<br/>(e.g. SearchApi)"]
    Start -->|General Web Pages<br/>Internal tools, small blogs, custom apps| WebBranch["Use Headless Browsers<br/>(Playwright, Puppeteer)"]

    SERPBranch --> S1["Fast, pre-parsed structured JSON"]
    SERPBranch --> S2["Zero proxy or CAPTCHA management"]
    SERPBranch --> S3["Guaranteed 99.9% uptime SLA"]

    WebBranch --> W1["Full UI interaction & custom click flows"]
    WebBranch --> W2["Session & SSO authentication handling"]
    WebBranch --> W3["Visual testing & PDF/screenshot generation"]

When to use headless browsers (Playwright, Puppeteer, Selenium)

End-to-end application testing: Automating user journeys, validating form submissions, and testing web applications in CI/CD pipelines.
Scraping authenticated enterprise portals: Logging into internal corporate dashboards or web apps behind custom Single Sign-On (SSO) login systems.
Visual regression testing and screenshots: Capturing full-page screenshots, rendering PDFs, and verifying UI visual consistency.
Low-security, dynamic niche sites: Extracting content from single-page web applications (SPAs) that do not enforce strict anti-bot measures.

When to use a dedicated SERP API (SearchApi)

SEO rank tracking and SERP monitoring: Tracking organic rankings, featured snippets, and local search visibility for thousands of keywords daily.
AI agent web search and RAG pipelines: Providing real-time search capabilities and clean web context to Large Language Models (LLMs) without latency bottlenecks.
E-commerce price intelligence and Google Shopping: Extracting structured product pricing, seller ratings, and stock availability at scale.
Market research and competitor tracking: Monitoring news trends, brand mentions, and Knowledge Graph developments with guaranteed uptime.

Wrapping up

Building your own search engine scraper with Playwright feels like a quick win during prototyping. But as your project scales, the hidden costs of residential proxies, anti-bot defenses, fragile DOM selectors, and heavy server hardware turn a simple script into a continuous infrastructure burden.

Developer time is your team's most valuable asset. Spending hours debugging broken CSS selectors and configuring proxy rotation takes engineering focus away from building core features that deliver real customer value.

By shifting your search data collection to a managed service like SearchApi.io, you eliminate infrastructure complexity:

Replace 100+ lines of fragile automation scripts with a clean, single HTTP GET call.
Reduce cloud compute and proxy bandwidth expenses by over 80%.
Gain instant access to structured JSON data for organic results, People Also Ask questions, Knowledge Graphs, and local packs.
Enjoy guaranteed 99.9% uptime backed by enterprise-grade proxy rotation and automatic parser maintenance.

Stop wrestling with headless browser memory leaks and CAPTCHA blocks. Sign up for SearchApi, claim your free API credits, and start extracting reliable search engine data in minutes.

Building a real-time AI search agent with SearchApi and OpenAI

Emmanuel Uchenna — Fri, 21 Aug 2026 04:38:31 +0000

Large Language Models (LLMs) are remarkably capable, but they suffer from two fundamental flaws: knowledge cutoffs and hallucinations. Ask an offline model about a breaking news event, a shifting stock price, or a new library release that dropped this morning, and it will either confidently fabricate a wrong answer or admit it has no idea. To build production-grade AI tools, your models need eyes and ears on the live web; your models need to be in the know of happenings as they break.

While static Retrieval-Augmented Generation (RAG) works well for searching internal vector databases, it fails the moment an accurate answer requires a real-time Google search. That is where AI search agents come in. An autonomous search agent turns an LLM into a reasoning engine that can choose when to browse the web, pull down search engine results pages (SERPs), synthesize real-time data, and deliver accurate, fact-checked answers.

In this article, you will build a production-ready AI search agent from scratch using Python, OpenAI's tool-calling framework, and SearchApi.io as the live web data pipeline.

Why traditional web scraping fails for AI agents

When developers realize their AI agent needs Google data, the first instinct is usually to reach for raw browser automation tools like Playwright, Puppeteer, or BeautifulSoup. You write code to open a headless browser, navigate to Google, type a query, and scrape the HTML.

In a local testing environment, this approach works for about three requests. In production, it breaks immediately.

Building your own search engine scraper means constantly fighting multi-billion-dollar anti-bot security teams. Your script will quickly run into:

IP blocks and CAPTCHAs: Search engines rapidly flag and block data center IP addresses. Rotating your own proxy pool is expensive and time-consuming to maintain.
Fragile DOM structures: Search engines constantly shift their HTML layout and CSS classes. A script targeting div.g tonight will silently return empty data by tomorrow morning.
Compute overhead: Spinning up heavy browser environments (like headless Chromium) consumes significant server memory and CPU cycles, adding cost and latency to every query.

To keep AI agents fast and reliable, you need an abstraction layer that handles proxy rotation, bypasses CAPTCHAs, and returns structured data instantly. By offloading this infrastructure to SearchApi.io, you retrieve clean, pre-parsed JSON payloads using a lightweight HTTP request - at a fraction of the compute overhead.

System architecture

Before writing any code, it helps to understand how an autonomous tool-calling loop operates. This system does not blindly search the web on every turn. Instead, it follows a four-step conversation loop:

The intent check: The user asks a question. The LLM evaluates whether it can answer using its internal knowledge or whether it needs live web data.
The tool call trigger: If it needs live data, the LLM stops generating text and outputs a structured request to invoke the search_web tool.
The infrastructure layer: The Python script intercepts this request, queries SearchApi's Google engine, and returns the top organic search results as structured JSON.
The synthesis: The text snippets are injected back into the LLM's context window. The model reads the live web data and generates a grounded, hallucination-free response.

This architecture is efficient because the LLM acts as the decision-maker. It avoids unnecessary API calls for questions it can already answer accurately from its training data.

Prerequisites and project setup

To follow along with this tutorial, you will need three things:

Python 3.8+ installed on your machine.
An OpenAI API key to power the model's reasoning loop.
A SearchApi API key to handle live web access. You can sign up at SearchApi.io and get 100 free searches right out of the box - no credit card required.

Start by initializing the project folder and installing the required dependencies:

# Create and move into your project folder
mkdir ai-search-agent && cd ai-search-agent

# Install dependencies
pip install requests openai python-dotenv

Next, create a .env file in the root of your project directory to keep your API tokens secure:

OPENAI_API_KEY="your_actual_openai_api_key_here"
SEARCHAPI_API_KEY="your_actual_searchapi_api_key_here"

With the project environment ready, here is an overview of the full project structure you will build:

Building a Real-Time AI Search Agent/
├── ai-search-agent/           <- Python CLI Implementation
│   ├── search_helper.py       <- SearchApi wrapper function (Step 1)
│   ├── agent.py               <- OpenAI tool schema and ask_agent() (Step 2)
│   ├── run.py                 <- Orchestrator loop (Step 3)
│   ├── .env.example           <- Template for environment variables
│   └── requirements.txt       <- pip dependencies
└── realtime-search/           <- Production Next.js 14 Application
    ├── app/
    │   ├── api/
    │   │   ├── search/        <- Server-side SearchApi route handler
    │   │   └── synthesize/    <- Real-time OpenAI SSE streaming handler
    │   ├── globals.css        <- Design system tokens & glassmorphism
    │   ├── layout.tsx         <- Root layout & SSR theme script
    │   └── page.tsx           <- Agent dashboard container
    ├── components/            <- React UI components
    ├── hooks/                 <- Pipeline orchestration & history hooks
    ├── lib/                   <- Server utilities & TypeScript definitions
    └── README.md              <- Next.js application documentation

Step 1: Writing the search API helper function

The first module is a search_web() function that sends a query to SearchApi's Google Search engine and returns a compressed block of text containing the titles, snippets, and source links from the top organic results.

The key design decision here is to extract only three fields - title, snippet, and link - from SearchApi's organic_results array. This keeps the payload lightweight and prevents the LLM from wasting tokens on irrelevant metadata like ad blocks, knowledge panels, or navigation breadcrumbs.

Create a file called search_helper.py:

import os
import requests
from dotenv import load_dotenv

# Load API keys from environment variables
load_dotenv()
SEARCHAPI_KEY = os.getenv("SEARCHAPI_API_KEY")

def search_web(query: str, location: str = "United States") -> str:
    """
    Queries the SearchApi.io Google Search engine and returns a compressed
    string containing the titles, snippets, and links of organic results.
    """
    if not SEARCHAPI_KEY:
        return "Error: SearchApi API key missing from environment variables."

    url = "https://www.searchapi.io/api/v1/search"

    # Configure payload parameters according to the SearchApi documentation
    params = {
        "engine": "google",
        "q": query,
        "location": location,
        "api_key": SEARCHAPI_KEY
    }

    try:
        response = requests.get(url, params=params, timeout=10)
        response.raise_for_status()
        data = response.json()

        # Extract organic search results
        organic_results = data.get("organic_results", [])
        if not organic_results:
            return f"No relevant web search results found for: '{query}'"

        # Parse and stringify the results into a compact context block for the LLM
        formatted_results = []
        for index, result in enumerate(organic_results[:5], 1):  # Limit to top 5 results
            title = result.get("title", "No Title")
            snippet = result.get("snippet", "No Snippet Available")
            link = result.get("link", "")
            formatted_results.append(f"[{index}] {title}\nSnippet: {snippet}\nSource: {link}\n---")

        return "\n".join(formatted_results)

    except requests.exceptions.RequestException as e:
        return f"An infrastructure error occurred while querying the search API: {str(e)}"


# Quick standalone test execution
if __name__ == "__main__":
    test_query = "Who won the men's 100m sprint in the 2024 Paris Olympics?"
    print(f"Testing SearchApi wrapper with query: '{test_query}'...\n")
    print(search_web(test_query))

The organic_results[:5] slice is intentional. Feeding more than five results into the LLM context rarely improves answer quality and meaningfully increases token consumption and cost.

Step 2: Defining the LLM agent framework (tool calling)

With the data pipeline ready, you need to teach the LLM when and how to use it. This is done through OpenAI's tool calling (formerly called function calling) feature, which lets you define structured schemas that the model can invoke when it determines external data is required.

Create a file, and name it agent.py:

import os
import json
from openai import OpenAI
from search_helper import search_web
from dotenv import load_dotenv

load_dotenv()

# Initialize the OpenAI client
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Define the SearchApi tool schema for the model
search_tool_definition = {
    "type": "function",
    "function": {
        "name": "search_web",
        "description": (
            "Call this tool whenever you need up-to-date information, news, "
            "current events, or web data that occurred after your knowledge cutoff date."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": (
                        "The optimized search query string to look up on Google "
                        "(e.g. 'Tesla Model Y 2026 pricing specifications')."
                    )
                },
                "location": {
                    "type": "string",
                    "description": "Geographical location filter for search targeting. Defaults to 'United States'.",
                    "default": "United States"
                }
            },
            "required": ["query"]
        }
    }
}


def ask_agent(user_prompt: str):
    """
    Sends a query to the LLM along with the available search tool definition.
    The LLM determines autonomously whether it needs to execute a Google search.
    """
    messages = [
        {
            "role": "system",
            "content": (
                "You are a factual, precise AI search agent with live web access via SearchApi. "
                "Use the search_web tool when you need current or time-sensitive information."
            )
        },
        {"role": "user", "content": user_prompt}
    ]

    # Request model execution with the tools parameter exposed
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",        # Efficient, tool-capable model
        messages=messages,
        tools=[search_tool_definition],
        tool_choice="auto"          # Let the model autonomously decide whether to use the tool
    )

    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls

    if tool_calls:
        print("Agent decision: 'I need to use SearchApi to look up live information on the web.'")
        return {"status": "tool_call_required", "data": tool_calls, "message_history": messages}
    else:
        print("Agent decision: 'I can answer this question from my existing knowledge.'")
        return {"status": "completed", "data": response_message.content}

The tool_choice="auto" parameter is the key to the agent's autonomy. It tells OpenAI to let the model decide whether external search is necessary rather than forcing a tool call on every request. Simple questions like "What is Python?" get answered directly, while questions about current events correctly trigger a SearchApi call.

Step 3: Stitching the search agent loop together

The final piece is the orchestrator - a runtime loop that ties everything together. It interprets the model's decision, calls search_web() with the model's chosen arguments, injects the live results back into the conversation history, and requests a final synthesized answer from OpenAI.

Create a file called run.py:

import json
from agent import ask_agent, openai_client
from search_helper import search_web


def run_agent_loop(user_prompt: str):
    """
    The orchestrator runtime loop. It prompts the agent, detects tool requests,
    executes the local SearchApi call, updates the conversation history, and
    generates the final web-informed answer.
    """
    print(f"\nUser: {user_prompt}")

    # First turn: ask the agent what it wants to do
    result = ask_agent(user_prompt)

    # Case A: The LLM answered immediately without needing the web
    if result["status"] == "completed":
        print(f"\nFinal answer:\n{result['data']}\n")
        return

    # Case B: The LLM explicitly requested a SearchApi tool call
    if result["status"] == "tool_call_required":
        tool_calls = result["data"]
        messages = result["message_history"]

        # Append the model's tool request to keep the chat history balanced
        assistant_msg = {
            "role": "assistant",
            "content": None,
            "tool_calls": [
                {
                    "id": tc.id,
                    "type": "function",
                    "function": {
                        "name": tc.function.name,
                        "arguments": tc.function.arguments
                    }
                } for tc in tool_calls
            ]
        }
        messages.append(assistant_msg)

        # Process each tool call (handles potential parallel calls from the model)
        for tool_call in tool_calls:
            function_name = tool_call.function.name
            function_args = json.loads(tool_call.function.arguments)

            print(f"Invoking tool: {function_name}() with arguments {function_args}")

            if function_name == "search_web":
                search_query = function_args.get("query")
                search_location = function_args.get("location", "United States")

                # Fetch structured data live from SearchApi.io
                raw_search_results = search_web(query=search_query, location=search_location)

                # Append the search result payload back to the chat history
                tool_response_msg = {
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "name": function_name,
                    "content": raw_search_results
                }
                messages.append(tool_response_msg)
            else:
                print(f"Warning: model requested unknown tool: {function_name}")
                return

        # Second turn: send the full conversation history back to OpenAI
        # (This now contains: user query + agent tool call + SearchApi results)
        print("Synthesizing live search data into a comprehensive answer...")
        final_response = openai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages
        )

        print(f"\nFinal answer:\n{final_response.choices[0].message.content}\n")


# End-to-end system test
if __name__ == "__main__":
    # Test case 1: Requires SearchApi intervention (current events)
    run_agent_loop("What were the top 3 tech stock market trends in Q1 of 2026?")

    # Test case 2: Native knowledge execution (no web tool needed)
    run_agent_loop("Explain the fundamental difference between a SQL and NoSQL database in two sentences.")

Pay close attention to how the messages list is built across both turns. After the model requests a tool call, you must append the assistant's tool request and then the tool's response before sending the history back to OpenAI. Omitting either step causes the API to return an error - this is the most common breaking point for developers new to tool calling.

Live test drives

With all three files in place, run the agent with python run.py. Here is what a real execution looks like for each scenario.

Scenario A - Current events query:

User: What were the top 3 tech stock market trends in Q1 of 2026?

Agent decision: 'I need to use SearchApi to look up live information on the web.'
Invoking tool: search_web() with arguments {'query': 'top tech stock market trends Q1 2026'}
Synthesizing live search data into a comprehensive answer...

Final answer:
Based on recent web data, the top three tech stock market trends in Q1 2026 were:

1. AI infrastructure stocks surged as hyperscalers accelerated GPU procurement...
2. Semiconductor supply chain stabilization drove AMD and TSMC to multi-year highs...
3. Enterprise SaaS consolidation continued, with several major M&A announcements...

Scenario B - Native knowledge query:

User: Explain the fundamental difference between a SQL and NoSQL database in two sentences.

Agent decision: 'I can answer this question from my existing knowledge.'

Final answer:
SQL databases organize data into structured tables with predefined schemas and use
relational joins to link data across tables, making them ideal for consistent,
transactional workloads. NoSQL databases use flexible document, key-value, or graph
data models with no fixed schema, trading strict consistency for horizontal scalability
and performance on unstructured or rapidly evolving data.

The agent correctly identifies that the first query requires live web data and that the second can be answered from its own training knowledge - skipping an unnecessary API call entirely.

Demo: see the agent in action

The interactive demo below visualizes the exact agent loop described above - including the tool decision badge, the SearchApi result cards, and the synthesized answer panel.

Interactive Demos & Repositories:

🚀 Live Web Application: Test the full-stack agent live on Vercel at realtime-search.vercel.app.

🐍 Python CLI Repository: Clone the Python tutorial source code at Eunit99/ai-search-agent.

⚡ Next.js App Repository: Clone the production web application code at Eunit99/realtime-search.

Full-Stack Next.js Application: Located in realtime-search/ - features server-side API key isolation (/api/search), real-time OpenAI GPT-4o-mini SSE streaming (/api/synthesize), geo-targeting, and query history persistence.

Production best practices

Before shipping this agent to production, consider these refinements.

Filter low-quality snippets to save tokens

SearchApi's organic_results array occasionally includes results with very short or unhelpful snippets (e.g. login-gated pages or index-only entries). Add a simple length filter to skip any snippet under 40 characters:

if len(snippet) > 40:
    formatted_results.append(f"[{index}] {title}\nSnippet: {snippet}\nSource: {link}\n---")

Use geo-targeting for localized results

SearchApi supports a location parameter that localizes search results geographically. This is useful for agents that serve users across different regions, where price, news, or availability data differs by market. The search_web() function already exposes this parameter:

search_web(query="current petrol prices", location="Lagos, Nigeria")

Cache duplicate queries

In production, users frequently ask similar questions within a short time window. Caching the search_web() response avoids burning SearchApi credits on identical lookups. A Redis-backed wrapper keeps it simple:

import hashlib, redis

cache = redis.Redis(host='localhost', port=6379, db=0)
TTL_SECONDS = 300  # Cache results for 5 minutes

def search_web_cached(query: str, location: str = "United States") -> str:
    cache_key = hashlib.md5(f"{query}:{location}".encode()).hexdigest()
    cached = cache.get(cache_key)
    if cached:
        return cached.decode()
    result = search_web(query, location)
    cache.setex(cache_key, TTL_SECONDS, result)
    return result

Keep the system prompt specific

Your system prompt directly affects how often the model decides to call search_web. Be explicit about when the tool should and should not be used:

"Use the search_web tool for any question involving current events, prices, "
"people, companies, or anything that may have changed after your training cutoff. "
"Answer directly only for timeless facts, historical events, or technical concepts."

Wrapping up

You have built a production-ready AI search agent that solves the two fundamental limitations of offline LLMs: knowledge cutoffs and hallucinations on real-time data.

The architecture is intentionally minimal but extensible. The search_web() function can be swapped for any SearchApi engine (News, YouTube, Google Scholar, and more). The tool schema can be extended with additional tools for math, code execution, or database lookup. The agent loop can be wrapped in a web server, a Slack bot, or a voice assistant.

The key insight is that SearchApi handles the hard infrastructure work - proxy rotation, CAPTCHA solving, HTML parsing, and structured JSON delivery - so you can focus entirely on the reasoning layer on top of it.

Ready to build? Grab your free API key at SearchApi.io, check out the live web app on Vercel, clone the Python CLI repository from GitHub (ai-search-agent) or the full-stack web application from GitHub (realtime-search), and drop a comment describing what you build.

Have questions or improvements? Open an issue or submit a pull request on GitHub (Python CLI) or GitHub (Next.js App).

This post is sponsored by SearchApi.

How I automated hyper-local lead routing with Nextdoor Scraper and Slack

Emmanuel Uchenna — Tue, 04 Aug 2026 16:25:45 +0000

Picture this. A sales rep or local contractor sitting in their truck on a Tuesday morning, burning fuel driving 45 minutes across town to quote a home repair, only to discover the prospect is a renter who cannot authorize work. Meanwhile, three streets over, in a hidden, high-density homeowner neighborhood, local residents are actively posting on Nextdoor to ask for contractor recommendations.

Collecting hyper-local data into a weekly CSV export is like brewing a fresh pot of coffee and leaving it on the counter for two days before serving it to your sales team. By the time an analyst manually exports the dataset, filters target zip codes, and emails the report to reps, those golden neighborhood opportunities are cold and competitors have already booked the jobs.

However, extracting this hyper-local data from public sources has historically presented a severe operational challenge. Nextdoor is a rich ecosystem for local community context, but manually researching across hundreds of city pages is painfully slow. Worse still, traditional web scraping workflows often deposit extracted data into static CSV files or database tables that sit idle until someone manually reviews them days later. In high-intent lead generation, every hour of delay degrades lead momentum and lowers conversion rates.

To solve this problem, I built two complementary lead routing workflows around Nextdoor Scraper on the Apify platform. The first is a no-code Slack notification wired directly inside Apify Console. The second is an AI agent workflow that connects Nextdoor Scraper to Claude or Cursor via the Apify MCP server, letting an AI agent run the scraper, filter results, and push alerts to Slack through a single conversational prompt.

In this article, I will walk you through the extraction setup, the qualification filter logic, both Slack routing approaches, and exactly what each option looks like in practice.

1. The gap between data collection and sales action

For home service providers, such as plumbers, roofers, HVAC technicians, landscapers, and solar installers, speed and geographic precision are everything. Identifying an ideal target neighborhood is only half the battle. If your sales team receives demographic reports 48 hours after a data crawl finishes, the opportunity to run timely local direct mail campaigns, localized digital ad pushes, or door-to-door outreach in high-demand pockets has already started to slip.

+-------------------+      +--------------------+      +----------------------+
|  Public Nextdoor  | ---> |  Nextdoor Scraper  | ---> |  Data Dump (CSV/DB)  |
|    City Pages     |      |  (Manual Crawl)    |      |  (Sits Idle Days)    |
+-------------------+      +--------------------+      +----------------------+
                                                                  |
                                                                  v
                                                       [ Delayed Sales Action ]

The primary friction point in modern web scraping is not extracting the data - it is the latency between data acquisition and business action. Traditional workflows decouple extraction from execution:

A crawler runs and outputs thousands of raw records into key-value stores or CSV files.
An analyst exports the CSV and manually applies spreadsheet filters to isolate target zip codes.
The analyst formats a report and emails it to the sales lead.
The sales lead assigns territories to reps who finally begin outreach.

This multi-step, manual chain creates massive operational friction. By the time reps receive the target list, days have elapsed.

To eliminate this gap, I built two notification paths that both start from the same Nextdoor Scraper run, and both deliver results to Slack, but in different ways depending on your team's setup.

+-------------------+      +--------------------+      +---------------------+
|  Public Nextdoor  | ---> |  Nextdoor Scraper  | ---> |  Apify Dataset      |
|    City Pages     |      |  (Apify Platform)  |      |  (Structured JSON)  |
+-------------------+      +--------------------+      +---------------------+
                                                                  |
                          +--------------------------+------------+
                          |                          |
                          v                          v
             [ Path A: No-code ]        [ Path B: MCP server ]
             Slack notification         AI agent (Claude/Cursor)
             via Actor integration      calls Nextdoor Scraper,
             inside Apify Console       filters leads, posts to Slack

2. The pipeline architecture

The automated lead routing system operates across two stages: extraction and post-run notification.

sequenceDiagram
    autonumber
    participant ND as Public Nextdoor Pages
    participant NS as Nextdoor Scraper (Apify Platform)
    participant DS as Apify Dataset
    participant SL as Slack (#local-leads)
    participant AI as AI Agent (Claude / Cursor)

    ND->>NS: Fetch public city & neighborhood page HTML
    NS->>DS: Store structured place data (centroid, income, homeowners)
    Note over DS: Run completes. Dataset is ready.

    alt Path A: No-code Slack notification
        DS->>SL: Actor integration fires on run finish
        SL-->>DS: Notification delivered
    else Path B: MCP server + AI agent
        AI->>NS: call-actor via Apify MCP server (mcp.apify.com)
        NS->>DS: Actor runs and stores results
        DS->>AI: AI agent reads dataset items
        AI->>AI: Filter by homeowners >= 50%, income >= $40k
        AI->>SL: Post formatted alert via Slack bot or Slack MCP tool
    end

Stage 1: Extraction

The workflow begins by launching Nextdoor Scraper on the Apify platform. The Actor targets public Nextdoor city and neighborhood pages, bypassing the need for complex browser automation setups or raw HTML parsing scripts. It parses visible structured data and metadata, transforming disorganized web content into clean JSON records that include geographic coordinates, demographic metrics, resident counts, and references to community activity.

Stage 2: Notification routing

Once the Actor run finishes and the dataset is populated, there are two ways to get that data into Slack. The first requires no code at all. The second turns the entire pipeline into something an AI agent like Claude or Cursor can trigger, filter, and broadcast with a single prompt.

3. Step 1: Extracting targeted neighborhood demographics

To get started, we configure Nextdoor Scraper to target public city pages. Nextdoor public city pages display valuable summary metrics without requiring user logins or private authentication.

Input configuration

In Apify Console, open the Input tab for Nextdoor Scraper and supply your target city URLs along with crawl limits. Here is the JSON input configuration used for this pipeline:

{
  "startUrls": [
    { "url": "https://nextdoor.com/city/north-miami--fl/" },
    { "url": "https://nextdoor.com/city/bellaire--oh/" },
    { "url": "https://nextdoor.com/city/bellbrook--oh/" },
    { "url": "https://nextdoor.com/city/belle-center--oh/" }
  ],
  "downloadStartUrlsToHtmlSources": false,
  "maxRequestsPerCrawl": 20,
  "maxConcurrency": 2,
  "maxItemsPerSection": 25,
  "followNeighborhoodPages": true,
  "followBusinessDirectoryPages": false,
  "followAdditionalResourcePages": false,
  "saveHtmlSnapshots": false
}

Key demographic fields extracted

When Nextdoor Scraper processes a city page, it returns normalized fields ready for immediate analysis:

Field	Type	Description	Sample Value
`cityName`	String	Name of the target city or municipality	`"North Miami"`
`state`	String	Two-letter state abbreviation	`"FL"`
`centroid`	Object	Geographic center latitude and longitude	`{"latitude": 40.016, "longitude": -80.747}`
`residentsCount`	Integer	Total estimated residents in the area	`59854`
`averageAge`	Integer	Average resident age	`35`
`averageIncome`	String	Public average household income indicator	`"$41k"`
`percentageHomeowners`	String	Share of residents who own their homes	`"56%"`
`affordabilityScore`	Integer	Calculated local affordability index	`99`
`friendlinessScore`	Integer	Community friendliness score	`86`
`faqs`	Array	Structured FAQ question and answer pairs	`[{"question": "...", "answer": "..."}]`
`neighborhoods`	Array	Discovered public neighborhood links	`[{"title": "128th St", "url": "..."}]`

Sample output JSON

Here is a snippet of the structured dataset item generated by Nextdoor Scraper:

{
  "url": "https://nextdoor.com/city/north-miami--fl/",
  "finalUrl": "https://nextdoor.com/city/north-miami--fl/",
  "pageType": "city",
  "title": "North Miami, Florida | Nextdoor",
  "description": "Public city summary extracted from Nextdoor.",
  "cityName": "North Miami",
  "state": "FL",
  "centroid": {
    "latitude": 25.8901,
    "longitude": -80.1867
  },
  "residentsCount": 59854,
  "averageAge": 38,
  "averageIncome": "$48k",
  "percentageHomeowners": "54%",
  "affordabilityScore": 82,
  "friendlinessScore": 88,
  "neighborhoodCount": 26,
  "businessDirectoryCount": 54,
  "faqCount": 7,
  "accessStatus": "public"
}

Cost efficiency with Pay-Per-Event pricing

One major advantage of using Nextdoor Scraper on the Apify platform is its pricing structure. Nextdoor Scraper operates on a Pay-Per-Event (PPE) model.

Unlike traditional compute-hour billing where you pay for raw server uptime regardless of success, PPE charging means you are billed strictly for successfully extracted place records. If a network fluctuation or access block occurs on a page, you are not billed for that failed attempt. This ensures predictable margins when executing large-scale, automated lead discovery campaigns across thousands of zip codes.

4. Step 2: Filtering for contractor accountability and intent

To understand how conditional filtering works in practice, let us examine a real-world implementation executed during a regional contractor accountability platform pilot with Lake Rock Ventures.

The Lake Rock Ventures pilot scenario

Lake Rock Ventures, a venture growth studio, was launching a pilot platform designed to connect vetted local roofing and HVAC contractors with high-intent homeowners. The platform needed to identify suburban communities with high enough homeownership rates to justify targeted direct-response marketing. Sending contractor representatives into renter-heavy neighborhoods resulted in wasted labor costs and low conversion rates.

To solve this, we defined strict qualification parameters for acceptable lead routing:

Homeowner ratio (percentageHomeowners): Minimum of 50%. Homeowners make long-term property decisions; renters rarely purchase roof replacements or HVAC upgrades.
Community scale (residentsCount): Minimum of 1,000 residents to guarantee adequate market volume.
Income indicator (averageIncome): Must meet or exceed $40k to ensure purchasing power for home improvement projects.

Writing the mid-run evaluation logic

We implemented a custom evaluation function in JavaScript that processes each dataset record emitted by Nextdoor Scraper. The script parses formatted strings (such as "56%" or "$48k") into clean numerical values and tests them against our threshold rules.

/**
 * Evaluates whether an extracted Nextdoor dataset item qualifies as a high-intent local lead.
 * @param {Object} item - Structured record from Nextdoor Scraper.
 * @returns {Object} Qualification result containing boolean status and parsed metrics.
 */
function evaluateLeadQualification(item) {
    // Extract numerical value from percentage string e.g., "56%" -> 56
    const homeownerRate = item.percentageHomeowners 
        ? parseFloat(item.percentageHomeowners.replace('%', '')) 
        : 0;

    // Extract numerical value from income string e.g., "$48k" -> 48000
    let incomeValue = 0;
    if (item.averageIncome) {
        const rawIncome = item.averageIncome.replace('$', '').toLowerCase();
        if (rawIncome.includes('k')) {
            incomeValue = parseFloat(rawIncome.replace('k', '')) * 1000;
        } else {
            incomeValue = parseFloat(rawIncome);
        }
    }

    const residentCount = item.residentsCount || 0;

    // Qualification thresholds
    const MIN_HOMEOWNER_PERCENT = 50;
    const MIN_INCOME = 40000;
    const MIN_RESIDENTS = 1000;

    const isQualified = 
        homeownerRate >= MIN_HOMEOWNER_PERCENT &&
        incomeValue >= MIN_INCOME &&
        residentCount >= MIN_RESIDENTS;

    return {
        isQualified,
        metrics: {
            homeownerRate,
            incomeValue,
            residentCount,
            affordabilityScore: item.affordabilityScore || 'N/A',
            friendlinessScore: item.friendlinessScore || 'N/A'
        }
    };
}

By embedding this evaluation logic directly into our execution pipeline, every crawled page is screened within milliseconds of extraction. If isQualified evaluates to true, the pipeline proceeds to dispatch a real-time Slack notification.

5. Step 3 (Path A): Sending a Slack notification with no code

Apify Console includes a built-in Slack notification integration for every Actor. It requires no custom code, no webhook endpoint, and no token management. You set it up once directly on the Nextdoor Scraper Actor page and it fires automatically after every run.

Setting up the Slack notification in Apify Console

Open Nextdoor Scraper in Apify Console and click the Integrations tab in the Actor's top navigation.
In the Search integrations bar, type slack. You will see a Send notification via Slack option listed under Generic integrations.
Click Send notification via Slack. A panel appears with a Connect workspace button.
Click Connect workspace and authorize the Apify app to access your Slack workspace. Select the target channel, such as #local-leads.
Configure when the notification fires. The most useful trigger is Run succeeded, which posts a Slack message automatically each time a Nextdoor Scraper run completes successfully.
Save the integration. From this point forward, every completed run posts a notification to your chosen Slack channel without any additional code.

The notification includes the run status, a direct link to the dataset in Apify Console, and the Actor name. Your sales team gets an immediate signal that fresh neighborhood data is ready to review, without having to log in and check manually.

+--------------------+      +---------------------+      +------------------------------+
| Nextdoor Scraper   | ---> | Run succeeds        | ---> | Slack: #local-leads          |
| (Apify Console)    |      | (Apify Platform)    |      | "Your run finished. View      |
|                    |      |                     |      |  results in Apify Console."  |
+--------------------+      +---------------------+      +------------------------------+

This is the fastest path to getting Slack alerts from Nextdoor Scraper. It takes under two minutes to configure and works on every run automatically, including scheduled runs and saved tasks.

6. Step 4 (Path B): Using the Apify MCP server with an AI agent

Path A sends a basic completion notification. Path B goes further: it lets an AI agent such as Claude or Cursor actually run Nextdoor Scraper, read and filter the extracted dataset, and post a formatted, actionable Slack alert with specific neighborhood metrics - all from a single conversational prompt.

This is possible because Apify hosts a remote Model Context Protocol (MCP) server at https://mcp.apify.com. Once you connect an MCP client to that server, your AI agent gains access to tools such as call-actor, search-actors, and fetch-actor-details. It can run Nextdoor Scraper on demand and process the results directly inside the chat context.

What is the Apify MCP server?

The Apify MCP server exposes Apify Actors as callable tools to external AI clients. It is a programmatic interface that works with any MCP-compatible client, including Claude for Desktop, Cursor, VS Code with MCP support, and others.

This is distinct from MCP connectors, which work in the opposite direction (an Actor calling an external service). The Apify MCP server makes Actors available as tools your AI agent can invoke from outside Apify entirely.

Connecting the Apify MCP server to your AI client

You authorize once via OAuth browser flow. After that, your AI client can call Nextdoor Scraper directly.

Cursor - create or update .cursor/mcp.json in your project:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
    }
  }
}

VS Code - add to .vscode/mcp.json or run MCP: Add Server from the Command Palette:

{
  "servers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
    }
  }
}

Claude for Desktop - edit claude_desktop_config.json:

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
      ]
    }
  }
}

The ?tools=actors,docs,eunit/nextdoor-scraper query parameter scopes the tools list to the Nextdoor Scraper Actor specifically, alongside general Actor search and documentation tools. On first connection, a browser OAuth flow opens so you sign in to your Apify account and approve access. No API token is pasted into the chat.

Running the pipeline from a prompt

Once configured, the entire extraction and filtering workflow becomes a single AI prompt. Here is an example of what you would type in Claude or Cursor:

Use Apify to run the Nextdoor Scraper on these four city pages: north-miami--fl, bellaire--oh, bellbrook--oh, and belle-center--oh. Once finished, read the dataset results. Filter for any city where percentageHomeowners is 50% or above and averageIncome is $40k or above. For each qualifying city, format a Slack message with the city name, homeowner share, average income, resident count, and affordability score, then post it to #local-leads.

The AI agent handles the full sequence:

Calls call-actor with the Nextdoor Scraper Actor ID and input configuration.
Waits for the run to complete and fetches the dataset items.
Applies your qualification filter logic across every result.
Formats a structured message for each qualifying neighborhood.
Posts the alert to Slack using either a connected Slack MCP tool or a Slack bot token you provide in the chat context.

+------------------+      +----------------------+      +------------------+      +------------------+
| AI Agent         | ---> | Apify MCP server     | ---> | Nextdoor Scraper | ---> | Apify Dataset    |
| (Claude/Cursor)  |      | mcp.apify.com        |      | (Actor run)      |      | (JSON results)   |
| call-actor tool  |      | (OAuth, no token     |      |                  |      |                  |
|                  |      |  in chat)            |      |                  |      |                  |
+------------------+      +----------------------+      +------------------+      +------------------+
         ^                                                                                  |
         |                           reads & filters dataset                               |
         +-----------------------------------------------------------------------------------+
         |
         v
+------------------+
| Slack #local-leads|
| Formatted alert  |
| per qualifying   |
| neighborhood     |
+------------------+

Why this approach works well for local lead routing

The AI agent approach is especially effective because filtering logic does not need to be written in advance or maintained as code. You adjust the criteria conversationally. If the roofing company decides to lower the homeowner threshold to 45% for a quieter week, you change one line in the prompt. No code deployment required.

The Apify MCP server also supports output schema inference, meaning Claude or Cursor understands the structure of the Nextdoor Scraper dataset before the run completes. That makes it easier for the agent to write accurate filter logic against fields like percentageHomeowners, averageIncome, and residentsCount without needing to inspect a raw JSON file first.

7. What the end result looks like in both paths

Path A: What the no-code Slack notification looks like

When a Nextdoor Scraper run finishes, the built-in Slack integration posts a notification automatically to your chosen channel. The message includes:

The Actor name and run status
A direct link to the dataset in Apify Console
The run duration and the number of items stored

Your sales team opens Slack, sees the notification, and clicks through to the dataset to review results. No CSV downloads, no email chains, no delayed handoffs.

For teams that run the scraper on a schedule (e.g. every Monday morning before the weekly sales meeting), this turns a manual check-in into a reliable, automated briefing.

Path B: What the AI agent prompt workflow looks like

With the Apify MCP server connected to Claude or Cursor, the agent runs the full pipeline from one prompt. Here is what a real output message from Claude might look like after processing a Nextdoor Scraper dataset with a 50% homeowner filter:

+--------------------------------------------------------------+
|  #local-leads                                       [Slack]  |
+--------------------------------------------------------------+
|  🏡 High-intent lead: North Miami, FL                        |
|  Homeowner share: 54%  |  Avg income: $48k                   |
|  Residents: 59,854     |  Affordability: 82/100              |
|  Nextdoor page: https://nextdoor.com/city/north-miami--fl/   |
|                                                              |
|  🏡 High-intent lead: Bellbrook, OH                          |
|  Homeowner share: 61%  |  Avg income: $52k                   |
|  Residents: 7,009      |  Affordability: 91/100              |
|  Nextdoor page: https://nextdoor.com/city/bellbrook--oh/     |
+--------------------------------------------------------------+

In the Lake Rock Ventures pilot, switching from manual CSV review to the Apify MCP server + Claude workflow reduced the time from data collection to campaign briefing from 48 hours to under 10 minutes, while narrowing outreach to high-ownership areas only.

7. Wrapping up

Static data collection is no longer sufficient for modern local marketing and lead generation operations. By replacing manual CSV exports with an automated pipeline powered by Nextdoor Scraper and Apify MCP connectors, you can bridge the gap between web data extraction and real-time sales execution.

With this architecture, you benefit from:

Hyper-local precision: Extracting structured resident counts, homeowner ratios, and income signals from public Nextdoor pages.
Predictable cost scaling: Leveraging Pay-Per-Event pricing to pay strictly for successful record extractions.
Zero glue-code overhead: Eliminating third-party integration tools and AWS Lambda scripts by using Apify MCP connectors.
Instant lead velocity: Routing high-intent local opportunities straight to your sales team's Slack channel in seconds.

Ready to automate your local lead routing? Try Nextdoor Scraper in the Apify Store, connect your Slack workspace via Apify MCP connectors, and start turning public neighborhood signals into real-time sales momentum today.

How to download public YouTube videos at scale with an Apify Actor

Emmanuel Uchenna — Thu, 18 Jun 2026 12:30:13 +0000

You open a YouTube page. The video is public. You need that video - or its transcript, or its metadata, or just a clean download link you can pass to the next step in your workflow.

So you search for a YouTube video downloader. You find a browser extension, a sketchy website, maybe a command-line tool. You paste the URL. It works - once. But you have twenty more URLs to process. Or you need to run this same task every Monday morning. Or three people on your team need to do the same thing without using three different methods and getting three different outputs.

That is where every basic downloader starts to fall apart.

YouTube Video Downloader is an Apify Actor built to solve exactly this problem. It processes public YouTube watch URLs, YouTube Shorts, and live replay links in a structured, repeatable, automatable way. You get download format URLs, video metadata, thumbnails, channel data, and optional transcript text - all in one clean dataset output per run.

This guide will show you exactly how it works, how to configure it, and why it is a better fit than generic one-off tools if your workflow involves more than one video or one person.

What the YouTube Video Downloader Actor does

Most downloader tools treat the task as simple: give them a URL and they give you a file. That is fine if that is truly all you need.

But most teams need a bit more. They need the title and description to track what they downloaded. They need the thumbnail to display in a content board. They need the transcript to feed into a summarization or search tool. And they need all of this in a format that works with the rest of their stack - not a browser download that lands in someone's desktop folder.

YouTube Video Downloader handles all of that in one run. Here is what it can return for each URL you give it:

Direct download URLs for available MP4 and WebM formats
Progressive (video + audio combined) and adaptive (video-only or audio-only) stream metadata
Video title, description, tags, and view count
Channel name and channel ID
Duration in seconds
Thumbnail URL (including the high-resolution version when available)
Media type - regular video, Short, or live replay
Public transcript text when available and enabled
A saved file in the run's key-value store when you use file export mode

The Actor runs on the Apify platform, which means you are not limited to a single manual session. You can run it from Apify Console, call it via API from your own application, schedule it on a recurring basis, or trigger it with a webhook. The output lands in a structured dataset you can export in JSON, CSV, or Excel.

Who this Actor is actually useful for

Before you decide whether this tool is worth using, it helps to know whether your situation matches how it was designed.

Content and editorial teams are a natural fit. If your team regularly works with public video assets - for clipping, repurposing, archiving, or editorial reference - you likely need more than the video file itself. You need the metadata alongside it. The Actor gives you both in a consistent format, run after run.

Agencies and operations teams benefit most from the repeatability. When different people handle different requests manually, the output is inconsistent. One person uses one site, another uses a different tool. The Actor standardizes the intake: one input format, one output structure, every time.

Developers and product teams often want a YouTube video downloader API - something they can call from inside their application or automation, not something they have to operate manually. The Actor exposes a full REST API and is also available through the JavaScript and Python client libraries. You can integrate it into any stack in a few lines of code.

Researchers and analysts who work with transcripts, topic metadata, or structured video data will find the Actor useful for building consistent datasets from public YouTube content. Instead of scraping each video manually, you can pass a list of URLs and get back a single structured export.

Why browser-based downloaders are not enough

It is worth being direct about this, because plenty of people reading this article have already tried the obvious alternatives.

Browser-based YouTube downloader sites work for one-off tasks. You paste a link, download a file, and you are done. But they are built for individual, manual use. They do not give you metadata. They do not store transcripts. They do not support API access. They do not let you schedule a run or trigger one programmatically. And they definitely do not output a structured dataset you can hand to the next step in a pipeline.

There is also the reliability issue. Many of these sites change frequently, break without warning, or quietly stop working after YouTube updates. If you build a team process around one of them, you are building on unstable ground.

The YouTube Video Downloader Actor approaches the problem differently. Direct format URLs returned by YouTube are time-limited - they expire. The Actor solves this by giving you two options: use the expiring URL for immediate access, or switch to save-best-progressive mode and have the Actor store a durable copy in Apify's key-value store. From there, you can fetch it via a stable URL for as long as you need it.

Step-by-step: how to download public YouTube videos with Apify

Here is exactly how to run the Actor and get results.

Step 1: Open the Actor in Apify Console

Go to the YouTube Video Downloader page in Apify Store. If you do not have an Apify account yet, you can sign up for free - no credit card required to start.

Once you are in, click Try for free to open the Actor in Apify Console. You will land on the Input tab, which is where you configure everything before running.

Step 2: Add your YouTube URLs

In the YouTube video URLs field, paste one or more public YouTube links. The Actor supports:

Standard watch links: https://www.youtube.com/watch?v=...
YouTube Shorts: https://www.youtube.com/shorts/...
Live replay links for completed live streams

You can add up to 100 URLs per run (controlled by the Max videos input field, which defaults to 10). If you have a longer list, you can run multiple jobs or call the Actor programmatically in batches.

Step 3: Choose your download mode

This is where the Actor becomes more powerful than a generic downloader. Under Download mode, you have two options:

metadata-only - The Actor returns format URLs, metadata, and optional transcript data. No file is saved. This is the default mode and the fastest way to get structured output. Use this when you only need the download link or metadata to pass downstream.
save-best-progressive - The Actor selects the best available progressive format (video + audio combined), downloads it, and stores the file in the run's default key-value store. You can then fetch it via a stable Apify URL.

Set your preferred quality (best, 720p, 1080p, etc.) and container (mp4, webm, or any) to guide which format gets selected when saving.

Step 4: Configure the rest of your input

A few more fields worth knowing:

Include transcript - Toggle this on if you need the public transcript text. When available, it gets included in the dataset output for each video. Useful for search indexing, summarization, and AI enrichment.
Include progressive formats - Returns combined video + audio formats in the output. On by default.
Include adaptive formats - Returns video-only and audio-only streams separately. On by default. Useful if you are working with adaptive streaming workflows.
Max formats per video - Limits how many format entries appear in the output. Default is 8. Increase this if you want to see the full list of available streams.
Max saved file size (MB) - A safety cap when using save-best-progressive mode. Defaults to 150 MB. Increase if you are working with longer videos and have confirmed the file will fit.

Here is an example input that covers most use cases:

{
  "startUrls": [
    { "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ" },
    { "url": "https://www.youtube.com/shorts/abc123" }
  ],
  "maxVideos": 10,
  "preferredQuality": "720p",
  "preferredContainer": "mp4",
  "includeProgressiveFormats": true,
  "includeAdaptiveFormats": false,
  "downloadMode": "metadata-only",
  "includeTranscript": true,
  "maxDownloadSizeMb": 150
}

Step 5: Start the run

Click Start in Apify Console. The Actor processes each URL in sequence, resolves the available formats, collects the metadata, and pushes one item to the dataset for each URL it handles.

You do not need to keep the browser open. Apify Console shows run status in real time, and you can come back when it is done. For large batches or scheduled runs, you will not be watching at all - the output will just be there when you need it.

Step 6: Review and use the output

Once the run finishes, open the Dataset tab. You will see one row per input URL. Each row has:

Field	What it gives you
`status`	`ready`, `downloaded`, or `error`
`videoId`	The normalized YouTube video ID
`title`	The video title from YouTube
`mediaType`	Video, Short, or live replay
`durationSeconds`	Runtime in seconds
`viewCount`	Public view count at the time of the run
`thumbnailUrl`	High-resolution thumbnail URL
`formats`	All returned format entries with stream URLs and quality labels
`transcript`	Public transcript text (if enabled and available)
`selectedDownload`	Key-value store key and public URL of saved file (if applicable)

You can export the dataset from the Export button as JSON, CSV, or Excel. If you saved files using save-best-progressive mode, they appear in the Storage tab under the run's key-value store.

How to integrate the Actor into your own code

If you want to call the Actor programmatically - for example, from an internal tool, a script, or a larger automation - the easiest way is through the official Apify client libraries.

JavaScript / TypeScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('eunit/youtube-video-downloader').call({
  startUrls: [{ url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' }],
  downloadMode: 'metadata-only',
  includeTranscript: true,
  preferredQuality: '720p',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("eunit/youtube-video-downloader").call(run_input={
    "startUrls": [{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}],
    "downloadMode": "metadata-only",
    "includeTranscript": True,
    "preferredQuality": "720p",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Install the client with pip install apify-client (Python) or npm install apify-client (JavaScript/TypeScript). Your API token is in Apify Console under Settings > Integrations.

A note on legal and responsible use

Any good guide on YouTube video downloading should address this clearly.

The YouTube Video Downloader Actor is built for public YouTube content - videos that are publicly visible without login. That does not automatically mean every public video is fair game to download, archive, or redistribute.

YouTube's terms of service generally prohibit downloading videos without explicit permission from the platform or the content owner. There are exceptions for content you own, content licensed for download, and content in the public domain - but those are the exceptions, not the default.

If you are asking how to download YouTube videos legally, the clearest answer is: work with content you own, content you have been licensed to use, or content in the public domain. If you are unsure about a specific piece of content, check with a qualified legal advisor, not a downloader tool's FAQ.

For teams, this matters even more. A repeatable workflow that processes video at scale creates a larger footprint. Good compliance means having clear answers to: what content are we allowed to process, where does the output go, and who approved it.

The Actor helps you build a consistent, auditable workflow. The permissions question is something you need to answer before you run it.

Common use cases where the Actor earns its cost

If you are comparing this Actor against free browser-based tools, the question is not whether the free tool works for one video. It usually does. The question is what your workflow actually costs.

Content repurposing pipelines. If your team regularly turns public video into clips, summaries, or knowledge-base articles, you need the transcript and metadata alongside the media file. The Actor gives you all three in one run, in a structured format that fits your next tool.

Approved asset management. Agencies and media teams that manage approved public video assets across campaigns or clients need consistency. The Actor standardizes the output, so every URL processed the same way, by anyone on the team, gives the same fields.

AI and research workflows. Transcripts combined with metadata - title, tags, description, view count, duration - are a useful input for classification, topic modeling, or AI summarization pipelines. The Actor gives you a clean starting point for that kind of enrichment.

Product integration. If you are building an internal tool or product feature that involves YouTube video data, the Actor is a faster path than building and maintaining your own downloader infrastructure. You call an API, you get structured output, and you handle the hard part (your product logic) rather than YouTube format resolution.

Why automation changes the value equation

Here is a way to think about whether the Actor is worth paying for.

If you download five YouTube videos a month manually, the cost of manual work is low. A free tool is probably fine.

If you process fifty URLs every week across a team of three people, the math changes. Manual work is time, and time is money. If each URL takes five minutes to process by hand (finding the right tool, pasting the link, downloading the file, renaming it, noting the metadata, passing it to the next step), that is over four hours of manual work per week. At scale, or across a team, that adds up fast.

The real cost of manual tools is not the tool's price. It is the time spent, the inconsistency in output, and the fragility when the tool breaks or the website changes.

For teams that process video data regularly, the automation pays for itself quickly. And for teams that need scheduled or API-triggered runs - the kind of workflow that simply cannot be done with a manual tool - the comparison is not really about price at all.

Frequently asked questions

Can this Actor download YouTube Shorts?

Yes. The Actor supports public YouTube Shorts URLs. They are handled as a supported media type and included in the dataset output with the mediaType field set to Short.

Does it return direct download links?

Yes. When available, the Actor returns direct format URLs in the formats array. These URLs are time-limited, so use them promptly or switch to save-best-progressive mode to get a durable stored copy.

Can it save the video file, not just the URL?

Yes. Set downloadMode to save-best-progressive and the Actor will select the best available progressive format, download it, and store it in the run's key-value store. You can then access it via a stable public URL from the selectedDownload field.

Can it extract transcripts?

Yes, when the video has a public transcript and you enable the includeTranscript option. Not all videos have public transcripts, and in those cases the Actor returns a structured item without transcript data rather than failing the run.

Does it work for live replays?

Yes. Completed live streams that are publicly available on YouTube are supported as a media type. The Actor handles them the same way as regular videos.

Can I run this on a schedule or trigger it with a webhook?

Yes. Scheduled runs and webhook triggers are both available from Apify Console. You can also call the Actor via the REST API or the official client libraries to integrate it into any external system.

Why use this instead of a free downloader?

Because free downloaders are built for one-off, manual, single-user tasks. They do not return structured metadata. They do not support transcripts. They do not give you API access, scheduled runs, or structured exports. And they break without warning. If your workflow involves more than one video, more than one person, or more than a one-time task, a structured Actor on a stable platform is a significantly better fit.

Wrapping Up

If you have a workflow that involves downloading public YouTube videos - or extracting metadata, transcripts, thumbnails, or stream format data from them - YouTube Video Downloader is built for exactly that.

You can start with a single public URL, review the dataset output, and see in one run how much cleaner and more useful the output is compared to what you get from a browser-based tool.

From there, you can scale up to batches, schedule recurring runs, or connect it to your own code using the Apify JavaScript or Python clients.

Open YouTube Video Downloader on Apify →

How to download Instagram videos and Reels at scale with Apify

Emmanuel Uchenna — Wed, 17 Jun 2026 11:27:09 +0000

Picture this. Your marketing team just wrapped a three-month creator campaign. Thirty public Reels, a handful of IGTV videos, and dozens of short posts - all published, all public, and all living exclusively on Instagram. Now someone asks: "Can we archive these for the brand library?"

You open your browser, paste a URL into a random download site, and... nothing. Or worse, you get an ad-laden page with a broken download button. You try four more sites. Two work. One gives you a 240p file. And you've just spent twenty minutes manually processing what should take seconds.

Sound familiar? This is the reality for anyone managing Instagram video content at even a modest scale. The good news is there's a better way.

Instagram Video and Reel Downloader is an Apify Actor built specifically for this. It handles public Instagram Reels, /p/ video posts, and IGTV (/tv/) links in a repeatable, automation-ready workflow - giving you direct video URLs, metadata, thumbnails, and optional stored files, all in clean structured output.

This article walks you through exactly how it works, who it's for, and how to go from zero to a working download workflow in under ten minutes.

Why downloading Instagram videos is harder than it looks

On the surface, saving an Instagram video sounds trivial. The content is public. The URL is right there. How hard can it be?

Harder than expected, it turns out.

Instagram doesn't expose direct video file URLs through any official public API. The platform serves video content dynamically from its content delivery network, and the URLs it generates are session-bound, expire quickly, and are deeply embedded in the page's internal data structures. What you see in your browser isn't what a simple HTTP request sees.

On top of that, Reels in particular behave differently from regular posts. Some pages expose enough data in the static HTML for a fast scraper to pick up the video URL immediately. Others don't - they require a realistic browser session to load the required JavaScript and resolve the actual media delivery request before the video URL becomes visible.

This is why so many free Instagram downloader websites are inconsistent. They work sometimes, fail silently other times, and give you no real explanation when something goes wrong. For a one-off personal download, that's annoying but acceptable. For a business workflow, it's a real problem.

Instagram Video and Reel Downloader was designed around this reality. It uses a two-path extraction approach: try the fast static route first, and fall back to a browser-based session for harder Reel pages that don't expose the video URL in their initial HTML. The result is a more dependable tool for real public content workflows.

What Instagram Video and Reel Downloader actually does

Let's be clear about what this Actor is and isn't.

It is not a browser extension, not a mobile app, and not a one-click website. It's an Apify Actor - a serverless cloud program that accepts structured input, performs a task, and returns structured output. You run it from Apify Console, call it through the API, or trigger it from an automation. It keeps a record of every run, and you can reuse the same configuration whenever you need it.

More specifically, for each public Instagram URL you give it, the Actor can return:

a direct videoUrl pointing to the resolved video file
a thumbnailUrl for the post's preview image
the post title and description when available
the canonical URL, shortcode, and mediaType (reel, post, or tv)
optional file references if you choose to store the actual video or image in Apify's key-value store
structured error data (with an errorCode and errorMessage) if a URL can't be resolved - so the run stays auditable even when something goes wrong

That last point matters more than it might seem. Most free downloader tools just... fail. No error message, no record, no way to know what happened. This Actor keeps a dataset item for every URL it processes, whether it succeeded or not. For teams tracking a batch of URLs, that's a meaningful difference.

Ready to try it? Open Instagram Video and Reel Downloader in Apify Console and run it directly in your browser - no code required.

Who actually needs this?

Before getting into the walkthrough, it's worth being honest about who this tool is and isn't for.

If you occasionally want to save a single Instagram Reel for personal use, a free browser extension or a consumer downloader site will probably do the job. This Actor isn't aimed at that use case.

Where it earns its place is in repeatable, team-level, or integration-heavy workflows.

Social media managers and content teams often need downloadable copies of published Reels - for brand libraries, cross-platform repurposing, or simply keeping a backup of content that lives only on Instagram. Running a batch of URLs through this Actor takes minutes instead of an afternoon.

Agencies working with multiple clients or creator rosters regularly need to pull public video assets into approval workflows, reporting decks, or content management systems. A structured JSON output with direct video URLs and thumbnails is far easier to feed into those systems than manually downloading files and uploading them elsewhere.

Developers and automation builders can use the Actor as a building block. The Apify platform gives you API access, webhooks, and scheduling - so a workflow that starts with "download these public Instagram videos" can chain directly into "save to Google Drive," "send for review," or "push metadata to a spreadsheet."

Researchers and operations teams sometimes need video URLs and metadata for tracking or reporting purposes - not necessarily the files themselves. The dataset output covers that without requiring extra storage overhead.

Step-by-step: how to use the Actor from Apify Console

You don't need to write a single line of code to get your first result. Here's the full walkthrough.

Step 1: Open the Actor

Go to apify.com/eunit/instagram-video-and-reel-downloader.

If you don't already have an Apify account, you can create one for free. Once you're in, you'll land on the Actor's overview page. Read through the README - it's short and covers the key things to know before you run anything. Then click the Input tab.

Step 2: Collect your public Instagram URLs

The Actor supports three URL formats:

/reel/ - standard Instagram Reels
/p/ - regular photo and video posts
/tv/ - IGTV-style long-form video

Copy the public URLs you want to process. The critical word there is public. This tool doesn't attempt to access private accounts, private posts, or restricted content. If the post isn't publicly viewable without logging in, the Actor will return an error item for that URL rather than silently pretending it worked.

A quick way to check: open the URL in a private/incognito browser window. If you can see the video without logging in, the Actor can process it.

Step 3: Configure the input

The Input tab in Apify Console gives you a form to fill in. Here's what each field does:

startUrls - This is the only required field. It's the list of Instagram URLs you want the Actor to process. You can add one or fifty.

storeVideoFiles - When this is enabled, the Actor doesn't just resolve the video URL - it also downloads the actual video file and stores it in Apify's key-value store, accessible via a public URL. Useful when you need the binary file, not just a link. Leave it off if the videoUrl in the dataset is sufficient for your workflow.

storeThumbnailFiles - Same idea, but for the post's thumbnail image. Enable it if you need the actual image file stored alongside the video reference.

includePageHtml - Saves the raw Instagram HTML from each fetched page to key-value store. You'll rarely need this, but it's a useful debugging tool if Instagram changes its page structure and the Actor stops resolving video URLs correctly.

filenamePrefix - If you're storing files, this prefix gets prepended to every stored filename. Something like campaign-june-2026 makes it easy to find related files later.

maxConcurrency - Controls how many pages the Actor processes in parallel. The default of 3 is a reasonable starting point. Keep this lower (1 or 2) if Instagram is rate-limiting your requests; only push it higher if speed is a priority and you're willing to accept some request failures.

maxRequestRetries - How many times the Actor retries a failed page request before marking it as an error. The default of 2 is fine for most use cases.

requestTimeoutSecs - The timeout for individual page fetches and optional file downloads. 30 seconds works well under normal conditions; increase it if you're on a slow connection or downloading large video files into storage.

proxyConfiguration - Optional proxy settings. Most users don't need this to start. If you run into consistent request failures or geo-blocking behavior, enabling Apify Proxy here can help distribute requests more effectively.

A realistic input configuration for a small batch might look like this:

{
  "startUrls": [
    { "url": "https://www.instagram.com/reel/ABC123xyz/" },
    { "url": "https://www.instagram.com/p/DEF456uvw/" },
    { "url": "https://www.instagram.com/reel/GHI789rst/" }
  ],
  "storeVideoFiles": true,
  "storeThumbnailFiles": false,
  "includePageHtml": false,
  "filenamePrefix": "campaign-june-2026",
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Step 4: Run the Actor

Click Run in the top-right corner.

The Actor starts processing your URLs immediately. For straightforward public posts and video pages, it extracts the video URL and metadata from the static page response - fast, efficient, no browser overhead. For trickier Reel pages that don't expose the video delivery URL in their initial HTML, it falls back to a browser-based path that can capture the actual media requests.

You don't need to manage any of this yourself. The Actor handles the routing internally.

You can watch the logs in real time as the Actor runs. Each URL gets processed, and any failures surface immediately with an error code and message rather than a silent skip.

Step 5: Review the dataset output

Once the run completes, click the Dataset tab.

Each URL you submitted produces one dataset item. A successful result looks like this:

{
  "status": "ready",
  "inputUrl": "https://www.instagram.com/reel/ABC123xyz/",
  "canonicalUrl": "https://www.instagram.com/reel/ABC123xyz/",
  "shortcode": "ABC123xyz",
  "mediaType": "reel",
  "title": "Instagram video",
  "description": "Caption text from the post",
  "videoUrl": "https://instagram.fcdn.example/v/t16/f1/video.mp4",
  "thumbnailUrl": "https://instagram.fcdn.example/v/t51/thumb.jpg",
  "videoStoreKey": "video-campaign-june-2026-abc123xyz.mp4",
  "videoPublicUrl": "https://api.apify.com/v2/key-value-stores/STORE_ID/records/video-campaign-june-2026-abc123xyz.mp4",
  "uploadDate": "2026-05-15T14:30:00.000Z",
  "fetchedAt": "2026-06-17T10:00:00.000Z",
  "errorCode": null,
  "errorMessage": null
}

A failed result is just as structured:

{
  "status": "error",
  "inputUrl": "https://www.instagram.com/reel/PRIVATE123/",
  "errorCode": "NO_VIDEO_URL",
  "errorMessage": "Could not resolve a public video URL from this page."
}

You can export the dataset in JSON, CSV, Excel, or HTML format - whichever fits your downstream workflow. If you enabled file storage, you can access all stored video and image files directly from the run's Storage tab.

Step 6: Put the output to work

This is where the Actor starts earning its value beyond a simple file download.

If your workflow is "download these URLs occasionally and review them manually," exporting the dataset as CSV and opening it in a spreadsheet is enough.

If your workflow is more continuous - weekly campaign archives, automated content pipelines, or team-level review processes - the Apify platform gives you several paths to build on this run:

API calls: Trigger the Actor from any script using the Apify JavaScript or Python client, or a direct HTTP request. The output is the same structured dataset.
Schedules: Run the Actor on a weekly or daily cadence with a fresh list of URLs.
Webhooks: Fire a webhook when the run completes to trigger downstream steps in Make, Zapier, n8n, or your own backend.
MCP server: If you're building AI workflows, the Actor is available as a tool through Apify's Model Context Protocol (MCP) server.

Integrating the Actor with code

If you're a developer or want to trigger the Actor programmatically, the setup is straightforward.

JavaScript / TypeScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const input = {
  startUrls: [
    { url: 'https://www.instagram.com/reel/ABC123xyz/' }
  ],
  storeVideoFiles: true,
  filenamePrefix: 'my-archive'
};

const run = await client.actor('eunit/instagram-video-and-reel-downloader').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

items.forEach((item) => {
  if (item.status === 'ready') {
    console.log('Video URL:', item.videoUrl);
  }
});

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "startUrls": [{"url": "https://www.instagram.com/reel/ABC123xyz/"}],
    "storeVideoFiles": True,
    "filenamePrefix": "my-archive"
}

run = client.actor("eunit/instagram-video-and-reel-downloader").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("status") == "ready":
        print("Video URL:", item["videoUrl"])

Install the client with npm install apify-client (JavaScript) or pip install apify-client (Python), then drop in your Apify API token from Apify Console under Settings > Integrations.

When it makes sense to pay for a reliable Actor

Let's address the obvious question: why pay for this when free downloader sites exist?

The honest answer is that free downloader sites are built for individual, one-off use. They're not designed to process batches, return structured data, integrate with APIs, or recover gracefully when Instagram changes something. They also come with ad loads, pop-ups, and opaque data practices that you probably don't want in a business workflow.

This Actor uses a pay-per-event model at $90.00 per 1,000 processed videos. That pricing structure means you pay for actual completed work - not a flat subscription that bills you whether you use it or not.

For context: if a marketing manager spends 30 minutes manually downloading and organizing 10 Instagram videos per week, that's roughly 26 hours of manual work per year. The same batch run through this Actor costs under $1 and completes in a few seconds.

The economics shift further when you factor in reliability. A workflow that fails halfway through a batch, returns corrupted files, or gives you no error visibility isn't really "free" - it's just deferring the cost into your time and frustration budget.

Paying customers here are buying dependability, structured output, and the ability to build workflows that run the same way every time.

A few things to keep in mind

This Actor is built for public Instagram content, and it's worth being clear about what that means in practice.

Private content won't work. If a post requires you to be logged in to view it, the Actor can't access it, and it will return an error item for that URL.

Instagram can and does change things. The page structure, video delivery logic, and anti-bot behavior can shift with Instagram updates. When that happens, the extraction may stop working for some URL types until the Actor is updated. The structured error output means you'll know which URLs failed rather than getting a silent empty result.

You're responsible for compliance. Only download content you own, have explicit permission to use, or have a legitimate business reason to process. Respect copyright, platform terms, and local laws. The tool does what it's designed to do - how you use it is your responsibility.

Wrapping Up

Most Instagram video download tools are consumer utilities. They're built for "save this one Reel right now" - not for "process this batch of public videos, give me structured output, and let me automate it next week too."

If your work touches Instagram video content at any real scale - campaign archives, content pipelines, creator workflows, or automated reporting - that distinction matters a lot.

Instagram Video and Reel Downloader is built for the second use case. It gives you direct video URLs, structured metadata, optional file storage, a reliable fallback for difficult Reel pages, and a platform underneath it that supports APIs, scheduling, webhooks, and integrations.

Try it from Apify Console with a few public URLs. The first run takes about few seconds to set up, and you'll see immediately whether it fits the workflow you're building.

Get started with Instagram Video and Reel Downloader on Apify →

How to scrape Nextdoor for hyper-local demographics and community sentiment

Emmanuel Uchenna — Sat, 30 May 2026 07:05:14 +0000

Neighborhood-level data is where many high-value business decisions start. Citywide averages are useful, but they often hide the details that matter when you are choosing where to advertise, where to invest, or how to understand what local communities care about. If you want to find areas with strong homeownership, identify places where residents discuss common service needs, or analyze how people describe a neighborhood in public community pages, you need hyper-local signals.

That is exactly why so many teams look for a reliable way to scrape Nextdoor. The challenge is that Nextdoor is a relatively closed ecosystem. It does not offer an easy, public, developer-friendly path for extracting the kind of structured data marketers, analysts, founders, and researchers want. Manual collection is slow, inconsistent, and impossible to scale across hundreds of locations.

This is where Nextdoor Scraper helps. Built as one of the Apify Actors for the Apify platform, it turns public Nextdoor city-page data into structured output you can use. Instead of copying information by hand, you can collect geographic coordinates, resident counts, income indicators, homeowner ratios, affordability signals, FAQ content, and local community references in a repeatable workflow. If you need a practical Nextdoor API alternative for public pages, this is the faster path.

What is the Nextdoor Scraper?

Nextdoor Scraper is a scraping and data extraction Actor for collecting structured data from public Nextdoor pages, especially city pages. It is designed for people who want usable output, not raw HTML. That means you get cleaned fields that can go straight into a spreadsheet, dashboard, CRM, enrichment pipeline, or reporting workflow.

In simple terms, the Nextdoor Scraper visits public Nextdoor pages, extracts the visible and structured page data, and returns it in machine-readable formats such as JSON, XLSX, and CSV. That makes it easier to analyze neighborhood-level trends at scale without building your own browser automation stack from scratch.

The extracted fields are especially useful for local market analysis. Depending on the public page and the available structured data, the Actor can capture:

Geographic coordinates, such as a centroid with latitude and longitude
City and state details
Resident counts
Average age
Average income
Percentage of homeowners
Affordability and friendliness scores
FAQ entries
Neighborhood references
Business directory references
Local groups and additional community resources

For example, a public city page can expose useful signals such as a centroid of 40.016, -80.747, a resident count of 3,878, an average age of 35, an average income of $41k, and a homeowner share of 56%. It can also include livability indicators such as an affordability score of 99 and a friendliness score of 86. Those values are much more actionable than a plain list of URLs because they help you compare locations quickly and prioritize where to spend time and budget.

The result is a practical way to scrape Nextdoor data for research, local service lead generation, geographic analysis, and community sentiment discovery, while staying focused on public information.

Top 4 profitable use cases for Nextdoor data

The biggest reason to use a Nextdoor scraper is not the scraping itself. It is what the data helps you decide, launch, test, or sell afterward. Here are four high-value ways to use hyper-local demographics and community sentiment from public Nextdoor pages.

1. Building regional contractor and reputation platforms

If you are building a regional directory, a home services marketplace, or a contractor reputation platform, you need more than a list of businesses. You need neighborhood context.

Public Nextdoor data helps you map cities and surrounding local structures in a way that feels more grounded in how residents actually organize their communities. The FAQs, neighborhood references, and local groups can help you understand how people describe an area, what kinds of issues come up repeatedly, and which categories of services matter most.

That is valuable when you are building:

A local contractor comparison site
A regional reputation database
A neighborhood services recommendation engine
A local SEO content network for service categories

Instead of launching broad, generic pages, you can prioritize neighborhoods with meaningful public signals and create location-aware service experiences. That makes your product more relevant, and it can improve both engagement and conversion.

2. Lead generation for home services

This is one of the clearest commercial use cases. Home service businesses such as roofers, landscapers, plumbers, painters, pest control companies, and HVAC providers need to know where homeowners live, where disposable income is high, and where local demand is likely to justify the cost of outreach. Nextdoor city-page data can help you build exactly that picture.

For example, if a location shows 3,878 residents, an average income of $41k, and 56% homeowners, that is immediately more useful than running a generic campaign across an entire metro area. It gives you a better basis for deciding where to target direct mail, paid social, local landing pages, sales outreach, and neighborhood-specific offers.

You can use the data to:

Find areas with stronger homeowner density
Compare income signals across nearby cities
Prioritize communities for seasonal campaigns
Match marketing messages to neighborhood concerns surfaced in FAQs

A landscaping company may care about homeowner-heavy areas. A roofing business may want to focus on suburban communities with stable ownership and strong local trust signals. A cleaning or handyman service may use local groups and neighborhood references to decide where to pilot a new offer. That is the real advantage of scraping Nextdoor for lead generation. You are getting closer to the local context behind the customer.

3. Real estate investment and market research

Real estate teams already use public data from many sources, but the hyper-local context is often fragmented. That is where public Nextdoor data becomes interesting.

When a city page includes livability indicators such as an affordability score of 99 and a friendliness score of 86, you gain another layer of context for evaluating a market. These values do not replace formal due diligence, but they can help analysts screen locations faster before moving into deeper research.

A real estate investor, brokerage, or proptech company can use the data to:

Compare neighborhoods and cities at a glance
Identify areas that look affordable relative to nearby markets
Spot locations with strong community-oriented signals
Enrich internal market reports with public local context
Create lead magnets and market snapshots for clients

Geographic coordinates also matter. A centroid such as 40.016, -80.747 makes it easier to map results, visualize city clusters, and combine the output with other location-based datasets. That is useful for market scoring, territory planning, and neighborhood-level dashboards.

If you are producing local real estate data products, public Nextdoor scraping can work as a lightweight enrichment layer that adds community context to your pipeline.

4. Community sentiment and trend analysis

Community sentiment is often hard to quantify because it spans many channels. Nextdoor is valuable because it is built around neighborhood-level discussion, recommendations, local concerns, and community interaction.

Even when you focus only on public pages, FAQ sections, local groups, business directory references, and related resources can still reveal what a place emphasizes. That helps you understand whether a community leans toward family life, local commerce, safety, neighborhood organization, pet activity, home improvement, or other recurring themes.

This is useful for:

Local market researchers
Urban policy teams
Media analysts
Economic development groups
Brands planning geo-targeted campaigns

For example, if you see repeated references to neighborhood groups, practical local services, and community FAQs, that can point to a highly engaged local audience. If business directories are prominent, it may suggest strong local commercial activity. If certain topic clusters appear across multiple nearby pages, you may have found a regional trend worth tracking over time. In other words, the value is not just in scraping Nextdoor data once. It is in building a repeatable view of how communities present themselves publicly.

Key features of the Nextdoor Scraper

The appeal of Nextdoor scraper is that it is not just a one-off script. It is packaged as one of the Apify Actors, so it fits into a repeatable workflow on the Apify platform.

Speed and scale

Instead of manually opening pages and copying details into a spreadsheet, you can run automated extraction across many public URLs in one job. That saves hours of repetitive work and makes side-by-side comparison possible.

Structured output for analysis

The Nextdoor scraper returns normalized data fields, not just raw page content. That makes it easier to filter, group, sort, enrich, and export your results for business use.

Browser-based extraction on the Apify platform

Nextdoor pages are modern web pages, so browser automation matters. The Nextdoor scraper handles the browser-driven extraction flow for you on the Apify platform, which is much easier than maintaining your own extraction stack and local tooling.

Cost-effective automation

Building and maintaining your own pipeline for public-page collection can get expensive fast. Using Apify Actors reduces engineering overhead, shortens time to value, and gives you a practical way to pay for execution instead of building everything yourself.

Export formats that fit real workflows

Once the data is collected, you can work with it in formats your team already uses. JSON is ideal for developers and downstream automation. CSV is ideal for analysts, operations teams, and quick spreadsheet reviews.

Step-by-step tutorial: how to scrape Nextdoor data

Using Nextdoor Scraper on the Apify platform is straightforward.

Step 1: Access the Actor

Open Nextdoor Scraper in Apify Store. On the Actor page, click Try for free or Start to open the input form in Apify Console.

Step 2: Configure the target URLs

In the Input tab, add one or more public Nextdoor URLs. A common starting point is a city page, such as:

{
  "startUrls": [
    { "url": "https://nextdoor.com/city/north-miami--fl/" }
  ],
  "maxRequestsPerCrawl": 10,
  "maxConcurrency": 2,
  "maxItemsPerSection": 25
}

Step 3: Run the Actor

Click the Start button at the bottom of the page. The Actor initializes, processes the target pages, extracts structured data fields, and stores the results in a dataset.

Depending on the page, the output can include fields such as:

cityName
state
centroid
residentsCount
averageAge
averageIncome
percentageHomeowners
affordabilityScore
friendlinessScore
faqs
localGroups

Step 4: Export the dataset

Once the run finishes, open the Storage tab to inspect the extracted records. You can then export the dataset in the format that best fits your workflow.

Here is a simplified example of the kind of output you can expect:

[
  {
  "url": "https://nextdoor.com/city/example-city/",
  "cityName": "Example City",
  "state": "OH",
  "centroid": {
    "latitude": 40.016,
    "longitude": -80.747
  },
  "residentsCount": 3878,
  "averageAge": 35,
  "averageIncome": "$41k",
  "percentageHomeowners": "56%",
  "affordabilityScore": 99,
  "friendlinessScore": 86,
  "faqCount": 7,
  "accessStatus": "public"
},
...
]

From there, you can download the results as JSON or CSV and move directly into analysis, enrichment, reporting, or outreach planning.

Integrating Nextdoor data into your workflows

The value of scraped Nextdoor data increases when it flows into the tools your team already uses.

On the Apify platform, you can connect dataset outputs to automations and downstream systems. That includes sending results to Google Sheets for review, using Make for no-code workflows, or routing data into Zapier for notifications and follow-up actions. You can also plug the JSON output into your own analytics stack, internal dashboards, CRMs, or lead scoring systems.

This is especially helpful when you are monitoring multiple cities over time. Instead of exporting data manually after each run, you can make Nextdoor scraping into a repeatable process that supports lead generation, market tracking, and local intelligence workflows.

Is scraping Nextdoor legal?

Scraping should always be done responsibly. The safest approach is to focus on publicly available data, respect applicable rules and terms, and avoid collecting private or personal information behind login walls.

Nextdoor Scraper is designed around public page extraction. That means the workflow is intended for public city-page data and other publicly accessible content, not private profiles, direct messages, or restricted user data. You should also review your own legal obligations, compliance needs, and usage policies before running large-scale data collection.

Wrapping up

If you want to scrape Nextdoor for hyper-local demographics, local service lead generation, neighborhood research, or community sentiment analysis, structured public-page extraction gives you a strong starting point. Instead of relying on manual research, you can collect repeatable location data such as resident counts, income signals, homeowner ratios, affordability indicators, friendliness scores, FAQs, and local group references in a format your team can actually use.

Nextdoor Scraper turns that process into a scalable workflow on the Apify platform. Open the Actor in Apify Store, run your first extraction, and start building better local market insights today.

How to get YouTube data for a growth strategy: video and channel analysis

Emmanuel Uchenna — Thu, 09 Apr 2026 17:02:04 +0000

YouTube has more than 2 billion logged-in users every month and processes over 500 hours of uploaded video every minute. It is the second-largest search engine in the world. Yet the majority of creators and brands still build their content strategy the old-fashioned way: watching what competitors post, guessing which topics will resonate, and hoping the algorithm cooperates.

That approach works early on. But as your niche becomes more competitive and your publishing schedule more demanding, manual research stops scaling. You can't audit 20 competitor channels by hand every week, track which video formats are trending across five countries simultaneously, and still have time to actually make content.

What changes everything is structured data. When you can collect trending video metrics, popular channel signals, and keyword data programmatically - across countries, categories, and time - YouTube growth becomes a repeatable system rather than a creative gamble.

This article shows you how to obtain that data using two purpose-built Apify Actors, and how to turn it into a growth strategy grounded in what the platform is actually rewarding right now.

Why YouTube data matters for growth strategy

YouTube operates on two distinct engines: search and recommendations. Search surfaces content in response to specific queries. Recommendations push content to viewers who weren't actively searching - but whose watch history signals a relevant interest. A strong growth strategy needs both, and both require different types of data.

The data that matters most:

Trending videos by category and country - what the recommendation algorithm is actively amplifying, which formats and topics are getting pushed to new audiences
Popular channels and their recent content - which creators the platform is currently favoring, and what they're publishing right now
Popular keywords - what viewers are actively searching for at this moment, not six months ago
Engagement metrics (views, likes, comments) - signals of content-audience fit that go deeper than raw view counts

The official YouTube Data API was not designed for this kind of research. Its quota limits cap how much data you can pull per day, it has no trending channels endpoint, and accessing granular trend data by country and category requires complex workarounds that hit limits quickly. For recurring competitive research, the API becomes an obstacle rather than a tool.

Purpose-built scraping Actors solve this. The YouTube Trending Videos by Categories Scraper and YouTube Popular Channels Scraper extract exactly what you need - structured, export-ready data - without API keys, quota management, or engineering overhead.

If you want a step-by-step scraping walkthrough before diving into strategy, read How to scrape YouTube trends and popular channels first.

The growth data you can collect and why it matters

Before looking at how to use the data, it helps to understand exactly what each Actor returns and which growth question each data point answers.

Data point	Source Actor	Growth use case
Trending video rank, title, views, likes	YouTube Trending Videos by Categories Scraper	Spot viral formats and topics before your competitors do
`published_time_text` (e.g., "Yesterday", "3 hours ago")	YouTube Trending Videos by Categories Scraper	Identify how quickly content trends and optimal publishing windows
Video thumbnail URL	YouTube Trending Videos by Categories Scraper	Analyze thumbnail patterns that correlate with high click-through rate
Channel name, recent videos, video URLs	YouTube Popular Channels Scraper	Identify which channels are dominating your niche right now
Popular keywords	YouTube Popular Channels Scraper	Discover high-intent search terms for SEO and content ideation
Category breakdown per country	Both Actors	Understand regional audience preferences for localized content strategies
Engagement ratio (likes / views)	YouTube Trending Videos by Categories Scraper	Filter for content that converts viewers, not just attracts them

Together, these two Actors give you a full picture view of what YouTube is amplifying in your space - the videos rising to the top and the channels being pushed to new audiences.

Using trending video data for content strategy

Trending videos are the algorithm's endorsement. They represent the exact combination of topic, format, title structure, and production style that YouTube is choosing to amplify in a given country and category at a specific moment. Analyzing these patterns systematically - rather than browsing trending pages manually - reveals what's working before it becomes obvious.

How to collect trending video data

Go to YouTube Trending Videos by Categories Scraper on Apify. Create a free account if you don't have one.
Select your target Country (e.g., united-states, united-kingdom, india, brazil).
Select your Category (e.g., gaming, music, howto-and-style, science-and-technology).
Click Save & Start. The run completes in seconds.
Preview results in the table view or export as JSON, CSV, or Excel.

Each result includes: rank, video title, video URL, thumbnail URL, author (channel name), published_time_text, views, likes, and comments.

{
  "rank": "1",
  "video_title": "Marvel Television's Wonder Man | Official Trailer",
  "video_url": "https://youtube.com/watch?v=wHuWmjXsReU",
  "author": "Marvel Entertainment",
  "published_time_text": "Yesterday",
  "views": "5.2M",
  "likes": "113K",
  "comments": "4.9K",
  "country": "united-kingdom",
  "category": "All"
}

Run this Actor across multiple countries and categories, and you have a competitive intelligence layer that updates every time you trigger it.

Competitor channel analysis: who's winning and why

Channel analysis is where most creators focus - but most do it wrong. They look at subscriber counts. A channel with 800k subscribers might not have posted in two months and could be algorithmically invisible. The more useful signal is which channels are currently appearing as popular in trending data - because that reflects what YouTube is actively recommending to new viewers right now.

Why popular channel data beats subscriber counts

The YouTube Popular Channels Scraper returns the channels YouTube is currently surfacing as "buzzing" in a given country and category, along with their most recent videos. This is fundamentally different from a ranked list of the biggest channels. A channel with 80k subscribers appearing here is actively growing. A channel with 2 million subscribers that doesn't appear has lost algorithmic favor.

This distinction matters enormously for competitor research. Your real YouTube competitors are not necessarily your biggest industry rivals - they're the channels competing for the same viewer attention and winning right now.

What to analyze in popular channel data

Publishing frequency and recency. How many videos has each trending channel posted in the last 30 days? Channels maintaining a consistent 2-3 videos per week in trending data signal that the algorithm rewards their cadence. That's a publishing frequency benchmark, not just an aspiration.

Topic and title patterns. Across a channel's recent trending videos, which topics repeat? If a competitor's recent 6 trending videos are all about personal finance for under-30s, that sub-niche is clearly working for their audience. Look for the intersection between their most successful topics and gaps in your own content calendar.

Cross-country channel overlap. Run the Actor for the same category across the US, UK, and Australia. Channels appearing in all three are building genuine algorithmic momentum across English-speaking markets. These are the channels worth monitoring most closely.

How to collect popular channel data

Go to YouTube Popular Channels Scraper on Apify.
Select your Country and Category.
Click Save & Start. Results include channel names, recent video URLs, and popular keywords for the selected market.
Export and repeat for 3-5 target countries.
Build a simple tracking sheet: channel name, countries trending in, recent topic themes, and estimated posting frequency. Update it weekly.

{
  "country": "united-kingdom",
  "category": "all"
}

Building a multi-country growth intelligence system

A one-off data pull gives you a snapshot. A repeatable system gives you a competitive advantage that compounds over time. Here's how to build one without a development team.

The architecture

Collect - run both Actors weekly across your target countries (3-5) and category
Store - export results to Google Sheets or Airtable with a date column so you can track changes over time
Analyze - compare week-over-week: are the same channels trending, or new ones emerging? Are keywords shifting? Are engagement ratios moving up or down?
Act - translate the top 3 insights into your next 2 weeks of content: one trending topic, one underserved format, one keyword to target

Scheduling automatic runs

From Apify Console:

Configure your Actor run (country + category)
Click Save as a new task and give it a name (e.g., "US Gaming Trends - Weekly")
Go to Schedules in the left navigation and click Create new schedule
Add your task and set the frequency - weekly is a good starting cadence

No code required. The Actor runs automatically, and new results are appended to the dataset.

For developers: automate with Python

If you're building a dashboard, feeding a machine learning model, or integrating this data into a larger pipeline, the apify-client makes automation straightforward.

Install the client:

pip install apify-client

Run the YouTube Popular Channels Scraper programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "country": "united-states",
    "category": "gaming",
}

run = client.actor("eunit/youtube-popular-channels-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Swap eunit/youtube-popular-channels-scraper for eunit/youtube-trending-videos-by-categories to pull trending video data instead. The input schema is identical. You can run both Actors in sequence and merge the outputs into a single structured dataset for a complete weekly intelligence snapshot.

Your API token is in Settings > API & Integrations in Apify Console.

Pricing: what this costs in practice

Both Actors use a Pay-Per-Event pricing model. You are charged a fixed amount per result successfully scraped - not for platform compute time or server usage.

Actor	Price
YouTube Trending Videos by Categories Scraper	from $1.00 / 1,000 results
YouTube Popular Channels Scraper	from $1.90 / 1,000 results

In practice, running a weekly trend scan across 5 countries with 1 category each returns approximately 200-300 data points - costing under $0.30 per run, or roughly $1.20/month.

Traditional SaaS YouTube analytics platforms typically charge between $49 and $299/month for dashboards with less granular, less current data than you get from a direct scrape. The Pay-Per-Event model means you pay exactly in proportion to how much data you need. Scale up for a product launch. Scale down during quiet weeks. No subscriptions, no idle spend.

Legal and ethical considerations

Both Actors collect publicly available data - information that any user can see by visiting YouTube without logging in. They do not access private videos, user passwords, private messages, or any content that requires authentication.

Scraping public data for research, competitive intelligence, and informational purposes is generally considered acceptable in most jurisdictions. Operate within YouTube's Terms of Service, use the data for your own analysis, and avoid any use cases that involve republishing scraped content without appropriate attribution.

For Apify's broader guidance on ethical web scraping, see the Apify documentation.

Wrapping Up

YouTube growth stops being a guessing game when you treat it like a data problem. Trending videos tell you what the algorithm is amplifying right now. Popular channel data shows you who it's amplifying and why. Engagement ratios reveal which content is building real audience relationships versus passive viewership. And popular keywords hand you the exact search terms your potential viewers are using today.

The growth loop becomes repeatable: collect data weekly, identify the 2-3 highest-signal patterns, build your next content batch around them, and measure whether your own engagement metrics are converging with trending benchmarks.

The best channels in every niche are already doing this. The ones that aren't are still guessing.

Start collecting:

YouTube Trending Videos by Categories Scraper - from $1.00 / 1,000 results
YouTube Popular Channels Scraper - from $1.90 / 1,000 results

FAQ

Can I get YouTube trending data without the official API?

Yes. The YouTube Trending Videos by Categories Scraper extracts real-time trending data across 25+ countries and 13 categories with no API key required and no quota limits to manage. You get the same data you'd see by browsing YouTube's trending pages, in a structured, export-ready format.

How is "popular channel" data different from subscriber count?

Popular channel data reflects which channels YouTube is actively recommending to new viewers right now. A channel with 80k subscribers can appear here because it's publishing consistently and gaining algorithmic momentum. A channel with 2 million subscribers that hasn't posted recently won't. For competitive intelligence, current algorithmic favor matters far more than historical growth.

What categories can I track?

Both Actors support: All, Music, Gaming, Film & Animation, Autos & Vehicles, Sports, Pets & Animals, Comedy, Entertainment, News & Politics, How-to & Style, and Science & Technology - across 25+ countries including the US, UK, India, Brazil, Japan, Germany, Australia, and more.

How often should I run the scrapers?

Weekly is sufficient for most content strategies. Daily runs are useful during product launches, campaigns, or for fast-moving niches like Gaming or News & Politics where trends shift within hours.

Can I use this data for sentiment analysis?

Engagement ratios (likes-to-views, comment volume relative to views) are a reliable sentiment proxy at scale and require no comment scraping. For comment-level sentiment - what viewers are actually saying - pair these Actors with a YouTube Comments Scraper and feed the output to an LLM for summarization.

Is YouTube scraping legal?

Scraping publicly available YouTube data for research and informational purposes is generally legal in most jurisdictions. These Actors only collect data visible to any visitor without logging in. Always use scraped data in accordance with YouTube's Terms of Service and applicable privacy laws.

How to get Twitter (X) data for sentiment analysis

Emmanuel Uchenna — Fri, 03 Apr 2026 14:40:48 +0000

When a product fails publicly, Twitter (X) is usually the first place it shows up. Not the press. Not review sites. Twitter. Within hours, a single product flaw can spawn thousands of tweets, drive a hashtag into the top ten trends in three countries, and permanently shift how a brand is perceived.

That's why researchers, brand managers, journalists, and quantitative analysts all want the same thing: reliable Twitter data, fast, and at scale.

The problem is that getting that data has never been harder or more expensive. Twitter's own Application Programming Interface (API) now costs anywhere from $100 to $5,000 per month, and even then, it doesn't give you clean, structured trend data. Do-it-yourself scrapers require proxies, constant maintenance, and a tolerance for early-morning debugging sessions when Twitter changes its UI without warning.

This article is about a smarter path. You'll learn what kinds of Twitter data are most useful for sentiment analysis, how the main collection methods compare, and how to get structured, real-time Twitter trend data using the Twitter (X) Trends Scraper (PPE) on Apify - in under five minutes, with no setup.

What is Twitter sentiment analysis?

Sentiment analysis is the process of using natural language processing (NLP) or large language models (LLMs) to automatically classify the emotional tone behind a piece of text - typically as positive, negative, or neutral. When applied to Twitter, it turns raw social data into measurable public opinion signals.

Twitter (X) is uniquely valuable for this because people say what they actually think there. Unlike LinkedIn, where professional image management dominates, or product review sites, where feedback is structured and often incentivized, Twitter is where unfiltered reactions occur in real-time. A product launch, a policy announcement, a celebrity controversy - they all land on Twitter first.

Two distinct types of Twitter data matter for sentiment analysis:

Tweet content - the actual text of individual tweets, replies, quote-tweets, and threads posted about a topic, keyword, or brand. This is what most sentiment analysis tutorials focus on. You scrape tweets, run them through a classifier, and obtain a sentiment label for each tweet.
Trending topic data - the ranked list of hashtags and topics that are dominating conversation in a specific location at a specific point in time. This includes the trend rank, name, tweet count, and hourly history. I wrote an article on how to get Twitter (X) Trends Without Breaking the Bank.

Why trend data is an underrated input for sentiment analysis

Here's a question most sentiment analysis tutorials skip: what if you don't know which topic to analyze yet?

Tweet-level sentiment analysis works well when you already have a clear target - a brand name, a product, a specific hashtag. But trend data answers a different and often more valuable question: What are people talking about right now, and how intense is that conversation? That intensity is itself a sentiment signal.

Consider this: #ProductX is trending in São Paulo with 45k tweets and rising. You don't need to read a single tweet to know something significant is happening. The trend data alone indicate that a significant number of people are discussing this topic simultaneously, which almost always reflects a strong emotional reaction - positive or negative.

Here's what trend data actually reveals:

Tweet count as a proxy for emotional intensity. High tweet counts on a trending topic consistently correlate with strong emotional reactions. A hashtag with 80k tweets is generating far more sentiment "energy" than one with 2k. That count becomes your first filter for what's worth analyzing in depth.

Hourly trend history reveals sentiment momentum. The Twitter (X) Trends Scraper (PPE) captures trends across hourly snapshots, not just the current moment. A trend that appears at rank 40 with 5k tweets and climbs to rank 3 with 50k tweets in two hours signals something going viral. A trend that hits rank 1 and then rapidly falls often signals a controversy that burned hot and fast. These patterns are distinct and actionable.

Location-based trends reveal geographic sentiment clusters. A consumer product campaign trending in New York but not in London, or a political hashtag dominating Lagos but absent in Nairobi, gives you geographic precision that tweet-level analysis rarely provides without significant pre-processing.

Practical use cases where trend data drives sentiment decisions:

Brand monitoring: Catch a PR crisis developing before it peaks, while you still have time to respond.
Product launch tracking: See whether your campaign is gaining traction geographically, not just globally.
Competitor intelligence: Know the moment your competitor's brand starts trending - and with what intensity.
Academic research: Track regional public opinion on political events, elections, or policy announcements.
Content strategy: Identify topics with proven high engagement before committing creative resources to them.

So what does an effective data collection strategy actually look like?

Methods for getting Twitter (X) data

There are three realistic options for collecting Twitter data at scale. Here's an honest comparison.

The official X API

The X API is Twitter's own data access layer. It gives you structured, compliant access to tweets, user profiles, and some trend data via REST and streaming endpoints.

The problem is the pricing model. The free tier is limited to 1,500 tweets per month for write-only access - virtually useless for sentiment analysis. The Basic plan costs $100.00 per month and gives you limited read access. The Pro plan, which provides meaningful volume, costs $5,000.00 per month. Enterprise pricing isn't even listed publicly.

For trend data specifically, the API provides access to the trends/place endpoint, but it's been progressively restricted alongside the broader API changes since 2023.

Best for: Enterprise teams with compliance requirements who need guaranteed terms-of-service alignment and have the budget to justify it.

DIY web scraping

You could write your own scraper using tools like Selenium or Playwright to automate a browser, log in to Twitter, and extract trend data from the web interface.

The appeal is obvious: potentially free and fully flexible. The reality is less appealing. Twitter's front-end changes frequently, and every UI update risks breaking your scraper. To avoid IP blocks, you'll need rotating residential proxies, which can cost hundreds of dollars per month on their own. And all of that assumes you have the time and technical skill to build and maintain the scraper indefinitely.

Best for: Developers who treat scraper maintenance as a learning exercise rather than a cost.

Ready-made Apify Actors

Apify Actors are pre-built, cloud-hosted scrapers maintained by their developers and hosted on the Apify platform. They handle proxy management, authentication avoidance, output formatting, and cloud execution - so you don't have to.

The Twitter (X) Trends Scraper (PPE) is exactly this kind of tool. It was built specifically to extract real-time trending topic data from Twitter, supports granular location targeting across hundreds of countries and cities, and requires zero configuration beyond selecting your location.

Operating on a pay-per-event model (starting from $1.00 per 1,000 results), it is a fraction of the cost of the Pro API plan - and delivers trend data the API doesn't even expose cleanly. You only pay for the successful scrapes you actually perform.

Best for: Researchers, marketers, and developers who need reliable data without infrastructure overhead or expensive monthly subscriptions.

At a glance: method comparison

	Official X API	DIY scraping	Apify Actor (PPE)
Cost	$100–$5,000+/month	"Free" + proxy costs	From $1.00 per 1,000 results
Trend data quality	Limited, restricted since 2023	Variable (breaks on UI changes)	Structured, real-time, hourly history
Setup complexity	High (API keys, OAuth, rate limits)	Very high (proxies, parsing, maintenance)	None (click to run)
Maintenance burden	Low (API is stable until deprecated)	High (breaks on any UI update)	None (maintained by developer)
Location targeting	Country-level only	Depends on implementation	300+ countries and cities
Structured output	Yes (JSON)	Manual (requires parsing)	Yes (JSON, CSV, Excel, Sheets)
ToS compliance	✅ Fully compliant	⚠️ Gray area	✅ Public data only
Best for	Enterprise compliance-first teams	Developers who enjoy scraper maintenance	Researchers, marketers, and developers

How to use the Twitter (X) Trends Scraper (PPE) to get data for sentiment analysis

The Twitter (X) Trends Scraper (PPE) extracts real-time trending topics from Twitter (X) for any country or city in its location list. It captures trend rank, hashtag names, tweet counts, direct search links, and a rolling hourly history - making it directly useful as a sentiment intensity tracker rather than just a static snapshot tool.

Here's how to get your first dataset.

Step 1: Access the Actor

Go to Apify Store and search for "Twitter (X) Trends Scraper (PPE)", or navigate directly to apify.com/eunit/x-twitter-trends-ppe.

If you don't already have an Apify account, sign up for free. No credit card is required to start the trial. Once on the Actor page, click Try for free.

Step 2: Configure your input

The input form has a single field: Country / City. This simplicity is intentional - the Actor is designed to return complete trend data for your selected location in one run.

The dropdown supports a wide range of locations, from top-level country selections like Worldwide or united-kingdom, to city-level targeting such as united-states/new-york, nigeria/lagos, or india/mumbai.

For sentiment analysis purposes, location selection strategy matters:

If you're monitoring a brand or product, start with Worldwide to establish whether the topic is trending globally.
Then re-run with specific market locations (e.g., united-states, united-kingdom, australia) to understand where the conversation is concentrated.

The equivalent JSON input looks like this:

{
  "country": "united-states/new-york"
}

You can also pass this input directly via the API or Apify client libraries, which is covered in the automation section below.

Step 3: Run and inspect the output

Click Run. The Actor typically completes in seconds and returns a structured JSON dataset. Here's what each key field contains:

scraped_at - the UTC timestamp of the run. Critical for tracking data freshness and building time-series datasets.
country_input - the location string you submitted, useful for multi-location runs.
timeline - an array of hourly snapshots, each containing up to 50 ranked trends. Each trend includes its rank, name (the hashtag or topic), tweet_count, and a direct search link. This is the richest output for sentiment analysis.
tag_cloud - a broader set of trending terms for that location, useful for topic discovery.
table_data - the current top 50 ranked trends in a flat table format, ideal for quick dashboards.

A sample of the timeline output:

{
  "scraped_at": "2026-04-03T10:30:00.000Z",
  "country_input": "united-states/new-york",
  "timeline": [
    {
      "datetime": "2 minutes ago",
      "timestamp": "1743672600",
      "trends": [
        {
          "rank": 1,
          "name": "#TechLayoffs",
          "link": "https://twitter.com/search?q=%23TechLayoffs",
          "tweet_count": "82K"
        },
        {
          "rank": 2,
          "name": "New York Giants",
          "link": "https://twitter.com/search?q=%22New+York+Giants%22",
          "tweet_count": "14K"
        }
      ]
    }
  ]
}

Step 4: Export and use the data

Once the run completes, Apify Console gives you several export options directly from the dataset view: JSON, CSV, Excel, XML, and a direct Google Sheets integration.

For sentiment analysis pipelines, JSON or CSV are the most practical formats. JSON preserves the nested timeline structure (essential if you're tracking momentum across hourly snapshots), while CSV flattens the data for easy import into tools like Excel, Google Sheets, or pandas.

Tip: For a fully automated pipeline - where you scrape trends, then instantly analyze the sentiment of trending topics without writing any code - connect this Actor with LLM Dataset Processor directly in Apify Console. Point the LLM processor at your trend dataset, define a prompt like "Classify the likely sentiment associated with this trending hashtag as Positive, Negative, or Neutral based on context", and export the enriched results to Google Sheets automatically.

For developers: automating the pipeline

If you want to integrate live Twitter trend data into your own application or run it on a schedule, Apify provides client libraries for Python and JavaScript that make this straightforward.

First, install the Python client:

pip install apify-client

Then run the Actor programmatically:

from apify_client import ApifyClient

# Initialize the client with your Apify API token
client = ApifyClient("<YOUR_API_TOKEN>")

# Define your input
run_input = {
    "country": "united-states/new-york",
}

# Run the Actor and wait for completion
run = client.actor("eunit/x-twitter-trends-ppe").call(run_input=run_input)

# Fetch and process the results
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Trends for: {item['country_input']}")
    for trend in item["timeline"][0]["trends"][:10]:
        print(f"  #{trend['rank']} {trend['name']} - {trend['tweet_count']} tweets")

For JavaScript/Node.js projects, the pattern is identical using the apify-client npm package:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor("eunit/x-twitter-trends-ppe").call({
    country: "united-states/new-york",
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    item.timeline[0].trends.slice(0, 10).forEach(trend => {
        console.log(`#${trend.rank} ${trend.name} - ${trend.tweet_count}`);
    });
});

Scheduling for longitudinal datasets. The real power for sentiment analysis is in running the Actor on a regular cadence - say, every hour - and storing the results in a database or data warehouse. This builds a time-series dataset of trend rank and tweet count, which lets you track sentiment momentum over hours, days, or weeks. Apify's built-in scheduler handles this without external cron jobs or infrastructure.

AI agent integration. If you're building an AI agent that needs to be aware of current social trends, the Actor supports Model Context Protocol (MCP) server integration. Your agent can query live trend data and use it as context for downstream reasoning tasks.

Is it legal to scrape Twitter (X) trends?

This is a fair question, and it deserves a direct answer.

The Twitter (X) Trends Scraper (PPE) collects publicly available, factual data - specifically the list of trending hashtags and topics that Twitter displays to any unauthenticated visitor. It does not scrape private profiles, direct messages, content behind a login wall, or any personally identifiable information.

Scraping publicly available factual data is generally considered legal in most jurisdictions. The landmark hiQ Labs v. LinkedIn case in the United States established that scraping publicly available data does not constitute a violation of the Computer Fraud and Abuse Act (CFAA). Trend data - hashtag names, rankings, and aggregate tweet counts - is about as factual and public as data gets.

That said, you should always review the Terms of Service of any platform you interact with, and ensure your use case complies with applicable regulations. The General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) govern personal data, not aggregated metadata like trend rankings and hashtag names. As long as you're collecting and using trend data responsibly, you're on solid ground.

Wrapping Up

The hardest part of any Twitter sentiment analysis project isn't the analysis. It's getting clean, reliable, structured data to analyze. The official API is prohibitively expensive for most teams. DIY scrapers are fragile and require time-consuming maintenance. And most sentiment analysis tutorials skip past the data collection problem entirely.

The Twitter (X) Trends Scraper (PPE) fills that gap directly. It gives you real-time Twitter trend data - ranked hashtags, tweet counts, hourly history, and location-level granularity - in a structured format that plugs directly into your sentiment pipeline, whether that's a Python NLP script, an LLM-powered classifier, or a no-code Apify workflow.

The free trial requires no credit card. A single run takes about ten seconds. And the data you get back can be the difference between catching a PR crisis before it peaks and reading about it in the Monday morning report.

Try the Twitter (X) Trends Scraper (PPE) on Apify and start turning Twitter's real-time noise into a clear, actionable signal.

Frequently asked questions

How do you get data from Twitter (X) for sentiment analysis?

The three main options are the official X API (expensive, limited trend data), custom web scrapers (fragile, requires proxy infrastructure), and ready-made tools like Apify Actors. The Twitter (X) Trends Scraper (PPE) is the most practical option for sentiment-focused use cases, as it returns structured trend data, including tweet counts and hourly history with zero configuration.

What is the best tool for scraping Twitter trends in 2026?

The Twitter (X) Trends Scraper (PPE) on Apify is the most practical option for most use cases. It supports granular location data for hundreds of countries and cities, requires no proxy setup, and uses a fair pay-per-event pricing model (from $1.00 / 1,000 results) - a fraction of the cost of the official API.

Is it legal to scrape Twitter (X) for sentiment analysis data?

Scraping publicly available trend data is generally legal. Trend rankings and hashtag names are factual, public-facing data, not personal information subject to GDPR or CCPA. However, you should always review the platform's Terms of Service and ensure your use case complies with local regulations.

How much does it cost to get Twitter data for sentiment analysis?

The official X API starts at $100 per month for basic access and $5,000 per month for professional volume. The Twitter (X) Trends Scraper (PPE) on Apify operates on a pay-per-event basis, meaning you only pay for the data you successfully scrape. If you need tweet-level data in addition to trends, additional Actors on the Apify Store are available, each with their own pricing.

Can I get historical Twitter trend data?

Twitter does not provide historical trend data through its official API. The Twitter (X) Trends Scraper (PPE) captures trends at the time of each run, including the hourly timeline snapshot that Twitter maintains for the current period. To build a historical dataset, schedule the Actor to run at regular intervals (e.g., hourly) and store each run's output in your own database. Apify Console also retains your run history and datasets for later access.

How to scrape TikTok search results: A complete guide for 2026

Emmanuel Uchenna — Sat, 21 Mar 2026 14:58:52 +0000

How to scrape TikTok search results: A complete guide for 2026

TikTok has transformed from a simple video-sharing app into a global cultural engine. With over 1 billion active users, the platform dictates what we listen to, what we buy, and how we communicate. For businesses, marketers, and researchers, TikTok is not just entertainment; it is a repository of real-time consumer sentiment and trend data.

However, getting that data is a top challenge as TikTok employs some of the most sophisticated anti-scraping technologies in the world. In this article, we will explore why TikTok data is essential, the technical barriers to getting it, and how you can use the TikTok Search Scraper to automate your data collection at scale.

The importance of TikTok data in modern marketing

In the past, brands relied on slow-moving market research surveys. Today, they rely on TikTok in 2026 as a primary growth engine, utilizing authentic, low-polish content, behind-the-scenes storytelling, and trending sounds to drive viral engagement. With over 5 billion daily users, it is crucial for building brand loyalty and reaching audiences who spend a significant amount of time on social media. The speed at which a trend moves from a single video to a global phenomenon is unprecedented.

Trend tracking and cultural shifts

By scraping TikTok search results, you can identify emerging trends before they hit the mainstream. For example, a search for "sustainable fashion" might reveal specific local hashtags or challenges that are gaining momentum. Tracking these shifts allows you to adjust your content strategy in real-time, ensuring your brand remains relevant.

Audience behavior and sentiment analysis

Comments, view counts, and video descriptions provide insight into how users feel about specific topics. Analyzing these data points helps you understand the language your target audience uses, their pain points, and what kind of content they find engaging.

Competitor and influencer benchmarking

Who are the top creators in your niche? What music are they using? How often do they post? You can answer all these questions by scraping search results for relevant keywords. This data helps you benchmark your performance against competitors and identify potential influencers for future collaborations.

Key aspects of brand reliance on TikTok

Authenticity over production: Brands, particularly in fashion and beauty, succeed by using smartphone cameras to show raw "before and after" transformations rather than polished ads.
Driving growth: Small businesses in sectors like fashion, fintech, and beauty use TikTok as their primary marketing channel to reach new audiences.
Immersion & trends: Successful brands like Sandro Paris and Yahoo create immersive, trend-driven content to stay relevant in pop culture.
Massive reach & user-generated content (UGC): Major brands like ESPN hold millions of followers, while companies increasingly rely on UGC to build trust.
Strategic outsourcing: To keep up with the pace of trends, many companies outsource their TikTok creative to specialists

Why traditional scraping methods fail

If you have tried to build a simple Python script to scrape TikTok, you have likely encountered immediate blocks or endless captchas. TikTok's security infrastructure is designed to distinguish between automated bots and real human users.

Dynamic content and JavaScript heavy lifting

TikTok is a single-page application (SPA) that heavily relies on JavaScript to load content. A basic HTTP request will return an empty shell of a page. To get actual video data, you need a browser-based scraper that can render the DOM and handle infinite scrolling.

Advanced bot detection

TikTok checks for several technical signals to identify automation, among which include:

Navigator.webdriver: Many automated browsers leave this flag set to true, making them easy to spot.
Fingerprinting: Systems analyze your browser's font list, canvas rendering, and hardware identifiers to see if they match a standard device.
Client Hints: Modern anti-bot systems check sec-ch-ua headers. If your User-Agent indicates that you are on Chrome 123 but your headers say something else, you will be blocked.
Datacenter IP blocking: TikTok blocks entire IP ranges belonging to cloud providers (AWS, GCP, Apify, etc.) at the CDN level before your browser fingerprint is even evaluated. A request arriving from a datacenter IP receives an empty page shell regardless of how convincing the browser looks.

The solution: TikTok Search Scraper

The TikTok Search Scraper is a professional-grade Apify Actor designed to bypass these barriers. It does not just "read" the page; it interacts with it as a human would.

Key features for maximum resilience

Human-like behavior: The scraper mimics real user interaction. It does not just jump to the bottom of the page; it performs smooth, variable-speed scrolling. It even "pauses" to simulate a user watching a video, which significantly reduces the risk of triggering behavioral blocks.
Browser profile rotation: Every run can use a different browser profile. These profiles include curated User-Agents, realistic viewports, and synchronized Client Hints. This makes each request appear to be coming from a unique, legitimate device.
Stealth technology: By leveraging the stealth plugin, the Actor masks its automated nature, overriding the navigator.webdriver property and other common detection vectors.
Deep fingerprint evasion: Beyond the stealth plugin, each session generates a Bayesian browser fingerprint (canvas noise, WebGL vendor/renderer, AudioContext noise, WebRTC IP leak prevention) that is unique per run and consistent with the selected browser profile.
Session cookie injection: You can supply real cookies exported from your own browser directly into the actor's input. TikTok's scoring system grants significantly higher trust to sessions that arrive with authentic tokens like msToken and ttwid — the same ones a real browser accumulates after visiting TikTok normally.
Custom proxy URL: Instead of purchasing residential proxies, you can route the actor's traffic through your own machine. This makes TikTok see your home IP address rather than a cloud datacenter IP, which is often the single most effective unblock technique.
Automatic cookie persistence: After every successful run, the actor saves the live TikTok session cookies to Apify's storage. The next run automatically reloads them, so the session trust level builds up over time without any manual action.
Rehydration script fallback: TikTok stores its initial data in a JSON blob called a rehydration script. If the DOM structure changes or is temporarily hidden, the Actor can pull data directly from this script, ensuring high reliability even during platform updates.

Scraping TikTok without residential proxies

Residential proxies work reliably, but they come at a cost. The TikTok Search Scraper now includes two built-in techniques that let you bypass TikTok's datacenter block without purchasing a residential proxy plan.

Technique 1: Session cookie injection

When you browse TikTok in your regular Chrome or Firefox browser, TikTok sets a collection of session cookies on your machine — most importantly msToken, ttwid, and s_v_web_id. These tokens tell TikTok that the request is coming from a real, established browser session. When you inject them into the actor, TikTok extends that same trust to the automated run.

How to export your cookies:

Install the free Cookie-Editor extension for Chrome or Firefox.
Open https://www.tiktok.com and browse normally for about 30 seconds — scroll the feed, watch a video.
Click the Cookie-Editor icon in your browser toolbar.
Click Export → Export as JSON.
Copy the entire JSON array that appears.
Paste it into the Session Cookies field in the actor's input on Apify.

The actor normalizes the cookie format automatically, strips any extension-specific metadata, and injects the cookies before any request is sent. It also skips the homepage warm-up visit so that the higher-trust tokens you provided are not overwritten by datacenter-seeded ones.

Note: Cookies expire — typically within a few hours to a few days. If scraping stops returning results, re-export and update the field. The actor's automatic cookie persistence feature saves live cookies from each successful run, so subsequent runs start with increasingly fresh tokens even without a manual re-export.

Technique 2: Route traffic through your own machine

If you want every actor run to use your home IP address, you can run a lightweight SOCKS5 proxy on your local machine and expose it to the internet using a free tunneling service. The actor sends all its traffic through your tunnel, so TikTok sees your residential IP — the same one that works when you run the scraper locally.

Setup with microsocks and ngrok (free tier):

# Step 1 — Install and start a local SOCKS5 proxy
# macOS/Linux:
brew install microsocks   # or: sudo apt install microsocks
microsocks -p 1080

# Step 2 — Create a public TCP tunnel to your proxy
# (requires a free ngrok account at https://ngrok.com)
ngrok tcp 1080
# ngrok will print something like: tcp://0.tcp.ngrok.io:12345 → localhost:1080

# Step 3 — Paste the address into the actor's "Custom Proxy URL" field:
# socks5://0.tcp.ngrok.io:12345

Your machine must stay running and connected while the actor is active. The customProxyUrl field takes priority over the Proxy Configuration panel, so you do not need to configure anything else.

Tip: Combining both techniques — injecting real session cookies and routing through your home IP — gives the actor the best possible chance of success. TikTok sees a request from a known residential IP carrying an established session, which is indistinguishable from a real user.

What data can you extract?

The TikTok Search Scraper provides a comprehensive set of data points for every video result:

Field	Description
Video ID	The unique identifier for the video (`id`).
Video URL	The direct link to the TikTok video (`url`).
Description	The full caption, including hashtags and mentions (`description`).
View Count	The number of times the video has been played (`views`).
Likes	The number of likes the video has received (`likes`).
Comments	The number of comments on the video (`comments`).
Shares	The number of times the video has been shared (`shares`).
Author Info	Username (`uniqueId`), nickname, avatar URL, verification status (`verified`), and `followers` count (`author`).
Music Title	The song or sound used in the video (`music`).
Music Author	The creator of the music or sound used in the video (`musicAuthor`).
Hashtags	A list of hashtags included in the video (`hashtags`).
Duration	The length or duration of the video (`duration`).
Timestamp	The date and time the video was published (`timestamp`).
Scraped At	The exact date and time the data was extracted (`scrapedAt`).
Source	The method or source used to extract the data (`source`).

How to use the scraper

Here is a sample of the extracted JSON output:

 {
    "id": "7580436439..........",
    "url": "https://www.tiktok.com/@techuser112/video/7580436439..........",
    "description": "Comment \"tech\" if you want a road map on how to get into any of these careers",
    "views": "11K",
    "likes": "",
    "comments": "",
    "shares": "",
    "author": {
      "uniqueId": "techuser112",
      "nickname": "tech with Lesedi’s profile",
      "avatar": "https://p16-sign-sg.tiktokcdn.com/tos-alisg-avt-0068/9f813384bac2a.......",
      "verified": false,
      "followers": ""
 },
    "music": "Lesedi with Adrian’s 7AM - Slowed + Reverb",
    "musicAuthor": "",
    "hashtags": ["technology", "techjobs", "techreview", "techroadmap"],
    "duration": "2025-12-5",
    "timestamp": "",
    "scrapedAt": "2026-03-21T10:47:10.773Z",
    "source": "dom"
 }

How to use the scraper on the Apify platform

Running the TikTok Search Scraper on the Apify platform is a simple four-step process. You do not need any coding knowledge to get started.

Step 1: Set your queries

Once you are on the TikTok Search Scraper page, navigate to the Input tab. In the Search Queries field, enter an array of search terms you want to analyze. For example: ["tech review", "laptop unboxing", "coding tips"]. This screenshot below shows what you need to get started:

Step 2: Define your limits

Use the maxItems field to specify how many videos you want to collect per query. If you are conducting a deep dive, consider setting this to 500. For a quick pulse check, 20 results might be enough.

Step 3: Configure proxy settings

TikTok has strict regional and IP-based limits. To avoid being blocked, always use Apify Proxy. We highly recommend using Residential proxies, as they use IP addresses assigned to real households, making your scraping activity indistinguishable from regular traffic. Learn more about ApifyProxy from the Apify documentation.

Running without residential proxies: If you prefer not to purchase a proxy plan right now, the actor provides two alternative bypass options under the Advanced — Datacenter IP Bypass section of the input:

Session Cookies — Paste a JSON cookie array exported from your real browser (see Technique 1 above). The actor injects these before any request, causing TikTok to treat the session as a real, trusted browser.
Custom Proxy URL — Enter a socks5:// or http:// URL pointing to a proxy running on your own machine (see Technique 2 above). This makes TikTok see your residential IP.

Using both options together offers the highest success rate without any proxy subscription.

Step 4: Run and export

Click the Start button at the bottom of the screen. You can monitor the progress in the Log tab. Once the run is finished, go to the Storage tab to export your data. You can choose from several formats, e.g., JSON, CSV, Excel, or XML.

Optimization and cost efficiency with Pay-Per-Event

Many scrapers charge based on the computing power they consume, which can be unpredictable and vary significantly. The TikTok Search Scraper uses a more transparent model: Pay-Per-Event (PPE).

With PPE, you are charged a fixed price of $0.29 for every 1,000 video results. This means:

You only pay for the data you actually get.
Your costs are perfectly predictable.
You do not pay for the platform's overhead or idle time.

Use cases: Putting the data to work

Influencer marketing

Agencies use the scraper to find rising stars. By searching for niche keywords, they can identify creators who have high engagement but have not yet been "discovered" by major brands.

Market research

Pharma and retail companies scrape TikTok to see how people are using their products. Are they finding new use cases? Are they complaining about a specific feature? This qualitative data is often more honest than what you get in a focus group.

Trend capitalization

Media companies track trending hashtags to decide what topics to cover next. By seeing which videos are gaining the most views in a short period, they can jump on trends while they are still hot.

For Developer: Automate with JavaScript

If you are a developer looking to integrate the TikTok Search Scraper directly into your projects, you can use the official Apify API JavaScript/TypeScript client.

Installation

First, install the apify-client package in your project:

npm install apify-client

Usage Example

To get started, initialize the client with your Apify API token and pass your search queries. Here is an example script showing how to scrape data for the "AI coding" query:

import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
 token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "AI coding"
    ],
    "maxItems": 5,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    },
    // Optional: paste your browser cookies to bypass datacenter IP blocks for free.
    // Export from Cookie-Editor extension (Export as JSON), then paste the array here.
    // "sessionCookies": "[{\"name\":\"msToken\",\"value\":\"...\",\"domain\":\".tiktok.com\",...}]",

    // Optional: route traffic through your own machine instead of Apify's datacenter.
    // Run a local SOCKS5 proxy (e.g. microsocks -p 1080) and expose it with: ngrok tcp 1080
    // "customProxyUrl": "socks5://0.tcp.ngrok.io:12345",
};

// Run the Actor and wait for it to finish
const run = await client.actor("eunit/tiktok-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
 console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Legal and ethical considerations

When scraping TikTok, it is important to act responsibly.

Respect privacy: Only scrape publicly available data. Never attempt to access private accounts or unauthorized sections of the platform.
Avoid overloading: Use the scraper's built-in delays and human-like interactions to avoid putting unnecessary strain on TikTok's servers.
Comply with regulations: Ensure your use of the data complies with local regulations, such as GDPR in the EU or CCPA in California.

Wrapping Up

TikTok is a powerful source of real-time intelligence, but accessing it requires a professional approach. The TikTok Search Scraper on the Apify platform provides the stealth, speed, and reliability you need to turn raw social media activity into actionable business insights.

Start your first run today and stop guessing about the latest trends.

Happy automating!

How to actually start your fitness journey and stick to it (with the FitJourney.app platform)

Emmanuel Uchenna — Sat, 07 Mar 2026 12:46:36 +0000

You have been thinking about it for months. Maybe even years. That expensive gym membership is gathering dust, your workout clothes still have the tags on them, and you have a growing, endless list of reasons why “tomorrow” will be the perfect day to begin. Here is the honest, unfiltered truth about health and wellness: starting a fitness journey is absolutely the hardest part, but it is also the most transformative decision you will ever make for your body and your mind.

The first workout is always harder mentally than it is physically. We often build up incredible, insurmountable barriers in our minds. When you finally take that first step, you are not just beginning a workout routine - you are rewiring decades of self-doubt and breaking down established negative habits. However, taking that step can feel incredibly daunting because the modern fitness landscape is undeniably confusing. You have fitness influencers screaming conflicting advice, terrifying big box gyms full of confusing machines, and a dozen different apps you supposedly need to track your life.

This article is designed to cut through the noise. We are going to deeply explore why starting your health and fitness journey feels so overwhelming, the science-backed benefits of committing to a routine, and how to find the lasting fitness motivation you need. Most importantly, we will introduce the FitJourney platform, an all-in-one ecosystem that drastically simplifies progress tracking, fitness challenges, and educational content so you can finally stop planning and start doing.

Why starting your fitness journey feels so overwhelming

Obstacle 1: Analysis paralysis

You have probably researched every workout program, compared various gym memberships in your local area, and read countless articles about the ”best” way to get started. You spend hours watching videos on the optimal grip for a barbell deadlift, the exact macronutrient ratio for fat loss, or the perfect supplements to buy before you have even picked up a pair of dumbbells. Meanwhile, weeks turn into months of intense planning rather than taking any real action.

The truth about how to start a fitness journey is simple: the best program is the one you will actually follow consistently. Analysis paralysis keeps you stuck on the starting line, waiting for perfect conditions that will never arrive. You don't need to know everything to begin. You need to take the first step and learn as you go.

Obstacle 2: Fragmentation

Relying on scattered, disconnected tools completely kills your momentum. Right now, you might be using a basic notes app for tracking your sets and reps, a separate application for logging your meals and counting calories, another website for learning how to perform exercises safely, and a group chat for accountability. This fragmentation makes the simple act of working out feel like a stressful, overwhelming part-time job.

When your fitness tools are scattered across the wind, it is incredibly easy to drop the ball on one of them, which often leads to dropping the whole routine out of pure frustration. A unified, dedicated fitness tracking platform is essential for keeping everything organized and keeping you focused on the actual physical work.

Obstacle 3: Fear of judgment and unrealistic expectations

Walking into a new gym feels intimidating, especially when you are convinced that everyone else there has it perfectly figured out. Big box gyms can dramatically amplify this fear of judgment. You might worry that people are watching you struggle with a machine, or you might feel deeply embarrassed because you do not look like the fitness models prominently featured on your social media feed.

It is vital to remember that social media only shows the highlight reel, not the messy, sweaty middle of the journey. Furthermore, unrealistic expectations set you up for immediate failure. You might expect to drop 10 pounds in your first week and get discouraged when the scale barely moves. Sustainable fitness is about showing up consistently, embracing the learning curve, and rejecting the pressure to be perfect from day one.

It is vital to remember that social media only shows the highlight reel, not the messy, sweaty middle of the journey

The core benefits of committing to your fitness

Drawing inspiration from real science and established medical research, it becomes clear that committing to regular exercise is the ultimate game-changer for your entire life, far beyond just looking good in the mirror.

Physical health

The most obvious benefits of a health and fitness journey are physical. Exercise helps control weight by helping you burn calories and build active, lean muscle mass. But beyond the aesthetics, regular physical activity is a powerhouse for your internal, functional health. It vigorously combats chronic health conditions and diseases. Whether you are worried about hereditary heart disease or hoping to naturally prevent high blood pressure, being active boosts your high-density lipoprotein (HDL) cholesterol—often known as the “good” cholesterol—and decreases unhealthy triglycerides.

This one-two punch keeps your blood flowing smoothly, which significantly lowers your risk of cardiovascular diseases, type 2 diabetes, metabolic syndrome, and even certain types of cancer. Additionally, as you build muscle strength and greatly boost your endurance, you will find you have significantly more physical energy to tackle your daily chores, play with your children, and live life without getting easily winded.

Mental well-being

If you currently need a significant emotional lift or a reliable way to lower stress after a difficult, demanding day at work, a solid workout session can help tremendously. Physical activity stimulates various powerful brain chemicals, including endorphins and dopamine, that leave you feeling happier, significantly more relaxed, and noticeably less anxious. Exercise is a potent, natural antidepressant that helps you regulate cortisol levels in your body.

Longevity

Fitness is a lifelong marathon, not a frantic 30-day sprint. It is the single best investment you can make in your future self and your long-term independence. By maintaining your muscle mass and increasing your bone density through dedicated resistance training, you actively prevent the risk of debilitating falls and arthritis as you age. Getting fit is not just about extending the total years of your lifespan; it is about drastically improving your health span. This ensures that your later years are vibrant, active, and fulfilling, free from the extreme physical limitations that plague so many older adults.

How to define your "why" and set realistic goals

Before nervously lacing up your running shoes or paying for an expensive gym membership, you absolutely need to understand the psychology of goal setting and intrinsic motivation.

Finding your motivation

To continually sustain your health and fitness journey through the tough days, you must clearly define your underlying "why". Why do you genuinely want to get fit? Your motivation must be unique to you. If your initial goal is "to look good in a swimsuit," your motivation will likely plummet to zero the moment you put on a cozy sweater in the winter.

You need to look beyond surface-level aesthetics to understand your deepest, most personal reasons for getting fit. Do you deeply want to have the boundless energy to play with your children without losing your breath? Do you desperately want to reverse a frightening pre-diabetes diagnosis so you can live a long, full life? Do you want to build the physical confidence to hike a steep mountain on your next vacation? Write down your deeply rooted reasons on paper, and revisit them whenever your motivation wavers. Knowing your true "why" gives your fitness journey unshakeable, relentless purpose.

To continually sustain your health and fitness journey through the tough days, you must clearly define your underlying "why". Why do you genuinely want to get fit?

Goal setting

Establishing clear, intelligent, and achievable goals is crucial for maintaining constant motivation. However, there is a massive conceptual difference between outcome goals and behavioral goals.

An outcome goal might be something like "I want to lose 20 pounds by summer." While this is a perfectly fine desire, many complex biological factors influence the actual, literal outcome, which is outside of your direct control, such as your shifting metabolism, hormonal water weight fluctuations, and daily stress levels.

A behavioral goal is an actionable step that is 100% within your complete control, such as "I will work out three times a week for exactly 45 minutes" or "I will eat a serving of green vegetables with every single dinner." Focus your mental energy entirely on mastering behavioral goals. Celebrate each small behavioral victory, because these repeatable actions serve as tangible evidence of your commitment and will inevitably lead to the desired long-term outcome.

Consistency over perfection

Consistency beats perfection every single time. It is the golden, unbreakable rule of fitness. You don't need to execute perfectly to start a fitness journey, and you absolutely do not need to have a flawless diet to see amazing results. Life happens to all of us. Work gets crazy, the kids get sick, and natural motivation fluctuates from day to day.

If you miss a scheduled workout, it does not mean you have failed. If you eat a piece of cake at a birthday party, your progress is not magically ruined forever. Remind yourself constantly that taking a brisk 15-minute walk on a bad, stressful day is infinitely better than doing nothing at all. The people who ultimately succeed in fitness are not those who never make mistakes; they are the ones who refuse ever to quit.

Enter the FitJourney platform: your all-in-one fitness ecosystem

Now that you have a comprehensive understanding of the mental challenges, the profound medical benefits, and the resilient mindset explicitly required, you desperately need the right tools to execute your plan flawlessly. This is where the FitJourney platform comes in to revolutionize your approach.

Introducing the FitJourney.app platform

We built the FitJourney platform to permanently eliminate the confusing guesswork from your health and wellness goals. Remember the detrimental obstacle of fragmentation we discussed earlier? The FitJourney platform solves this critical issue by providing a comprehensive, all-in-one ecosystem. You no longer need five different, annoying applications to manage your health. Our integrated fitness tracking platform brings your scheduled workouts, your ongoing education, your nutrition guidelines, and your vital progress data into a single, beautiful, and highly intuitive interface.

Structured challenges

One of the biggest hurdles for nervous beginners is bravely walking into a gym and thinking, "What in the world do I do now?" The FitJourney platform removes this paralyzing anxiety entirely through our structured, expertly designed programs. By navigating to the Challenges section of the platform, you can quickly enroll in fitness challenges purposefully tailored to your specific experience level.

Whether you are tentatively stepping into a gym for the very first time in your life or you are an intermediate lifter looking to break through a plateau aggressively, our comprehensive challenges provide step-by-step guidance. You will know exactly which exercises to do, how many sets and reps to complete properly, and how to progress safely week after week.

Education and practicals

We firmly believe that education is the ultimate form of empowerment. We do not just want you to follow a generic workout routine; we desperately want you to understand exactly how your body works fully. The FitJourney platform features built-in, highly detailed educational articles and interactive practical exercises that teach you exactly how to train effectively and eat perfectly right for your dynamic goals.

A practical is a hands-on learning module that rigorously tests your knowledge in a real-world setting. You will thoroughly learn about the scientific principles of progressive overload, how to properly fuel your precious body before a grueling workout, and how to proactively prevent common injuries. This deep, expanding well of knowledge strongly ensures that you are building sustainable, intelligent habits that will truly last a lifetime, rather than just seeking an impossible 30-day quick fix.

All-in-one tracking

Seeing clear, tangible data is unequivocally the most reliable source of endless fitness motivation. When you can objectively see on a chart that you are getting stronger, running faster, and becoming healthier, everything falls into place wonderfully.

The FitJourney platform makes tracking completely effortless and highly rewarding. Your personalized Dashboard provides a clear, bird's-eye view of your current active streaks, your upcoming workouts, and your recent achievements. Meanwhile, Your Progress Tracker allows you to intimately log the specific, granular details of your journey, securely storing your daily workout data, body measurements, and athletic performance milestones over time. Seeing your personal line graph steadily trending in the right direction provides the incredibly powerful psychological reinforcement you need to keep showing up every single week.

3 simple steps to kickstart your journey today

Your fitness journey is quietly waiting for you. Not next Monday. Not after the busy holidays. Not when life miraculously gets less busy. Right today. The objectively hardest part is truly just making the bold decision that you are worth the time and physical investment. To help you aggressively overcome analysis paralysis, here are three simple, actionable steps you can confidently take right now to begin successfully.

Step 1: Get set up

Your very first action is to swiftly create your account on the FitJourney platform. The entire sign-up process is remarkably quick, intentionally. Once you are successfully logged in, take a few dedicated minutes to explore your newly personalized Dashboard. Familiarize yourself with the clean interface, set your initial user preferences, and complete your basic profile. Setting up your digital environment is a crucially important psychological cue to your brain that you are officially beginning a completely new chapter of your life.

Step 2: Join a challenge

Once your account is fully set up, purposefully navigate to the Challenges tab located in the main menu. Enthusiastically browse through the available, structured programs and select a beginner-friendly challenge that perfectly aligns with your current fitness level and your personal goals. Do not overthink it; the primary goal right now is to build the consistent, daily habit of working out.

Step 3: Track and learn

When it is finally time for your critical first workout, open up the platform, diligently follow the detailed instructions safely provided in your chosen challenge, and give it your absolute best effort. Immediately after finishing, triumphantly jump into Your Progress Tracker to carefully log your completed workout. Openly celebrate this massive, foundational win! Then, take 10 minutes to read your first educational article properly or complete an engaging practical on the platform.

Wrapping up

Actively starting your health and fitness journey is undoubtedly the hardest part of the entire process. It demands that you aggressively overcome deep-seated self-doubt, push decisively past paralyzing analysis paralysis, and bravely step into the vast unknown. But the staggering rewards—boundless daily energy, profound mental clarity, and exceptional long-term health—are ultimately worth infinitely more than the brief initial discomfort.

To predictably succeed, you must define your deeply personal "why", prioritize relentless consistency over impossible perfection, and set behavioral goals that you can easily and directly control. Furthermore, strategically leveraging the right tools is paramount. By actively using the FitJourney platform, you permanently eliminate the distracting confusion of fragmented apps and systematically replace it with a focused, all-in-one fitness ecosystem that tracks your progress and educates your mind.

Ready to definitively stop planning and start doing? Create your free account on the FitJourney platform and join your very first challenge today. Your future self will profoundly thank you for the incredible courage you showed right now.

DEV Community: Emmanuel Uchenna

How to Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion

The evolution of programmatic indexing

Moving from passive crawling to active push pipelines

Stage 1: Automated URL discovery with Fast Sitemap Generator

Configuring the URL crawler

Key configuration settings explained

Output storage and Dataset generation

Stage 2: Programmatic submission via Google Cloud infrastructure

Step 1: Setting up Google Cloud credentials

Step 2: Granting Google Search Console access

Bypassing DIY limitations: Python vs. Apify Actors

Why custom scripts break at scale

The solution: Google Indexer & Instant SEO Submitter

Stage 3: The missing link: Logging to Notion via MCP Connectors

Understanding MCP connectors

Setting up the Notion database

Authorizing the Notion MCP connector

Connecting the Notion Uploader Actor

Orchestrating the full "set and forget" pipeline

Step 1: Creating saved Actor tasks

Step 2: Chaining tasks using Actor-to-Actor integrations

Chain 1: Sitemap Generator → Google Indexer

Chain 2: Google Indexer → Notion Uploader

Step 3: Scheduling automated pipeline execution

Best practices for automated SEO indexing

1. Submit only modified or new content

2. Match canonical URLs strictly

3. Use Test Mode during initial setup

4. Monitor quota usage in Google Cloud Console

Wrapping up

Read more

How to Automate Instagram Reels to Notion via MCP

The content archiving nightmare

The pipeline architecture: how MCP connectors change the game

1. Dual-path extraction engine

2. Solving CDN URL expiration via Key-Value Store

3. Automated dataset push via Notion Uploader and MCP connectors

4. Zero-trust token security and governance

Step 1: Setting up the Notion database

Step 2: Configuring Instagram Video and Reel Downloader

Input JSON configuration

Key configuration settings explained

Step 3: Wiring the Notion Uploader integration in Apify Console

1. Creating your Notion MCP connector (or API key)

2. Adding the Notion Uploader integration to your Actor

3. Mapping fields and configuring Notion Uploader input

Step 4: Executing the run and verifying the content calendar

Watching automated workflow execution

Extending the workflow with API triggers and scheduling

Programmatic execution via JavaScript / TypeScript

Programmatic execution via Python

Automated set-and-forget scheduling

Wrapping up

Headless browsers vs. SERP APIs: Why your Playwright scraper fails in production

The real cost of doing it yourself

1. The proxy infrastructure tax

2. Fragile HTML selector maintenance

3. Massive server resource drain

4. Advanced anti-bot fingerprinting and CAPTCHAs

The abstracted alternative: Dedicated SERP APIs

Side-by-side implementation: Playwright vs. SearchApi

Building a custom Playwright scraper in Python

The streamlined SearchApi approach

Code walkthrough and key differences

Technical performance and cost showdown

Detailed architectural comparison

Real-world cost breakdown at scale

Handling advanced search engine layouts and rich snippets

The challenge with DIY extraction of interactive elements

Extracting structured SERP data effortlessly

Decision matrix: Choosing the right tool for your project

When to use headless browsers (Playwright, Puppeteer, Selenium)

When to use a dedicated SERP API (SearchApi)

Wrapping up

Further reading and references

Building a real-time AI search agent with SearchApi and OpenAI

Why traditional web scraping fails for AI agents

System architecture

Prerequisites and project setup