Programmatic Traffic Generation with Python and Apify

Data pipelines often involve more than just raw data extraction. Sometimes, the data you push downstream is not records to be processed, but actions to be taken, or simulated user behavior to measure impact. This is where a tool like the Traffic Generator Actor becomes useful. It is designed to generate realistic page views, scrolls, and interactions, which is helpful for SEO validation, website monitoring, or analytics testing.

Running this Actor manually in the console is straightforward, but integrating it into a resilient, programmatic pipeline requires careful consideration of platform limits, state management, and potential failure modes. This guide details how to orchestrate the Actor programmatically, handle platform constraints, and build robust automated workflows.

How can I prevent double-processing on re-run?

To prevent double-processing on re-run, manage state by tracking Apify run IDs and processing results only from successfully completed, new runs. Store each unique run ID with its status in a persistent database before triggering downstream actions. Before a new run, check the completion status of previous runs and the last processed run ID. If a run fails, re-process its data after a successful retry, or initiate a new run, discarding the failed attempt's incomplete output.

When you trigger the traffic-generator Actor, it provides a unique run ID. This identifier is crucial for tracking. Since a run might complete, fail, or be aborted, you need a mechanism to only act on new, successful data. For example, if you are pushing visit metrics to a warehouse, you only want to ingest a visit record once.

Consider a Python orchestration using the Apify Client. After starting a run, you should store the metadata immediately.

import os
from apify_client import ApifyClient

# Initialize the ApifyClient with your API token
# This token should be stored securely, e.g., in an environment variable
apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_id = "crawlerbros/traffic-generator"

# Input for the Traffic Generator Actor
run_input = {
    "startUrls": [{"url": "https://example.com"}],
    "mode": "PAGEVIEW",
    "enableCrawling": False,
    "waitOnPage": 15,
    "endAfterSeconds": 180
}

print(f"Starting run for actor {actor_id} with input: {run_input}")

# Start the Actor run
run = apify_client.actor(actor_id).call(run_input=run_input)

print(f"Actor run started: {run['id']}")

# In a real pipeline, you would store run['id'] in a database
# with a status (e.g., 'started', 'completed', 'failed')
# and a timestamp. Then, upon completion, update the status.
print(f"Run status: {run['status']}")

The output of the traffic-generator is a dataset, accessible via run['defaultDatasetId']. Each record in this dataset represents a single page visit. To prevent double-processing, store the run['id'] and the run['defaultDatasetId'] after a successful run. When a subsequent run completes, compare its ID to your database of processed runs. If it is new, fetch the dataset items. If it is a retry of a previously failed run, replace the old, incomplete data with the new successful data.

How can I manage long-running Actor jobs?

For jobs exceeding 300 seconds, you must trigger the Actor asynchronously and then either poll for completion or handle webhook notifications. The Apify synchronous run endpoint has a hard-cap at 300 seconds and returns an HTTP 408 gateway timeout beyond that limit. If you need to generate traffic with long dwell times, run deep crawls, or set long session durations, synchronous execution will fail.

The traffic-generator Actor's endAfterSeconds input parameter can be configured to 0 for unlimited runtime, allowing the Actor to run until all target URLs are fully processed. Because this easily bypasses the 300-second synchronous limit, an asynchronous orchestration strategy is required. A reliable pattern involves kicking off the run via a non-blocking API call and relying on a webhook to notify a downstream service (such as n8n, a custom API gateway, or a serverless function) when the run finishes.

Here is how you can trigger a long-running job and configure a webhook payload template:

import os
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_id = "crawlerbros/traffic-generator"
webhook_url = "https://your-custom-api-endpoint.com/webhook-receiver"

run_input = {
    "startUrls": [{"url": "https://apify.com/blog"}],
    "mode": "PAGEVIEW",
    "enableCrawling": True,
    "maxPagesPerUrl": 5,
    "waitOnPage": 60,
    "endAfterSeconds": 0,  # 0 indicates unlimited runtime
    "enableAdvancedFingerprinting": True
}

print(f"Starting long-running run for actor {actor_id}")

# Start the actor run without blocking
run = apify_client.actor(actor_id).start(
    run_input=run_input,
    webhooks=[
        {
            "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED", "ACTOR.RUN.ABORTED"],
            "requestUrl": webhook_url,
            "payloadTemplate": """
            {
                "runId": {{run.id}},
                "status": "{{run.status}}",
                "defaultDatasetId": "{{run.defaultDatasetId}}",
                "startedAt": "{{run.startedAt}}",
                "finishedAt": "{{run.finishedAt}}"
            }
            """
        }
    ]
)

print(f"Actor run initiated: {run['id']}. Webhook will notify {webhook_url} upon completion.")

By decoupling the execution from the API request thread, your pipeline avoids timeout exceptions and handles execution times of any length.

How can I schedule recurring traffic generation?

To schedule recurring traffic generation, create an Apify Schedule with a 6-field cron expression, ensuring that the target Actor has been run manually at least once before creating the schedule. When you create a new schedule on the Apify platform, it is created disabled by default. You must explicitly enable the schedule, either through the Apify Console UI or by setting the enablement property via the API client.

The minimum interval for Apify schedules is 10 seconds. When setting up a recurring job to test websites or generate steady streams of pageviews, the schedule acts as an automated trigger.

import os
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_id = "crawlerbros/traffic-generator"
actor_task_id = "your-pre-configured-actor-task-id"

# The target Actor must have run at least once before a schedule can be created.
# Ensure that actor_task_id points to an existing, previously executed task.

schedule_input = {
    "startUrls": [{"url": "https://example.com"}],
    "mode": "PAGEVIEW",
    "waitOnPage": 20,
    "endAfterSeconds": 300,
    "enableCrawling": False
}

# Cron expression with 6 fields: second, minute, hour, day of month, month, day of week
cron_expression = "0 0 * * * *"  # Every hour on the hour

print(f"Creating schedule for task {actor_task_id} with cron '{cron_expression}'")

schedule = apify_client.schedules().create(
    name="hourly-traffic-generation",
    cronExpression=cron_expression,
    isEnabled=True,  # Explicitly enable the schedule, as they default to disabled
    actorTaskId=actor_task_id,
    input=schedule_input
)

print(f"Schedule '{schedule['name']}' created with ID: {schedule['id']}. Enabled status: {schedule['isEnabled']}")

If you do not explicitly activate the schedule during creation or through the console, the job will remain dormant. Additionally, ensure your task inputs are specified directly; platform features like input schema prefill values are only visible in the console UI and are not applied to API calls.

Programmatic execution and dynamic behaviors

To simulate organic human interaction across multiple categories of web pages, you can design your automation script to build dynamic inputs. By modifying parameters like mode, waitOnPage, and enableCrawling, you can target different behavior patterns in a single script.

For example, you might want a default PAGEVIEW run for normal blogs, a deeper browsing session for documentation, and a specific video playback session for YouTube URLs.

import os
import random
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])
actor_id = "crawlerbros/traffic-generator"

def generate_traffic_input(url, behavior_type="normal"):
    base_input = {
        "startUrls": [{"url": url}],
        "enableAdvancedFingerprinting": True
    }

    if behavior_type == "normal":
        base_input.update({
            "mode": "PAGEVIEW",
            "enableCrawling": True,
            "maxPagesPerUrl": random.randint(3, 7),
            "waitOnPage": random.randint(20, 40),
            "endAfterSeconds": 300
        })
    elif behavior_type == "deep_engagement":
        base_input.update({
            "mode": "PAGEVIEW",
            "enableCrawling": True,
            "maxPagesPerUrl": 1000,  # Maximum supported value is 1,000
            "waitOnPage": random.randint(60, 120),
            "endAfterSeconds": 0,  # Run until completion
            "crawlingLinkSelector": "a[href]"
        })
    elif behavior_type == "youtube_view":
        base_input.update({
            "mode": "PAGEVIEW",
            "enableYoutube": True,
            "waitOnPage": 180,
            "endAfterSeconds": 240
        })

    return base_input

# Executing a run with deep engagement configuration
deep_input = generate_traffic_input("https://example.com/docs", "deep_engagement")
print(f"Triggering deep engagement run...")
run = apify_client.actor(actor_id).call(run_input=deep_input)
print(f"Run completed with dataset ID: {run['defaultDatasetId']}")

This structural flexibility allows you to mimic different profiles or adjust tests programmatically without hardcoding inputs.

Understanding platform cost metrics

The Traffic Generator Actor uses a real Chromium browser behind the scenes. This means it has a larger resource footprint than HTTP-only scrapers. Resource usage on Apify is measured in Compute Units (CUs), calculated using the formula:

CU = (memory_mb / 1024) * duration_hours

Your billing tier dictates your per-CU rate. Under the standard platform tiers:

• Free tier ($0/mo) costs $0.20 per CU.
• Starter tier ($19/mo) costs $0.20 per CU.
• Scale tier ($199/mo) costs $0.16 per CU.
• Business tier ($999/mo) costs $0.13 per CU.

For runs requiring geographic targeting or IP protection, you can configure the proxyConfiguration field. When utilizing Apify's residential proxies, data usage is charged per gigabyte:

• Free and Starter tiers charge $8/GB for residential bandwidth.
• Scale tier charges $7.50/GB.
• Business tier charges $7/GB.

Because the Actor runs a full headful browser session, your costs scale directly with the waitOnPage setting and the number of pages navigated via enableCrawling. To control runaway costs on long crawls, set the endAfterSeconds limit to act as a runtime gate. You can also pass the maxTotalChargeUsd query parameter on your API run endpoints, which maps to the environment variable ACTOR_MAX_TOTAL_CHARGE_USD inside the running container. While this serves as an effective budget safety switch, note that container termination is not instantaneous; resource consumption may continue briefly after the threshold is breached.

Managing proxy sessions and geographic targeting

When simulating local user traffic, geographic targeting is essential. The Apify platform allows geographic targeting with US state granularity via specific proxy routing configurations, such as using the group name format country-US_XX where XX is the state abbreviation.

However, proxy session persistence is a crucial variable in long crawls. Datacenter proxy sessions persist for up to 26 hours, whereas residential proxy sessions persist for approximately 30 minutes. If you configure a long crawl with high waitOnPage values that extends past 30 minutes, your residential IP session will rotate mid-run. This can cause subsequent requests to show a different exit IP address or location.

Here is an example input configuration targeting a specific region:

{
    "startUrls": [
        { "url": "https://www.etsy.com/shop/YourShopName" }
    ],
    "mode": "PAGEVIEW",
    "waitOnPage": 45,
    "endAfterSeconds": 300,
    "enableCrawling": true,
    "maxPagesPerUrl": 10,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "groups": ["RESIDENTIAL"],
        "countryCode": "US"
    }
}

If your workflow requires continuous, multi-hour crawls originating from a single, static location, you must break the session down into a sequence of smaller runs under 30 minutes each. This avoids mid-run session loss and ensures geographic consistency across all page views.

Downstream processing of the visit dataset

Once a run Succeeded, the output data is stored in a dataset. Each item represents a single page visit and contains detailed navigation metrics.

Here is the JSON schema structure of a standard output record from the dataset:

{
    "url": "https://example.com/product-1",
    "status": "success",
    "pageTitle": "Product One Title",
    "loadTimeMs": 620,
    "timeOnPageSec": 35.4,
    "scrollDepthPercent": 75,
    "linksFound": 12,
    "linksFollowed": 1,
    "referrer": "https://example.com/shop",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36",
    "viewport": "1920x1080",
    "timestamp": "2026-03-23T10:30:00.000Z"
}

When integrating with automation tools like n8n, you can use the Apify Trigger node. Since n8n's trigger node fires immediately on run completion, you do not need to construct manual polling logic. Once the trigger fires, you can pull the dataset using the defaultDatasetId and route the logs to databases or visualization systems.

Handling data retention limits and storage expiration

A common pitfall when building pipelines is relying on default, unnamed storages for historical reporting. When the Actor runs, it writes to an unnamed dataset unless a specific named dataset is defined in the execution settings.

The Apify platform enforces strict retention limits on unnamed storages. On the Free plan, only the 10 most recent runs are retained, and files are automatically deleted after 4 months. If you trigger the Actor frequently, older datasets will expire quickly, resulting in data loss.

To guarantee data persistence, you should immediately fetch and copy your dataset records to an external data warehouse upon run completion. Alternatively, you can use named storages, which are exempt from automatic retention deletions.

Storage operations also carry hard platform rate limits. Datasets and request queues are capped at 400 requests per second for item pushes and CRUD operations, while standard storage objects are limited to 60 requests per second. For high-volume parallel crawls, you must design your clients to stay within these thresholds to prevent throttling.

Technical limitations and critical failure modes

This Actor runs a real browser, which makes it more resource-intensive than simple request libraries. There are several structural limitations and failure points to design around:

1. CAPTCHA blocks: The Actor cannot solve CAPTCHAs. If a website forces a CAPTCHA challenge on load, the execution will stall, resulting in a timeout or a blocked status.
2. Empty start URLs: The startUrls array is marked as required in the input schema. Passing an empty array will trigger a validation error immediately, causing the execution to fail before the browser even launches.
3. Queue limitations: A request queue can only be processed by a single Actor or task run at a time. Trying to run multiple concurrent Actor runs off the exact same request queue for a fan-out style execution will fail; each concurrent run requires its own independent queue.
4. YouTube tracking limitations: YouTube employs complex internal anti-fraud algorithms. While the Actor simulates video playback, some views may still fail to register in YouTube's internal analytics dashboards.
5. No native cloud integrations: The Apify platform does not feature native AWS S3 or Slack integrations. If your pipeline requires exporting visit datasets to S3 or sending alerts to Slack, you must route those operations through webhooks, custom API scripts, or external automation engines.

Below is a Python snippet demonstrating how to programmatically analyze your output dataset to detect and handle failed or blocked visits:

import os
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# Fetching the items from a completed dataset run
dataset_id = "your-completed-dataset-id"
dataset_items = apify_client.dataset(dataset_id).list_items().items

for item in dataset_items:
    url = item.get("url")
    status = item.get("status")

    if status != "success":
        print(f"Alert: Visit to {url} failed with status: {status}")
        # Here you would trigger custom logic, such as switching proxies,
        # flagging the URL, or retrying with a different user agent.
    else:
        print(f"Successfully processed {url} | Load Time: {item.get('loadTimeMs')}ms")

By explicitly evaluating the status field of each item in the dataset, your downstream pipeline can automatically filter out failed page loads or raise alerts when targeting systems block your requests.

Checked against the Actor's input schema and Apify docs on 2026-09-12.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]