The challenge of scraping job boards at scale
Extracting job listings from major platforms is rarely a simple matter of fetching a page and parsing HTML. Platforms like ZipRecruiter employ aggressive defensive measures, such as Cloudflare protection, which instantly block standard datacenter IP addresses. Furthermore, job search engines present architectural hurdles: they often render duplicate listings in search panels, fail to provide full job descriptions on the search results page, and constantly update their listings, which can lead to double-processing the same jobs in downstream databases.
To build a reliable data pipeline, you must orchestrate your scraper as part of a larger stateful system. It is not enough to run a scraper and dump raw JSON into a folder. You need to handle proxy authentication, manage pagination boundaries, detect platform blocks, filter out duplicate records, and design a downstream ingestion process that remembers which jobs have already been processed.
This guide demonstrates how to build an orchestration pipeline around the ziprecruiter-scraper-pro Actor. We will look at how the Actor's input and output schemas dictate your pipeline design, how to handle the platform's constraints, and how to write the state-management logic in Python and pandas to ensure your database stays clean on subsequent runs.
What is the correct input schema for ziprecruiter-scraper-pro?
To run the Actor, pass a JSON payload specifying startUrls or a search keyword and location. The scraper requires a residential US proxy to bypass Cloudflare checks, which is pre-configured in the input schema. You must pass an explicit input payload because the console's prefill values are ignored by API calls; only schema defaults are applied automatically.
The input schema allows you to query using search keywords and location shortcodes, or target specific curated URLs directly. Because ZipRecruiter blocks datacenter traffic, the input schema hardcodes a residential proxy configuration. Do not attempt to override or disable this configuration in your payload, as residential US proxies are required to bypass the platform's Cloudflare checks and fetch the detail pages.
Here is an example of a JSON input payload using direct search URLs with custom filters:
{
"startUrls": [
"https://www.ziprecruiter.com/jobs-search?search=software+engineer&location=New+York%2C+NY"
],
"maxItems": 50,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
}
If you prefer to let the Actor construct the search URL for you using internal shortcuts, you can provide keyword, location, and specific filters like employment type and distance radius. Here is an alternative payload that leverages these fields:
{
"search": "nurse",
"location": "New York, NY",
"jobType": "any",
"daysPosted": 0,
"remoteOnly": false,
"maxItems": 50,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
}
How do you trigger the Actor programmatically in Python?
Trigger the Actor asynchronously via POST to its runs endpoint, then poll for completion or use webhooks. Running the Actor synchronously is limited by a hard-cap of 300 seconds, which returns an HTTP 408 response past that limit. Gathering full job descriptions requires visiting each detail page individually, so your run will likely require more than 5 minutes.
To prevent infinite runs from consuming your platform budget, you can set the maxTotalChargeUsd query parameter on your API call. This parameter is exposed to the Actor code as the environment variable ACTOR_MAX_TOTAL_CHARGE_USD. When this limit is reached, the platform will initiate a run termination. Note that this is not an instant process, and the run will continue to consume resources briefly during the shutdown phase.
The following Python script starts an asynchronous run, polls for completion, and fetches the resulting dataset. Checked against the Actor's input schema and Apify docs on 2026-09-09.
import os
import time
import requests
APIFY_TOKEN = os.environ.get("APIFY_TOKEN")
ACTOR_ID = "crawlerbros/ziprecruiter-scraper-pro"
run_input = {
"search": "software engineer",
"location": "New York, NY",
"maxItems": 30,
"proxyConfiguration": {
"useApifyProxy": True,
"apifyProxyGroups": ["RESIDENTIAL"]
}
}
# Start the run asynchronously, setting a maximum charge cap
run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?token={APIFY_TOKEN}&maxTotalChargeUsd=5.0"
response = requests.post(run_url, json=run_input)
response.raise_for_status()
run_data = response.json()["data"]
run_id = run_data["id"]
default_dataset_id = run_data["defaultDatasetId"]
print(f"Run started. ID: {run_id}. Polling for completion...")
# Poll the run status until finished
status_url = f"https://api.apify.com/v2/actor-runs/{run_id}?token={APIFY_TOKEN}"
while True:
status_response = requests.get(status_url)
status_response.raise_for_status()
status_data = status_response.json()["data"]
status = status_data["status"]
if status == "SUCCEEDED":
print("Run completed successfully.")
break
elif status in ["FAILED", "ABORTED", "TIMED-OUT"]:
raise RuntimeError(f"Actor run failed with status: {status}")
time.sleep(15)
# Fetch the items from the default dataset
items_url = f"https://api.apify.com/v2/datasets/{default_dataset_id}/items?token={APIFY_TOKEN}"
items_response = requests.get(items_url)
items_response.raise_for_status()
jobs = items_response.json()
print(f"Retrieved {len(jobs)} job listings.")
What is the expected output structure?
The Actor outputs individual JSON records containing the unique job card ID, parsed salary ranges, location details, and the full job description. Duplicate job cards are resolved by the Actor internally, ensuring each unique job listing is emitted only once. Missing fields are omitted entirely from the output object rather than returned as null values.
Each output record contains parsed metadata, location flags, salary ranges, and the full job description text if the detail page was successfully fetched. Fields that do not have source data are omitted from the payload rather than being returned as null or zero, which prevents you from storing empty or misleading values in your database.
Here is an example of a successfully parsed output record returned by the dataset endpoint:
{
"id": "abc123_unique_card_id",
"jid": "f4a8b2c1d",
"inputUrl": "https://www.ziprecruiter.com/jobs-search?search=software+engineer&location=New+York%2C+NY",
"url": "https://www.ziprecruiter.com/c/Acme-Corp/Job/Senior-Software-Engineer",
"title": "Senior Software Engineer",
"company": "Acme Corp",
"companyUrl": "https://www.ziprecruiter.com/c/Acme-Corp",
"location": "New York, NY (Hybrid)",
"city": "New York",
"state": "NY",
"isRemote": false,
"isHybrid": true,
"salary": "$130K - $160K/yr",
"salaryMin": 130000,
"salaryMax": 160000,
"salaryPeriod": "year",
"description": "We are looking for a senior software engineer to join our team and build scalable microservices.",
"scrapedAt": "2026-09-09T12:34:56Z"
}
How can we write a robust, duplicate-safe pandas ingestion pipeline?
To run this pipeline daily, you must track previously scraped jobs. If you simply append every run's output to your database, you will end up with multiple copies of the same job postings. We must build an ingestion step that checks for existing records using a unique identifier and updates or discards records accordingly.
In our schema, each job has two unique identifiers: id (the ZipRecruiter card token) and jid (the parsed short ID). We will use id as our primary key. Our pipeline must load the historical dataset, filter out incoming jobs that we have already stored, and append the new records.
This Python example uses pandas to implement a stateful local storage database. It manages missing fields cleanly using pandas' native handle-empty methods and saves only unique new listings:
import os
import json
import pandas as pd
DB_FILE = "historical_jobs.parquet"
def ingest_new_jobs(raw_dataset_json):
# Parse the incoming JSON data
new_records = json.loads(raw_dataset_json)
if not new_records:
print("No new records to process.")
return
# Convert to DataFrame
df_new = pd.DataFrame(new_records)
# Ensure primary key exists in incoming records
if "id" not in df_new.columns:
print("Error: Input data lacks primary 'id' keys.")
return
# Drop any internal duplicates within the current run batch
df_new = df_new.drop_duplicates(subset=["id"], keep="first")
# Load historical database if it exists
if os.path.exists(DB_FILE):
df_old = pd.read_parquet(DB_FILE)
print(f"Loaded {len(df_old)} existing records from database.")
# Filter out records that are already present in the database
df_new_unique = df_new[~df_new["id"].isin(df_old["id"])]
else:
print("No historical database found. Creating a new one.")
df_old = pd.DataFrame()
df_new_unique = df_new
if not df_new_unique.empty:
print(f"Found {len(df_new_unique)} new unique job listings to ingest.")
# Combine old and new records
df_combined = pd.concat([df_old, df_new_unique], ignore_index=True)
# Save updated database
df_combined.to_parquet(DB_FILE, index=False)
print(f"Database updated. Total records: {len(df_combined)}")
else:
print("All scraped jobs already exist in the database. No updates performed.")
# Mock incoming data containing one new job and one duplicate
mock_payload = """
[
{
"id": "abc123_unique_card_id",
"jid": "f4a8b2c1d",
"title": "Senior Software Engineer",
"company": "Acme Corp",
"scrapedAt": "2026-09-09T12:34:56Z"
},
{
"id": "xyz987_unique_card_id",
"jid": "a1b2c3d4e",
"title": "Data Engineer",
"company": "Beta Systems",
"scrapedAt": "2026-09-09T12:35:10Z"
}
]
"""
# Run the ingestion twice to demonstrate state retention and duplicate blocking
print("--- First Run ---")
ingest_new_jobs(mock_payload)
print("\n--- Second Run (Simulating duplicate execution) ---")
ingest_new_jobs(mock_payload)
How do you route the output to n8n for downstream automation?
If you want to send scraped jobs directly to Slack, an internal email newsletter, or an external API, you can route the results using webhooks to n8n. Apify platform webhooks support exactly one action: sending an HTTP POST request to a target URL upon run completion.
When setting up your workflow, n8n provides an official Apify node. This node includes a Trigger option that listens for run completions, removing any need to write manual polling loops in your workflow engine. If you are running self-hosted n8n, you can authenticate using your Apify API key. If you are using n8n Cloud, you can use standard OAuth2 credentials.
Here is an example of an n8n webhook workflow definition. This JSON payload can be copied directly into the n8n canvas. It listens for the completed run, fetches the dataset items, and routes them to a subsequent node:
{
"nodes": [
{
"parameters": {
"httpMethod": "POST",
"path": "apify-job-ingest",
"options": {}
},
"type": "n8n-nodes-base.webhook",
"typeVersion": 1,
"position": [250, 300],
"id": "webhook-trigger",
"name": "Apify Webhook Trigger"
},
{
"parameters": {
"url": "={{$json.body.resource.defaultDatasetId}}",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Authorization",
"value": "=Bearer Your_Apify_API_Token_Here"
}
]
},
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4,
"position": [470, 300],
"id": "fetch-dataset-items",
"name": "Fetch Dataset Items"
}
],
"connections": {
"Apify Webhook Trigger": {
"main": [
[
{
"node": "Fetch Dataset Items",
"type": "main",
"index": 0
}
]
]
}
}
}
What are the real limitations and failure modes of this scraper?
This scraper is designed to handle standard search and extraction patterns, but several technical constraints and platform edge cases can disrupt execution. Specifically, residential proxy sessions can die after roughly 30 minutes, request queues cannot be shared across multiple parallel runs, and Cloudflare blocking can cause early exits with sentinel records.
First, residential proxies are prone to session loss. While datacenter proxy sessions can persist for 26 hours, residential proxy sessions typically expire and die after about 30 minutes. If you run large scrapes with high maxItems values, session disconnects will occur mid-run. Your pipeline must be built to accept partial datasets when a connection drops, rather than discarding the entire container's progress.
Second, the scraper utilizes storage request queues. On the Apify platform, a request queue can only be processed by one Actor run or task run at a time. If you design a fan-out architecture trying to run multiple parallel scraping runs using a shared, global request queue, the runs will block each other or error out. Each run must operate on its own independent request queue.
Third, if Cloudflare completely blocks all scraper sessions during execution, the Actor is designed to exit gracefully with code 0. Rather than throwing an error, it emits a sentinel record into your dataset of this shape:
{
"type": "job_ziprecruiter_blocked",
"message": "All sessions blocked. Exiting run to save resources."
}
If your downstream data processing code assumes that every record matches the job schema, this sentinel record will crash your ingestion loop. You must explicitly look for this sentinel block and halt downstream ingestion when it appears:
def validate_and_filter_sentinels(records):
clean_records = []
for r in records:
# Detect the sentinel block indicating a platform-wide session block
if r.get("type") == "job_ziprecruiter_blocked":
print("Warning: Run terminated early due to Cloudflare block.")
continue
# Verify that the record is an actual job card
if "id" in r and "title" in r:
clean_records.append(r)
return clean_records
Finally, be aware of storage retention limits. If you do not name your datasets, they are created as unnamed storages. On the Apify Free plan, unnamed storages expire. The platform only retains the 10 most recent runs, and they are deleted after 4 months. To prevent loss of history, your pipeline must write to named datasets, or you must run the Actor on a paid tier where retention policies are longer.
How do you estimate and calculate execution costs?
Platform costs are calculated using three variables: compute unit usage, platform pricing tier rates, and residential proxy bandwidth consumption. Compute units (CU) are calculated based on memory allocation and execution time using the formula CU = (allocated_memory_mb / 1024) * run_duration_hours.
If you double the memory allocation of your run, the CU consumption remains neutral only for runs that leverage platform autoscaling. However, autoscaling only applies to solutions running multiple tasks or processing URLs for at least 30 seconds each.
Your base compute cost depends on your active subscription tier. The cost per compute unit is structured as follows:
- Free Plan: $0.20 per CU
- Starter Plan: $0.20 per CU
- Scale Plan: $0.16 per CU
- Business Plan: $0.13 per CU
Residential proxies are billed based on data throughput. Datacenter proxies are included in standard plans, but because this scraper requires residential US proxies to bypass Cloudflare, you are charged per gigabyte transferred. The cost scales based on your subscription tier:
- Free and Starter Plans: $8 per GB
- Scale Plan: $7.50 per GB
- Business Plan: $7 per GB
Because the Actor must fetch individual detail pages to parse the full description text, the total bandwidth consumed scales linearly with the number of jobs scraped. For example, scraping 500 jobs will consume significantly more bandwidth than scraping 50 search results, since the Actor makes 500 separate residential connections to individual job detail pages. You can control this consumption directly using the maxItems input parameter.
How do you schedule runs and maintain pipeline state?
To set up a recurring daily import, you can schedule the Actor using Apify's cron-based schedules. Schedules support a 6-field cron syntax (where seconds are optional), with a minimum allowed running interval of 10 seconds.
There are two critical platform rules to remember when setting up a schedule. First, the Actor must have been executed successfully at least once before you can attach it to a schedule. Second, all newly created schedules are set to a DISABLED state by default. After creating your schedule, you must make a programmatic call or use the UI console to explicitly enable it.
Here is an example payload to create a daily cron schedule using the Apify API:
{
"name": "daily-ziprecruiter-scrape",
"cronExpression": "0 0 12 * * *",
"isEnabled": true,
"actorId": "crawlerbros/ziprecruiter-scraper-pro",
"runInput": {
"search": "data engineer",
"location": "Dallas, TX",
"maxItems": 100,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": ["RESIDENTIAL"]
}
}
}
If a scheduled run fails due to network degradation or proxy session loss, you do not need to start a fresh run from scratch. The platform allows you to resurrect a run. Resurrecting a run restarts the original container using the same storage system (preserving your request queues), excludes the downtime from your running time calculations, and restarts the timeout clock.
If you need to stop a running container, you can issue an abort command. The platform grants a graceful abort window of 30 seconds, during which the Actor can finalize its database transactions, emit final dataset records, and exit cleanly without data loss.
The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com
Top comments (0)