Why Wellfound Scraper Runs Drop Items After 300 Seconds

Data engineering pipelines often start with a simple task: fetch data from a source. But as soon as you move beyond local scripts, the real work begins: orchestration, state management, error handling, and making sure your data is fresh without blowing your budget.

Using a pre-built Actor like the Wellfound Jobs Scraper (wellfound-scraper) on Apify is a common starting point for job market analysis or competitive intelligence. It extracts job IDs, titles, compensation, remote status, location, company name, and logo from Wellfound.com without requiring login. But simply running the Actor isn't a pipeline. The challenge lies in wiring it into a downstream process: whether that's a webhook to n8n, a direct load into pandas, or staging data in a warehouse, while managing state to prevent double-processing on re-runs.

This article focuses on the integration, not the scraping itself. We'll explore how to handle real-world challenges like synchronous run limitations, proxy session persistence, and proper cost management when building production-ready data flows with wellfound-scraper.

How Does the Apify Synchronous Run Endpoint Impact Data Ingestion?

The Apify synchronous run endpoint imposes a hard limit of 300 seconds (5 minutes) before timing out and returning an HTTP 408 error. If your wellfound-scraper run is expected to take longer, you must use the asynchronous API by POSTing to /v2/acts/<actor>/runs and then polling for completion or setting up a webhook. Failing to account for this limit means losing any data the Actor generated after the 5-minute mark, resulting in incomplete datasets and downstream pipelines that appear to drop records.

The wellfound-scraper Actor provides several input fields that directly influence run duration, primarily maxItems. The default maxItems is 50, but it can be set as high as 500. While 500 items might complete within the 5-minute synchronous window under ideal conditions, requesting a large number of jobs, especially with complex filters or if the Actor needs to escalate to residential proxies for deep-filter URLs, can push the execution time beyond this threshold.

Here's an example of running wellfound-scraper synchronously with a low maxItems value. This is fine for quick tests or small datasets, but not for comprehensive pipeline steps.

import os
from apify_client import ApifyClient

# Initialize the ApifyClient with your API token
apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# Prepare the Actor input for a quick, synchronous run
actor_input = {
    "startUrls": ["https://wellfound.com/jobs?remote=true&role=engineering"],
    "remoteOnly": True,
    "jobTitle": "Data Engineer",
    "maxItems": 50,
}

print("Starting synchronous wellfound-scraper run...")
# Run the Actor and wait for it to finish (up to 300 seconds)
# The run() method defaults to synchronous if not explicitly set to wait=False
run = apify_client.actor("crawlerbros/wellfound-scraper").call(
    run_input=actor_input
)

print(f"Run finished with status: {run['status']}")
print(f"Collected {run['output']['dataset']['itemCount']} items.")

# Fetch and print the results
print("Fetching dataset items...")
for item in apify_client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

For longer runs, you must switch to an asynchronous approach. This involves initiating the run and then either polling the run status periodically or configuring a webhook to notify your downstream system when the run completes. Polling adds latency and overhead, so webhooks are generally preferred for production pipelines.

import os
import time
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_input = {
    "startUrls": ["https://wellfound.com/jobs"],
    "keyword": "machine learning engineer",
    "remoteOnly": True,
    "maxItems": 400,
    "experience": "mid",
}

print("Starting asynchronous wellfound-scraper run...")
# Start the Actor without waiting for it to finish
run = apify_client.actor("crawlerbros/wellfound-scraper").call(
    run_input=actor_input,
    wait_for_finish=False # Crucial for asynchronous execution
)

run_id = run["id"]
print(f"Actor run started with ID: {run_id}. Polling for completion...")

# Polling loop (in a real pipeline, you'd use webhooks or a more robust scheduler)
while True:
    run_status = apify_client.run(run_id).get()
    print(f"Current run status: {run_status['status']}")
    if run_status["status"] in ["SUCCEEDED", "FAILED", "ABORTED"]:
        break
    time.sleep(30) # Wait 30 seconds before polling again

print(f"Run finished with status: {run_status['status']}")
if run_status["status"] == "SUCCEEDED":
    print(f"Collected {run_status['output']['dataset']['itemCount']} items.")
    # Process results here
    # Example: apify_client.dataset(run_status["defaultDatasetId"]).iterate_items()

How Does Proxy Session Persistence Affect Scrape Reliability?

The wellfound-scraper automatically handles proxy selection, using residential proxies for DataDome-protected deep-filter URLs (e.g., /role/l/) and direct connections for others. While this simplifies input configuration, it introduces a reliance on proxy session persistence. Datacenter proxies typically persist for 26 hours, but residential proxies, which are used for more complex filtering, only persist for approximately 30 minutes.

This 30-minute residential proxy session limit means that if your wellfound-scraper run, especially one configured to hit DataDome-protected URLs, extends significantly beyond this duration, it might encounter proxy session invalidation mid-run. This can lead to increased retries, slower performance, and potentially missed data as the Actor attempts to acquire new residential proxy sessions or encounters temporary blocks.

To mitigate this, for runs expected to be long and use deep-filter URLs, it's often better to break them into smaller, more manageable runs, each completing within the 30-minute window. This can be achieved by carefully segmenting startUrls or using the maxItems parameter across multiple Actor calls. For example, instead of a single run for all job titles, you might run the Actor once per job title, each within its own short session.

When constructing startUrls, be aware that URLs like https://wellfound.com/jobs (default) are generally stable, but deep-filter paths like https://wellfound.com/role/l/data-engineer/san-francisco will trigger residential proxy usage.

{
  "startUrls": [
    "https://wellfound.com/jobs?remote=true",
    "https://wellfound.com/role/l/software-engineer/london",
    "https://wellfound.com/role/l/product-manager/new-york"
  ],
  "remoteOnly": false,
  "jobTitle": "engineer",
  "maxItems": 50
}

This input, especially with multiple deep-filter startUrls, could push a run's duration beyond the residential proxy session limit, leading to diminished reliability. If you need to scrape a large set of deep-filtered roles, consider orchestrating multiple independent runs, each focusing on a subset of the roles/locations.

What's the Cheapest Way to Filter Wellfound Jobs by Salary Range?

Using minSalary and maxSalary in the Actor input is the cheapest way to filter Wellfound jobs by salary range, as you only pay for results that match. This avoids paying for all jobs and then filtering them downstream.

The wellfound-scraper offers client-side filtering via minSalary and maxSalary input parameters. Using these filters is significantly cheaper than retrieving all jobs and then filtering them downstream in your own application or warehouse. This is because the Actor's pricing model is "PAY_PER_EVENT," with the primary charge event being "result" at $0.002 per item in the default dataset. By filtering at the source, you pay only for the items that match your criteria, rather than paying for every job listing found before filtering.

For example, if you need jobs with a minimum salary, setting minSalary in the Actor input will prevent results below that threshold from ever be emitted. This directly reduces the number of "result" events you're charged for. Conversely, fetching all jobs and then filtering them in pandas or a data warehouse would incur the full cost for every scraped item, plus any downstream processing costs. Always check the input schema's default values, as includeNoSalary defaults to true.

{
  "startUrls": [
    "https://wellfound.com/jobs"
  ],
  "remoteOnly": true,
  "jobTitle": "Fullstack Developer",
  "maxSalary": 0,
  "includeNoSalary": false,
  "maxItems": 50
}

In this example, only remote Fullstack Developer jobs with salaries would be returned, minimizing the number of "result" events. Omitting includeNoSalary: false would mean you'd still pay for jobs that don't specify a salary, even if they might not meet your implicit compensation criteria.

How to Orchestrate Wellfound Data for Downstream Systems?

Once wellfound-scraper completes, the scraped data resides in a default dataset. For robust pipelines, you need to pull this data and push it into your own systems. This could be a webhook, direct API calls to dump to S3, or an integration with a tool like n8n.

Apify offers webhooks that fire on run completion. These webhooks POST a JSON payload to a specified URL, which can be an n8n workflow, an AWS Lambda endpoint, or any other HTTP receiver. This event-driven approach is superior to polling for run status, reducing latency and resource consumption.

For example, an n8n workflow can be triggered by an Apify webhook. The webhook payload will contain the defaultDatasetId from the completed run. Your n8n workflow can then use the Apify node to fetch items from that dataset and subsequently load them into a database, a spreadsheet, or another API.

{
  "eventTypes": [
    "ACTOR_RUN_SUCCEEDED",
    "ACTOR_RUN_FAILED"
  ],
  "requestUrl": "https://your-n8n-instance/webhook-path",
  "headers": {
    "Content-Type": "application/json",
    "Authorization": "Bearer your-n8n-api-key"
  },
  "payloadTemplate": "{\"runId\": {{run.id}}, \"datasetId\": {{run.defaultDatasetId}}, \"status\": \"{{run.status}}\", \"itemCount\": {{run.output.dataset.itemCount}}}"
}

This webhook configuration sends a concise payload, including the dataset ID, to your n8n workflow upon run success or failure. The n8n workflow would then look something like this:

1. Webhook Trigger Node: Receives the POST request.
2. Apify Node: Uses the datasetId from the webhook payload to Get All Dataset Items.
3. Data Processing Node: (e.g., Function node, Split In Batches) to clean, transform, or enrich the data.
4. Database/Service Node: (e.g., PostgreSQL, Google Sheets, HTTP Request) to store the processed job listings.

How to Avoid Duplicate Job Listings in Downstream Systems?

One critical detail for state management: wellfound-scraper does not track jobs it has previously scraped. If you run it on a schedule, you will likely get duplicate entries. To prevent double-processing, your downstream system must implement deduplication. The jobId field in the Actor's output is ideal for this. It's a stable, unique identifier for each job listing.

Here's a snippet of the typical output shape:

[
  {
    "type": "job",
    "jobId": "2498565",
    "title": "Senior Data Engineer",
    "slug": "senior-data-engineer-at-acme-corp",
    "jobUrl": "https://wellfound.com/jobs/2498565-senior-data-engineer",
    "compensation": "$150k – $180k",
    "remote": true,
    "locations": [
      "Remote"
    ],
    "companyId": "24503",
    "companyName": "Acme Corp",
    "companySlug": "acme-corp",
    "companyUrl": "https://wellfound.com/company/acme-corp",
    "companyLogo": "https://assets.wellfound.com/...",
    "postedAt": "2024-09-17T14:30:00.000Z",
    "scrapedAt": "2026-09-18T10:00:00.000Z"
  }
]

When loading these records into a database, you'd typically define jobId as a unique primary key or use an UPSERT (INSERT OR UPDATE) strategy to prevent duplicates.

When Should You Use Named Storages and When Do They Expire?

Apify manages data storage for Actor runs, but the default behavior for unnamed storages is ephemeral. On the Free plan, only the 10 most recent runs are retained, and they expire after 4 months. For wellfound-scraper, if you rely on the defaultDatasetId and are on a Free plan, your historical data will eventually disappear.

For any production pipeline that requires long-term data retention or specific data management, you must use named storages. Named datasets and key-value stores are exempt from deletion policies and are retained indefinitely until you explicitly delete them.

You can specify named storages in your Actor run input. For example, to store results in a dataset named my-wellfound-jobs:

import os
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_input = {
    "startUrls": ["https://wellfound.com/jobs?remote=true"],
    "remoteOnly": True,
    "maxItems": 50,
}

# Explicitly name the dataset
run = apify_client.actor("crawlerbros/wellfound-scraper").call(
    run_input=actor_input,
    dataset_name="my-wellfound-jobs" # Using a named dataset
)

print(f"Run finished, data stored in named dataset: {run['datasetName']}")

Using a named dataset ensures that your historical job data is always available for auditing, trend analysis, or re-processing, regardless of your Apify plan or the number of recent runs. This is crucial for building reliable historical data pipelines.

What Are the Limitations and Caveats of the Wellfound Scraper?

While wellfound-scraper is highly effective, it has specific limitations that developers must understand for robust pipeline design. The /jobs feed serves only a single page (approximately 47 jobs) without client-side pagination. This means to get more jobs from the main feed, you either need to use multiple startUrls with different filters (e.g., location, role) or employ deeper filter URLs if you are targeting specific roles and locations.

The Actor extracts high-level job metadata. If your pipeline requires detailed job descriptions, skills, or benefits, you'll need to hit individual /jobs/<id>-<slug> pages. These individual job pages are DataDome-protected, which means fetching them will also fall back to the residential proxy pool, impacting cost and potentially run duration, similar to deep-filter URLs. The current wellfound-scraper does not do this automatically. You would need to chain it with another Actor or custom code to visit these individual pages. Residential proxy sessions are relatively short-lived (approximately 30 minutes). If your scraping strategy relies heavily on deep-filter URLs and maxItems is high, runs may exceed this limit, leading to increased retries or slower performance. This isn't a hard failure but an efficiency and reliability caveat. Lastly, Apify does not offer native integrations with services like AWS S3 or Slack for run notifications or data export. For these, you must route through webhooks to an intermediary like n8n, Make, or Zapier, or implement custom logic in your application using the Apify API.

How Does Understanding the Pricing Model Save Costs When Using the Wellfound Scraper?

The wellfound-scraper Actor operates on a PAY_PER_EVENT pricing model. This means you are charged for specific named events that occur during its execution, not for compute time or a fixed subscription rate. Understanding these events and their costs is essential for managing your pipeline budget.

The charged events and their prices are:

• "result" (apify-default-dataset-item): $0.002 per event. This is the primary cost driver. Each item (job listing) pushed to the default dataset counts as one event.
  • Volume tiers apply: FREE $0.002, BRONZE $0.00167, SILVER $0.00133, GOLD $0.001, PLATINUM $0.001, DIAMOND $0.001.
• "Actor Start" (apify-actor-start): $0.005 per event, with the number of events being 1 per GB of memory allocated to the run (minimum 1 event). This is a one-time charge per run, scaling with memory usage.

From this, we can see that the cost scales directly with the number of job listings you extract. If you request maxItems: 500, and all 500 items are returned, your cost for results alone will be 500 * $0.002 (on the FREE tier), plus the "Actor Start" cost.

Input parameters like minSalary, maxSalary, remoteOnly, jobTitle, keyword, location, jobType, experience, companyCategories, includeCompanies, and excludeCompanies are client-side filters. Effectively using these filters to reduce the number of results returned directly lowers your "result" event count, and thus your overall cost. If you set maxItems: 500 but your filters only yield 50 matching jobs, you only pay for those 50 items. This reinforces the importance of using source-side filtering over fetching all data and filtering downstream.

The maxTotalChargeUsd parameter is also a crucial safety mechanism. You can pass this as a query parameter to your Actor run endpoint (which is then exposed to Actor code as ACTOR_MAX_TOTAL_CHARGE_USD). The ApifyClient conveniently allows you to pass this as a max_total_charge_usd keyword argument to call(). If the run's charges approach this limit, the run will terminate, preventing unexpected expenses. It's important to note that termination is not instantaneous, and some resources may still be consumed briefly after the cap is tripped.

import os
from apify_client import ApifyClient

apify_client = ApifyClient(os.environ["APIFY_API_TOKEN"])

actor_input = {
    "startUrls": ["https://wellfound.com/jobs"],
    "keyword": "frontend developer",
    "maxItems": 50 # Potentially expensive if all items match
}

# Run the Actor, setting a maximum charge limit via a run endpoint parameter.
# For ApifyClient, this is typically passed as a keyword argument to call().
run = apify_client.actor("crawlerbros/wellfound-scraper").call(
    run_input=actor_input,
    max_total_charge_usd=0.50, # Set a maximum budget for the run
    wait_for_finish=False
)

print(f"Actor run started with ID: {run['id']} with a max charge of $0.50.")
# Continue with polling or webhook integration

This example ensures that even if maxItems is set high, the run will stop before exceeding your defined budget, giving you control over unexpected costs.

Checked against the Actor's input schema and Apify docs on 2026-09-18.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]