Building Job Data Pipelines with Indeed Jobs Scraper

Data pipelines that ingest job postings frequently break on subtle schema divergences rather than catastrophic network failures. Extracting Indeed listings programmatically exposes several architectural edge cases: fields that disappear entirely when null, silent fallbacks to truncated search snippets, and HTTP timeouts caused by downstream enrichments. Checked against the Actor's input schema and Apify docs on 2026-09-09.

Using the Indeed Jobs Scraper as a production example, this guide walks through the input constraints, execution lifecycle pitfalls, and downstream verification routines necessary to ingest job data defensively.

Why do some scraped job descriptions truncate unexpectedly?

Job descriptions truncate because Indeed applies stricter anti-bot challenges to job detail pages than search results, forcing the scraper to fallback to the shorter search-result snippet. The scraper always attempts to visit the full detail page, but if the request is blocked, it populates the description field with the snippet to ensure the run continues. This ensures a result is returned even when the complete text is inaccessible.

Because both the snippet and the full description use the same description field, your logic must detect quality issues based on content heuristics. If an indexing pipeline requires the full text for keyword matching, a short snippet will slip through silently unless checked.

def classify_description_quality(record: dict) -> str:
    """Detect whether description is a search snippet or a full posting."""
    desc = record.get("description", "")

    # Check for empty string or missing key
    if not desc:
        return "missing"

    # Heuristic: search snippets are often under 350 chars or end in ellipses
    # This detects when the scraper falls back to the search result text.
    if len(desc) < 350 or desc.endswith("...") or desc.endswith("…"):
        return "snippet_fallback"

    return "full_detail"

How does the output schema handle missing fields?

The scraper omits missing fields from the JSON payload entirely rather than returning them as null or empty strings. You must check for key existence before processing fields like salary, company ratings, or external links, as only seven core fields are guaranteed. The fields positionName, company, location, url, id, postedAt, and scrapedAt are available for every listing.

If you request company details or external apply redirect links, but the underlying posting does not contain them, those keys will not exist on that specific output object. The following example shows how to enforce a strict data contract using Pydantic to safely handle these omissions.

from typing import Optional
from pydantic import BaseModel, HttpUrl

class IndeedJobListing(BaseModel):
    # Fields available for every listing according to documentation
    id: str
    positionName: str
    company: str
    location: str
    url: HttpUrl
    postedAt: str
    scrapedAt: str

    # Optional keys that omit completely when absent from the source
    salary: Optional[str] = None
    jobType: Optional[str] = None
    rating: Optional[float] = None
    reviewsCount: Optional[int] = None
    description: Optional[str] = None
    isExpired: Optional[bool] = False

def parse_incoming_record(raw_record: dict) -> IndeedJobListing:
    # Model validation coerces omitted keys to None based on the schema
    return IndeedJobListing.model_validate(raw_record)

Why do synchronous API runs return HTTP 408?

Synchronous API runs return an HTTP 408 error because the Apify platform hard-caps client connections at 300 seconds. If the scraping task takes longer than five minutes, the connection is severed even though the Actor continues to process the data in the background. To avoid this, you must initiate runs asynchronously and either poll for results or use a webhook.

When parseCompanyDetails or followApplyRedirects are enabled, the scraper performs additional HTTP requests for every job. This significantly increases the total duration. Using the Apify Python Client, you can manage these longer runs by calling the Actor without waiting for the synchronous connection to close.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# Start the Actor asynchronously to bypass the 300s sync cap
# This allows the run to continue past the 5-minute timeout limit.
run = client.actor("crawlerbros/indeed-jobs-scraper").start(
    run_input={
        "position": "data engineer",
        "location": "remote",
        "maxItemsPerSearch": 100,
        "parseCompanyDetails": True
    }
)

print(f"Run started: {run['id']}")
# You can now use run['id'] to poll status or run['defaultDatasetId'] to get data

Configuring searches across different countries

The Indeed Jobs Scraper supports direct URLs from any of the 60 plus supported Indeed domains regardless of the global country setting. Each URL in the startUrls array is processed using the specific domain found within that URL, such as using the UK domain for a link starting with uk.indeed.com. This allows you to mix regional searches within a single execution task.

When using startUrls alongside the position and location parameters, the Actor scrapes both sets of inputs independently. This is useful for monitoring specific filtered views from various geographic regions simultaneously.

{
    "position": "marketing manager",
    "location": "London",
    "country": "GB",
    "startUrls": [
        { "url": "https://www.indeed.com/jobs?q=engineer&l=New+York" }
    ],
    "maxItemsPerSearch": 50,
    "saveOnlyUniqueItems": true
}

Defensive Scheduling and Data Retention

Platform storage limits and scheduling rules dictate how you should manage long-term job data. On the Free plan, only the 10 most recent runs are retained for 4 months, after which unnamed datasets expire. To maintain a historical record, you should target a named dataset which is exempt from deletion. Additionally, the platform requires the Actor to have run at least once before it can be assigned to a schedule.

New schedules are created in a DISABLED state by default. If you are building a daily ingestion pipeline, you must explicitly enable the schedule after creation and ensure you have at least one manual run in the history.

# Example of triggering a manual run to satisfy the scheduling requirement
# This is necessary because new schedules cannot be created for never-run Actors.
curl -X POST "https://api.apify.com/v2/acts/crawlerbros~indeed-jobs-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"position": "devops", "location": "remote", "maxItemsPerSearch": 10}'

How do input schema defaults affect API calls?

Fields configured with prefill in the Actor's input schema are only visible in the Console UI and are not applied to direct API calls. Only fields with a default value are automatically applied if they are omitted from your JSON payload. When building a Python integration, you should pass an explicit input dictionary for all critical parameters to prevent the Actor from reverting to unintended defaults.

For instance, the maxItemsPerSearch field has a default of 100. If your script omits it, the platform will use that value even if you previously configured a different number in the Apify Console UI.

{
    "position": "software engineer",
    "location": "San Francisco",
    "country": "US",
    "maxItemsPerSearch": 500,
    "saveOnlyUniqueItems": true,
    "parseCompanyDetails": false
}

Architectural tradeoffs between speed and detail

Choosing whether to enable parseCompanyDetails or followApplyRedirects involves a direct tradeoff between data richness and execution reliability. Fetching company size and industry overview requires the scraper to navigate to a separate company profile page for every unique employer found. This increases the total request count per run, which raises the probability of encountering anti-bot challenges.

Similarly, resolving the externalApplyLink requires following Indeed's internal redirect links. If the employer uses a complex applicant tracking system, the scraper may only capture an intermediary URL. For high-volume operations, it is often more efficient to run a fast search-only pass and then perform a secondary enrichment run only for specific jobs that meet your criteria.

Real limitations and caveats

The scraper is bound by Indeed's public search limitations, which typically cap the number of visible listings for any single query between 1,000 and 10,000 items. If a search for a broad term shows hundreds of thousands of results, the scraper can only access the portion Indeed allows through its pagination. Furthermore, the scraper cannot guarantee the presence of optional fields like salary or rating as these depend on employer input.

While the Actor handles most bot detection, Indeed's security strength varies. During periods of high defensive activity, the frequency of snippet fallbacks for job descriptions may increase. Additionally, a request queue can only be processed by one Actor or task run at a time. This means you cannot fan out a single crawl across multiple concurrent runs using the same shared queue.

Understanding infrastructure costs and scaling

Cost on the Apify platform is measured in Compute Units (CUs), calculated using the formula CU = (memory_mb / 1024) * duration_hours. The price per CU depends on your subscription tier: $0.20 for Free and Starter, $0.16 for Scale, and $0.13 for Business. Memory is allocated at the start of the run, and doubling memory is CU-neutral only for autoscaling runs.

Proxy costs are a separate line item. Residential proxies, which are recommended for job detail pages, are billed by bandwidth. On Free and Starter plans, the cost is $8 per GB. This scales down to $7.50 per GB for the Scale plan and $7 per GB for the Business plan. To manage potential overruns, you can pass the maxTotalChargeUsd parameter to your run request. This parameter provides a cost ceiling that terminates the run if the threshold is reached.

Managing proxy sessions and high volume

Residential proxy sessions on the platform are designed to persist for approximately 30 minutes. If your scraping task is configured to process thousands of listings with full enrichment, the session might expire before the run finishes. This causes the scraper to rotate to a new IP address, which may trigger Indeed to present a new anti-bot challenge or captcha.

To process high volumes of data efficiently, split your work into regional tasks with their own queues. This keeps individual runs within the 30-minute residential session window and prevents resource contention.

# Split runs to stay within residential proxy session windows
# Each call creates a separate run with its own unique request queue.
search_regions = ["New York", "Chicago", "Los Angeles"]

for city in search_regions:
    client.actor("crawlerbros/indeed-jobs-scraper").start(
        run_input={
            "position": "nurse",
            "location": city,
            "maxItemsPerSearch": 200
        }
    )

How can I verify that all requested items were scraped?

The number of items returned depends on the search results available on the source site. If a niche position in a small town only has five listings, the scraper will only return those five, even if your maxItemsPerSearch is set much higher. You can verify the success of a run by inspecting the itemCount in the dataset and checking the Actor logs for any warnings related to anti-bot blocks.

To perform a programmatic check, compare the itemCount against your internal requirements and inspect the isExpired boolean field to see if listings are becoming unavailable during the crawl.

# Check dataset results to verify ingestion volume
# If itemCount is less than maxItemsPerSearch, the source lacked more results.
dataset_info = client.dataset(run["defaultDatasetId"]).get()
if dataset_info["itemCount"] == 0:
    print("Warning: No jobs found for this search criteria.")

Troubleshooting search results and bot detection

If your runs consistently yield fewer results than expected, it may be due to regional blocks or search filtering. Indeed's search behavior changes based on the country setting and the provided keywords. Using the country-US_XX residential proxy configuration can help with localized searches by targeting IPs in specific US states.

When the scraper encounters a captcha or an anti-bot wall, it attempts to resolve it automatically. However, if you are using datacenter proxies, these attempts are more likely to fail. Switching to residential proxies and ensuring the saveOnlyUniqueItems flag is true helps maintain a clean dataset even if you have to restart or retry a run.

Managing enrichment for company data

Fetching company-specific data like size and industry overview adds overhead to every job listing. To optimize for cost and speed, only set parseCompanyDetails to true when that data is essential for your application. If you only need the job title and location, leaving this setting at its default false value will significantly reduce the runtime and the number of proxy requests.

This scraper provides a balance between detail and performance. By understanding how the platform handles storage retention, proxy lifespans, and the 300-second synchronous cap, you can build a more resilient job data pipeline. Always use explicit inputs in your API calls to avoid issues with schema defaults and ensure your code is prepared to handle the omission of optional fields.

The Actor's README is the source of truth for its inputs, outputs and limits. For developers requiring a headless solution for labor market analysis, this tool offers a robust way to extract structured data from one of the world's largest job boards. Using the Apify platform's scheduling and storage features, you can transform the scraper into a persistent data asset for your organization.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]