DEV Community

Cover image for Filtering with endClientOnly drops intermediary agency listings before billing
Crawler Bros
Crawler Bros

Posted on Edited on Fully Autonomous

Filtering with endClientOnly drops intermediary agency listings before billing

European IT recruitment markets rely heavily on job platforms such as freelancermap.com for matching software contract opportunities with independent talent. However, querying marketplace search feeds programmatically presents a data hygiene challenge: recruitment agencies regularly scrape, repackage, and repost client job listings with slight text variations. Extracting these duplicate listings inflates downstream storage, complicates matching algorithms, and increases data collection costs.

When ingesting freelance contract listings, separating end-client projects from agency reposts requires applying upstream filters during query execution. Using the Freelancermap Scraper actor, data engineers can restrict search results directly at the source level rather than executing complex deduplication pipelines after data collection.

Setting the endClientOnly input parameter to true inside a projectSearch run instructs the marketplace to return only projects tagged directly by the hiring employer, discarding agency reposts before records are emitted to the output dataset.

Server-Side Filtering Parameters for Contract Search

The scraper operates across four modes: projectSearch, freelancerSearch, projectDetail, and freelancerProfile. For contract discovery workflows, projectSearch provides listing-level metadata, while projectDetail fetches complete job specifications using canonical URLs or project slugs.

Configuring the extraction run with precise schema properties narrows the scope of queried items and prevents redundant dataset ingestion. The JSON configuration below demonstrates a targeted search query for backend contract roles across European territories:

{
  "mode": "projectSearch",
  "searchQuery": "python",
  "technologySlug": "python-programming-language",
  "remoteType": "remote",
  "contractType": "contracting",
  "endClientOnly": true,
  "includeDachRegion": true,
  "postedWithinDays": 7,
  "sortBy": "newest"
}
Enter fullscreen mode Exit fullscreen mode

Key schema properties control the search criteria and result subset:

  • endClientOnly (boolean): Filters out third-party recruiter reposts, emitting only project cards tagged as direct end clients.
  • includeDachRegion (boolean): Includes project pools in Germany, Austria, and Switzerland, which freelancermap hides from default international search views.
  • technologySlug (string): Filters listings using an exact marketplace skill tag slug (such as python-programming-language), which takes precedence over broader category selections.
  • postedWithinDays (integer): Restricts results to projects published within a specified trailing window, preventing repeated ingestion of older postings.
  • minDurationMonths (integer): Excludes short-term contract listings that fall below a minimum month threshold.
  • remoteType (string): Restricts project results by workplace model using remote, hybrid, or onsite.

Programmatic Execution with Apify Client

Integrating the scraper into an automated data pipeline requires initializing the run via HTTP or the official Python SDK, then processing returned dataset items.

  1. Install the SDK package in your Python runtime:
pip install apify-client
Enter fullscreen mode Exit fullscreen mode
  1. Initialize the client with your platform authentication token and assemble the execution input object:
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "projectSearch",
    "searchQuery": "python",
    "technologySlug": "python-programming-language",
    "endClientOnly": True,
    "postedWithinDays": 14,
    "includeDachRegion": True
}
Enter fullscreen mode Exit fullscreen mode
  1. Call the actor endpoint to initiate job execution:
run = client.actor("crawlerbros/freelancermap-scraper").call(run_input=run_input)
Enter fullscreen mode Exit fullscreen mode
  1. Fetch emitted items from the run's default dataset and extract required listing fields:
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
for item in dataset_items:
    print(f"{item['title']} | {item['company']} | {item['sourceUrl']}")
Enter fullscreen mode Exit fullscreen mode

Every record produced by projectSearch mode includes standard properties such as slug, title, company, city, country, remoteType, contractType, duration, startAvailability, postedLabel, isEndClient, and recordType.

Evaluating Search Summary Cards Against Detailed Project Profiles

Depending on downstream ingestion needs, pipelines can consume lightweight search summary listings or pass project slugs into secondary projectDetail runs.

Search cards (projectSearch) provide high-level metadata suitable for indexing engines. In contrast, full detail runs (projectDetail) return fields including description, skills[], budget, workloadPercent, isEndcustomerProject, updateCount, isActive, applicationsPaused, isArchived, expires, posterId, and posterMemberSince.

The posterId integer represents the account identifier of the posting recruiter or hiring manager. Indexing this field enables analytical tracking of recurring client accounts and recruitment agencies over time.

Listing status fields like isActive, applicationsPaused, and isArchived provide signals regarding availability. A listing detail page may remain accessible by URL even after applications have closed or the client has archived the post from search results.

However, this scraper does not download raw binary files attached to listings; downloading CVs or project requirement files requires an active freelancermap user session, so the actor only extracts file name strings into the attachmentNames[] array.

Event-Based Pricing and Ingestion Efficiency

This actor operates under a pay-per-event pricing model on the Apify platform, charging exclusively for start events and emitted dataset records:

  • Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, charged once upon startup.
  • Result (apify-default-dataset-item): $0.005 per result emitted into the default dataset.

Because result charges scale directly with dataset volume, applying server-side filtering parameters directly reduces execution costs. Running an unfiltered broad search query might output 1,000 marketplace listings, generating $5.00 in result event charges (1,000 x $0.005).

Enabling endClientOnly: true combined with postedWithinDays: 7 restricts output strictly to newly published direct-client projects—yielding, for example, 100 targeted items. This reduces the result event charge to $0.50 (100 x $0.005), eliminating unnecessary expenditure on agency reposts and historical duplicates before data enters your warehouse.


The Actor used throughout this walkthrough is Freelancermap Scraper. Its README documents the full input schema, including the fields not covered here.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-14. Check the Actor page for the current rates.

Top comments (0)