DEV Community

Cover image for How to Scrape Sortlist Agency Profiles and Budgets with Python
ReventaData
ReventaData

Posted on Originally published at apify.com

How to Scrape Sortlist Agency Profiles and Budgets with Python

#python #webscraping #apify #automation

By the end of this tutorial, you will have a working Python script that extracts public B2B digital agency directories from Sortlist into structured records: agency name, website, client rating, review counts, team size, office locations, languages, service budgets, and verified badges.

Sortlist is a directory where businesses find marketing, software development, web design, and branding agencies. Evaluating agency partners manually across multiple service and country listings is tedious.

In this guide, we use Python and the Apify Actor Sortlist Scraper (reventadata/sortlist-scraper). You will define services and countries, configure client-side filters like minimum ratings or office locations, and export the resulting agency records into clean JSON. The sample row below represents a real agency record from a capture dated 2026-10-21, parsed with the Actor's own parsing logic.

⏱️ Time estimate: About 10 minutes to install one package, configure an API token, and run the Python extraction script.

Prerequisites

To follow this tutorial, you will need the following tools:

  1. Python 3.11+ installed locally (apify-client 3.x requires Python 3.11 or newer).
  2. The Apify API client for Python installed in your environment.
  3. An Apify account with an API token, accessible from the API & Integrations settings page in Apify Console.

Install the official client using your package manager:

pip install -U "apify-client~=3.0"
Enter fullscreen mode Exit fullscreen mode

Save your API token into an environment variable so your script can read it safely:

export APIFY_API_TOKEN="your_apify_token_here"
Enter fullscreen mode Exit fullscreen mode

What you'll achieve

You will turn Sortlist directory listings into structured rows of agency data. Each row contains the agency identifier, public profile URL, official website, tagline, description, office locations, languages spoken, stated team size, client rating score, review count, service budget range, overall budget range, services offered, industry sectors, awards, and partner verification flags.

Note on Data Access: Sortlist Scraper reads the public listing pages at /s/<service>/<country>. It extracts the structured data already embedded in the listing and does not fetch individual agency profile pages or private contact details.

Example use case: Suppose you want web application agencies in Belgium with a stated team of at least 5 people and a client rating of at least 4.5. Setting services: ["web-application"], countries: ["belgium-be"], minTeamSize: 5, and minRating: 4.5 keeps only agencies that pass both thresholds, up to maxItems.

Step 1: Access Sortlist Scraper

Navigate to the Sortlist Scraper page on Apify Store.

On the Actor page, you can review the input schema, description, and output dataset fields. In this tutorial, we drive the Actor programmatically via the Python API client, so everything from configuration to data extraction happens in code.

Step 2: Configure services and countries

The Actor requires listing criteria to know which directories to browse. You provide services and countries together as slug lists.

A service slug represents an expertise category, such as web-application, branding, marketing, or software-development. A country slug consists of the English country name and its two-letter ISO code, such as belgium-be, germany-de, or france-fr.

Two parameters control the volume of data collected:

  • maxItems: the maximum number of agency rows saved across all searched listings (default is 100, prefill is 20).
  • services and countries: every service is searched across every specified country.

Here is the setup script that configures web-application in belgium-be for a maximum of 20 agencies:

import os
from apify_client import ApifyClient

# 1. Initialize ApifyClient with your API token
client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# 2. Define the input: search web application agencies in Belgium
run_input = {
    "services": ["web-application"],
    "countries": ["belgium-be"],
    "maxItems": 20,
}
Enter fullscreen mode Exit fullscreen mode

💡 Tip on Slugs: Service and country inputs are validated up front. If you provide an invalid format, the validation raises a ValueError before any network requests are sent.

Step 3: Add optional agency filters

No server-side filter for these was confirmed on the listing pages (query parameters such as ?minRating= changed nothing), so the Actor fetches the listing entries and applies post-fetch filters before pushing rows:

Parameter Type What it does
minRating number Minimum client rating (0 to 5). Drops unreviewed agencies without ratings.
minReviews integer Minimum number of client reviews required.
minTeamSize integer Minimum stated team size (headcount as an integer).
languages array Keeps agencies supporting any requested language code (en, nl, fr, de).
locationContains string Case-insensitive and accent-insensitive match on English office city text.

For example, to keep agencies with an office in Antwerp, set locationContains: "Antwerp".

Step 4: Run the Actor

Start the extraction using client.actor("reventadata/sortlist-scraper").call(run_input=run_input). This method starts the Actor run on Apify and waits for execution to complete:

# 3. Start the Actor run and wait for it to finish
print("Starting Sortlist extraction...")
run = client.actor("reventadata/sortlist-scraper").call(run_input=run_input)
if run is None or run.status != "SUCCEEDED":
    status = run.status if run else "unknown"
    raise SystemExit(f"Run did not succeed (status: {status}).")

# 4. Fetch the saved agencies from the dataset
items = client.dataset(run.default_dataset_id).list_items().items
print(f"Retrieved {len(items)} agencies.\n")
Enter fullscreen mode Exit fullscreen mode

Behind the scenes:

  • The Actor constructs the target URL (https://www.sortlist.com/s/web-application/belgium-be?page=1) and fetches the page.
  • In testing on Apify platform, direct HTTP requests without proxy succeeded reliably (25 of 25 listing pages).
  • Page 1 delivers 20 agency listings; subsequent pages deliver up to 100 agencies each until the listing concludes.
  • Deduplication ensures that agencies appearing across multiple service directories are recorded only once.

Step 5: View and download results

You can inspect the extracted dataset rows directly in Python:

for item in items:
    print(item["name"])
    print(f"  Website: {item['website']}")
    if item["rating"] is not None:
        print(f"  Rating: {item['rating']} / 5 ({item['reviewsCount']} reviews)")
    else:
        print("  Rating: No reviews yet")
    print(f"  Team size: {item['teamSize']}")
    print(f"  Offices: {', '.join(item['locations'])}")
    print("-" * 40)
Enter fullscreen mode Exit fullscreen mode

Here is one record, a sample from a capture dated 2026-10-21 from the Belgium Web Application listing parsed with the Actor's own parser (scrapedAt omitted):

[
  {
    "id": "fa2146d2-016b-4c81-9e2f-af6f1bb347ea",
    "url": "https://www.sortlist.com/agency/code-the-kiwi",
    "name": "Code The Kiwi",
    "tagline": "Maatwerk software web & mobile",
    "description": "Bij Code The Kiwi zijn we gespecialiseerd in het creëren van robuuste, schaalbare oplossingen die zijn afgestemd op uw bedrijfsbehoeften. Onze expertise strekt zich uit over de nieuwste technologieën en ontwikkel standaarden, zodat elke applicatie die we bouwen is geoptimaliseerd voor prestaties en gebruikerservaring.",
    "website": "https://codethekiwi.be",
    "logo": "https://sortlist-core-api.s3.eu-west-1.amazonaws.com/58arpu4ux2cu0n1hqvrgipw0n43d",
    "locations": [
      "Sint-Katelijne-Waver, Belgium"
    ],
    "languages": [
      "nl-BE",
      "en",
      "nl"
    ],
    "teamSize": 5,
    "registrationNumber": "1013800151",
    "registrationCountry": "BE",
    "rating": 5.0,
    "reviewsCount": 1,
    "recentRecommendedReviews": 0,
    "worksCount": 10,
    "serviceWorksCount": 10,
    "serviceBudgetMin": 3000,
    "serviceBudgetMax": 1000000,
    "budgetMin": 1000,
    "budgetMax": 1000000,
    "services": [
      {
        "name": "Web Application",
        "worksCount": 10
      },
      {
        "name": "Software Development",
        "worksCount": 10
      },
      {
        "name": "Artificial Intelligence",
        "worksCount": 3
      },
      {
        "name": "Mobile App",
        "worksCount": 10
      },
      {
        "name": "Product Management",
        "worksCount": 6
      },
      {
        "name": "Innovation",
        "worksCount": 7
      },
      {
        "name": "Ergonomy (UX/UI)",
        "worksCount": 10
      }
    ],
    "sectors": [
      "Others",
      "E-commerce",
      "Transportation",
      "Software & Computer Services",
      "Industrial Goods & Services",
      "Construction",
      "Energy & Oil",
      "Pharmaceuticals & Biotech"
    ],
    "awards": [
      {
        "name": "Microsoft Partner",
        "category": "",
        "wonAt": "2022-04-01"
      },
      {
        "name": "Google Partner",
        "category": "",
        "wonAt": "2022-04-01"
      },
      {
        "name": "Mollie partner",
        "category": "",
        "wonAt": "2022-11-01"
      }
    ],
    "topAwarded": false,
    "trustedPartner": false,
    "verified": true,
    "claimed": true,
    "sponsored": true,
    "updatedAt": "2026-09-30T13:32:07.483Z",
    "searchService": "Web Application",
    "search": "web-application/belgium-be"
  }
]
Enter fullscreen mode Exit fullscreen mode

Datasets can be exported directly from Apify Console or the API in JSON, CSV, XML, Excel, HTML table, RSS, or JSONL formats.

Understanding your results

Each dataset row provides detailed agency metadata extracted from Sortlist's directory page:

  • rating: client score out of 5, calculated as reviews_rating_total / reviews_count * 5 and rounded to two decimal places. The public site displays this truncated to one decimal place. If an agency has zero client reviews, rating is null.
  • teamSize: integer headcount provided by the agency (for example, 5 or 14), rather than the display range bracket shown on the website.
  • serviceBudgetMin and serviceBudgetMax: budget limits specifically listed for the queried service category. Values are numbers; they were shown as EUR on the Belgium, Germany and Switzerland listings checked, and other markets were not checked.
  • budgetMin and budgetMax: overall agency project budget range. Many agencies show 1000000 as the upper value; the site does not say what it means.
  • locations: array of office locations in English. The headquarters city is listed first.
  • languages: list of ISO language codes supported by the agency. Regional variants like nl-BE are preserved alongside standard two-letter codes.
  • website: direct link to the agency's primary website. Sortlist directory listings do not include email addresses or phone numbers.

Pro tips and advanced usage

Here are several strategies for optimizing your agency data extraction workflows:

  1. Batch services and countries together: Sortlist Scraper iterates over every pair of services and countries in a single run. Instead of launching separate tasks, pass ["web-application", "branding"] and ["belgium-be", "germany-de"] to search four directories sequentially.
  2. Filter by location for regional targeting: If you are building a local partner network, use locationContains (for example, "Brussels" or "Munich") to retain only agencies with physical presences in your target metro areas.
  3. Use rating thresholds carefully: In a 1,000-row Belgium run (web-application and marketing listings), 719 agencies had no reviews and therefore a null rating. Setting minRating will exclude all unreviewed agencies regardless of portfolio size.
  4. Automate via webhooks: Connect your runs to external pipelines using Apify webhooks. Setting up an ACTOR.RUN.SUCCEEDED trigger delivers a payload containing defaultDatasetId directly to your endpoint.
  5. Deduplication is automatic: The Actor records the agency slug and skips duplicates when an agency ranks across multiple service categories.

Troubleshooting common issues

If your run behaves unexpectedly, check these common causes:

  • HTTP 404 on page 1: Sortlist uses specific English slug conventions. Slugs like united-states-us or united-kingdom-gb do not exist on /s/ directories. If the site returns a 404 on page 1, the Actor logs a warning and moves to the next listing.
  • Run stops with a ValueError before starting: The Actor validates slugs before any network request: a service or country that is not shaped like a Sortlist slug (for example a country without its two-letter suffix, such as belgium), services given without countries (or the reverse), or a listingUrls entry that is not a https://www.sortlist.com/s/<service>/<country> address raises a ValueError at startup.
  • Run fails with "No rows pushed": If your input filters are too strict (for example, setting minRating: 4.9 alongside minTeamSize: 50), every candidate agency may be rejected. The Actor raises a RuntimeError when zero rows are pushed so you can adjust your filters.

Next steps

Automate your lead discovery by scheduling regular extractions in Apify Console using standard cron expressions (such as 0 8 * * * for daily runs at 8 AM). You can stream results directly into your CRM or Google Sheets using Apify's API integrations.

For current pricing and details, see the Store page.

What kind of agencies are you looking to discover on Sortlist? Drop a comment below! 💬

Top comments (0)