DEV Community

Gio Rich
Gio Rich

Posted on Edited on

Building agency lead lists from Sortlist with Python

#python #webscraping #api #leadgeneration

I built this actor; it's a paid tool on Apify with a free trial credit.

If you sell to agencies (white-label services, SaaS for marketers, partnerships), you need a list of agencies with their size, budget range and website. Clutch has that list but sits behind Cloudflare. Sortlist, a European agency marketplace, has the same kind of directory for marketing, design and IT agencies in most countries, and it is open.

I'm an 18-year-old engineering student, and I built Sortlist Scraper to turn a directory like "advertising agencies in the United States" into one clean JSON row per agency, over plain HTTP. This post covers how it works, calling it from Python and Node, what a row looks like, and a small project: a partnership shortlist.

How it works

Sortlist is a Next.js site, so every page carries its data as JSON in a __NEXT_DATA__ script tag. A directory page (/<service>/<location>) lists the agencies with their website, team size, review count, address, languages, registration number, budget and services. Page 1 shows 20 agencies and later pages show 100, so a directory of 3,600 agencies takes about 38 requests.

With agency pages switched on, it opens each /agency/<slug> page and adds the founding year, revenue, every office with coordinates, social profiles, per-service budgets, awards and reviews. Ratings are stored from 0 to 1 and returned on a 5-point scale. Filters (must have a website, minimum reviews / rating / team size / budget) run before anything is charged.

A lesson from the first cloud run: some agencies have an AI review summary stored as an object, and Apify's dataset schema check rejected those rows. The actor now flattens it into text plus a reviewKeyPoints list, and I validate a real local dataset against the schema before every push.

Calling it from Python 

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/sortlist-scraper").call(run_input={
    "startUrls": [
        "https://www.sortlist.com/digital-marketing/united-kingdom-gb",
        "https://www.sortlist.fr/seo/paris-ile-de-france-fr",
    ],
    "maxResultsPerDirectory": 200,
    "requireWebsite": True,
    "minTeamSize": 10,
})

for a in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(a["name"][:35], "|", a["website"], "|", a["teamSize"], "|", a["city"], "|", a["founded"])
Enter fullscreen mode Exit fullscreen mode

The same from Node:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('rel8ble/sortlist-scraper').call({
    startUrls: ['https://www.sortlist.com/advertising/united-states-us'],
    maxResultsPerDirectory: 0, // the whole directory
    scrapeDetails: false, // directory data only: 100 agencies per request
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'advertising agencies');
Enter fullscreen mode Exit fullscreen mode

What a row looks like

From a real run, trimmed:

{
    "name": "Goliaence",
    "website": "https://goliaence.com",
    "tagline": "Make them notice. Digital marketing, creative and applied AI.",
    "address": "1 Chisholm Trail Road, Round Rock, TX, USA",
    "countryCode": "US",
    "latitude": 30.5140931,
    "longitude": -97.6890449,
    "teamSize": 5,
    "founded": 2025,
    "budgetMin": 1000,
    "budgetMax": 1000000,
    "services": ["Advertising", "Online Advertising"],
    "industries": ["Industrial Goods & Services", "Construction"],
    "languages": ["en"],
    "socialLinks": ["https://www.linkedin.com/company/goliaence/"],
    "projectsCount": 7,
    "verified": true,
    "worksRemotely": true,
    "url": "https://www.sortlist.com/agency/goliaence",
    "directory": "Top Advertising Agencies in the United States",
    "position": 2
}
Enter fullscreen mode Exit fullscreen mode

In a 451-agency test (website and 1-review filters on), every row had a website, address, coordinates, team size and rating; 97% had a founding year and social links, 87% a budget range and 53% revenue.

Project: a partnership shortlist

Say you offer white-label SEO. The best partners are established agencies that sell digital marketing but do not list SEO, with a team big enough to resell.

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

def fit(a):
    services = {s.lower() for s in a["services"]}
    if any("seo" in s for s in services):
        return 0                                   # already does SEO
    s = 0
    s += 2 if (a["teamSize"] or 0) >= 10 else 0
    s += 2 if a["founded"] and a["founded"] <= 2018 else 0
    s += 1 if (a["reviewCount"] or 0) >= 5 else 0
    s += 1 if a["budgetMin"] and a["budgetMin"] >= 5000 else 0
    return s

shortlist = sorted(rows, key=fit, reverse=True)[:20]
for a in shortlist:
    print(f"{a['name'][:30]:30} {fit(a)}  {a['teamSize']:>4}  {a['city'] or '':15} {a['website']}")
Enter fullscreen mode Exit fullscreen mode

The socialLinks column usually includes the agency's LinkedIn page, which is where the outreach starts.

Pricing and limits

$0.75 per 1,000 agencies, no start fee, same price with or without agency pages. Filtered agencies, duplicates, dead directories and failed requests are never charged. No e-mails or phone numbers (Sortlist routes contact through its own briefs; pair it with a website contact extractor), revenue and awards only when the agency publishes them, results in Sortlist's own order. Reviewer names are personal data, so treat them accordingly.

Store page: https://apify.com/rel8ble/sortlist-scraper

This article was drafted with AI and published by me.

Top comments (0)