Calling Clutch Scraper from MCP Agents in Python

Exposing Clutch Data to MCP Agents

Autonomous software agents require reliable access to external tools when researching B2B vendors, evaluating software agencies, or qualifying prospective service partners. The Model Context Protocol (MCP) provides an open standard for exposing remote tools to language models without building custom integration layers for every downstream agent framework.

When you expose an extraction pipeline to an agent, the core challenge is not simply invoking a container. The challenge is ensuring the model understands the schema signature, passes valid profile targets, handles synchronous execution boundaries gracefully, and operates within strict cost controls.

The Clutch.co B2B Agency Scraper provides an ideal test case for an MCP tool interface. The Actor accepts target profile URLs, bypasses Cloudflare security mechanisms using Chrome TLS impersonation over residential proxies, and extracts twenty-six structured fields per company. Exposing this capability through the Apify MCP server allows any MCP-compatible agent to request verified agency profiles directly within its reasoning loop.

Checked against the Actor's input schema and Apify docs on 2026-09-09.

Configuring the Scoped MCP Tool URL

The Apify MCP server operates at https://mcp.apify.com. By default, an unconfigured MCP connection exposes generic discovery tools such as search-actors, fetch-actor-details, search-apify-docs, and fetch-apify-docs. These four endpoints operate unauthenticated, but running an Actor requires an API token.

Allowing an AI model to search and select from thousands of public Actors introduces hallucination risks, excessive prompt token consumption, and non-deterministic tool selection. You can scope the MCP server directly to a single tool by passing the Actor identifier in the tools query parameter.

{
  "mcpServers": {
    "clutch-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-fetch",
        "https://mcp.apify.com/v1?tools=crawlerbros/clutch-scraper"
      ],
      "env": {
        "APIFY_TOKEN": "your_apify_api_token_here"
      }
    }
  }
}

When scoped with ?tools=crawlerbros/clutch-scraper, the MCP endpoint presents only the tool definition generated from the Actor's build metadata. This minimizes the context window payload delivered to the agent during initialization.

# Verify the scoped MCP server endpoint using curl
curl -s -X GET "https://mcp.apify.com/v1?tools=crawlerbros/clutch-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN"

How Does the Actor Input Schema Map to MCP Tool Parameters?

The Actor input schema translates directly into a JSON Schema object containing companyUrls as a required array and maxItems as an optional integer. The MCP server registers these properties so the calling agent generates structured arguments matching the exact parameter names. Note that console prefill values are ignored during API invocation, so agents must supply all required fields explicitly.

The input schema for clutch-scraper defines two properties:

1. companyUrls: An array of strings representing direct Clutch profile URLs. This property is marked as required in the schema.
2. maxItems: An integer representing the maximum number of agency profiles to scrape, defaulting to 20.

When the MCP server translates this definition for an LLM tool call, it produces a standard JSON Schema tool signature.

{
  "name": "crawlerbros_clutch-scraper",
  "description": "Extract agency profiles, ratings, reviews, and verified client data from Clutch.co. Get ratings, pricing, services, location, and reviews for B2B service providers.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "companyUrls": {
        "type": "array",
        "items": {
          "type": "string"
        },
        "description": "Direct Clutch.co profile URLs to scrape (e.g., https://clutch.co/profile/toptal)."
      },
      "maxItems": {
        "type": "integer",
        "default": 20,
        "description": "Maximum number of profiles to scrape."
      }
    },
    "required": ["companyUrls"]
  }
}

A critical distinction on the platform involves schema fields: the prefill property configured in an Actor console interface only applies to the web UI. API invocations and MCP tool executions ignore prefill values completely and only respect declared default values. An agent calling this tool must always supply an explicit array under companyUrls.

{
  "companyUrls": [
    "https://clutch.co/profile/toptal",
    "https://clutch.co/profile/cleveroad"
  ],
  "maxItems": 2
}

Handling Sync Timeout Limits and HTTP 408 Responses

AI agent frameworks typically expect synchronous tool execution: the model emits a tool call, the runtime executes the action, and the result returns immediately within the reasoning turn.

Apify provides synchronous run endpoints, but they enforce a hard platform limit of 300 seconds (5 minutes). If scraping a list of companyUrls exceeds 300 seconds, the gateway terminates the HTTP connection and returns an HTTP 408 Request Timeout.

[Agent Tool Execution]
       |
       v
POST /v2/acts/crawlerbros~clutch-scraper/run-sync-get-dataset-items
       |
       +---> Duration <= 300s ---> Returns HTTP 200 + Dataset JSON
       |
       +---> Duration >  300s ---> Gateway Timeout: HTTP 408

If an agent needs to process multiple company URLs in a single tool call, relying solely on synchronous execution introduces intermittent failures. You should implement a client wrapper that initiates an asynchronous run via POST /v2/acts/crawlerbros~clutch-scraper/runs, polls the run status, and retrieves the dataset items once complete.

import os
import time
import requests

APIFY_TOKEN = os.environ.get("APIFY_TOKEN")
ACTOR_ID = "crawlerbros~clutch-scraper"
HEADERS = {"Authorization": f"Bearer {APIFY_TOKEN}"}

def run_clutch_tool_async(company_urls: list[str], max_items: int = 20) -> list[dict]:
    run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs"
    payload = {
        "companyUrls": company_urls,
        "maxItems": max_items
    }

    response = requests.post(run_url, json=payload, headers=HEADERS)
    response.raise_for_status()
    run_data = response.json()["data"]
    run_id = run_data["id"]
    default_dataset_id = run_data["defaultDatasetId"]

    poll_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs/{run_id}"
    while True:
        status_resp = requests.get(poll_url, headers=HEADERS)
        status_resp.raise_for_status()
        status = status_resp.json()["data"]["status"]

        if status in ["SUCCEEDED", "FAILED", "ABORTED", "TIMED-OUT"]:
            break
        time.sleep(5)

    if status != "SUCCEEDED":
        raise RuntimeError(f"Actor run ended with non-success status: {status}")

    dataset_url = f"https://api.apify.com/v2/datasets/{default_dataset_id}/items"
    items_resp = requests.get(dataset_url, headers=HEADERS)
    items_resp.raise_for_status()
    return items_resp.json()

How Do You Prevent Agent Runaway Costs with maxTotalChargeUsd?

You pass the maxTotalChargeUsd query parameter in your execution request to cap spending. When the Actor reaches this spending threshold, execution terminates automatically instead of running indefinitely. While container teardown is not instantaneous, setting this parameter prevents autonomous agents from generating unbounded bills during loop errors or broad scrape lists.

Autonomous agent loops can malfunction by issuing hundreds of tool calls or passing massive URL lists. You can enforce hard spending limits on the Apify platform by passing maxTotalChargeUsd as a query parameter when starting the Actor run.

Inside the container, this limit is surfaced through the ACTOR_MAX_TOTAL_CHARGE_USD environment variable. When the accumulated compute and proxy cost reaches this value, the platform initiates a run termination.

import os
import requests

APIFY_TOKEN = os.environ.get("APIFY_TOKEN")
ACTOR_ID = "crawlerbros~clutch-scraper"

def run_bounded_clutch_tool(urls: list[str], budget_usd: float = 0.50) -> dict:
    url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?maxTotalChargeUsd={budget_usd}"
    headers = {"Authorization": f"Bearer {APIFY_TOKEN}"}
    payload = {
        "companyUrls": urls,
        "maxItems": len(urls)
    }

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()
    return response.json()["data"]

Because the run continues consuming resources for a brief window during termination, set your maxTotalChargeUsd slightly below your absolute tolerance limit.

Validating Structured Profile Fields in Python Agent Workflows

The Actor extracts twenty-six distinct attributes per agency by parsing JSON-LD LocalBusiness structures, DOM elements, and embedded JavaScript objects (window.chartPie and window.serviceLines).

The output contains:

• Core identity: name, url, tagline, website, logo, description
• Ratings and pricing: rating, reviewCount, verificationStatus, minProjectSize, averageHourlyRate, priceRange, employees, foundingDate
• Location: country, city, region, postalCode, streetAddress, telephone
• Weighted focus arrays: services, focus, industries, clientSizes
• Reviews: reviews (up to 10 parsed recent reviews)
• Metadata: scrapedAt

To prevent schema mismatch errors in downstream tools, the agent runtime should validate the shape of returned dataset items using typed models.

from typing import Optional, List
from pydantic import BaseModel

class WeightedItem(BaseModel):
    name: str
    percent: float

class FocusGroup(BaseModel):
    title: str
    values: List[WeightedItem]

class ReviewItem(BaseModel):
    name: Optional[str] = None
    datePublished: Optional[str] = None
    author: Optional[str] = None
    rating: Optional[float] = None
    reviewBody: Optional[str] = None

class ClutchAgencyProfile(BaseModel):
    name: str
    url: str
    tagline: Optional[str] = None
    website: Optional[str] = None
    logo: Optional[str] = None
    description: Optional[str] = None
    rating: Optional[float] = None
    reviewCount: Optional[int] = None
    verificationStatus: Optional[str] = None
    minProjectSize: Optional[str] = None
    averageHourlyRate: Optional[str] = None
    priceRange: Optional[str] = None
    employees: Optional[str] = None
    foundingDate: Optional[str] = None
    country: Optional[str] = None
    city: Optional[str] = None
    region: Optional[str] = None
    postalCode: Optional[str] = None
    streetAddress: Optional[str] = None
    telephone: Optional[str] = None
    services: List[WeightedItem] = []
    focus: List[FocusGroup] = []
    industries: List[WeightedItem] = []
    clientSizes: List[WeightedItem] = []
    reviews: List[ReviewItem] = []
    scrapedAt: str

Validating output with Pydantic ensures that downstream LLM prompts receive strictly typed data structures, preventing unexpected key errors or formatting crashes during multi-step reasoning chains.

How Does an Agent Handle Missing Category Scraping?

The Actor only accepts direct agency profile URLs and cannot parse directory listing pages directly. An agent must supply pre-discovered Clutch profile links within the companyUrls array rather than passing category or search URLs. If an invalid or redirecting profile link is passed, the Actor skips the target and logs a warning without returning junk records to the dataset.

Because clutch-scraper does not support directory navigation or search result pagination, an agent must construct target URLs adhering to the format https://clutch.co/profile/<agency-slug>. If an agent extracts links from a general search engine or an external list, it must filter those URLs before calling the tool.

import re

def filter_clutch_profile_urls(raw_urls: list[str]) -> list[str]:
    profile_pattern = re.compile(r"^https?://(?:www\.)?clutch\.co/profile/[a-zA-Z0-9\-_]+/?$")
    valid_urls = []

    for url in raw_urls:
        clean_url = url.strip()
        if profile_pattern.match(clean_url):
            valid_urls.append(clean_url)

    return valid_urls

Passing a directory URL such as https://clutch.co/developers/artificial-intelligence will cause the scraper to encounter unexpected page structures or redirects. The Actor code detects redirects back to listing pages or the homepage and skips them, returning zero items for that invalid URL.

Managing Residential Proxy Sessions and Resiliency

Clutch.co enforces strict Cloudflare challenges that block datacenter IP addresses. The scraper overcomes this by routing requests through Apify residential proxies combined with curl_cffi for Chrome TLS fingerprint impersonation.

Apify residential proxy sessions persist for approximately 30 minutes. If an Actor run processes a large batch of profile URLs and extends past the 30-minute window, the underlying proxy pool will automatically assign new IP sessions. The Actor handles this session rollover internally without requiring user intervention.

When designing custom orchestrations, be aware that proxy sessions behave differently across pool types:

[Datacenter Proxy Session]  ---> Persists up to 26 hours
[Residential Proxy Session] ---> Persists ~30 minutes

For platform resilience, Apify supports graceful aborts and container resurrection. When a run receives an abort signal, the container receives a 30-second window to clean up connections and flush remaining dataset items. If an Actor run is resurrected after being aborted or terminated, the container restarts with the exact same storage, excludes downtime from its billed duration, and restarts the timeout clock.

Integrating Clutch Scraper with Asynchronous Webhook Handlers

When building event-driven agent architectures, polling the Apify run endpoint repeatedly introduces unnecessary network overhead. The platform provides webhooks as its sole event-driven primitive. Webhooks support exactly one action: sending an HTTP POST payload to a designated URL upon run completion.

You can configure a webhook directly in the API call that starts the Actor run. This pattern allows your Python backend or agent orchestrator to receive notifications when the run reaches the SUCCEEDED status.

import os
import requests

APIFY_TOKEN = os.environ.get("APIFY_TOKEN")
ACTOR_ID = "crawlerbros~clutch-scraper"

def trigger_clutch_run_with_webhook(urls: list[str], webhook_receiver_url: str) -> dict:
    url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs"
    headers = {"Authorization": f"Bearer {APIFY_TOKEN}"}

    payload = {
        "companyUrls": urls,
        "maxItems": len(urls)
    }

    webhook_config = [
        {
            "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
            "requestUrl": webhook_receiver_url,
            "payloadTemplate": "{\n  \"runId\": {{eventData.actorRunId}},\n  \"datasetId\": {{eventData.defaultDatasetId}}\n}"
        }
    ]

    response = requests.post(
        url,
        json=payload,
        headers=headers,
        json_webhooks=webhook_config
    )
    response.raise_for_status()
    return response.json()["data"]

Once the webhook endpoint receives the payload, the backend can fetch the dataset records directly from https://api.apify.com/v2/datasets/<datasetId>/items. Storage rate limits permit up to 60 requests per second per storage object and 400 requests per second for dataset item pushes, providing ample throughput for ingestion pipelines.

Documented Limitations and Known Edge Cases

When deploying this Actor in production pipelines, keep the following documented operational boundaries in mind:

1. No Category Scraping: The Actor does not crawl category directories or search results. It accepts only direct profile URLs passed through companyUrls.
2. 10 Recent Reviews Cap: The Actor extracts up to 10 recent reviews per agency from structured JSON-LD data. The total historical review count is accessible in reviewCount, but older reviews beyond the first 10 cannot be extracted.
3. Queue Concurrency: A request queue on the Apify platform can only be processed by one Actor or task run at a time. Multiple runs may push items into a shared queue, but fan-out processing across a single shared queue is not supported.
4. Storage Expiration on Free Tiers: Unnamed datasets and key-value stores expire automatically. On the Free plan tier, only the 10 most recent runs are retained, for up to 4 months. Named storages are required if your agent workflow must retain profile datasets indefinitely.
5. Schedules Require Prior Execution: When scheduling recurring extraction tasks, Apify cron schedules require a 6-field cron expression (with a minimum interval of 10 seconds). The target Actor must have run at least once before a schedule can be created, and all new schedules are created disabled by default.

Platform Pricing and Compute Unit Consumption

Cost on the Apify platform scales based on Compute Unit (CU) consumption and residential proxy bandwidth usage.

A Compute Unit represents container resource consumption defined by the formula:

CU = (memory_mb / 1024) * duration_hours

For Actor runs, memory allocation determines the rate of CU consumption over time. Doubling container memory is only CU-neutral for autoscaling runs, and platform autoscaling applies exclusively to solutions running multiple tasks or URLs that run for at least 30 seconds each.

Compute unit rates depend on your Apify subscription tier:

• Free Plan ($0/month): $0.20 per CU
• Starter Plan ($19/month): $0.20 per CU
• Scale Plan ($199/month): $0.16 per CU
• Business Plan ($999/month): $0.13 per CU

Because Clutch.co requires residential proxy routing to bypass anti-scraping filters, residential proxy bandwidth incurs additional platform charges:

• Free and Starter Plans: $8/GB
• Scale Plan: $7.50/GB
• Business Plan: $7/GB

When orchestrating agency research through Clutch.co B2B Agency Scraper, ensure your agent controls batch sizes and monitors execution parameters to stay aligned with your plan tier limits and bandwidth allocations.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]