DEV Community

Datawright Labs
Datawright Labs

Posted on

Give your AI agent web-data tools with Apify's MCP server (Python, no SDK, 2026)

Most AI agents can now talk to remote MCP servers. Apify runs one at https://mcp.apify.com, and any public Actor in Apify Store can be exposed through it as a tool your agent can call: SSL checks, DNS audits, broken-link crawls, screenshots, keyword research and thousands more.

This post shows what actually happens on the wire, with a ~50-line stdlib Python client, so you can see exactly what your agent sees and pays for.

1. Pick the tools with the URL

?tools= takes a comma-separated list of Actors:

https://mcp.apify.com?tools=datawright-labs/ssl-certificate-checker,datawright-labs/email-domain-dns-checker
Enter fullscreen mode Exit fullscreen mode

Auth is your Apify API token as Authorization: Bearer <APIFY_TOKEN> (free account; Actors bill per result on your own account).

2. The client

"""Call Apify Store Actors as MCP tools from plain Python (stdlib only).
Usage: APIFY_TOKEN=... python3 mcp_agent_tools.py
"""
import json, os, urllib.request

TOOLS = "datawright-labs/ssl-certificate-checker,datawright-labs/email-domain-dns-checker"
URL = f"https://mcp.apify.com?tools={TOOLS}"
HEADERS = {
    "Authorization": "Bearer " + os.environ["APIFY_TOKEN"],
    "Content-Type": "application/json",
    "Accept": "application/json, text/event-stream",
}

def rpc(body, session=None):
    h = dict(HEADERS)
    if session:
        h["mcp-session-id"] = session
    req = urllib.request.Request(URL, data=json.dumps(body).encode(), headers=h)
    with urllib.request.urlopen(req, timeout=180) as r:
        text = r.read().decode()
        sid = r.headers.get("mcp-session-id") or session
    # Streamable HTTP may answer as SSE ("data: {...}") or plain JSON
    msgs = [json.loads(l[5:]) for l in text.splitlines() if l.startswith("data:")]
    return (msgs[-1] if msgs else (json.loads(text) if text.strip() else None)), sid

# 1. Handshake
_, sid = rpc({"jsonrpc": "2.0", "id": 1, "method": "initialize",
              "params": {"protocolVersion": "2025-03-26", "capabilities": {},
                         "clientInfo": {"name": "tutorial", "version": "1"}}})
rpc({"jsonrpc": "2.0", "method": "notifications/initialized"}, sid)

# 2. What tools does the agent see?
tools, _ = rpc({"jsonrpc": "2.0", "id": 2, "method": "tools/list"}, sid)
for t in tools["result"]["tools"]:
    print(f"- {t['name']}: {t.get('description', '')[:90]}")

# 3. Call one, exactly like an agent would
name = next(t["name"] for t in tools["result"]["tools"] if "ssl-certificate-checker" in t["name"])
res, _ = rpc({"jsonrpc": "2.0", "id": 3, "method": "tools/call",
              "params": {"name": name, "arguments": {"domains": ["example.com", "github.com", "expired.badssl.com"]}}}, sid)
for c in res["result"]["content"]:
    print(c.get("text", "")[:1500])

# 4. The call returns a run summary; the rows come from get-dataset-items (the agent's next step)
run = json.loads(res["result"]["content"][0]["text"])
ds = run["storages"]["datasets"]["default"]["id"]
items, _ = rpc({"jsonrpc": "2.0", "id": 4, "method": "tools/call",
                "params": {"name": "get-dataset-items",
                           "arguments": {"datasetId": ds, "limit": 20,
                                         "fields": "host,status,daysUntilExpiry,issuerOrganization,problems"}}}, sid)
for c in items["result"]["content"]:
    print(c.get("text", "")[:2000])
Enter fullscreen mode Exit fullscreen mode

3. What the agent sees

tools/list returns one tool per Actor (datawright-labs--ssl-certificate-checker, datawright-labs--email-domain-dns-checker) plus four helpers: get-actor-run, get-dataset-items, get-key-value-store-record, abort-actor-run.

4. The two-step call (the part most guides skip)

Calling the Actor tool runs it and returns a run summary, not the rows: status, run time, item count, the field list and a nextStep hint telling the agent to call get-dataset-items. Good agents follow that hint; if yours stops at "SUCCEEDED", tell it to fetch the dataset items. Passing fields= keeps the tokens down:

{"host":"example.com","status":"ok","daysUntilExpiry":63,"problems":[]}
{"host":"github.com","status":"ok","daysUntilExpiry":37,"problems":[]}
{"host":"expired.badssl.com","status":"expired","daysUntilExpiry":-4212,"problems":["Certificate expired","Not trusted: CERT_HAS_EXPIRED"]}
Enter fullscreen mode Exit fullscreen mode

That whole round trip took about 9 seconds and cost $0.003 (3 hosts at $1 per 1,000).

5. Plug the same URL into your agent

Any client that supports remote Streamable HTTP MCP servers with a header takes the same two values (URL + Bearer header). Two configs we tested end to end for connection and tool discovery:

  • Hermes (pip install "hermes-agent[mcp]"), ~/.hermes/config.yaml: mcp_servers: { datawright: { url: "...", headers: { Authorization: "Bearer ${APIFY_TOKEN}" } } }, then hermes mcp test datawright.
  • OpenClaw: openclaw mcp add datawright --no-probe --url "..." --transport streamable-http --header 'Authorization=Bearer ${APIFY_TOKEN}', then openclaw mcp probe datawright (the pre-save probe doesn't expand ${APIFY_TOKEN}, hence --no-probe).

6. Gotchas

  • Agents only see the top 10 results of Apify's search-actors, so pin the Actors you want with ?tools= instead of letting the agent search.
  • /sse is gone; use the Streamable HTTP endpoint above.
  • Keep the token in an env var or your client's secret store, never in a prompt.

Disclosure: the two example Actors are ours (Datawright Labs). Any Store Actor works the same way.

Top comments (2)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

In an MCP web-data setup the bill often sits in the tool result, not in the call: a scraped page can weigh many times the answer the agent needed. Does Apify's server cut the result down before it reaches the context?

Collapse
 
alexcodebytes profile image
Oleksandr •

The "two-step call" explanation is the most valuable part of this post. Most MCP tutorials show a simple tools/call and assume it returns the final data, but Apify's architecture means the agent gets a run summary and has to actively fetch the dataset items next. If an agent isn't explicitly instructed to follow that nextStep hint, it'll just sit there thinking it succeeded without the actual data.

How are you handling long-running Actors, though? If a heavy scraper takes 5 minutes to finish, does the MCP server block the connection, or does the agent just get the run ID immediately and have to poll get-actor-run until it's done?