DEV Community: Apify

I gave my YC jobs scraper write access to Notion without ever holding a Notion token

Artem Lazarev — Sat, 03 Oct 2026 09:06:38 +0000

TL;DR

Apify MCP connectors use the Model Context Protocol (MCP) to let an Actor write into a user's connected apps without ever touching that app's credentials. I used one to push my scraped Y Combinator jobs straight into Notion and delete my CSV export step. Below: why a token field was never an option for a published Actor, the documented Python snippet that couldn't run on any SDK version, and five steps covering connector setup, tool discovery, field mapping, safe writes, and what the whole thing costs per run.

The workflow I wanted

My Y Combinator jobs scraper has run happily for months. It walks the YC directory, pulls every company that's hiring, and returns salaries, equity ranges, founder profiles, and full job descriptions. The last full sweep did 1,429 companies and 3,305 jobs in about an hour and fifty minutes.

Then the data sat there. An 18 MB JSON file in a dataset. What I actually wanted was boring and specific: a running list of who's hiring, at what salary, in which batch, that I could sort and annotate and come back to a week later. So every time, I exported a CSV, opened it in a spreadsheet, and filtered by hand. The scraper was finished. My workflow wasn't.

Apify MCP connectors closed that gap. They let an Actor call an external service's allowed tools through an Apify-managed proxy, using the user's own authorized account.

The Actor is published on Apify Store, charges per result, and has 120 users I've never met.

After I added connector support, it writes scraped jobs into a Notion database on its own. All it took was a connector field in the input schema, a declaration of the Notion tools it needs, and the code that turns scraped jobs into Notion pages.

The full implementation is in the repo: github.com/artem-lazarev/apify-ycombinator-jobs-scraper.

For users of Apify Actors. To use an MCP connector, the Actor's developer has to add support for it first. Once that's in place, open Settings > API & Integrations in your own Apify account and authorize the connector you need, such as Notion. Then open the Actor's input form, select that connector, and fill in the destination details. For my YC Jobs Scraper, that's the Notion data source ID. Run it, and the scraped jobs land in your Notion database.

Why I couldn't just ask for a Notion token

The obvious option was to add a Notion API token field to the input schema and call the Notion API directly. A Notion integration token is a single secret that grants access to everything you've shared with that integration: pages, databases, all of it.

Asking those users to paste a Notion token into my Actor means asking them to trust that my code won't log it or send it elsewhere. I wouldn't paste my own token into someone else's Actor, so I don't think it's reasonable to ask.

MCP connectors invert that. The user authorizes Notion once, in their Apify account. My Actor receives a connector ID, which is just an opaque string. At runtime it authenticates to the Apify MCP proxy with the run's own Apify token, and the proxy adds the Notion credential server-side before forwarding anything upstream. That credential never enters my container. There's nothing for me to leak and nothing anyone has to trust me about.

Users still need to check which tools an Actor declares, because it can read and write through them. Apify treats the Actor runtime as untrusted precisely because it runs developer-supplied code. I'm the untrusted party here, and I'd rather be.

What you need to follow along

By the end of this walkthrough your Actor will finish a scrape and push the results into Notion on its own. No CSV export, no import step, no Notion token anywhere in your code.

Before any of that works, one thing has to be true: your Notion database has to exist already, with the columns you actually want. This Actor creates pages. It doesn't create databases and it doesn't create columns. Step 3 covers what happens when you gloss over that, which in my case was a batch of jobs landing in Notion with nothing in them but titles.

So build the destination first. Start with a title column and add the fields you plan to fill.

The rest of the list is short:

An Apify account.
A Python Actor you can build and run in Apify Console.
A Notion workspace you can authorize. Apify provides the managed OAuth client for Notion, so you don't need to register your own OAuth app.
apify-cli 1.8.0 or newer, if you deploy from the command line.

The connection code targets Python 3.10 or newer, the Apify SDK, and mcp 2.x. One packaging detail worth knowing up front: mcp 2.x depends on httpx2, which is a separate package from httpx. Install and import that one.

The changes land in three places: inputs go in .actor/input_schema.json, the Notion helpers live in src/notion_sync.py, and the sync gets called after results are saved in src/main.py. The repository has the complete scraper; the excerpts below are the integration only.

1. Declare the Apify MCP connector and its allowed tools

When I opened my authorized Notion connector, it offered 28 tools: notion-search, notion-update-page, notion-create-database, and 25 more. My Actor can call two of them. The proxy rejects everything else before it ever reaches Notion.

Setting up that allowlist is what this step is for, and it lives in .actor/input_schema.json. Here's the notionConnector field inside the existing schema's properties object:

{
  "properties": {
    "notionConnector": {
      "title": "Notion connector",
      "type": "string",
      "description": "Optionally write jobs to Notion through an authorized connector.",
      "resourceType": "mcpConnector",
      "nullable": true,
      "sectionCaption": "Notion output",
      "mcpServers": [
        {
          "url": "https://mcp.notion.com/mcp",
          "tools": {
            "required": [
              "notion-create-pages",
              "notion-fetch"
            ]
          }
        }
      ]
    },
    "notionDataSourceId": {
      "title": "Notion data source ID",
      "type": "string",
      "description": "Destination data source UUID or collection:// reference.",
      "editor": "textfield",
      "nullable": true
    }
  }
}

resourceType: "mcpConnector" turns the input into a connector picker in Console. After building the Actor, authorize Notion under Settings > API & Integrations, then select it in the Notion output section.

The second input, notionDataSourceId, chooses where the pages go. Keep both optional so dataset-only runs still work. Step 3 shows how to fetch the destination reference once the connection is up.

The connector picker rendered in the Actor input form, with the Notion connector selected.

The tools.required list is the part I'd skim past in someone else's article, so let me be specific about it. It isn't documentation and it isn't a hint. The proxy enforces it: it filters tools/list down to what you declared, and rejects tools/call for anything outside it.

Here's the first line my Actor logs on every run, straight from a real run log:

🔧 Tools allowed through the connector: ['notion-create-pages', 'notion-fetch']

So a jobs scraper can't rummage through your Notion. Not because I promise it won't, but because the proxy won't pass the call. You can check that yourself without reading a line of my code, which is about the only kind of security promise worth anything in a public Actor store.

One thing the allowlist doesn't do: it limits which operations the Actor can perform, not which content those operations can reach. notion-fetch isn't restricted to one database. What it can read still depends on the Notion account and permissions sitting behind the connector.

MCP connectors under Settings > API & Integrations, showing the authorized Notion connector.

The connector's Edit dialog, showing all 28 Notion tools it could expose.

2. Connect and read the tool descriptions before you write anything

The documented snippet couldn't run on either SDK version

The first thing I did was copy the Python snippet out of Apify's connector docs. It deployed, and then:

ValueError: not enough values to unpack (expected 3, got 2)

The snippet unpacked three values from the transport:

async with streamable_http_client(...) as (read, write, _):

In the mcp 2.x SDK, streamable_http_client yields exactly two. I checked the type rather than guessing:

>>> from mcp.client.streamable_http import TransportStreams
>>> TransportStreams
tuple[ReadStream[SessionMessage | Exception], WriteStream[SessionMessage]]

Three values is the 1.x shape, where the third element was a session-ID callback. But 1.x has no function called streamable_http_client at all. It's streamablehttp_client, with no underscore between streamable and http, and it takes a headers argument rather than an injected HTTP client. I downloaded the 1.9.0 source to confirm:

# mcp 1.9.0 - mcp/client/streamable_http.py
async def streamablehttp_client(
    url: str,
    headers: dict[str, Any] | None = None,
    timeout: timedelta = timedelta(seconds=30),
    sse_read_timeout: timedelta = timedelta(seconds=60 * 5),
    terminate_on_close: bool = True,
)

So the documented snippet was a hybrid: the 2.x import name with the 1.x unpacking. It couldn't run on either version. On 2.x the import resolves and the unpacking fails. On 1.x the import fails before you get that far.

Apify has since fixed that page. It now unpacks two values, and it carries a version callout spelling out the same distinction: on mcp 1.x the transport function is named streamablehttp_client, takes headers instead of http_client, and yields a third value. If you're reading this with a working snippet in front of you, that's why.

Save the tool descriptions before you build a payload

The notion-create-pages description is 5,857 characters long. Console truncated it in my logs, so for the first few attempts I was coding against maybe a third of the contract.

The fix is one extra block, and it's the trick I'd start with on any MCP service: save the tool descriptions and input schemas into the run's key-value store, then read them properly.

Which means your first connection should do nothing but discover tools. It confirms authorization and hands you the service's real contract before you build a single payload. Each tool description explains what the tool does, and its input schema defines the arguments it accepts.

Open a session from Python

The Actor needs three values: the selected connector ID, the MCP proxy's base URL, and the Apify run token. Apify supplies the last two in every platform run.

Install these in the Actor build:

apify>=2.0.0
mcp>=2.0.0
httpx2>=2.5.0,<3

import asyncio
import os

import httpx2
from apify import Actor
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client


async def main() -> None:
    async with Actor:
        actor_input = await Actor.get_input() or {}
        connector_id = actor_input.get('notionConnector')
        proxy_url = os.environ.get('ACTOR_MCP_CONNECTOR_BASE_URL')
        token = os.environ.get('APIFY_TOKEN')
        if not connector_id or not proxy_url or not token:
            Actor.log.warning(
                '⚠️ No connector or no proxy credentials - skipping Notion sync. '
                'MCP connectors only resolve in a platform run, not a local one.'
            )
            return
        try:
            async with httpx2.AsyncClient(
                headers={'Authorization': f'Bearer {token}'},
                timeout=httpx2.Timeout(60.0),
                follow_redirects=True,
            ) as http_client:
                async with streamable_http_client(
                    f"{proxy_url.rstrip('/')}/{connector_id}",
                    http_client=http_client,
                ) as (read, write):
                    async with ClientSession(read, write) as session:
                        await session.initialize()
                        tools = (await session.list_tools()).tools
                        Actor.log.info(
                            f'🔧 Tools allowed through the connector: '
                            f'{[tool.name for tool in tools]}'
                        )
        except Exception:
            Actor.log.exception('Notion connector check failed')
            raise


if __name__ == '__main__':
    asyncio.run(main())

Then, immediately after the tools assignment and inside the same initialized session, add the block that saves everything:

payload = [
    {
        'name': tool.name,
        'description': tool.description,
        'inputSchema': tool.input_schema,
    }
    for tool in tools
]
await Actor.set_value('NOTION_TOOL_SCHEMA', payload)

Open the run's default key-value store and read NOTION_TOOL_SCHEMA. One naming detail that tripped me up: the SDK attribute is input_schema, while inputSchema above is just the JSON key I picked for the saved file. Inspect the allowed arguments for notion-fetch and notion-create-pages before going further, because the parent ID and the property formats you need in step 3 are both buried in there.

Push to a dev build tag, not latest. I have paying users on latest, and I wasn't going to test a new network call in front of them. There's a second reason to work this way: ACTOR_MCP_CONNECTOR_BASE_URL only exists in a platform run, so apify run on your laptop can't reach a connector at all. My whole loop was push to a dev tag, run on the platform, read the log.

3. Prepare the Notion destination and map its fields

This is the step that cost me the most time, and the worst failure in it was the silent one.

A batch of pages went into Notion with nothing but titles. No company, no salary, no equity. The log cheerfully reported that it was sending all columns. It wasn't lying, exactly: my schema parser had returned an empty dict, and the row builder read {} as "this database has a title column and nothing else", so it filtered every other field out on the way through.

Two things have to be right before you write anything: the destination reference, and the column map.

Get the data source reference

My first mistake here was simpler. I sent the database ID, because that's what the Notion URL hands you, and the tool rejected it.

A Notion database contains one or more data sources, and the write needs the ID of the source that should receive the jobs, not the database itself. A database URL on its own isn't the destination value either. Fetching the database is what makes the distinction obvious.

Set database_url to the URL copied from your Notion database. Add this after session.initialize() in the connection example, at the same indentation as the other statements inside that session:

result = await session.call_tool(
    'notion-fetch', arguments={'id': database_url}
)
if result.is_error:
    raise RuntimeError('Notion could not fetch the database')
await Actor.set_value(
    'NOTION_DATABASE',
    result.model_dump(mode='json', by_alias=True),
)

Open NOTION_DATABASE in the run's key-value store and find the collection:// reference for the data source you want. Put that reference, or its bare UUID, into notionDataSourceId. The sync strips collection:// before sending parent: {"data_source_id": source_id}.

Read the schema before building rows

Back to the empty dict. But there's a reason I was reading the schema at all, and it wasn't the bug.

I had it working against my own database and I nearly shipped that. But it needed to work for my users' databases too. My database has a column called Salary Min. Theirs might call it Salary (min), or have no salary column at all. The title column is usually Name, but Notion lets you rename it to anything. Since one unknown column kills an entire batch, an Actor that assumes my layout works for exactly one person.

So the Actor reads the target before writing to it. That's what the second declared tool, notion-fetch, is for.

In src/notion_sync.py, _fetch_data_source_schema calls notion-fetch for source_id and passes the response to parse_data_source_schema. Those helpers, and logger, are defined in that file. The sync uses their result like this:

schema = await _fetch_data_source_schema(session, source_id) or None
if schema:
    logger.info(f'🗂️ Target columns: {sorted(schema)}')
else:
    logger.warning('⚠️ Could not read the data source schema - sending all columns')

That or None is the fix for the silent failure. An empty dict is falsy, but it isn't None, and the row builder treated {} as a real schema with no optional columns. Collapsing it to None forces a failed or empty lookup down the fallback path: send the expected fields and let Notion report the mismatch out loud. You can still end up with a rejected batch, but you'll know about it instead of finding fifty half-empty pages later. And either way the complete scraped data is already sitting in the Apify dataset.

The parsing itself has a wrinkle. notion-fetch doesn't return JSON. It returns a document wrapped in a JSON envelope: some prose, a <data-source-state> block, and a SQLite CREATE TABLE rendering of the same schema. The parser has to unwrap the envelope before it can read the schema at all. This excerpt handles the outer layer; the repository's parse_data_source_schema handles the remaining formats:

import json

try:
    envelope = json.loads(payload)
    if isinstance(envelope, dict) and isinstance(envelope.get('text'), str):
        payload = envelope['text']
except json.JSONDecodeError:
    pass

Then the row builder sends the job title to the column with type title and filters the other fields against the discovered names. This excerpt uses _title_column and the candidates dictionary from build_job_rows in the same file:

title_key = _title_column(schema) if schema else 'Name'
...
for column, value in candidates.items():
    if value is None:
        continue
    if schema is not None and column not in schema:
        continue
    properties[column] = value

The nice side effect: point the Actor at a brand-new Notion database with only a title column and it still works. It writes titles. Add a Salary Min column and the next run fills it in, with no code change.

Use the MCP tool's property formats

One more trap, and this one produced the least helpful error message of the entire build:

{"code":"validation_error","status":400,
 "message":"Properties {propertyKeys} not found in the data source.\n
            Date property {propertyKeys} not found in the data source.\n
            All editable property keys: {editableProperties}."}

Those are literal {propertyKeys} placeholders. The template never got interpolated, so the error tells you that some property is wrong while withholding which one and what the valid names are. Notion knows both. And one unknown column rejected all 20 pages in the batch, because the offending page doesn't fail on its own.

Reading the saved tool description got me there faster than guessing from that message ever would have. It also handed me the encoding rules I'd otherwise have found by trial and error. Properties take scalar values rather than the Notion REST API's nested property objects. Send numbers as numbers. Checkboxes are the literal strings __YES__ and __NO__. And date properties split across prefixed keys, so a column called "Scraped At" is written like this, where properties is the row's property dictionary and scraped_at is its date value:

properties['date:Scraped At:start'] = scraped_at

Not properties['Scraped At']. That one would have taken me an hour on my own.

Match column names exactly and give them compatible types. A Number column named Salary Min will take the scraper's salary value; Salary (min) gets skipped. The Actor discovers the title column, but it won't translate other names for you or validate every column type.

4. Save the scrape, then write pages in batches

My first connection attempt broke, and the run still finished with exit code 0 and a complete dataset.

That wasn't luck, it's the entire reason for the ordering in this step. Save the scrape to the dataset first, then sync, and keep the sync inside its own try/except. A connector failure should never throw away work the scraper has already finished, and on a per-result Actor that's work the user has already paid for.

Here's the call site in src/main.py, after the results are saved:

# Dataset first - this must succeed before anything touches Notion
await Actor.push_data(jobs)
notion_connector = actor_input.get('notionConnector')
notion_data_source_id = actor_input.get('notionDataSourceId')
if notion_connector and notion_data_source_id:
    try:
        created = await sync_jobs_to_notion(
            jobs,
            connector_id=notion_connector,
            data_source_id=notion_data_source_id,
        )
        Actor.log.info(f'✅ Wrote {created} jobs to Notion')
    except Exception:
        Actor.log.exception('Notion sync failed - dataset is unaffected')
else:
    Actor.log.info('No Notion connector selected - dataset output only')

The repository has the surrounding Actor lifecycle and job collection. The two guards that matter are both here: only call the sync when notionConnector and notionDataSourceId are both supplied, and never let it raise past the try.

Check tool responses as well as exceptions

A successful network request can still carry a failed tool result. In the Python SDK the flag is is_error, snake case, not the isError that most existing MCP writing shows, because most of that writing is TypeScript. Exceptions cover transport and protocol failures; is_error covers everything else, and you have to check it after every call. Miss it and every write looks like it worked while nothing appears in Notion.

Define this helper in src/notion_sync.py before the batch-writing function:

from typing import Any, Optional


def _tool_error_text(result: Any) -> Optional[str]:
    if not getattr(result, 'is_error', False):
        return None
    for block in getattr(result, 'content', None) or []:
        text = getattr(block, 'text', None)
        if text:
            return text
    return 'unknown error'

Once the session is initialized and the rows are built, call notion-create-pages with the data source ID as parent. This excerpt from sync_jobs_to_notion uses _chunk and PAGE_BATCH_SIZE (20) from the same file; CREATE_PAGES_TOOL is notion-create-pages, and created starts at zero:

for batch_no, batch in enumerate(_chunk(rows, PAGE_BATCH_SIZE), 1):
    response = await session.call_tool(
        CREATE_PAGES_TOOL,
        arguments={
            'parent': {'data_source_id': source_id},
            'pages': batch,
        },
    )
    error_text = _tool_error_text(response)
    if error_text:
        logger.error(f'Notion batch {batch_no} failed: {error_text}')
        continue
    created += len(batch)

Log the failed batches and count only the successful ones. And since the run can exit 0 with a broken sync, check the sync logs, not just the overall run status.

The Notion database after a run, populated with YC jobs.

Start with a small run and check a few pages against the dataset: job title, company, salary, destination. If your database only has a title column you'll get title-only pages, which is the same symptom as the empty-schema bug in this step with a far more boring cause. Add the optional columns before you expect to see values in them.

5. Verify the result and measure the overhead

Two seconds and about 5% more compute. That's what the Notion write cost me.

I ran the Actor over 20 companies at 512 MB, with and without the connector:

	Runtime	Compute units	Jobs	Notion pages
Without connector	38.7 s	0.00537	50	0
With connector	40.7 s	0.00566	50	50

The Apify run log showing tool discovery, the target columns, and the batch writes.

Because this Actor charges per result, that extra compute comes out of my margin. It doesn't raise the per-result price my users pay. At 5% I'll take that trade happily, but it's worth knowing which side of the ledger the overhead lands on before you ship a connector on a per-result Actor.

The 50 pages went out in three notion-create-pages calls, batched 20, 20, and 10. Those were the writes only; tool discovery and schema fetching were additional requests. One call per job would have meant 50 round trips and a much less pleasant table. It's a small comparison, so treat it as the overhead I measured rather than a guarantee for larger runs or other services.

The scrape output had the same shape in both runs: 20 companies, 50 jobs, 28 founders. The connector is optional. If nobody selects one, the Actor makes no MCP calls at all, and my existing users see exactly what they saw last month.

Before you schedule it, deal with duplicates

One thing I haven't solved yet. Every run appends new pages, so a daily schedule will re-add jobs that are already in the database.

There's no deduplication in the current implementation. If you want a persistent job tracker rather than a snapshot, comparing job URLs before writing is the next change to make, and I'd make it before setting up any schedule.

What I'd do differently

I'd read from Notion, not just write to it. Right now the filters live in the Actor input. It'd be better to keep a Notion database of the batches, industries, and locations I care about, have the Actor read its own task list at the start of a run, and write results back. A loop instead of a one-way pipe. The connector already permits it; I just haven't built it.

I'd also like to try two connectors on one Actor: Notion for the archive, Slack for a message when a job matches a saved search. The input schema accepts an array of connector IDs, with one MCP session per connector.

Reuse the Apify MCP connector pattern in another Actor

For a different destination, keep the same order: declare the tools, connect and save their descriptions, inspect the destination, map your results, and write only once the dataset is safe.

Another service will have its own connector, its own tool names, and its own payload format. Which is exactly why the discovery step in step 2 is worth doing first. It hands you those requirements up front instead of making you reverse-engineer them from error messages with {propertyKeys} in them.

FAQ

Can I add a connector to any existing Actor?

The Actor has to declare the connector input and contain code that calls it. For this scraper, select Notion and supply notionDataSourceId. Adding a connector in your account settings alone doesn't make an unrelated Actor write to Notion.

Can the Actor see my Notion token?

No. It authenticates to the Apify MCP proxy with its own run token, and the proxy adds the Notion credential server-side. The connector session ends with the run.

Can I restrict what the Actor can do?

Yes. Access has to satisfy the service's authorization permissions, any connector-level tool allowlist, and the Actor's mcpServers[].tools declaration. A tool allowlist limits operations, while the upstream service controls which content those operations can reach.

Can I develop this locally?

Not really. ACTOR_MCP_CONNECTOR_BASE_URL only exists in a platform run. I test row building and response parsing locally, then use a dev build to verify authorization and real tool calls.

Which SDK version do I need?

mcp 2.x for Python, which pulls httpx2 rather than httpx. The 1.x API differs in both the transport function's name and its signature.

Is this the Apify MCP server?

No. Here the Actor calls Notion through an outbound MCP connector. The Apify MCP server is the other direction: it lets external AI clients call Actors.

Code: the connector implementation is in src/notion_sync.py.

The Actor: YC Jobs Scraper on Apify Store.

Further reading

How to Get Reliable Options Chain Data in Python (When yfinance Keeps Failing)

Ahmed Jasarevic — Mon, 21 Sep 2026 14:57:30 +0000

TL;DR — Getting stock prices in Python is easy. Getting options chain data (strikes, expirations, implied volatility, open interest, volume) reliably and at scale is a different problem. This guide shows you how to pull clean, structured options data into a pandas DataFrame and build an options screener and IV dashboard on top of it — without fighting rate limits or rewriting your data layer every few weeks.

The problem nobody warns you about

You've probably built this pipeline before:

import yfinance as yf

aapl = yf.Ticker("AAPL")
chain = aapl.option_chain("2026-10-17")
calls = chain.calls
puts = chain.puts

It works beautifully in a notebook. Then you ship it.

And then, three weeks later, it returns empty DataFrames. Or throws a 429. Or works for 40 tickers and silently fails for the other 460. Or your nightly job just… doesn't have data this morning.

If you're building anything that depends on options data — a screener, a volatility dashboard, an alert bot, a backtest — the data layer is where projects die. Not the strategy. Not the model. The data layer.

This article is about fixing that layer once, so you can get back to building the interesting part.

Why options data is much harder than stock prices

Stock quotes are one number. Options chains are hundreds of contracts per ticker that each carry their own:

Strike price and expiration date
Contract type (call / put)
Implied volatility (IV)
Open interest and volume
Bid / ask / last price
In-the-money status
The underlying stock price at the moment of the snapshot

That's a lot of surface area. And unlike a simple price feed, options data is:

Large — a single liquid ticker can have thousands of contracts across all expirations.
Time-sensitive — IV and open interest shift throughout the trading day.
Painful to fetch at scale — you need hundreds of requests for a realistic watchlist.

Most free options data sources were never designed for this. They're fine for a demo with three tickers. They fall over the moment you point them at a real universe.

Why the free route breaks in production

If you've used yfinance or yahoo_fin, you already know the pattern. These libraries are excellent for exploration and research, and I genuinely recommend them for that. But they share a few characteristics that make production usage painful:

What you need	The free/DIY route
Predictable results at scale	Rate limiting appears as you scale up
Consistent schema across tickers	Fields shift; empty responses happen
Hundreds of tickers per run	You manage retries, throttling and pacing yourself
Fresh data on a schedule	Silent failures — you only notice when data is missing
Zero maintenance	Breaks when upstream changes

The core issue isn't that these tools are bad. It's that they're not a data pipeline. They're a convenience wrapper. When you build on top of them, you inherit 100% of the reliability burden — retries, backoff, session handling, proxy rotation, schema normalization, and monitoring.

That's a full-time engineering project. And it has nothing to do with the options strategy you actually want to build.

What "reliable" options data actually looks like

Before choosing a tool, get clear on the contract. You want each row to look like this — one row per option contract:

Field	Meaning
`ticker`	Underlying symbol (e.g. `AAPL`)
`symbol`	Full options contract symbol
`type`	`call` or `put`
`strike`	Strike price
`expiration`	Expiration date
`price`	Option contract price
`stockPrice`	Underlying stock price at scrape time
`iv`	Implied volatility
`volume`	Contracts traded
`openInterest`	Open contracts outstanding
`itm`	Whether the contract is in-the-money
`scrapedAt`	Timestamp of the snapshot

That's a clean, flattened schema — one row per contract, ready for a DataFrame, ready for CSV, ready for a database, ready for a chart. No nested JSON to unpack, no per-ticker field drift.

The goal is simple: you ask for tickers, you get back a tidy table.

Getting the data (in about 15 lines)

Instead of maintaining a scraper, you can call a managed data source that returns exactly the schema above. Here's the whole integration — a plain HTTP call, so it works from Python, Node, a cron job, or an agent:

import requests
import pandas as pd

APIFY_TOKEN = "YOUR_APIFY_TOKEN"
ACTOR = "ahmed_jasarevic~yahoo-finance-options"

resp = requests.post(
    f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
    params={"token": APIFY_TOKEN},
    json={
        "tickers": ["AAPL", "TSLA", "NVDA"],
        "limitPerTicker": 50,
    },
    timeout=300,
)
resp.raise_for_status()

df = pd.DataFrame(resp.json())
print(df.head())
print(f"{len(df)} contracts across {df['ticker'].nunique()} tickers")

That's it. No retry logic, no session management, no proxy setup, no per-ticker error handling.

Input options:

Field	Type	Required	Default	Notes
`tickers`	array	✅	—	Symbols to fetch, e.g. `["AAPL", "TSLA"]`
`limitPerTicker`	integer	❌	`20`	Number of call + put contracts per ticker
`proxyConfiguration`	object	❌	Apify Proxy	Residential proxy recommended for larger runs

A couple of things worth knowing before you build on it:

limitPerTicker controls cost and speed. Start at 20–50 while developing, raise it when you go live.
The output is one row per contract, so 50 tickers × 50 contracts = 2,500 rows. Size your DataFrame accordingly.
Pricing is per result (roughly $1.20 per 1,000 contracts at the free tier, and lower as your Apify plan tier goes up). A 10-ticker daily screener is fractions of a cent per run.

You can also run it interactively from the Store page if you just want to eyeball the output first:
👉 Yahoo Options — Chains, IV, Live

Use case 1 — Build an options screener

Now the fun part. With a clean DataFrame, a screener is genuinely a few lines.

Let's find liquid calls — high open interest, real volume, and an IV band that filters out both dead contracts and lottery-ticket noise:

import pandas as pd

# Normalise types
df["iv"] = pd.to_numeric(
    df["iv"].astype(str).str.replace("%", "", regex=False),
    errors="coerce",
)
df["openInterest"] = pd.to_numeric(df["openInterest"], errors="coerce")
df["volume"] = pd.to_numeric(df["volume"], errors="coerce")

calls = df[df["type"].str.lower() == "call"].copy()

screened = calls[
    (calls["openInterest"] > 500) &      # real liquidity
    (calls["volume"] > 100) &            # actually trading today
    (calls["iv"].between(15, 60)) &      # avoid dead + lottery contracts
    (calls["itm"] == False)              # out-of-the-money only
].sort_values("openInterest", ascending=False)

print(screened[["ticker", "strike", "expiration", "iv", "openInterest", "volume"]].head(10))

That table is the beginning of a real tool. From here you can add your own filters: distance from spot, days-to-expiration windows, spread width, IV rank against history — whatever your strategy needs.

Because the schema is stable, your screener logic never has to change when the data source does. That's the whole point.

Use case 2 — Track implied volatility over time

IV is the single most-watched number in options. But IV only becomes useful when you have history — today's IV means nothing without yesterday's to compare it to.

The pattern is: snapshot on a schedule → store → chart.

# Append each run to a CSV (or swap in SQLite / Postgres / DuckDB)
df["scrapedAt"] = pd.to_datetime(df["scrapedAt"], errors="coerce")
df.to_csv("iv_history.csv", mode="a", header=False, index=False)

Then compute a simple average IV per ticker over time:

history = pd.read_csv("iv_history.csv")
history["scrapedAt"] = pd.to_datetime(history["scrapedAt"])

trend = (
    history.groupby(["ticker", "scrapedAt"])["iv"]
    .mean()
    .reset_index()
    .sort_values("scrapedAt")
)

# A 30-point IV jump on a single name is worth looking at
latest = trend.groupby("ticker")["iv"].last()
previous = trend.groupby("ticker")["iv"].nth(-2)
spikes = (latest - previous).sort_values(ascending=False)
print(spikes.head())

You've now got the foundation of a volatility alert system — the thing that actually tells you when something interesting is happening in the market, instead of you refreshing a page all day.

Use case 3 — Automate it and stop thinking about it

The real value kicks in when this runs without you. Two options:

Scheduling: Apify supports cron-style schedules, so you can run the same configuration every market day at a fixed time — pre-market, at the open, or at the close.

Piping the output anywhere: Because the output is clean JSON, you can push it straight into Google Sheets, a webhook, Make/Zapier, your own database, or an AI agent that summarises what changed.

A practical setup for a watchlist monitor:

Schedule a run each trading morning with your ticker list.
Append results to your storage.
Diff against yesterday's snapshot.
Alert when IV, open interest or volume moves beyond a threshold you care about.

That's a monitoring product you'd otherwise spend a month building. It's now a config file and a schedule.

How this compares to a DIY pipeline

Capability	DIY free library	This managed actor
Get options chains	✅	✅
Flattened, stable schema	⚠️ varies	✅ one row per contract
Scale to hundreds of tickers	⚠️ rate limits	✅ built for it
Retries / throttling handled	❌ you build it	✅
Runs unattended on a schedule	⚠️ fragile	✅
Maintenance when upstream changes	❌ ongoing	✅ none
Cost per 1,000 contracts	"free" + your time	~$1.20

The honest framing: free libraries are great for learning and one-off research. A managed data source is what you use when the data needs to be there every single time — because you're shipping a product on top of it.

Frequently asked questions

Is there a free Yahoo Finance options API?

Yahoo shut down its official public API in 2017. Everything you see today — including yfinance and similar libraries — relies on unofficial endpoints, which is exactly why reliability varies over time. For production use, most developers move to a managed data source rather than maintaining an unofficial one themselves.

Why does yfinance return empty options data?

It usually comes down to rate limiting, throttling, or upstream changes. When you request many tickers quickly, you can hit limits that return empty or partial results instead of a clear error — which is the worst kind of failure, because your pipeline looks healthy while quietly producing nothing.

How do I get options chain data in Python?

Two routes: use a library that wraps an unofficial source (fast to start, fragile in production), or call a managed API that returns structured JSON you load straight into a DataFrame. The second route removes the maintenance burden entirely — the snippet above is the whole integration.

Can I get implied volatility and open interest, not just prices?

Yes — and those are usually the fields that matter most. Implied volatility tells you what the market expects, and open interest tells you where the real money is positioned. Any useful options tool is built on those two numbers far more than on raw contract price.

How much does it cost to pull options data at scale?

The actor is billed per result, roughly $1.20 per 1,000 contracts at the free tier and cheaper at higher plan tiers. A daily 10-ticker screener at 50 contracts each is 500 contracts — well under a cent per run.

Can I use this with an AI agent or LLM?

Yes. Because the output is structured JSON and the integration is a single HTTP call, it works cleanly as a tool for an LLM agent — useful for things like "summarise the biggest IV moves in my watchlist today."

Does it work for any ticker?

It's built for US-listed tickers with options chains. Set tickers to your watchlist and run it.

The takeaway

Options data doesn't have to be the fragile part of your project.

The pattern that works:

Get a stable, flattened schema — one row per contract.
Build your logic on top of it — screeners, IV tracking, alerts.
Schedule it so it runs without you.
Never maintain a scraper again.

Your edge is in what you do with the data. Everything upstream of that should be boring, predictable and boring again.

👉 Get started: Yahoo Options — Chains, IV, Live on Apify

Run it once in the UI to see the output, then drop the 15-line snippet into your project and get back to building.

If this saved you an afternoon of rate-limit debugging, drop a reaction — and tell me in the comments what you're building with options data. Screener, alert bot, or something weirder?

How I Built a Linktree, Beacons & Bio Email Scraper with Apify

Ahmed Jasarevic — Sun, 20 Sep 2026 20:42:11 +0000

1. Introduction

Collecting emails from Linktree, Beacons, and other bio links is often a tedious and time-consuming task. As a developer and freelancer working with lead generation projects, I faced this problem firsthand: manually navigating hundreds of profiles just to extract a few emails is inefficient and error-prone.
To solve this, I built a Linktree, Beacons & Bio Email Scraper Actor using Apify. It automates the process, reliably collects emails from multiple platforms, and provides structured outputs for marketing, data engineering, and business intelligence purposes.
This article details the journey: the challenges I faced, the design of the Actor, lessons learned, and tips for anyone looking to build production-grade scraping solutions.

2. Problem Context & Motivation

Manual email collection from bio links presents several challenges:

Dynamic pages: Platforms like Linktree often use JavaScript to load content asynchronously. Simple HTML scraping fails here.
Anti-scraping measures: Repeated requests can trigger rate limits or captchas.
Scale: For real lead-generation projects, you often need hundreds or thousands of emails daily—manual collection is not an option. I needed a solution that was fast, scalable, and reliable, and that could handle these technical challenges without breaking.

3. Actor Overview

Architecture and Workflow
The Actor is built on Apify and Crawlee, combining headless browser automation with structured data extraction. Here’s a high-level workflow:

Input: A list of Linktree, Beacons, or other bio URLs.
Navigation: Actor launches a headless browser to visit each URL.
Email Extraction: Uses DOM selectors and regex patterns to identify valid email addresses.
Anti-blocking: Rotates proxies and applies request throttling to avoid detection.
Output: Saves results in JSON or CSV with details: URL, email, timestamp.

Input, Output, and Configuration Options

Input: List of profile URLs (CSV, JSON, or manually typed).
Output: JSON, CSV, or Google Sheets integration.
Configurable Options: Maximum pages per run Timeout per request Proxy rotation (on/off) Output file format This flexibility allows the Actor to handle both small projects and large-scale campaigns.

4. Implementation Details

Handling Dynamic Content
Linktree and Beacons often load content via JavaScript. Initially, I tried simple HTTP requests, but emails were missing in the HTML response. Switching to Crawlee with Playwright solved the issue:

import { PlaywrightCrawler } from 'crawlee';

await crawler.run(['https://linktr.ee/example']);

This approach ensures the Actor captures all visible emails, even on dynamic pages.
Anti-Blocking Strategies
During early tests, some accounts triggered rate limits. To fix this, I added:
Proxy rotation
Randomized delays between requests
Error retries for failed pages
These measures increased the success rate to 99.6% over thousands of URLs.

5. Metrics & Results

To evaluate the Actor in a real-world scenario, I ran it against a dataset of 1,800+ URLs. The results showed strong reliability while keeping the cost of each run predictable:

99.6% success rate
1,816 emails collected
$14.47 total cost per run
$23.76 profit generated per run from the collected leads

These results demonstrated that the Actor could process large batches of URLs reliably while remaining cost-effective for lead-generation workflows.

This shows the Actor is not only technically reliable but also economically valuable for lead-generation workflows.

6. Lessons Learned

Building this Actor taught me several important lessons:
JavaScript rendering matters: Always test pages in a headless browser when dealing with dynamic content.
Anti-blocking is critical: Even simple rotation and throttling drastically improve success rates.
First-person debugging insights: Logging actual page content during development helped identify hidden issues.
Scalable design: Structuring input/output for batch processing makes the Actor production-ready.

7. Who Can Benefit

This Actor is useful for:

Developers looking to automate repetitive data collection tasks
Marketers and sales teams needing up-to-date email lists
Data engineers building pipelines that integrate multiple data sources
The architecture can also be adapted for other bio link platforms or email collection projects with similar challenges.

8. Conclusion

Automating email extraction from Linktree, Beacons, and bio links is no longer a manual nightmare. By using Apify and Crawlee, I built a reliable, scalable, and cost-effective Actor that delivers real-world results.
If you plan to build your own Actor, remember: focus on handling dynamic content, preventing blocks, and structuring your data pipeline. Sharing these lessons ensures other developers can build production-grade automation with confidence.

9. Next Steps

Test the Actor on new bio platforms
Add AI-powered validation to filter incorrect emails
Explore integrations with CRMs and email marketing tools

A default MCP connection hands an agent 11 tools. None of my 23 Actors is one of them.

Ai-Q Labs — Fri, 18 Sep 2026 00:10:09 +0000

A default MCP connection hands an agent 11 tools. None of my 23 Actors is one of them.

I have 23 audit Actors on the Apify Store. They all do a version of the same thing: take a public record, check whether what it still claims is true, and write the verdict into a dataset.

I have also written six articles about what happens when an AI agent calls one of them through the Apify MCP server. Output fields that arrive empty. Input schemas that read differently to a form and to a tool. A default that quietly fills a required field. A connector that shows 44 tools and grants four.

MCP is the Model Context Protocol, the interface that turns an Actor into a tool that clients like Claude and Cursor can call on their own. Every one of those six articles assumed the agent had that tool in hand. Last week I checked the assumption for the first time. It does not hold by default.

The connection string I never read

I found the problem in my own machine before I found it anywhere else. This is the Apify entry in my Claude config, verbatim except that there is no token in it:

"url": "https://mcp.apify.com?tools=aiqlabs/github-repository-audit"

I wrote that line months ago to test one Actor in isolation, and then left it. My editor session had five Apify tools in it. One of them was an Actor, and it was the one named in that string.

So I had spent six articles reasoning about how agents see my catalogue. The session I was reasoning from had been narrowed to a single Actor, by me, on purpose, and then forgotten. The first thing worth measuring was what a connection with nothing in it returns.

What a default connection actually sends

Twenty lines, no dependencies. This is the whole test:

// reach.mjs — what does a default connection to the Apify MCP server actually send?
// Run: APIFY_TOKEN=... node reach.mjs
const URL = `https://mcp.apify.com/?token=${process.env.APIFY_TOKEN}`;
const HEAD = { 'content-type': 'application/json', accept: 'application/json, text/event-stream' };

async function rpc(body, sid) {
    const res = await fetch(URL, { method: 'POST', body: JSON.stringify(body),
        headers: sid ? { ...HEAD, 'mcp-session-id': sid } : HEAD });
    const text = await res.text();
    // the server answers as SSE, so the JSON sits on a "data:" line
    const line = text.split('\n').find((l) => l.startsWith('data:'));
    return { sid: res.headers.get('mcp-session-id') ?? sid, json: line ? JSON.parse(line.slice(5)) : null };
}

const { sid } = await rpc({ jsonrpc: '2.0', id: 1, method: 'initialize', params: {
    protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'reach', version: '1' } } });
await rpc({ jsonrpc: '2.0', method: 'notifications/initialized' }, sid);

const { json } = await rpc({ jsonrpc: '2.0', id: 2, method: 'tools/list' }, sid);
const tools = json.result.tools;
console.log(`tools: ${tools.length}`);
for (const t of tools) console.log(' ', t.name);

Its output, on 12 August:

tools: 11
  search-actors
  fetch-actor-details
  call-actor
  get-actor-run
  get-dataset-items
  get-key-value-store-record
  abort-actor-run
  search-apify-docs
  fetch-apify-docs
  report-problem
  apify--rag-web-browser

Ten of those are the server's own tools. One is an Actor: RAG Web Browser, which belongs to Apify. Of the 23 Actors I have published, zero are in the list.

Nothing is being hidden. Add ?actors= to the same URL and name all 23 slugs. The same server then returns 27 tools: my 23 plus four storage tools. The Actors are reachable. They are not default.

The default is also defensible. The Store holds thousands of Actors. A server that turned every one of them into a tool definition would blow out the context window of every client that connected. Eleven tools is a budget, not a snub.

But it does mean that everything I had written was conditional on a step I had never measured.

Three ways an Actor reaches an agent

Route	Who decides	What an author can do about it
A — named in the connection URL (`?actors=` / `?tools=`)	whoever configures the agent	Almost nothing. They have to know your slug before they can type it.
B — included in the default set	Apify	Nothing. Today that set contains one Actor.
C — found at runtime by `search-actors`	Store search ranking	This one.

A and B are not work I can do. C is the entire surface, so I measured it.

Measuring the third route

The first two attempts failed on the parameter name. {"search": …} and {"query": …} are both accepted and both silently ignored. The server echoes Search query: back empty and returns its default list. Rather than guess a third time, I read the tool's own inputSchema out of the JSON-RPC reply to tools/list above:

keywords : string   default ""
limit    : integer  default 5, max 10
offset   : integer  default 0

The parameter is keywords. But the line that matters more is limit. It defaults to 5 and caps at 10. That is the window. An agent that runs one search and picks from it is choosing among five Actors out of the whole Store.

The second trap is in the response. Each result set is introduced by a line like **Number of Actors found:** 7, which reads as a total and is not one. It is the size of that page:

Instagram keeps producing fresh results at offset 0, 10, 20 and 30 — 39 distinct Actors, no repeats. One of those four pages came back with nine rather than ten, which is the second thing to know: the pages are ragged.
Chrome extensions returns 7 at offset 0 and a different 6 at offset 10. A short page does not mean the list ended.

My first pass treated the short page as the end of the list, so it never looked past page one. It reported seven of my Actors as absent when they were sitting at ranks 12, 16, 31, 41, 52, 55 and 57. Every number below comes from the corrected run. That run pages to rank 100 whatever the page count says, and computes rank as offset + position in page, because the pages come back ragged.

What it found

I paired each of 23 Actors with the query it was built for — 28 queries in all, since several Actors got both a specific phrase and the broader term a user might type instead. Then I asked how deep in the results that Actor sits.

Where my Actor lands	queries	share
inside the default 5 the agent receives	6	21%
inside the maximum 10	7	25%
rank 11–100: indexed, past the window	7	25%
not in the first 100	14	50%

On the queries these Actors were written for, an agent doing a default search finds one of them about one time in five. Half of them are not in the first hundred results at all.

That was worse than I expected, but it was not the finding. The finding is which half.

The queries I can win are the queries nobody is making

Every search-actors result carries the Actor's usage with it, in a line like **Stats:** 20 total users, 3 monthly users. So the demand on a query is readable from the same response that gives you the ranking: take the 30-day user count of whatever sits at rank 1.

my rank	queries	median 30-day users of that query's rank-1 Actor
reachable (top 10)	7	6
unreachable (past 10, or absent)	21	23

The unreachable side is where the users are. App Store — 311. Google Play — 295. article extractor — 271. Shopify — 186. My Actors for the first three of those are not in the first hundred results.

And the one query where I rank #1 is sitemap checker, where the rank-1 Actor has 0 users in the last thirty days. That Actor is mine. I am first on a shelf with no traffic, and invisible on every shelf with traffic.

I had measured the same shape once before, from the other side. On 2 August I sampled the Store itself and found that Actors naming a well-known platform had a median of 5 monthly users against 2 for the rest. I read that as an argument for naming platforms. This measurement says the platform-named shelves are exactly the ones where I cannot be seen — 1 reachable query out of 14, against 6 out of 14 for the generic ones. Both results are true and they are not in conflict: platform names are where demand collects, demand attracts competitors, and competitors fill a window five deep. But I predicted the wrong one.

The tool tells the agent to broaden the search, and broadening removes me

Every search-actors response ends with this, addressed to the agent:

IMPORTANT: You MUST always do a second search with broader, more generic keywords (e.g., just the platform name like "TikTok" instead of "TikTok posts") to make sure you haven't missed a better Actor.

That is good advice for the user. It is also an instruction to run the query I do worst on. Same Actor, specific phrase against the broadened phrase:

specific	rank	broadened	rank
`sitemap checker`	#1	`sitemap`	not in first 100
`PDF tables`	#2	`PDF`	not in first 100
`PDF text markdown`	#2	`PDF`	not in first 100
`Chrome extensions`	#16	`Chrome Web Store`	not in first 100
`Shopify apps`	#41	`Shopify`	not in first 100
`GitHub repository`	#52	`GitHub`	not in first 100
`App Store apps`	absent	`App Store`	absent
`Google Play apps`	absent	`Google Play`	absent

Six worse, none better, two unchanged. The clearest one is the first: sitemap checker puts me at #1, and deleting one word puts me past rank 100. Whatever advantage a precise name buys, the agent is under instructions to take a second look without it.

What decides the order

Not popularity, at least not alone. Across every first page I collected, 157 of 238 adjacent pairs are in descending order of monthly users — 66%. A pure popularity sort would be 100%. Relevance carries real weight, which is how an Actor with no users reaches #1 on sitemap checker.

Apify documents what does decide it, and says so for both surfaces at once. From Actor quality score:

Actors with higher quality scores tend to rank higher on both surfaces, though no specific position is guaranteed.

The second surface named on that page is "the Apify MCP server search-actors tool used by external AI agents". The score aggregates eight categories: reliability, popularity, feedback and community, ease of use, pricing transparency, trustworthiness, history of success, and congruency of texts.

I have moved it once and can report the size of the move. On 2 August I added dataset output schemas across the catalogue; the score on the Actor I watched went from 74 to 78, and after that pass the whole catalogue sat between 78 and 81. There is no API for the number — I tried four endpoints and got 404 from each — so the only place to read it is Console > Insights > Actor quality.

Here is that panel today, for the Actor this article keeps returning to:

79 out of 100. Better than 71% of Actors on the platform, better than 99% on reliability, and one suggestion left in the panel.

That is the pairing worth sitting with. Console rates this Actor above 71% of the platform and has one cosmetic item left to suggest. The same Actor is absent from the first hundred results on every query with real demand behind it, and its one first place is on a query with none. Both readings are correct, and the documentation is careful enough to allow it: a higher score tends to rank higher, with no position guaranteed. On a crowded shelf, tending is not the same as landing in the five results the agent receives.

Which leaves the honest part. Three of the eight categories — popularity, feedback and community, history of success — are things you get from having users. For an Actor with none, part of the ranking that would bring users is held shut by not having them yet. That is not a complaint about the design; a store that ranked unproven tools first would be worse for everyone using it. But it is the actual shape of the problem, and no amount of description polishing changes it.

What I would tell an Actor author

Check what your connection sends before you tune anything. The 20 lines above answer it. If your Actor is not in the list, none of your tool-definition work is reaching an agent yet.
Measure your rank on the query you were built for, with limit: 10, paging past the first short page. Anything past rank 10 is in the index and outside the window.
Read the demand off the same response. The rank-1 Actor's monthly users are printed next to it. If that number is 0, being #1 buys nothing — I have the receipt.
Expect the broad term to be worse. It is also where the demand is. Decide which of those two facts you are building for, rather than discovering the trade after you publish.

What I got wrong

Three things, in the order I found them.

I narrowed my own MCP connection to one Actor and then wrote six articles about how agents see my catalogue. The narrowing was in a config file I had not opened in months.

I read a page count as a total, and my first pass reported seven Actors as unreachable that rank between 12 and 57. Had I not checked whether offset kept returning results, this article would have carried seven false claims of the most flattering kind — the kind where the platform looks worse than it is.

And I expected the platform-named Actors to be the findable ones. They are the least findable ones I own.

None of that makes the earlier six articles wrong. It reorders them. An input schema that misleads an agent, an output schema that arrives empty, a default that fills a required field — all of those are real, and all of them start to matter at the moment the agent receives the tool. For 23 out of 23 of mine, that moment does not happen by default.

I pinned my Actors to the MCP URL, the way the docs recommend. The agent lost every way to see what a call costs.

Ai-Q Labs — Fri, 18 Sep 2026 00:09:29 +0000

I pinned my Actors to the MCP URL, the way the docs recommend. The agent lost every way to see what a call costs.

I publish 23 Actors on the Apify Store. All 23 are pay-per-event.

Last week I measured how an AI agent actually reaches them. The answer was uncomfortable. A
default MCP connection hands an agent 11 tools. None of mine is one of them.

So you pin. You name your Actors in the connection URL. The
Apify MCP documentation tells you to:

For production deployments, explicitly specify which tools to load rather than relying on
defaults. This ensures consistent behavior across updates.

That is good advice for reliability. I followed it. Then I measured what the agent receives
afterwards, and found something the docs do not mention.

Pinning removes the agent's ability to see what a call costs. It also removes its ability to
cap what a call spends.

Below are the measurements. Then the three tools that disappear. Then the half of this that is
my own fault rather than the platform's.

What pinning actually changes

I connected three ways and listed the tools each time. Same account, same day, 2026-08-12.
Every listing came from a tools/list call over the
MCP HTTP transport. That is what the client
library hands the model.

Default connection, no parameters. Eleven tools:

search-actors          get-key-value-store-record   report-problem
fetch-actor-details    abort-actor-run              apify--rag-web-browser
call-actor             search-apify-docs
get-actor-run          fetch-apify-docs
get-dataset-items

Pinned with two of my Actors, using the documented parameter
?tools=aiqlabs/seo-audit-tool,aiqlabs/pdf-inspector. Six tools:

aiqlabs--seo-audit-tool    get-actor-run                 abort-actor-run
aiqlabs--pdf-inspector     get-dataset-items
                           get-key-value-store-record

Pinned with all 23. Twenty-seven tools: my 23, plus the same four.

The four survivors are all post-run tools. Each one reads a run, reads a dataset, reads a
key-value record, or aborts something already running.

Every tool that helps an agent decide before it calls is gone. Three of the missing ones
matter for money.

The three tools that carried the price

I searched the full JSON of each default tool definition for pricing language. These are the
strings that came back.

search-actors returns pricing in its results. Its own description says so:

- **Pricing:** Details with pricing link

Its output schema enumerates the models. FREE, PRICE_PER_DATASET_ITEM,
FLAT_PRICE_PER_MONTH, and a pricePerUnit field.

fetch-actor-details takes a flag whose description reads, in full:

Include pricing model and costs.

call-actor is the one that actually protects the caller. It accepts maxTotalChargeUsd:

Pay-per-event Actors only — ignored otherwise.
Caps total USD billed; does NOT limit work. Prefer the Actor's own input fields to bound work.

That is a spending ceiling. The caller sets it, at call time. Note the second sentence. I will
come back to it, because it is the sentence that indicts me.

All three tools are in the default set. None of them survives pinning.

An agent holding my Actors directly cannot look up what they cost. It cannot put a ceiling on
what it spends. It can only call them.

What my own definitions tell the agent about price

Nothing. I checked, and I checked wrong the first time.

My first pass searched each tool definition for a dollar sign. It reported a hit on 23 of 23.
For about a minute I believed my descriptions already carried prices.

They do not. The dollar signs were $id, the JSON Schema keyword:

"required":[],"$id":"https://apify.com/mcp/aiqlabs--app-store-audit"

I searched again for a dollar sign followed by a digit. An actual amount. The count was
0 of 23.

The words price and cost do appear in a few of my definitions. Every one of them is about
the subject matter, not the invoice.

One Actor notes that prices and availability differ by storefront. Another explains that each
archived URL costs a request to the live site.

Those are useful sentences. Neither tells an agent what calling the tool will cost.

Meanwhile every one of the 23 is metered. Each has two charge events, configured the way
Apify's monetization docs
describe:

Event	Price
`apify-actor-start`	$0.00005
`apify-default-dataset-item`	$0.002 – $0.01 depending on the Actor

The per-item price is the one that moves.

I wanted to be sure those events fire on their own. My Actors never call a charge function. So
I read a real run record rather than assuming:

run thLehE7kk3NVRAvqm  SUCCEEDED
chargedEventCounts: {"apify-actor-start":4,"apify-default-dataset-item":608}

They fire. Another run in the same batch recorded 1,000 dataset items. At my highest per-item
price, one call like that is ten dollars.

The part that is my fault

Go back to that sentence in call-actor:

Prefer the Actor's own input fields to bound work.

Apify is right about this. A billing cap stops the invoice, not the work. The real bound
belongs in the
input schema,
where the author controls it.

So I checked whether my own Actors carry that bound.

All 23 have a field for it. maxUrls, maxPages, maxApps, maxPdfs, maxVehicles,
maxOrganizations, maxStories. Twenty-three out of twenty-three.

Not one of them declares a JSON Schema maximum.

pdf-inspector        maxPages(none), maxPdfs(none), maxFileMb(none), maxLinksToCheck(none)
sitemap-checker      maxUrlsToCheck(none), maxSitemaps(none), maxDepth(none)
http-status-checker  maxUrls(none), maxRedirects(none), maxConcurrency(none)

I named the fields as if they were limits. I never made them limits.

An agent can pass maxUrls: 1000000. The schema will accept it. I told the schema that any
integer is fine.

So the failure has two halves. Only one of them is the platform's.

Pinning removes the caller-side ceiling. That is the platform's shape. My Actors have no
author-side ceiling. That is mine, in 23 files I wrote myself.

An agent holding my pinned tools has no cap available from either direction.

What I got wrong on the way here

I started this measurement expecting a different problem.

My hypothesis was time. Actor runs are slow. Agents time out waiting. Authors ship tools an
agent cannot practically await.

I pulled the run history for all 23 and computed durations from the API.

	Value
Median of per-Actor medians	4.05 s
Slowest single run observed	58.0 s
Actors with a median above 60 s	0 of 23

There is no timeout problem. My hypothesis was wrong.

I dropped that half of the article rather than stretching the numbers to fit it. Only five of
the 23 had enough run history to produce a median, and that limit is worth stating. But nothing
in the data pointed toward latency. I would rather report a dead hypothesis than a decorated
one.

What to change, concretely

Three changes. The first two are the author's job. The third is a question for the platform.

1. Put a real ceiling in the input schema.

A field named maxPages should refuse an absurd value. The
maximum keyword does
this before your code runs:

"maxPages": {
  "title": "Max pages per PDF",
  "type": "integer",
  "default": 50,
  "minimum": 1,
  "maximum": 2000,
  "description": "Hard cap. Each page produces one billed result row."
}

default is not a limit. prefill is not a limit either, and the two are easy to confuse in
Console. That distinction bit me once already, from the correctness side. This is the same
lesson arriving from the money side.

You can verify the deployed schema rather than trusting the file:

curl -s https://api.apify.com/v2/acts/<user>~<actor> \
  | grep -o '"maximum":[0-9]*' | head

2. Say the price in the description the agent reads.

Once you pin, the agent has no other source. One clause is enough:

Each result row is one billed event ($0.002). A 500-URL input bills roughly $1.00.

That sentence costs you nothing. It is the only pricing signal a pinned agent will ever see.

3. The platform question.

call-actor exists in the default set and carries maxTotalChargeUsd. Pinned tools are
invoked directly, so no wrapper carries it.

A per-call spending cap that survived pinning would close the gap. It could be an optional
argument on every metered tool, or a connection-level parameter. Either way it would spare each
author from rediscovering this alone.

I do not know whether that is on anyone's roadmap. I am reporting the shape I measured.

Where this leaves an Actor author

Two connection modes, and a cost to each.

Leave the default. The agent can search, can read your pricing, can cap its spend. It will also
almost certainly never receive your Actor. The default set is 11 tools and none of them is
yours.

Pin your Actors. The agent receives them and calls them directly. It does so without a price
and without a ceiling.

I do not think either mode is wrong. I think the second one quietly moves a responsibility onto
the Actor author. The docs that recommend it do not mention the move.

Until they do, the fix is the schema you already control. Mine was missing it in all 23 files.
I only found out because I went looking for a different problem entirely.

Measurements were taken on 2026-08-12 against mcp.apify.com with a single account. Tool
listings came from tools/list over the MCP HTTP transport. Run durations and charge counts
came from
Apify API run records. No Actor was executed to
produce these numbers, so nothing here cost anything to measure.

I removed the under maintenance label from my Actor twice. It came back in four hours, then in one.

Ai-Q Labs — Fri, 18 Sep 2026 00:08:48 +0000

I removed the under maintenance label from my Actor twice. It came back in four hours, then in one.

I publish 23 Actors on Apify. Twenty-two of them take a public record and check whether what it
still claims is true.

The twenty-third is different. It reads an audit dataset and opens a GitHub issue for each finding.
It reaches GitHub through an Apify MCP connector,
so the Actor never handles my token.

That difference is the whole story. Two days after I published it, Apify flagged it as under
maintenance. I spent an afternoon removing the label by hand and learned that the label is not a
setting. It is a conclusion.

The email

The notification arrived at 10:08 on 8 August. My Actor had failed the automated tests.

Apify runs a daily test on every Actor.
The documentation is precise about what it does:

The test runs the Actor with its default input (defined by the prefill option in the input schema
file) and expects it to finish with a Succeeded status and non-empty default dataset within 5
minutes of the beginning of the run.

My Actor cannot do that. Not on a bad day — on any day.

Why this Actor and not the other 22

My other Actors take a URL, a domain, or a package name. Every one of those is a string. A string
goes in the prefill of the input schema, the test picks it up, and the run succeeds.

An authorized connector is not a string. It is a grant, tied to an account, held by the platform.
There is no value I can put in prefill that gives the test account access to my GitHub
installation. Nor should there be.

So the run fails at the first step, every day, forever.

The documentation says this outright, in a section I had not read:

Actors that require some sort of authentication will always fail the tests despite being fully
functional.

I want to be exact about what this means, because I got it wrong at first. I had assumed the cost
belonged to my design choice of failing fast on empty input. It does not. Twenty-two of my Actors
fail fast on empty input and none of them were flagged. The cost belongs to a narrower category:
Actors whose only successful path requires a credential the test account cannot hold.

If that describes your Actor, the daily test is not a quality signal about your code. It is a
structural mismatch, and it will not resolve itself.

What I did instead of reading the docs

I went to Console. Under Publication → Display information there is an Actor status control with an
"Under maintenance" switch. I turned it off and saved. The banner disappeared. The public page
stopped showing the badge.

I recorded it as fixed. It was not fixed. It was hidden.

At 17:08 the same day, about four hours after I cleared it, the second notification arrived. The
label was back.

I removed it again. At 18:08 — inside an hour — the third notification arrived.

Two removals, two returns, and the interval got shorter. That second number is what made me stop
and read.

The label is not a setting

Here is the mechanism, and it is stated plainly in the same documentation page I had skipped:

If the Actor fails to complete successful runs for three consecutive days, the developer will be
notified, and the Actor will be labeled under maintenance until it is fixed.

And the other direction:

The best course of action is to fix the Actor and rebuild it. The automatic testing system will
pick this up within 24 hours and mark it as healthy. In some cases, your Actor might break because
of issues with the target website. In such a case, if your Actor passes the majority of test runs
in the next 7 days, it will be marked as healthy automatically.

Read those two together. The label is derived from your run history, and the system recomputes
it. The switch in Console writes to a field that an evaluator overwrites on its own schedule.

For a normal broken Actor, the switch is honest and useful. You fix the bug, you rebuild, you clear
the label, and the next evaluation agrees with you. Your manual action and the computed state
converge.

My case had no convergence. The failures were still in the window and would be there tomorrow too.
Every removal I made was a claim the evidence contradicted, so the evaluator restored it.

I also stopped for a second reason, which has nothing to do with mechanics. Repeatedly toggling a
flag that an automated system keeps restoring is not a fix. It is an argument with a robot, in
public, on my own store listing. I did not want to find out how that reads from the other side.

The path that actually worked

The documentation tells you where to go:

If that's the case with your Actor, contact support and explain your specific use case that
justifies why the Actor should be excluded from the automated tests.

There is also a form. While my Actor was flagged, Console opened a skip-test request straight from a
query parameter:

https://console.apify.com/actors/<actorId>/publication?showSkipTestForm=true

One caveat I can only report, not explain. Two days later, with the exemption granted, that same URL
renders the ordinary publication page for me. So treat it as something that is there while you are
flagged, not as a permanent entry point. If it does not open, the documented route is support, and
that is the route the docs tell you to take anyway.

I filled it in with three sentences: what the Actor does, why the test account cannot authorize the
connector, and the sentence from the docs that describes exactly my situation. Quoting their own
documentation back to them felt lazy. It was the right call — it turned a request for special
treatment into a request to apply a rule that already existed.

The reply came in about 31 hours:

After reviewing your situation, we've approved your request. Your Actor will now not be checked by
our tests.

Then I cleared the label a third time. This one held, because there is no longer an evaluator
producing a verdict to overwrite it.

The order matters and I had it backwards. Get the exemption first. Clear the label second. Doing
it the other way round produces exactly what I got: a label that returns on a schedule you do not
control.

How to check the flag without lying to yourself

While I was doing this I needed a reliable way to answer "is it flagged right now?" I tried three
and only one of them is trustworthy.

The Console banner is fine, but it is one Actor at a time and it needs a browser.

The public Store page is not reliable. After my third removal I fetched the page and grepped for
the phrase. The badge was still in the HTML, on a page that was already correct in Console. When I
checked again later the same day, it was gone. So the page is not wrong — it lags, and on that day
the lag was hours. If you grep the page right after a change, you will read your own stale render
and conclude the change failed.

The API is the source of truth. The Actor object carries a top-level notice field:

curl -s "https://api.apify.com/v2/acts/aiqlabs~dataset-to-github-issues" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['notice'])"
# NONE

The values I have observed are UNDER_MAINTENANCE and NONE. This endpoint answers for public
Actors without a token, which means you can check anyone's — including, before you fork it, the one
you are about to depend on.

I did not find notice documented as the maintenance flag. I found it by reading the object.

A check you can run on everything you publish

After this, I stopped trusting my memory of which Actors were healthy. This prints the flag for a
list of slugs:

#!/usr/bin/env bash
# Print the maintenance flag for every Actor you name. No token needed for public Actors.
USER="aiqlabs"
for slug in dataset-to-github-issues github-repository-audit pdf-table-extractor; do
  notice=$(curl -s "https://api.apify.com/v2/acts/${USER}~${slug}" \
    | python3 -c "import json,sys; d=json.load(sys.stdin).get('data') or {}; print(d.get('notice','MISSING'))")
  printf '%-32s %s\n' "$slug" "$notice"
done

I ran it across the Actors I care about after the approval. All NONE, including the twenty-two
that were never at risk. That last part matters: it confirmed the flag had not spread from a shared
cause.

The trap next door

One warning that cost me three rounds of rework, because it sits on the same Console screen.

The Actor status switch lives under Publication → Display information. Saving that section dropped
one of my categories, three times in a row.

I had set three categories through the API. After each save, OPEN_SOURCE was gone, and I put it
back through the API.

Three for three is not an accident. But I want to be exact about the limit of what I know: when I
opened that same form today to write this, all three categories were sitting in the field. So I can
report the outcome I measured and not the cause.

So if you go anywhere near this screen, re-apply your categories through the API afterwards, and
read them back:

curl -s "https://api.apify.com/v2/acts/aiqlabs~dataset-to-github-issues" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['categories'])"
# ['DEVELOPER_TOOLS', 'AUTOMATION', 'OPEN_SOURCE']

Do not confirm this in the UI you just used. Confirm it in the API.

What it costs to ignore this

The label is not cosmetic, and the timeline is published:

After another 14 days of failing runs, you will receive another notification. Finally, if the runs
continue to fail after yet another 14 days, the Actor will be deprecated.

Three days of failures to get labeled. Roughly a month of them to get deprecated. For an Actor that
structurally cannot pass, that clock starts on the day you publish and never stops. Nothing in your
code will change it, which is precisely why manual removal feels like it works and is the most
expensive thing you can do with the time.

The rule I now apply

Before I publish an Actor that needs an authorized connector, I ask one question: is there any
input the test account can supply that produces a successful run?

If the answer is yes, I make sure that path is what prefill describes, and the daily test becomes
a free health check I get for nothing.

If the answer is no, I file the skip request in the same session as the publish, before the first
notification arrives. Then I never touch the status switch, because by then there is nothing left
for it to argue with.

The Actor in this story is dataset-to-github-issues.
Its source is public under ISC at
github.com/ai-q-labs/dataset-to-github-issues,
and the connector it uses speaks the Model Context Protocol.
The rest of what I publish is at apify.com/aiqlabs.

I gave my Actor a GitHub MCP connector. It could see 44 tools and use 4.

Ai-Q Labs — Fri, 18 Sep 2026 00:08:07 +0000

I gave my Actor a GitHub MCP connector. It could see 44 tools and use 4.

I publish 22 audit Actors on Apify. They all answer one question: is this public record still true?

The registry serves a package with no warning on it. Its repository has been archived. My Actors
find that gap and write it into a dataset.

Then nothing happens. A dataset is a place findings go to be correct in private.

So I built the missing half. It reads an audit dataset, decides what deserves attention, and opens
a GitHub issue for each finding.

It reaches GitHub through an
Apify MCP connector. That is a Model Context
Protocol (MCP) server, wired into the Actor's input by the platform. My Actor never touches my
token.

I expected the interesting part to be the writing. It was not. The interesting part was how little
the platform let my Actor do, and how much the connector appeared to offer.

At authorization, Console listed 44 tools. My run could see four. Both numbers are correct,
and the gap between them is the whole design.

Here is what I built, where in the run the connector fires, and the four things that surprised me.
One of them broke an assumption the Actor was designed around.

What I built, and why it is a separate Actor

The obvious move was to add issue-filing to
github-repository-audit directly. I did not,
for a boring reason: that Actor was under an unrelated measurement, and changing its build would have
destroyed the data.

The boring reason turned out to be the right architecture. Filing issues has nothing to do with
auditing. It needs a dataset, a tracker, and a policy. It does not need to know what npm is.

So dataset-to-github-issues takes a dataset ID and a connector, and nothing about my audit is
hard-coded into it.

The input schema is the interesting file:

{
    "datasetId": {
        "title": "Audit dataset",
        "type": "string",
        "editor": "resourcePicker",
        "resourceType": "dataset",
        "resourcePermissions": ["READ"]
    },
    "githubConnector": {
        "title": "GitHub connector",
        "type": "string",
        "resourceType": "mcpConnector",
        "mcpServers": [
            {
                "url": "https://api.githubcopilot.com/mcp/",
                "tools": {
                    "required": ["issue_read", "issue_write", "list_issues", "search_issues"]
                }
            }
        ]
    }
}

Two declarations sit side by side. One names what the run may read. The other names what it may call
through the connector. The platform enforces both. My code enforces neither.

That symmetry is easy to miss when you are writing it. It matters later.

Setting up the connector: the docs and Console disagree

The MCP connectors documentation says:

Supported services include Notion, Slack, GitHub, Sentry, and Supabase.

The Add new MCP connector dropdown in Console offers three presets. I scrolled to be sure. The
list does not grow.

Preset	URL
Sentry	`https://mcp.sentry.dev/mcp`
Notion	`https://mcp.notion.com/mcp`
Linear	`https://mcp.linear.app/sse`

Slack, GitHub, and Supabase are not presets. Linear is a preset and is not in that sentence.

This is not a bug. The URL field accepts free text, so the presets are a convenience, not an
allowlist. I typed the URL of
GitHub's own MCP server and waited.

https://api.githubcopilot.com/mcp/

About four seconds later the field turned green and an auth panel appeared. Apify had gone and asked
the server what it supports. It offered API key, selected by default. It rendered OAuth
grayed out, with this text:

This server doesn't support dynamic client registration. Your own OAuth client is recommended.

I like that the platform probes rather than assumes. It also told me exactly which path to take on a
free account: a personal access token used as a bearer credential.

The token I actually issued

I used a fine-grained personal access token,
scoped as tightly as GitHub allows:

Scope element	Value
Repository access	Only select repositories → one repo
Permissions	Issues: read and write
Permissions	Metadata: read-only (added automatically, marked Required)
Expiry	30 days

No contents write. No administration. No second repository. Remember this table; it is the point of
the next section.

The trap: the URL reverted while I was not looking

Between filling in the dialog and pasting the API key, the server URL changed under me. It went back
to https://mcp.sentry.dev/mcp, the first preset in the list. The field label also changed from
"MCP server URL" to "MCP server".

My API key field kept its contents. The URL showed a green validation check. Save was enabled.

If I had saved that, a GitHub token would have been registered as the bearer credential for
Sentry's MCP server. Nothing in the dialog would have complained.

Re-typing the GitHub URL and picking it from the suggestion fixed it, and the key survived the
change.

I have a rule now: read the server URL again immediately before you save. Typing it once is not
enough. A connector dialog holds a live credential, and it deserves the same paranoia as a payment
form.

44 tools discovered, 4 of them visible

Saving the connector produced an ID and a list. Console showed the tools it had discovered at
authorization time. All 44:

add_comment_to_pending_review  add_issue_comment  add_reply_to_pull_request_comment
create_branch  create_or_update_file  create_pull_request  create_repository  delete_file
fork_repository  get_commit  get_file_contents  get_label  get_latest_release  get_me
get_release_by_tag  get_tag  get_team_members  get_teams  issue_read  issue_write
list_branches  list_commits  list_issue_fields  list_issue_types  list_issues
list_pull_requests  list_releases  list_repository_collaborators  list_tags
merge_pull_request  pull_request_read  pull_request_review_write  push_files
request_copilot_review  run_secret_scanning  search_code  search_commits  search_issues
search_pull_requests  search_repositories  search_users  sub_issue_write
update_pull_request  update_pull_request_branch

Read that list against my token's permissions. create_repository is there. delete_file is there.
merge_pull_request, push_files, fork_repository, run_secret_scanning — all there, all far
outside a token that can only read and write issues in one repository.

The discovered tool list describes the server, not the token. It is a menu, not a grant. Every
one of those calls would die upstream, at GitHub itself.

That is a fine outer boundary. It is also the worst possible place to learn about it. So I looked at
what my run actually got. From the run log:

INFO  Read 12 row(s) from dataset iBbLEMf3NUSJcU0cZ.
INFO  The proxy exposes 4 tool(s) to this run: issue_read, issue_write, list_issues, search_issues

Four. Exactly the four names in mcpServers[0].tools.required, and nothing else. I changed nothing
about the connector between those two observations. The
input schema declaration
is what narrowed it.

The docs put it in one sentence, and now I have watched it happen:

The proxy enforces that an Actor can only call tools it explicitly declared in its input schema.

So there are three layers, and they do different jobs:

The token. What the upstream service will honor. Enforced by GitHub, discovered on failure.
The connector. What server this credential belongs to. Chosen once, in Console.
The input schema. What this Actor may call. Enforced by the proxy, before the request leaves Apify.

Layer 3 is the one that ties a tool to a specific Actor. It is also the only one a reader of my Actor
can see. Anyone can open my input schema and learn the worst thing my code can do to their GitHub
account.

I value that as much as the enforcement itself. A limit nobody can inspect is a promise.

Where the connector fires, and why not at the end

The easy design writes at the end. Audit, decide, file, done.

I fire the connector in the middle, twice, and the first call is a read. The Actor lists the open
issues in the tracker before it decides anything:

// This is the whole reason the connector fires here and not at the end: the decision of
// whether to open an issue depends on what is already open.
const openKeys = new Set();
for (let page = 1; page <= 5; page += 1) {
    const res = await call(client, readTool.name, argsFor(readTool, {
        method: methodValue(readTool, /list/),
        owner,
        repo,
        state: 'open',
        perPage: 100,
        page,
    }));
    if (!res.ok) { readError = res.error; break; }
    const list = Array.isArray(res.data) ? res.data : (res.data?.items ?? res.data?.issues ?? []);
    if (!Array.isArray(list) || list.length === 0) break;
    for (const issue of list) {
        // a title like "[dep-drift] npm:left-pad - deprecated..." gives back "npm:left-pad"
        const key = keyFromTitle(issue?.title ?? '');
        if (key) openKeys.add(key);
    }
    if (list.length < 100) break;
}

Identity lives in the issue title. Every issue my Actor opens is tagged [dep-drift], and the
finding key follows the tag. A later run recovers the key by parsing titles it wrote itself. No
state file, no external store, nothing to fall out of sync.

I proved it by running the same input twice, live, with the same settings.

First run:

INFO  0 finding(s) already have an open issue (read 0 title(s) over 1 page(s)).
INFO  Opened 3 issue(s).

Second run, identical input:

INFO  3 finding(s) already have an open issue (read 3 title(s) over 1 page(s)).
INFO  Opened 3 issue(s).
INFO  Done. skipped: 7, filed: 3, deferred: 2.

The Actor recognized its own three issues from the first run and skipped them. It filed the next
three. Two more hit the per-run ceiling, and it reported them as deferred instead of dropping
them.

Every row says why, in the dataset, whether or not it became an issue:

This is what a connector buys that a separate script does not. A write-only integration would
have re-filed the same three issues. Then it would do it again every scheduled run, and the tracker
becomes noise inside a week. Reading and writing in the same run, through the same authorization,
is what makes re-running safe.

The Actor also refuses to write blind. If the read fails, it stops:

if (readError) {
    await Actor.fail(
        `Reading the existing issues failed: ${readError}. Stopping before any write, because `
        + 'without that list this run cannot tell a new finding from one it filed last time.',
    );
}

Duplicates in someone's tracker cost more than a failed run.

The assumption that broke

My audit answers in three states. It found a defect. It found nothing. Or it could not check —
usually because GitHub's hourly allowance ran out mid-run.

The third state is the one I care about. Filing it turns "I don't know" into an alarm. Dropping it
silently turns an unanswered question into a clean bill of health.

So decide() gives it its own outcome:

export function decide(row, { minRiskLevel = 'high', openKeys = new Set() } = {}) {
    const key = findingKey(row);
    const real = realCodes(row);              // codes that mean "we found something"
    const unverified = unverifiedCodes(row);  // codes that mean "the check did not run"

    if (real.length === 0) {
        if (unverified.length) {
            return { decision: 'withheld', reason: `not verified (${unverified.join(', ')})`, key };
        }
        return { decision: 'skipped', reason: 'nothing to report', key };
    }
    if (openKeys.has(key)) {
        return { decision: 'skipped', reason: 'an open issue already covers this', key };
    }
    return { decision: 'file', reason: `${row.riskLevel}: ${real.join(', ')}`, key };
}

To exercise that path honestly, I needed an audit that really ran out of allowance. So I pointed the
audit Actor at gatsby's package.json — 166 dependencies. It duly ran out:

{
  "github":   { "requestsMade": 56, "distinctRepositories": 131,
                "repositoriesNotChecked": 75, "rateLimited": true },
  "findings": { "not_checked": 76, "repository_not_on_github": 76, "repo_moved": 7,
                "deprecated_on_registry": 4, "repo_archived": 2, "silent_abandonment": 1 }
}

Seventy-six unverified rows. I expected a dry run to report dozens of withheld findings. It reported:

INFO  Done. skipped: 161, would-file: 5.
0 finding(s) were withheld because the audit could not verify them

Zero.

I read the rows instead of guessing at the cause. Every single not_checked row also carried
repository_not_on_github. Not one row in 166 had an unverified code and nothing else.

That kills the assumption. I had built the decision around "a finding is either established or
unestablished". Real data says a row is usually both: something the audit established, plus a
caveat about a check that did not run. The whole-row case turns out to be the rare one.

The behavior that actually mattered was the other one. The caveat has to travel with the finding,
into the issue body. Here is issue #11, read back from the public GitHub API:

**npm:string-similarity** — risk: `high`

### What the audit found
- `deprecated_on_registry` (high) — The registry marks this deprecated: "Package no longer supported..."
- `repository_not_on_github` (low) — The registry links to git://github.com/aceakash/string-similarity.git,
  which is not a GitHub repository.

### What the audit could not check
- `not_checked` — aceakash/string-similarity was not checked: GitHub's hourly allowance ran out.

These are open questions, not clean results. Re-run the audit to close them.

A reader cannot mistake that last section for a clean result. The design goal held. The code path I
built to demonstrate it never fired.

I am leaving withheld in, with its unit tests, and saying plainly that no live audit has produced
one yet. It is designed behavior. I have never watched it happen.

The reply that looked complete and was not

Run 4 opened three issues and then crashed:

INFO  Opened 3 issue(s).
ApifyApiError: Schema validation failed   clientMethod: DatasetClient.pushItems
  instancePath: '/issueNumber'  message: 'must be integer'

The issues were real. I checked them from outside the run with an unauthenticated curl. The
report is what failed, because issueNumber came back null.

I had stored the first raw reply in the key-value store on purpose, so I could look instead of
guess. This is issue_write's answer, verbatim:

{"id":"5078084249","url":"https://github.com/ai-q-labs/github-repository-audit/issues/4"}

There is no number field. id is a string, and it holds GitHub's internal identifier. The
issue number — 4 — exists only inside the URL.

Reaching for id would have recorded issue "5078084249". Nothing would have errored. My dataset
would have looked complete and been wrong in a way no type check catches.

The fix reads the number where it actually lives, and refuses to invent one:

function findIssueRef(reply) {
    const out = { number: null, url: null };
    if (!reply) return out;
    if (typeof reply === 'object') {
        const o = reply.issue ?? reply.data ?? reply;
        const n = o?.number ?? o?.issue_number ?? o?.id;
        if (Number.isInteger(n)) out.number = n;    // note: a string id fails this on purpose
        const u = o?.html_url ?? o?.htmlUrl ?? o?.url;
        if (typeof u === 'string') out.url = u;
        if (out.number !== null && out.url !== null) return out;
    }
    const text = typeof reply === 'string' ? reply : JSON.stringify(reply);
    const m = /https?:\/\/github\.com\/[\w.-]+\/[\w.-]+\/issues\/(\d+)/.exec(text);
    if (m) {
        out.url ??= m[0];
        if (out.number === null) out.number = Number(m[1]);
    }
    return out;
}

Number.isInteger is doing real work there. It rejects the string id that would otherwise sail
through.

If you build against a connector, capture the first reply from every tool you call. A tool result is
whatever the upstream server decided to send. Assuming a field name and discovering the truth in
production is exactly the failure this prevents.

Two more failures worth designing for

A run cannot open an arbitrary dataset. My very first run died here:

ACTOR: Running under "LIMITED_PERMISSIONS".
ERROR Could not read dataset iBbLEMf3NUSJcU0cZ: Insufficient permissions for the dataset.

Same account, same owner, still refused. The fix is not a token — it is the resourceType and
resourcePermissions declaration from the schema at the top of this article. The schema names the
thing being read and the thing being written through. The platform grants exactly that, and no more.

A run that changed the outside world must not fail on its own bookkeeping. That crash after
"Opened 3 issue(s)" marked the whole run FAILED. FAILED reads as "nothing happened". That is the
opposite of the truth, and it sends the next run straight back at the same findings.

I now guard each dataset write on its own. Anything that fails lands in the summary instead of
killing the run:

for (const d of decisions) {
    try {
        await Actor.pushData(d);
    } catch (err) {
        reportProblems.push({ key: d.key, error: String(err?.message || err).slice(0, 300) });
    }
}

Order the work so the irreversible part happens last, and make everything after it non-fatal.

Seven runs got me here, and the first two are the ones worth reading:

The bug my new Actor found in my old one

Look again at issue #11. The registry link is git://github.com/aceakash/string-similarity.git.

That is on GitHub. The same row resolves aceakash/string-similarity and queues it for a GitHub
lookup. One row, two statements, contradicting each other.

My first explanation was that the URL parser rejects the git:// scheme. I ran the parser to check,
and it does not. It accepts git://, git+ssh://, git@host:path and github:owner/name alike.

The real cause is one line further on. The finding fires when the GitHub record is missing:

if (registryFound && !pkg.repoUrl) {
    add('no_repository_link', 'The registry record does not link to any source repository, so nothing can be verified against it.');
} else if (registryFound && pkg.repoUrl && !repo) {
    add('repository_not_on_github', `The registry links to ${String(pkg.repoUrl).slice(0, 120)}, which is not a GitHub repository.`);
}

The record is also missing when the lookup never ran. GitHub allows an unauthenticated caller 60
requests an hour. Past that, my Actor stops asking and stores null:

if (state.rateLimited) { state.skipped += 1; return null; }

So !repo means two different things, and the code reads only one of them. Every row cut off by the
rate limit is reported as a repository that is not on GitHub. That is why all 76 unverified rows were
paired. The pairing tracks the rate limit. The URL scheme has nothing to do with it.

It is the same mistake as the one in my first article, made a fifth time. A missing answer is not a
negative answer.

I have not fixed it. github-repository-audit is inside an unrelated observation window, and
changing its build would destroy that measurement. It is recorded, and it goes in next.

I did not expect the reader of an Actor's output to become the best test of that Actor. It is
obvious in hindsight. A dataset you only look at is a dataset nobody checks.

What it cost, and what I would do differently

Eleven issues, none duplicated, across four live runs and two audit datasets. All of it on the
free plan. When I stopped, my account usage read $0.61 against the $5 monthly credit. MCP
connectors are not a paid feature, which surprised me enough to check twice.

Three things I would tell myself before starting:

Declare the smallest tool set you can name. Not for safety theater — because the declaration is public. Anyone can read what my Actor is allowed to do without reading my code.
Fire the connector where the decision is, not where the output is. Read before you write, and stop if the read fails.
Store the first reply from every tool. The one that cost me a run was shaped nothing like I assumed. It failed quietly, in the one direction a type check cannot catch.

And one I got wrong: I designed a decision path around a data shape I had never seen. A branch, a
reason string, and unit tests, for a row that has not appeared once in 178 audited packages. Next
time I look at the data before I write the branch.

Run it yourself

The full source of the Actor in this article is at
ai-q-labs/dataset-to-github-issues —
the input schema, the decision function, the connector client, and the unit tests, including the one
covering the case that has never fired.

You will need three things to run it: an audit dataset with riskLevel and issueCodes on each
row, an MCP connector authorized against GitHub, and a repository you are willing to file issues in.
A free Apify account covers the rest.

Where the findings came from:
github-repository-audit is public and free.
Everything it produced through the connector is open in the
tracker repository, tagged
[dep-drift], including the one that exposed my own parser bug.

Full-size versions of every screenshot in this article sit in the source repository above, under
docs/screenshots.

The shortest version of the whole thing: your connector shows you a menu. Your Actor declares what it
eats. The proxy is what makes the difference real.

I connected my audit Actor to Claude, and it audited three packages nobody asked for

Ai-Q Labs — Fri, 18 Sep 2026 00:07:21 +0000

I connected my audit Actor to Claude, and it audited three packages nobody asked for

I maintain 22 audit Actors on Apify. They all do a version of the same thing. They take a public
record, and they check whether what it still claims is true.

Last week I connected one of them to Claude through the Apify Model Context Protocol (MCP) server. I
expected the interesting part to be the agent. It was not. The interesting part was my own input
schema. It had been lying to every non-human caller since the day I published it.

Then I made a second assumption. I decided I knew how an agent would read the result. I gave the
question to four of them to prove it, and all four proved the opposite.

This is what I found, what the agents did with it, and what I changed.

What the Actor does

github-repository-audit takes package names or
repository names. For each one it asks two sources the same question and compares the answers.

The registry says a package points at a repository. The repository says whether it is archived, moved,
or relicensed. Those two records disagree more often than you would think.

The finding I built it for is the quiet one. Here is real output from a run:

npm still serves cross-env@10.1.0 with no deprecation notice, while its repository
kentcdodds/cross-env has been archived. Installing it looks completely normal.

Nothing in your terminal tells you. npm install prints no warning. The archive banner sits on
the repository where nobody installing the package will
look. The maintainer said goodbye in the only place they could. The registry never passed the message
on.

Connecting it through the Apify MCP server

The Apify MCP server turns an Actor into a tool an
agent can call. The connection itself took one command:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=aiqlabs/github-repository-audit"

The ?tools= parameter matters more than it looks. Without it, the server exposes its whole surface.
With it, the agent sees exactly one tool. I wanted a clean experiment, so I scoped it to one Actor.

Authentication runs over OAuth with dynamic client registration and PKCE. I never handled a token.
The consent screen is worth reading rather than clicking through. Apify prints the honest version:

This application was registered dynamically, and wasn't verified by Apify. Make sure you trust it.
It's allowed to redirect you to following URL(s): http://localhost/callback

That is the right warning to show. Dynamic registration means nobody vetted the client.

The detour

My first three attempts failed, and the reason had nothing to do with Apify.

I was driving the CLI from a non-interactive shell. claude mcp login prints the authorization URL,
starts waiting, and then gives up:

Waiting for authorization… (^C to cancel)
Couldn't complete authentication: stdin isn't a terminal.

The login flow wants a real terminal even when the callback would arrive over the network. I tried
winpty next. winpty refused for the same reason one level down. It needs a console of its own.

What worked was a batch file, launched in a fresh console window, with stdout sent to a file. The
console satisfies the terminal check. The redirect lets me read the authorization URL. Then I opened
that URL, approved it, and the callback landed on localhost.

Write that down if you automate this. The blocker is the terminal, not the network.

The first run returned four rows for a one-repository question

Before handing the tool to an agent, I wanted to see the smallest possible call. So I sent one field
and nothing else:

{"repos":["facebook/create-react-app"]}

That is 39 bytes. The run recorded an inputBodyLen of 328.

Something had grown my input by a factor of eight. This is what the platform actually stored:

{
  "repos": ["facebook/create-react-app"],
  "packages": ["npm:request", "npm:babel-eslint", "npm:left-pad"],
  "defaultRegistry": "npm",
  "manifestType": "",
  "manifestGroups": ["dependencies"],
  "staleAfterDays": 365,
  "includeContributors": false,
  "includeReleases": false,
  "onlyIssues": false,
  "maxTargets": 200,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 30
}

Three packages I had not mentioned were now part of the run. The results came back like this:

`input`	`source`	`riskLevel`	`issueCodes`
`facebook/create-react-app`	`repos`	medium	`repo_moved`, `stale_no_push`
`npm:request`	`packages`	high	`deprecated_on_registry`, `stale_no_push`
`npm:babel-eslint`	`packages`	critical	`deprecated_on_registry`, `repo_archived`
`npm:left-pad`	`packages`	critical	`deprecated_on_registry`, `repo_moved`, `repo_archived`, `license_mismatch`

I asked about one repository. I got four rows. Every high and critical row belongs to something the
caller never mentioned.

request and
left-pad are not neutral filler either. They are two of the
most famously abandoned packages in the npm registry. They produce findings by design.

prefill and default are not the same word

The cause is one line in my own input schema, written months earlier.

{
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] },
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] }
}

I had used the two keys interchangeably. I thought both were examples for the form. The
input schema specification
is unambiguous, and I had simply never read this paragraph carefully:

Default — If the user omits the value when starting the Actor via any means (API, CLI,
scheduler, or user interface), the platform automatically passes the Actor this default value.

Prefill — this field is only used in the user interface but does not affect the Actor
functionality and API.

prefill is a suggestion. default is a promise the platform keeps on your behalf.

There is a second edge to this, and I only saw it once the Actor was a tool. The MCP server turns
the input schema into a JSON Schema and hands it to the model. Here is what arrives:

{
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] },
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] }
}

default is a JSON Schema keyword. It means "the value used when this is omitted", and a model
reading the tool definition can act on it. prefill is not a JSON Schema keyword at all. It passes
through as an unrecognised key.

So the field nobody asked about announces itself in the language of the specification. The field that
answers the question carries a word the model has no rule for. I had put the standard keyword on the
wrong field.

My Actor's own code is innocent here. It destructures with packages = []. The injection happens
above it, before Actor.getInput() ever returns.

What I assumed would happen next

Open the Actor in Apify Console and the bug is visible immediately. The Packages box has three values
sitting in it. You delete them, or you leave them, and either way you decided.

The form is doing something important. It shows you the whole input, including the parts you did not
supply.

An agent never sees a form. It sends the fields it decided to send. It receives rows.

My rows do carry a source field, so the information needed to separate them exists. An agent that
reads it can tell the four rows apart. I want to be exact about that. It is the part I got right by
accident, because I did not add source for this reason.

But an agent does not read fifty-four fields per row back to a user. It reads the summary. So I
opened the two records my Actor writes for exactly that purpose.

SUMMARY counts the run:

{
  "checked": 4,
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 },
  "licenseComparison": { "comparable": 3, "mismatchRate": 0.333, "notComparable": 1 }
}

Two critical and one high, for a question about one repository. No count in that block says which
findings belong to the caller's target. The mismatch rate is worse. It is a fraction whose
denominator the caller never chose.

ACTION_LIST is the record I named for what to do next. Its entries carry target, registry,
repo, riskLevel and issues. It has no source field at all. It is sorted by severity:

#	`target`	`riskLevel`	did the caller ask for it?
1	`babel-eslint`	critical	no
2	`left-pad`	critical	no
3	`request`	high	no
4	`facebook/create-react-app`	medium	yes

The one entry that answers the question sits at the bottom. Above it are three the caller never
mentioned. The record offers no field that would tell them apart.

That is the shape of the trap. Attribution survives in the raw rows, which an agent skims. It
disappears from both records built to be read instead.

So I had a prediction, and it was a tidy one. Ask an agent whether create-react-app is safe to depend
on. It will read critical: 2 and pass that on.

I was wrong four times in a row

I had already read the schema, so my own answer proved nothing. I gave the question to four fresh
agents instead. None had seen any of this. The Actor was still unfixed.

Is facebook/create-react-app safe to depend on?

All four got it right. Every one of them reported medium risk for create-react-app, which is the
correct answer. Not one passed on the critical count.

Three of the four went further and told the user my tool was broken.

One thing worth flagging about the tool: my first run asked about a single repository but produced
4 rows. The extra 3 are the Actor's built-in defaults for the packages field […] Those three
carry the alarming findings — 2 critical, 1 high […] Reading that summary at face value would
have produced a badly wrong answer.

Another said the same thing in different words, then added a line I did not want to read:

when you pass only repos to this Actor, you have to set packages to empty explicitly or
unrelated rows get mixed in.

That is a workaround for my Actor, written by an agent, addressed to my user.

The trap still cost something

Nobody was misled. I want to be careful not to soften that. But four correct answers is not the same
as no harm.

Two of the four ran the audit twice. Double the compute units, double the GitHub requests against
a 60-per-hour unauthenticated allowance, double the wait. The caller paid to undo my mistake.

One told the user the tool was defective. That paragraph is now part of what my Actor looks like
to somebody deciding whether to use it.

One said nothing at all. The fourth agent gave a clean, correct answer and never mentioned the
three rows it had discarded. The caller was billed for four rows and got the use of one. Nothing in
the exchange told them.

And the wrong answer was sitting there the whole time. byRiskLevel still reads critical: 2 with
nothing beside it to say whose. Four agents declined to take it. That is not the same as it not being
there.

One of them found a second bug I had missed

The agent that ran the audit twice explained why:

I ran the audit a second time because the repo-only run left registryDeprecated, licenseMatch
and silentAbandonment all null — it never consulted npm, so it couldn't have answered "safe
to depend on" as asked.

It is right. A repository has no registry side, so three checks cannot run on it. One of them is
silent_abandonment, the finding this Actor exists for. A caller who names a repository never gets
it, and nothing said so. The fields just came back null.

I had written the rule against this in my own README. Return "broken", "fine" and "could not check"
as three different values. Then I returned the third as a bare null and let it read like the second.

Two of the four agents worked around it the same way, without being asked. That is a design gap, not
a coincidence.

What I changed

Five changes, in order of how much they mattered.

I removed default from every field that names a target. Sample values moved to prefill, which
keeps the Console form useful and never reaches the API. A field that decides what to audit must
come from the caller. A field that decides how to audit can have a default, and staleAfterDays
still does.

I made the empty call fail loudly. With no targets the Actor now stops with a message that names
the three ways in. It used to be impossible to reach that path, because the default guaranteed there
was always something to audit. That was the bug hiding the bug.

I moved the relevance rule into the description. The source field is now documented where an
agent reads it, not only in the README.

That last one had a second layer I only found while writing this. source did not appear anywhere in
my dataset_schema.json either. So the default Console table did not show it, and neither did the
"Problems only" view.

Opening that table made it worse. My view leads with packageName, and the row I asked for came from
repos, so it has no package name. The Console printed it as null.

#	Package Name	Registry	Risk	Resolved Repo
1	null	null	medium	react/create-react-app
2	request	npm	high	request/request
3	babel-eslint	npm	critical	babel/babel-eslint
4	left-pad	npm	critical	left-pad/left-pad

The one row that answers the question looks like the broken one. The three rows nobody asked for look
authoritative. I had built a view that ranked my own output by how little the reader wanted it.

Both views now lead with the two columns that answer "did I ask for this row?": input and source.

I made the summaries carry attribution too. This is the change I would have missed if I had
stopped at the schema. Removing the default stops this particular injection. It does not make the
output attributable.

ACTION_LIST now begins each entry with input and source. SUMMARY now carries a bySource
block beside the flat counts:

{
  "checked": 4,
  "bySource": {
    "repos":    { "checked": 1, "critical": 0, "high": 0, "medium": 1, "low": 0, "ok": 0 },
    "packages": { "checked": 3, "critical": 2, "high": 1, "medium": 0, "low": 0, "ok": 0 }
  },
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 }
}

The flat line still says two critical. Now something next to it says whose.

That block is eight lines in src/audit.js, inside the function that builds SUMMARY:

// A single call can mix targets the caller typed with targets that arrived some
// other way - a manifest that expanded into forty dependencies, or a schema
// default. A flat count of "2 critical" cannot be acted on, because it does not
// say whose.
const bySource = {};
for (const row of rows) {
    const key = row.source ?? 'unknown';
    bySource[key] ??= { checked: 0, critical: 0, high: 0, medium: 0, low: 0, ok: 0 };
    bySource[key].checked += 1;
    if (row.riskLevel in bySource[key]) bySource[key][row.riskLevel] += 1;
}

ACTION_LIST took two lines: input and source moved to the front of each entry, above
target. That ordering is the whole change. A reader who cannot answer "did I ask for this?"
cannot act on the row.

This matters beyond the bug that started it. My Actor takes repositories, packages and a manifest in
one call. A manifest URL can expand into forty dependencies from a single field. Any of those
mixes produces a count the caller cannot take apart. The default was one way in. It was not the only
one.

Going through the rest of the output found one more. LICENSE_REPORT ends with a list of licences
that could not be resolved. That list is an instruction to open files by hand. It held bare names. So
I gave those entries input and source as well. Sending someone to read a licence for a dependency
they never named wastes the same afternoon a false finding does.

I made "could not check" say so out loud. This is the one an agent found for me. A
repository-only row now produces a note in SUMMARY:

1 target(s) were given as repositories, so no registry was consulted for them. registryDeprecated,
licenseMatch and silentAbandonment are null on those rows because they could not be checked, not
because they came back clean. Pass the package name to check them.

The same sentence went into the repos field description, where an agent reads it before calling.
null was already the honest value. It was not a legible one.

Four tests hold this. Three check the new fields on the three records. The fourth is a control. A run
from a single source must report one group, and its numbers must equal the flat totals. So the
breakdown cannot invent structure that is not there. The suite went from 48 to 52.

What actually changed, measured the same way

I pushed build 0.1.8 and sent the same one-field call again.

	before	after
rows returned	4	1
stored `INPUT`	328 bytes	267 bytes
`byRiskLevel`	`critical 2, high 1`	`critical 0, high 0`
`bySource`	absent	present
first `ACTION_LIST` entry	`babel-eslint`, critical	the repository I asked about

Then I ran the agent trial again, same question, same plain prompt, two fresh agents.

Both answered correctly, as before. Neither mentioned a defect in the tool. Before the change,
three of four had. That is the entire measurable result, and it is a modest one. Nobody was ever
given a wrong answer, so nothing about correctness improved.

What improved is that my Actor stopped asking its callers to compensate for it.

One thing did not change. Both agents still ran the audit twice, once by repository and once by
package. That is now the right behaviour, and the output asks for it in writing. I should say that my
new note may be causing the second call rather than merely permitting it. One agent made the same
second call before the note existed, which argues against that. With four trials I cannot separate
the two.

Then I checked the other twenty-one

One bad field is a typo. I wanted to know whether it was a habit. So I read every input schema I
have published and sorted the defaults into two piles.

A default is safe when it decides how the work is done. It is dangerous when it decides what
the work is done to.

Twenty-two published Actors. Every one of them sets a default somewhere. Ten of those defaults
still name a target, and I had just removed an eleventh.

Actor	field	what it audits when the caller says nothing
`seo-ai-visibility-auditor`	`startUrls`	`apify.com`
`bulk-domain-checker`	`domains`	`apify.com`
`domain-availability-checker`	`domains`	`apify.com`
`sitemap-checker`	`domains`	`apify.com`
`tech-stack-detector`	`domains`	`apify.com`
`dead-link-checker`	`domains`	`docs.apify.com`
`http-status-checker`	`urls`	`apify.com/store`
`pdf-inspector`	`pdfUrls`	a US tax form
`pdf-to-text-markdown`	`pdfUrls`	the same tax form
`hacker-news-link-rot`	`list`	the Hacker News `topstories` list

Call any of the first nine with an empty input and you get a finished report. It covers Apify's own
website, or a US tax form. Nothing in the response says it is a sample. An agent has a well-formed
answer to a question nobody asked.

Then I noticed the part that embarrassed me most. Nine of those ten fields also appear under
required in the same schema.

I had been reading required as a promise that the caller named the target. So I tested that
reading. I called seo-ai-visibility-auditor with an empty object.

echo '{}' > empty.json
apify call GuuMKUiWcaUUqGGhG --input-file=empty.json

The run succeeded and audited apify.com. Here is what the platform stored as my input:

{
  "startUrls": [{ "url": "https://apify.com" }],
  "crawlSite": false,
  "maxPages": 10,
  "checkBrokenLinks": false,
  "maxLinksToCheck": 50,
  "proxyConfiguration": { "useApifyProxy": false }
}

I sent {}. The Actor received a target, and nothing rejected the call.

The specification
treats the two settings as alternatives rather than as a pair. required is for fields that "don't
have a reasonable default". default is passed by the platform whenever the caller omits the field,
"via any means". Put both on one field and the second one decides. required survives as a note to
whoever is reading the form.

The other twelve Actors are fine. country, outputFormats, robotsAgent, manifestGroups —
omitting those does not invent work.

One row in the table made me think harder. hacker-news-link-rot defaults its list field to
topstories, and that does pick the target. But the Actor has no other way in. An empty call has to
mean something. It is also the one row I never marked required.

So the rule is not "never use default". It is narrower than that:

A caller who says nothing must not receive results they cannot tell apart from results they asked
for.

hacker-news-link-rot satisfies that with one sentence in its description. The other nine did not
satisfy it at all.

The same mistake, four times, before an agent was involved

Here is what stung. This was not a new class of error for me. It was the fourth time.

Every audit Actor I have written has produced a confident finding that was really a gap in my own
knowledge. Each time, the fix was the same shape: split one value into two.

A rate limit that looked like deletion. My Chrome extension auditor read 130 listings from
Google's own sitemap, one every 600 milliseconds. Ninety-three came back as "the store has never
heard of this ID". Every one of them was a healthy extension I had listed minutes earlier.

Google serves its rate-limit interstitial as a redirect away from the store. A reader that only asks
"did I land on a listing page?" sees exactly what a deleted extension looks like. The Actor now checks
for that first. It never treats the interstitial as a fact about the extension. It stops the run
instead of producing ninety-three more.

A 404 that was an advertising server refusing. My podcast auditor reported a 1.7% episode death
rate. Nine of those failures came from one host, and all nine shows were running fine.

A per-listener redirect service was serving that audio, and stitching in advertising as it went. It
would not build a redirect for an automated request. So it answered 404, with the body Missing redirect URL. I had a real death rate of 0.5% and a fake one three times larger.

Those cases became undetermined, not dead. The finding text says what I actually know:

Whether the audio is there cannot be established without behaving like a listener, which this
Actor does not do.

A licence warning on lodash. GitHub could not match lodash's LICENSE to a standard licence, so it
returned NOASSERTION. My first version called that license_non_standard and raised it.

lodash, jQuery UI and UglifyJS all land there. They are ordinary MIT and BSD projects whose LICENSE
carries an extra paragraph. A warning that fires on healthy rows does not add information. It buries
the rows that matter. The code comment I left says it better than I can paraphrase:

Raising this would put a warning on healthy rows and bury the ones that matter.

A severity I set before I measured anything. I once shipped "no update in three years" as a high
severity finding. Then I measured the base rate across 1,312 App Store apps. It is 18.4%.

A finding that fires on one row in five describes the ecosystem. It does not describe a problem with
your dependency. It sits at medium now.

For the Shopify auditor I dropped the check entirely. Shopify publishes a launch date and no update
date. Building the check anyway would have meant guessing.

The rule I ended up with

Every one of these is the same rule, arrived at four times the slow way.

Return "broken", "fine", and "I could not check" as three different values.

I used to justify that by saying an agent cannot tell the third from the second. My own trials say
otherwise. Four agents pulled the distinction out of a source field I had not documented. Two
worked out on their own that a repository-only row never touches the registry.

So the reason is not that agents cannot cope. It is what coping costs.

Every ambiguity you leave in your output is work you have handed to the caller. Sometimes they pay it
in a second run against a rate-limited API. Sometimes they pay it by writing a paragraph explaining
your tool's quirk to their user. Sometimes they pay it silently, by throwing away three quarters of
what you charged them for.

None of that shows up as an error. It shows up as your Actor being slightly more expensive and
slightly less trusted than the one next to it.

The input schema is the other half. If a field can change what gets audited, the caller must set it.
Silence has to mean silence.

What to check in your own schema

Five things, in the order that cost me the most time.

Search your schema for "default". For each hit, ask what happens when an API caller omits that field. If the answer changes what the Actor works on, move it to prefill. Listing the field under required will not do this for you.
Call your Actor with the minimum viable input, then read the stored INPUT record. Do not read the input you sent. Read what the platform saved. That is what your code receives.
Make sure an empty call fails. If your Actor can always find something to do, you cannot tell an empty call from a real one. I checked what that costs, because the platform runs published Actors on its own schedule. Twelve of my twenty-two already have nothing to do on an empty call, three of them because a required field carries no default at all. All twenty-two show zero failed runs across 159 platform runs in the last thirty days.
Write your limits into description, not the README. The description is what an agent reads. Mine now says why pyproject.toml is unsupported: half-parsing a manifest produces findings about dependencies you do not have.
Hand the tool to an agent and read what it says about you. This found more than my own review did. An agent that works around your quirk will usually explain the quirk to its user, in writing, in the answer. That paragraph is a free bug report. It is also what your Actor looks like to a prospective user.

Code

The full source for the Actor in this article is on GitHub:
ai-q-labs/github-repository-audit. That is
the code behind the published Actor at
apify.com/aiqlabs/github-repository-audit. It
includes the input schema this article is about, 52 unit tests, and a live check against the real
GitHub, npm and PyPI APIs.

Clone it, run npm install, then npm test for the unit tests or npm run test:live for the live
check. The live check needs no key, but GitHub allows unauthenticated callers 60 requests an hour, so
set GITHUB_TOKEN if you run it more than once.

Full-size versions of every screenshot in this article sit in the same repository, under
docs/screenshots.

My Actor ranked #1 in an agent's search. No one else's agent could see it at all.

Ai-Q Labs — Fri, 18 Sep 2026 00:07:12 +0000

My Actor ranked #1 in an agent's search. No one else's agent could see it at all.

Four days ago I published a number I was pleased with. I had measured how often an AI agent finds one of my 23 Actors when it searches the Apify Store through the Apify MCP server. The answer came out at roughly one query in five, and on sitemap checker my Actor came back first.

Both figures were wrong. On the queries I re-tested this morning, an agent that is not mine finds my Actors zero times out of six.

The arithmetic was fine. The problem was the connection I measured through, and it is a problem that any Actor author can have without noticing, because the platform gives you no obvious way to measure it any other way.

The measurement I trusted

My script for that article built one URL:

const url = `https://mcp.apify.com/?token=${encodeURIComponent(token)}`;

Then it called search-actors with a keyword, paged through the results and recorded where my Actor landed. sitemap checker put it at rank 1. PDF tables put it at rank 2. I wrote both numbers down and reasoned from them for a week.

I never asked whose search that was.

The same query, with the token and without

search-actors needs a token, so I could not run it anonymously. The Store also has a plain REST endpoint that serves the same shelf, and that one takes the Authorization header or does without it. So I ran the same query twice:

// store-auth-diff.mjs — is my Actor in the results, or only in *my* results?
const TOKEN = process.env.APIFY_TOKEN;

async function search(query, withAuth) {
    const url = `https://api.apify.com/v2/store?search=${encodeURIComponent(query)}&limit=100`;
    const res = await fetch(url, { headers: withAuth ? { authorization: `Bearer ${TOKEN}` } : {} });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const { data } = await res.json();
    const ids = data.items.map((a) => `${a.username}/${a.name}`);
    const at = ids.findIndex((id) => id.startsWith('aiqlabs/'));
    return { total: data.total, returned: ids.length, mineAt: at < 0 ? null : at + 1 };
}

for (const q of ['sitemap checker', 'pdf table extractor', 'github repository audit']) {
    console.log(q, '\n  auth:', await search(q, true), '\n  anon:', await search(q, false));
}

Its output on 16 August:

sitemap checker
  auth: { total: 339, returned: 78, mineAt: 1 }
  anon: { total: 339, returned: 77, mineAt: null }
pdf table extractor
  auth: { total: 757, returned: 86, mineAt: 2 }
  anon: { total: 757, returned: 84, mineAt: null }
github repository audit
  auth: { total: 417, returned: 71, mineAt: 1 }
  anon: { total: 417, returned: 69, mineAt: null }

Read the columns in order. total is the same number with and without the token — 339 stays 339. The count of items actually returned is one to three higher when the token is present. The extra items are mine.

I ran it over five queries. With the token my Actors sit at ranks 1, 2, 1, 1 and 5. Without it, none of them is anywhere in the first hundred results.

Same endpoint, same query, one HTTP header apart.

The MCP tool reads the same shelf

That only bears on the earlier article if search-actors ranks the same way the REST endpoint does. It does. Six queries through both, comparing the top five:

query	`search-actors` (token)	`/v2/store` (token)	`/v2/store` (no token)	top 5 identical
`sitemap checker`	#1	#1	absent	5/5
`PDF tables`	#2	#2	absent	5/5
`GitHub repository`	absent to 30	#40	absent	5/5
`broken links`	absent to 30	#29	absent	5/5
`RSS feed`	absent to 30	#47	absent	5/5
`tech stack`	absent	absent	absent	5/5

The top five agree on every query. Where my Actor appears in both channels, the rank is the same integer. So the first place I published — first place on sitemap checker, for an AI agent — was first place for an agent carrying my token. There is one such agent and I built it.

One query parameter says why

At this point I assumed the token was doing something to the ranking. It is not. The Store endpoint documents a parameter that tells you exactly what is happening, and you can use it without any credentials at all.

From the GET /v2/store reference, on includeUnrunnableActors, describing what the default excludes:

Actors from developers who haven't passed KYC, or full-permission Actors without a large user base

So I ran every query anonymously, twice — once plain, once with that flag on:

// unrunnable-probe.mjs — no credentials anywhere in this file
const q = 'github repository audit';
const base = `https://api.apify.com/v2/store?search=${encodeURIComponent(q)}&limit=100`;

for (const url of [base, `${base}&includeUnrunnableActors=true`]) {
    const { data } = await (await fetch(url)).json();
    const ids = data.items.map((a) => `${a.username}/${a.name}`);
    const at = ids.findIndex((id) => id.startsWith('aiqlabs/'));
    console.log(url.endsWith('true') ? 'unrunnable ON ' : 'default      ',
        'total:', data.total, 'mine at:', at < 0 ? '—' : at + 1);
}

default       total: 417 mine at: —
unrunnable ON total: 502 mine at: 1

Across five queries, unauthenticated both times:

query	default	`includeUnrunnableActors=true`
`sitemap checker`	absent, total 339	#3, total 352
`pdf table extractor`	absent, total 757	#2, total 877
`github repository audit`	absent, total 417	#1, total 502
`google play audit`	absent, total 5,102	#1, total 5,386
`http status checker`	absent, total 3,189	#6, total 3,379

Both calls are unauthenticated. The only difference is one documented query parameter.

The totals move. Between 13 and 284 more Actors enter each result set, and mine come back near the top of them. My Actors were never ranked badly. They were filtered out before ranking, for everyone except me.

Which of the two exclusions is mine

The documented filter covers two groups, and it is worth knowing which one you are in, because only one of them is fixed by paperwork. The permission level of any public Actor is readable without a token:

const r = await fetch('https://api.apify.com/v2/acts/aiqlabs~sitemap-checker');
const { data } = await r.json();
console.log(data.actorPermissionLevel, data.isPublic);   // LIMITED_PERMISSIONS true

All five Actors I checked come back LIMITED_PERMISSIONS, so the full-permission half of the filter does not apply to them. That leaves the other half, and Apify Console states it plainly:

Insights → Payouts, 16 August 2026. Billing details are registered on the same screen; the identity check is not.

The document I hold keeps being rejected by the verification vendor, which is an open thread with Apify support and not this article's subject.

What matters here is the shape of it. An incomplete account setting removed my Actors from every search except my own, and eleven days of work on titles, categories and output schemas went into Actors that were not in the result set while I was tuning them. Every one of those changes was aimed at Actor quality score, which Apify documents as the ranking input for both Store search and search-actors. Ranking inputs do nothing for a row that never reaches the ranking stage.

The Actor works. That was never the problem.

It is worth being clear about what is and is not broken here, because "invisible in search" and "unusable as a tool" are different failures and only one of them applies.

Name the Actor in the connection string and an agent picks it up as a tool immediately. Mine is pinned into my editor's MCP config, so I asked the agent to audit two well-known repositories. It called the Actor itself, waited for the run, and read the dataset back:

run m3PNCMD8HAAIGFpAq   SUCCEEDED in 2.402s   0.0027 compute units   2 items

requested	resolved to	archived	moved	last push	risk	issues
`facebook/create-react-app`	`react/create-react-app`	no	yes	547 days	medium	`repo_moved`, `stale_no_push`
`babel/babel-eslint`	`babel/babel-eslint`	yes	no	1,823 days	critical	`repo_archived`

Two and a half seconds, a fraction of a cent, and the agent had something it could act on: one dependency whose repository has quietly moved owner while the old URL still answers, and one that is archived outright. That is the whole point of exposing an Actor over MCP — the agent reaches for it mid-task instead of asking me to go and look.

So the tool definition, the input schema and the output fields all do their job. An agent that has this Actor uses it fine. The failure is one step earlier and entirely invisible from here: an agent that does not already have the name will not find it.

Why your own MCP session cannot tell you

The obvious control is to connect to the MCP server without a token and search again. That is not available:

$ curl -s -X POST https://mcp.apify.com/ -H 'content-type: application/json' \
    -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'
{"error":"invalid_token","error_description":"Missing or invalid access token. Pass an Apify API
token in the Authorization: Bearer <token> header. ..."}

HTTP 401, with no token and with an empty ?token= alike. The server speaks JSON-RPC over HTTP, as MCP requires, and every session belongs to somebody.

This is the part I would ask you to sit with, because it is structural and not personal. If you publish an Actor and want to know whether an agent can find it, the natural move is to connect your own client and search. That test is authenticated by construction. It will show you your Actor whether or not anyone else can see it. Your test passes, and it passes for the one reason that cannot generalise.

Nothing in the MCP documentation warned me about this, and I do not think it should have to. Returning an author their own Actors is a reasonable thing for a store to do — it is what makes testing an unpublished or unreviewed Actor possible at all. The gap is not in the behaviour. It is that the only search surface an Actor author naturally reaches for is the one that cannot be run anonymously.

The four-line check to run before you tune anything

No token, no dependencies, no cost:

// visible.mjs — run: node visible.mjs your-username sitemap checker
const [user, ...words] = process.argv.slice(2);
const url = (extra) =>
    `https://api.apify.com/v2/store?search=${encodeURIComponent(words.join(' '))}&limit=100${extra}`;

const seen = async (extra) => {
    const { data } = await (await fetch(url(extra))).json();
    const at = data.items.findIndex((a) => a.username === user);
    return `${at < 0 ? 'absent' : `#${at + 1}`}  (total ${data.total})`;
};

console.log('as everyone sees it :', await seen(''));
console.log('including filtered  :', await seen('&includeUnrunnableActors=true'));

Run against my own account and against a publisher who is not filtered, on the same query:

$ node visible.mjs aiqlabs sitemap checker
as everyone sees it : absent  (total 339)
including filtered  : #3  (total 352)

$ node visible.mjs automation-lab sitemap checker
as everyone sees it : #1  (total 339)
including filtered  : #6  (total 352)

Two different diagnoses from the same two lines. Mine is absent until the filter comes off. The other publisher is first without it, and slips to sixth with it — which is what a healthy result looks like, because turning the filter on adds thirteen competitors to that shelf.

So: if the first line says absent and the second gives you a rank, your Actor is being filtered out of search for every user but you, and no amount of ranking work will change that until the filter lifts. If both lines say absent, you have an ordinary ranking problem. If the first line gives you a rank, you are visible and can trust the numbers you measure.

I would rather have run those four lines on 5 August than on 16 August.

What I got wrong

I published a reachability figure and a rank without asking whose view produced them. The script that produced them had ?token= written into the URL, by me.

It is the second time in two weeks that I have measured my own visibility from inside my own session. The first was a rank I read off the Store web UI in a browser I was logged into. I caught that one, corrected it privately, and then made the same mistake again in a script — which tells me the lesson is not "be careful with browsers" but something duller: if a measurement can only be taken while authenticated, the number it returns is a statement about you, not about your users.

The earlier article's other findings survive. A default MCP connection really does hand an agent eleven tools with a single Actor among them; limit really does default to 5 and cap at 10; the result pages really are ragged, so a short page is not the end of the list; and broadening a keyword really does push my Actor down — that comparison sits inside one channel, so its direction holds even though both ranks are token-local.

What has to be withdrawn is the headline. One query in five is what I could see. For anyone else's agent, on the queries I re-tested, it was none of them.

Your Actor should never see my credentials: the security case for MCP connectors

Jeffrey Turov — Thu, 17 Sep 2026 22:34:36 +0000

Every automation platform has the same dirty secret: the easiest way to connect a third-party service is to paste a token into an input field. It works on the first try. It is also how credentials end up in places nobody intended.

I learned this building lead-generation Actors on Apify. My first working version took a GitHub token as an Actor input. It shipped fast, and it was wrong. This piece is about the threat model behind that mistake, and how MCP connectors invert it: the Actor I run never sees a credential at all.

Everything below comes from real builds and real run logs, including the failures.

The threat model nobody writes down

Passing a service token as an Actor input creates three leak paths, and all three are boring:

Run logs. Inputs are echoed, logged, and retained. Anyone with whom you share a run (a teammate, a support ticket, a screenshot in a bug report) potentially shares the token with it.
Shared input JSON. Actors get cloned, forked, and re-run from saved inputs. A token inside an input object travels with every copy.
Third-party Actor code. The moment you run code you did not write (a Store Actor, a fork, a colleague's experiment), any credential you hand it is only as safe as that code's worst console.log.

Notice what these have in common: none of them require an attacker. The leak vector is the architecture itself. The token is simply in more places than it needs to be.

The inversion: credentials live with the connector, not the code

Apify's MCP connectors restructure where the secret sits:

You authorize the third-party service (GitHub, Notion, Slack) once, in Apify Console under Settings, Integrations. The OAuth credential is stored with the connector, on the platform side.
At run time, the Actor receives a connector ID: an opaque string like ebw4ThD4cQbEKzC2l. It is not a token, and it is useless outside the platform.
The Actor talks to the Apify MCP proxy at ${ACTOR_MCP_CONNECTOR_BASE_URL}/<connectorId>, authenticating with the run's own APIFY_TOKEN.
The proxy enforces the tool permissions the Actor declared in its input schema.

The practical consequence: you can publish the Actor's source, share its runs, and paste its input JSON into a forum. There is nothing in any of those artifacts worth stealing.

Least privilege is declared, not hoped for

The second half of the model is the mcpServers rule in the input schema. This is where the Actor states which tools it needs:

"githubConnector": {
  "title": "GitHub connector",
  "type": "string",
  "resourceType": "mcpConnector",
  "mcpServers": [
    {
      "url": "*",
      "tools": {
        "required": ["create_*", "push_*", "update_*", "write_*", "commit_*", "get_*"],
        "readOnly": false
      }
    }
  ]
}

Two layers constrain what the Actor can do: what the connector exposed at authorization time, and what the schema declares. The intersection is enforced by the proxy at run time, not by documentation.

Verified in a real run log. My Actor's schema used the declaration above. When it called list_tools() through the proxy, it saw exactly 7 tools: create_branch, create_or_update_file, create_pull_request, create_repository, push_files, update_pull_request, update_pull_request_branch. Nothing else existed as far as the Actor could tell. The constraint layer is not a promise on a marketing page; it is observable behavior.

The honest limits

A security argument that only lists strengths is marketing. Three limits I hit or verified:

1. The connector's tool set is frozen at authorization time. When you authorize a connector, the platform discovers the available tools once. If the upstream server later adds tools you want, you re-authorize. The set does not refresh itself. I lost time to this before reading it properly in the docs: it is "Layer 1" of their two-layer model, and it is by design (a silent expansion of your Actor's powers would be a security hole, not a feature).

2. The run still authenticates with a token: the starter's APIFY_TOKEN. The proxy trusts whoever started the run. That means access control moves to who can start your Actor with your connector, which is an Apify permissions question, not a code question. For Actors you share or publish, treat connector-enabled inputs as capability grants and review who can run what.

3. Connectors do not sanitize your payloads. The proxy enforces which tools the Actor calls, not what data flows through them. An Actor scraping personal data and pushing it to your repo is still your compliance problem. The connector solves credential custody, not data governance.

Pitfalls from real runs

Three failures I actually hit while building on this model, because they cost me runs:

SDK drift. The MCP Python SDK docs show streamable_http_client unpacking into three values (read, write, _). The version my Docker image installed yields two. ValueError: not enough values to unpack. Indexing the tuple (streams[0], streams[1]) works across versions. Same class of issue: the tool result error flag is result.is_error (snake_case), not the isError casing the TypeScript-flavored docs suggest.

Default branch assumptions. create_or_update_file failed with Branch main not found: my repo's default is master. The tool does not fall back to the repository default; it fails loudly, which is the correct behavior for a security boundary. Detect and retry with the other common name, or pass the branch explicitly.

Write-only connectors and the SHA problem. Updating an existing file on GitHub requires its current blob SHA. My write-scoped connector exposed no get_file_contents to fetch it. Rather than widening permissions, I switched the pipeline to immutable, timestamped snapshots: each run writes a new dated file, and Git itself becomes the history. The least-privilege constraint pushed me to a better design, which is what least privilege is supposed to do.

When to use what

Situation	Token as input	MCP connector
Local throwaway script, one run, you watch it	Fine	Overkill
Actor you will run on a schedule for weeks	Leak path	Right answer
Actor you publish or share	Irresponsible	Right answer
Running someone else's Actor with your services	Never	The only sane option

The last row is the one that convinced me. If you consume third-party Actors, connectors are the only way to grant access to your services where a sloppy or malicious print() cannot exfiltrate your credentials.

The takeaway

Security models for automation are usually about adding vigilance: rotate tokens, restrict scopes, audit logs. MCP connectors remove the attack surface instead: the credential never enters the Actor's world, so it cannot leak from it. You trade a small amount of convenience (one authorization step in Console, a frozen tool set) for the ability to share runs, publish source, and run untrusted code without a knot in your stomach.

The working example this article is drawn from is public: maps-to-stack on GitHub, and the build walkthrough with the scraping pitfalls is on dev.to. Connector documentation: docs.apify.com/integrations/mcp-connectors.

Build log details verified against run logs from August 2026: tool filtering (7 tools), branch fallback, SDK unpacking, and the immutable-snapshot pattern all occurred in real runs of the maps-to-stack Actor.

Investor prospecting lists from PitchBook and LinkedIn data

Crawler Bros — Thu, 17 Sep 2026 05:21:25 +0000

Investor prospecting lists from PitchBook and LinkedIn data

Fundraising research usually means either paying for a PitchBook or Crunchbase seat the founder only needs for a few months, or manually building a spreadsheet from investor websites and LinkedIn one profile at a time. Neither is a great use of the limited time founders have during an active raise.

Investor prospecting with public data means joining public investor profile data with public LinkedIn partner profiles into one targeted, sector- and stage-matched list, instead of paying for a full research-platform seat or building the list one profile at a time by hand.

Quick answer

Use PitchBook Investors Scraper to collect public investor profiles — firm name, focus areas, and stated investment criteria — without a PitchBook subscription. Use LinkedIn Profile Scraper to enrich individual partner or associate profiles with role, background, and current firm. Combine both into a prospecting list segmented by sector and stage fit.

The investor prospecting stack

The stack uses two CrawlerBros Actors:

PitchBook Investors Scraper for public investor profiles without a subscription.
LinkedIn Profile Scraper for public LinkedIn profile information without login.

The workflow:

\Sector and stage criteria -> PitchBook Investors Scraper (firm-level investor profiles) -> LinkedIn Profile Scraper (partner-level enrichment) -> segmented investor prospecting list \\

Key facts

Firm-level data answers "who invests in this"; partner-level data answers "who do I email." Both are needed for outreach that isn't generic.
Start from known profile URLs, not broad category browsing. Both Actors work best against specific, already-identified profile pages.
Focus fields go stale. A firm's stated sector/stage focus can change; re-verify before outreach, not just before building the list.
Fit beats volume. A well-matched list of 20-50 firms consistently outperforms an unfiltered list of hundreds.

Actor configuration that matters

For PitchBook Investors Scraper, start from firm or investor profile URLs discovered through sector research rather than trying to enumerate an entire investor category in one run — public profile pages are the reliable input; broad category browsing is not.

\json { "profileUrls": [ "https://pitchbook.com/profiles/investor/example-1", "https://pitchbook.com/profiles/investor/example-2" ], "includeInvestmentCriteria": true } \\

For LinkedIn Profile Scraper, feed it the specific partner or associate profile URLs found through the PitchBook data or firm websites, since public profile scraping works best against known URLs rather than broad search.

What the output looks like

The PitchBook row needs firm name, focus sectors, stage focus, and stated check size when available. The LinkedIn row needs name, title, current firm, and background summary. Joining the two gives a prospecting row that answers both "does this firm invest in what I'm building" and "who specifically should I reach out to."

Use case 1: targeted seed and Series A outreach

A founder raising a specific round size in a specific sector can filter PitchBook profiles for matching stage and sector focus, then use LinkedIn enrichment to identify the right partner to reach rather than sending a cold email to a generic firm inbox.

Use case 2: warm-intro path mapping

Once a target investor list exists, cross-referencing partner LinkedIn backgrounds against a founder's own network (previous companies, schools, mutual connections) helps prioritize which firms are reachable through a warm introduction versus cold outreach.

Use case 3: competitive fundraising intelligence

Tracking which investors a competitor has raised from, where that's publicly known, helps a founder understand which firms are already active in the category and how to differentiate outreach messaging for firms likely to be comparing the two companies.

Production notes

Start from known profile URLs, not broad scraping. Both Actors work best against specific, already-identified profile pages rather than attempting to enumerate an entire category of investors in one pass.

Respect what's public. Only collect information visible on public profile pages. Avoid combining this with any data source that would require login credentials or private access.

Re-verify stage and sector focus before outreach. Investor focus areas change; a profile scraped months ago may not reflect a firm's current thesis.

Prioritize fit over volume. A list of 20 well-matched investors outperforms a list of 200 firms scraped without sector or stage filtering.

Firm-level data alone produces low response rates. A firm-level list without partner-level context means outreach goes to generic inboxes. Adding LinkedIn partner enrichment makes outreach targeted enough to get meaningfully better response rates.

Cost comparison

Approach	Cost	Data freshness	Partner-level detail
PitchBook/Crunchbase subscription	Expensive, ongoing	High	Yes
Manual research	Free, very slow	Depends on effort	Inconsistent
Apify PitchBook + LinkedIn pipeline	Pay per run	On demand	Yes

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify replace a PitchBook subscription?

It can replace the workflow of researching public investor profiles for a specific list; it doesn't replicate PitchBook's full paid database and analytics.

Is scraping public LinkedIn profiles allowed?

LinkedIn Profile Scraper collects publicly visible profile information without login; always review current platform terms before large-scale collection.

How big should a first investor list be?

Most founders get better results from a well-matched list of 20-50 firms than an unfiltered list of hundreds.

What's the fastest way to try this?

Pick 10 investor firms whose public focus matches your sector and stage, then run PitchBook Investors Scraper and LinkedIn Profile Scraper against them.

Try it yourself

Pick 10 investor firms whose public focus matches your sector and stage. Run PitchBook Investors Scraper against their profile pages, then use LinkedIn Profile Scraper to find the right partner at each. A short, well-matched list beats a long, generic one.

Facebook Marketplace intelligence: resale pricing and comment sentiment

Crawler Bros — Wed, 16 Sep 2026 05:15:01 +0000

Facebook Marketplace intelligence: resale pricing and comment sentiment

Facebook Marketplace is one of the largest resale and local-commerce platforms, and one of the least monitored. Most price-intelligence tools skip it entirely because it's local, listing-heavy, and constantly turning over.

Facebook Marketplace intelligence means combining public listing data with public comment data to price resale inventory, spot unauthorized brand resale, and read how a community actually reacts to a local post — instead of relying on manual browsing or reaction-emoji counts alone.

Quick answer

Use Facebook Marketplace Scraper to collect listing prices, condition, and seller data for a product category or location. Use Facebook Comments Scraper to pull public reactions on posts, whether that's a Marketplace listing, a local business page post, or a community group thread. Combine both to price resale inventory and read local sentiment in the same workflow.

The Marketplace intelligence stack

The stack uses two CrawlerBros Actors:

Facebook Marketplace Scraper for listing data, pricing, and seller information.
Facebook Comments Scraper for public comments on posts and videos with metadata.

The workflow:

\Product category or location -> Facebook Marketplace Scraper (listing prices, condition, seller) -> Facebook Comments Scraper (reactions on flagged listings or posts) -> resale price and sentiment dataset \\

Key facts

Location beats keyword. Marketplace is a local marketplace by design — a keyword search without a location filter returns inconsistent results.
Condition explains price, not irrationality. Two identical-looking listings at different prices are usually explained by condition, not by one seller being unreasonable.
Listing velocity beats a single price. A category with fast-turning listings at a given price point signals real demand better than a handful of stale unsold listings.
"Public" has a specific meaning here. Marketplace listings and public post comments qualify; private group content doesn't.

Actor configuration that matters

For Facebook Marketplace Scraper, scope by category and location rather than a broad keyword search — Marketplace results are heavily location-weighted, so a keyword search without a location filter returns inconsistent results.

\json { "category": "furniture", "location": "Austin, TX", "maxItems": 150, "sortBy": "date_listed" } \\

For Facebook Comments Scraper, feed it the specific post or listing URLs you want reactions on rather than trying to crawl comments broadly — Facebook's comment threading and pagination behave differently across post types, so targeted URL lists produce more reliable results than open-ended searches.

What the output looks like

The Marketplace row needs title, price, condition, location, seller name, listing date, and listing URL. Price and condition together are what make resale comps useful — a "good condition" $150 listing and a "like new" $150 listing aren't the same data point.

The Comments row needs comment text, author, timestamp, and post URL, so a spike in complaints or praise can be traced back to the specific listing or post that triggered it.

Use case 1: resale price benchmarking

A reseller flipping furniture, electronics, or collectibles can pull active listings for a category and location, then price new inventory against what's actually moving rather than guessing from a handful of manually checked listings.

Use case 2: gray-market and unauthorized resale monitoring

A brand that sells through authorized retailers can search Marketplace for its product names to spot bulk resale, counterfeit listings, or unauthorized liquidation stock showing up at prices well below MSRP — a pattern that's hard to catch by browsing manually.

Use case 3: local business post sentiment

A local business posting about a new menu item, event, or policy change can pull comments on that post to see the actual reaction, rather than relying on the reaction-emoji count alone. Comment text often reveals why a post landed well or badly, which the emoji summary doesn't.

Production notes

Scope by location first. Marketplace is a local marketplace by design — location filtering matters more here than on almost any other platform in this batch.

Track condition alongside price. Two identical-looking listings at different prices are often explained by condition, not by one seller being irrational.

Watch listing velocity, not just price. A category with fast-turning listings at a given price point tells you more about real demand than a handful of stale listings sitting unsold.

Respect what "public" means here. Only collect what's visible without logging in or joining a private group. Marketplace listings and public post comments qualify; private group content does not.

Price without condition confuses more than it clarifies. Price without condition and listing age produces comps that look contradictory. Adding condition and days-listed as required fields fixes most of the confusion.

Cost comparison

Approach	Coverage	Local granularity	Sentiment data
Manual browsing	One category at a time	High, but slow	None
General resale-price tools	Limited or no Marketplace coverage	Low	None
Apify Marketplace + Comments pipeline	Category and location scoped	High	Yes

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify scrape Facebook Marketplace listings?

Yes. Facebook Marketplace Scraper extracts public listing data including price, condition, and seller information.

Can I pull comments from any Facebook post?

Facebook Comments Scraper works on public posts and videos; private group or profile content isn't accessible.

Is this useful outside of reselling?

Yes — brand protection, local sentiment tracking, and community research all use the same two-Actor pattern.

What's the fastest way to try this?

Pick one product category and city, run Facebook Marketplace Scraper for current listings, then run Facebook Comments Scraper on the most-commented listings.

Try it yourself

Pick one product category and one city. Run Facebook Marketplace Scraper for current listings, then run Facebook Comments Scraper on the two or three most-commented listings you find. The price spread plus the comment tone will tell you more about that local market than either dataset alone.