DEV Community: Ai-Q Labs

A default MCP connection hands an agent 11 tools. None of my 23 Actors is one of them.

Ai-Q Labs — Fri, 18 Sep 2026 00:10:09 +0000

A default MCP connection hands an agent 11 tools. None of my 23 Actors is one of them.

I have 23 audit Actors on the Apify Store. They all do a version of the same thing: take a public record, check whether what it still claims is true, and write the verdict into a dataset.

I have also written six articles about what happens when an AI agent calls one of them through the Apify MCP server. Output fields that arrive empty. Input schemas that read differently to a form and to a tool. A default that quietly fills a required field. A connector that shows 44 tools and grants four.

MCP is the Model Context Protocol, the interface that turns an Actor into a tool that clients like Claude and Cursor can call on their own. Every one of those six articles assumed the agent had that tool in hand. Last week I checked the assumption for the first time. It does not hold by default.

The connection string I never read

I found the problem in my own machine before I found it anywhere else. This is the Apify entry in my Claude config, verbatim except that there is no token in it:

"url": "https://mcp.apify.com?tools=aiqlabs/github-repository-audit"

I wrote that line months ago to test one Actor in isolation, and then left it. My editor session had five Apify tools in it. One of them was an Actor, and it was the one named in that string.

So I had spent six articles reasoning about how agents see my catalogue. The session I was reasoning from had been narrowed to a single Actor, by me, on purpose, and then forgotten. The first thing worth measuring was what a connection with nothing in it returns.

What a default connection actually sends

Twenty lines, no dependencies. This is the whole test:

// reach.mjs — what does a default connection to the Apify MCP server actually send?
// Run: APIFY_TOKEN=... node reach.mjs
const URL = `https://mcp.apify.com/?token=${process.env.APIFY_TOKEN}`;
const HEAD = { 'content-type': 'application/json', accept: 'application/json, text/event-stream' };

async function rpc(body, sid) {
    const res = await fetch(URL, { method: 'POST', body: JSON.stringify(body),
        headers: sid ? { ...HEAD, 'mcp-session-id': sid } : HEAD });
    const text = await res.text();
    // the server answers as SSE, so the JSON sits on a "data:" line
    const line = text.split('\n').find((l) => l.startsWith('data:'));
    return { sid: res.headers.get('mcp-session-id') ?? sid, json: line ? JSON.parse(line.slice(5)) : null };
}

const { sid } = await rpc({ jsonrpc: '2.0', id: 1, method: 'initialize', params: {
    protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'reach', version: '1' } } });
await rpc({ jsonrpc: '2.0', method: 'notifications/initialized' }, sid);

const { json } = await rpc({ jsonrpc: '2.0', id: 2, method: 'tools/list' }, sid);
const tools = json.result.tools;
console.log(`tools: ${tools.length}`);
for (const t of tools) console.log(' ', t.name);

Its output, on 12 August:

tools: 11
  search-actors
  fetch-actor-details
  call-actor
  get-actor-run
  get-dataset-items
  get-key-value-store-record
  abort-actor-run
  search-apify-docs
  fetch-apify-docs
  report-problem
  apify--rag-web-browser

Ten of those are the server's own tools. One is an Actor: RAG Web Browser, which belongs to Apify. Of the 23 Actors I have published, zero are in the list.

Nothing is being hidden. Add ?actors= to the same URL and name all 23 slugs. The same server then returns 27 tools: my 23 plus four storage tools. The Actors are reachable. They are not default.

The default is also defensible. The Store holds thousands of Actors. A server that turned every one of them into a tool definition would blow out the context window of every client that connected. Eleven tools is a budget, not a snub.

But it does mean that everything I had written was conditional on a step I had never measured.

Three ways an Actor reaches an agent

Route	Who decides	What an author can do about it
A — named in the connection URL (`?actors=` / `?tools=`)	whoever configures the agent	Almost nothing. They have to know your slug before they can type it.
B — included in the default set	Apify	Nothing. Today that set contains one Actor.
C — found at runtime by `search-actors`	Store search ranking	This one.

A and B are not work I can do. C is the entire surface, so I measured it.

Measuring the third route

The first two attempts failed on the parameter name. {"search": …} and {"query": …} are both accepted and both silently ignored. The server echoes Search query: back empty and returns its default list. Rather than guess a third time, I read the tool's own inputSchema out of the JSON-RPC reply to tools/list above:

keywords : string   default ""
limit    : integer  default 5, max 10
offset   : integer  default 0

The parameter is keywords. But the line that matters more is limit. It defaults to 5 and caps at 10. That is the window. An agent that runs one search and picks from it is choosing among five Actors out of the whole Store.

The second trap is in the response. Each result set is introduced by a line like **Number of Actors found:** 7, which reads as a total and is not one. It is the size of that page:

Instagram keeps producing fresh results at offset 0, 10, 20 and 30 — 39 distinct Actors, no repeats. One of those four pages came back with nine rather than ten, which is the second thing to know: the pages are ragged.
Chrome extensions returns 7 at offset 0 and a different 6 at offset 10. A short page does not mean the list ended.

My first pass treated the short page as the end of the list, so it never looked past page one. It reported seven of my Actors as absent when they were sitting at ranks 12, 16, 31, 41, 52, 55 and 57. Every number below comes from the corrected run. That run pages to rank 100 whatever the page count says, and computes rank as offset + position in page, because the pages come back ragged.

What it found

I paired each of 23 Actors with the query it was built for — 28 queries in all, since several Actors got both a specific phrase and the broader term a user might type instead. Then I asked how deep in the results that Actor sits.

Where my Actor lands	queries	share
inside the default 5 the agent receives	6	21%
inside the maximum 10	7	25%
rank 11–100: indexed, past the window	7	25%
not in the first 100	14	50%

On the queries these Actors were written for, an agent doing a default search finds one of them about one time in five. Half of them are not in the first hundred results at all.

That was worse than I expected, but it was not the finding. The finding is which half.

The queries I can win are the queries nobody is making

Every search-actors result carries the Actor's usage with it, in a line like **Stats:** 20 total users, 3 monthly users. So the demand on a query is readable from the same response that gives you the ranking: take the 30-day user count of whatever sits at rank 1.

my rank	queries	median 30-day users of that query's rank-1 Actor
reachable (top 10)	7	6
unreachable (past 10, or absent)	21	23

The unreachable side is where the users are. App Store — 311. Google Play — 295. article extractor — 271. Shopify — 186. My Actors for the first three of those are not in the first hundred results.

And the one query where I rank #1 is sitemap checker, where the rank-1 Actor has 0 users in the last thirty days. That Actor is mine. I am first on a shelf with no traffic, and invisible on every shelf with traffic.

I had measured the same shape once before, from the other side. On 2 August I sampled the Store itself and found that Actors naming a well-known platform had a median of 5 monthly users against 2 for the rest. I read that as an argument for naming platforms. This measurement says the platform-named shelves are exactly the ones where I cannot be seen — 1 reachable query out of 14, against 6 out of 14 for the generic ones. Both results are true and they are not in conflict: platform names are where demand collects, demand attracts competitors, and competitors fill a window five deep. But I predicted the wrong one.

The tool tells the agent to broaden the search, and broadening removes me

Every search-actors response ends with this, addressed to the agent:

IMPORTANT: You MUST always do a second search with broader, more generic keywords (e.g., just the platform name like "TikTok" instead of "TikTok posts") to make sure you haven't missed a better Actor.

That is good advice for the user. It is also an instruction to run the query I do worst on. Same Actor, specific phrase against the broadened phrase:

specific	rank	broadened	rank
`sitemap checker`	#1	`sitemap`	not in first 100
`PDF tables`	#2	`PDF`	not in first 100
`PDF text markdown`	#2	`PDF`	not in first 100
`Chrome extensions`	#16	`Chrome Web Store`	not in first 100
`Shopify apps`	#41	`Shopify`	not in first 100
`GitHub repository`	#52	`GitHub`	not in first 100
`App Store apps`	absent	`App Store`	absent
`Google Play apps`	absent	`Google Play`	absent

Six worse, none better, two unchanged. The clearest one is the first: sitemap checker puts me at #1, and deleting one word puts me past rank 100. Whatever advantage a precise name buys, the agent is under instructions to take a second look without it.

What decides the order

Not popularity, at least not alone. Across every first page I collected, 157 of 238 adjacent pairs are in descending order of monthly users — 66%. A pure popularity sort would be 100%. Relevance carries real weight, which is how an Actor with no users reaches #1 on sitemap checker.

Apify documents what does decide it, and says so for both surfaces at once. From Actor quality score:

Actors with higher quality scores tend to rank higher on both surfaces, though no specific position is guaranteed.

The second surface named on that page is "the Apify MCP server search-actors tool used by external AI agents". The score aggregates eight categories: reliability, popularity, feedback and community, ease of use, pricing transparency, trustworthiness, history of success, and congruency of texts.

I have moved it once and can report the size of the move. On 2 August I added dataset output schemas across the catalogue; the score on the Actor I watched went from 74 to 78, and after that pass the whole catalogue sat between 78 and 81. There is no API for the number — I tried four endpoints and got 404 from each — so the only place to read it is Console > Insights > Actor quality.

Here is that panel today, for the Actor this article keeps returning to:

79 out of 100. Better than 71% of Actors on the platform, better than 99% on reliability, and one suggestion left in the panel.

That is the pairing worth sitting with. Console rates this Actor above 71% of the platform and has one cosmetic item left to suggest. The same Actor is absent from the first hundred results on every query with real demand behind it, and its one first place is on a query with none. Both readings are correct, and the documentation is careful enough to allow it: a higher score tends to rank higher, with no position guaranteed. On a crowded shelf, tending is not the same as landing in the five results the agent receives.

Which leaves the honest part. Three of the eight categories — popularity, feedback and community, history of success — are things you get from having users. For an Actor with none, part of the ranking that would bring users is held shut by not having them yet. That is not a complaint about the design; a store that ranked unproven tools first would be worse for everyone using it. But it is the actual shape of the problem, and no amount of description polishing changes it.

What I would tell an Actor author

Check what your connection sends before you tune anything. The 20 lines above answer it. If your Actor is not in the list, none of your tool-definition work is reaching an agent yet.
Measure your rank on the query you were built for, with limit: 10, paging past the first short page. Anything past rank 10 is in the index and outside the window.
Read the demand off the same response. The rank-1 Actor's monthly users are printed next to it. If that number is 0, being #1 buys nothing — I have the receipt.
Expect the broad term to be worse. It is also where the demand is. Decide which of those two facts you are building for, rather than discovering the trade after you publish.

What I got wrong

Three things, in the order I found them.

I narrowed my own MCP connection to one Actor and then wrote six articles about how agents see my catalogue. The narrowing was in a config file I had not opened in months.

I read a page count as a total, and my first pass reported seven Actors as unreachable that rank between 12 and 57. Had I not checked whether offset kept returning results, this article would have carried seven false claims of the most flattering kind — the kind where the platform looks worse than it is.

And I expected the platform-named Actors to be the findable ones. They are the least findable ones I own.

None of that makes the earlier six articles wrong. It reorders them. An input schema that misleads an agent, an output schema that arrives empty, a default that fills a required field — all of those are real, and all of them start to matter at the moment the agent receives the tool. For 23 out of 23 of mine, that moment does not happen by default.

I pinned my Actors to the MCP URL, the way the docs recommend. The agent lost every way to see what a call costs.

Ai-Q Labs — Fri, 18 Sep 2026 00:09:29 +0000

I pinned my Actors to the MCP URL, the way the docs recommend. The agent lost every way to see what a call costs.

I publish 23 Actors on the Apify Store. All 23 are pay-per-event.

Last week I measured how an AI agent actually reaches them. The answer was uncomfortable. A
default MCP connection hands an agent 11 tools. None of mine is one of them.

So you pin. You name your Actors in the connection URL. The
Apify MCP documentation tells you to:

For production deployments, explicitly specify which tools to load rather than relying on
defaults. This ensures consistent behavior across updates.

That is good advice for reliability. I followed it. Then I measured what the agent receives
afterwards, and found something the docs do not mention.

Pinning removes the agent's ability to see what a call costs. It also removes its ability to
cap what a call spends.

Below are the measurements. Then the three tools that disappear. Then the half of this that is
my own fault rather than the platform's.

What pinning actually changes

I connected three ways and listed the tools each time. Same account, same day, 2026-08-12.
Every listing came from a tools/list call over the
MCP HTTP transport. That is what the client
library hands the model.

Default connection, no parameters. Eleven tools:

search-actors          get-key-value-store-record   report-problem
fetch-actor-details    abort-actor-run              apify--rag-web-browser
call-actor             search-apify-docs
get-actor-run          fetch-apify-docs
get-dataset-items

Pinned with two of my Actors, using the documented parameter
?tools=aiqlabs/seo-audit-tool,aiqlabs/pdf-inspector. Six tools:

aiqlabs--seo-audit-tool    get-actor-run                 abort-actor-run
aiqlabs--pdf-inspector     get-dataset-items
                           get-key-value-store-record

Pinned with all 23. Twenty-seven tools: my 23, plus the same four.

The four survivors are all post-run tools. Each one reads a run, reads a dataset, reads a
key-value record, or aborts something already running.

Every tool that helps an agent decide before it calls is gone. Three of the missing ones
matter for money.

The three tools that carried the price

I searched the full JSON of each default tool definition for pricing language. These are the
strings that came back.

search-actors returns pricing in its results. Its own description says so:

- **Pricing:** Details with pricing link

Its output schema enumerates the models. FREE, PRICE_PER_DATASET_ITEM,
FLAT_PRICE_PER_MONTH, and a pricePerUnit field.

fetch-actor-details takes a flag whose description reads, in full:

Include pricing model and costs.

call-actor is the one that actually protects the caller. It accepts maxTotalChargeUsd:

Pay-per-event Actors only — ignored otherwise.
Caps total USD billed; does NOT limit work. Prefer the Actor's own input fields to bound work.

That is a spending ceiling. The caller sets it, at call time. Note the second sentence. I will
come back to it, because it is the sentence that indicts me.

All three tools are in the default set. None of them survives pinning.

An agent holding my Actors directly cannot look up what they cost. It cannot put a ceiling on
what it spends. It can only call them.

What my own definitions tell the agent about price

Nothing. I checked, and I checked wrong the first time.

My first pass searched each tool definition for a dollar sign. It reported a hit on 23 of 23.
For about a minute I believed my descriptions already carried prices.

They do not. The dollar signs were $id, the JSON Schema keyword:

"required":[],"$id":"https://apify.com/mcp/aiqlabs--app-store-audit"

I searched again for a dollar sign followed by a digit. An actual amount. The count was
0 of 23.

The words price and cost do appear in a few of my definitions. Every one of them is about
the subject matter, not the invoice.

One Actor notes that prices and availability differ by storefront. Another explains that each
archived URL costs a request to the live site.

Those are useful sentences. Neither tells an agent what calling the tool will cost.

Meanwhile every one of the 23 is metered. Each has two charge events, configured the way
Apify's monetization docs
describe:

Event	Price
`apify-actor-start`	$0.00005
`apify-default-dataset-item`	$0.002 – $0.01 depending on the Actor

The per-item price is the one that moves.

I wanted to be sure those events fire on their own. My Actors never call a charge function. So
I read a real run record rather than assuming:

run thLehE7kk3NVRAvqm  SUCCEEDED
chargedEventCounts: {"apify-actor-start":4,"apify-default-dataset-item":608}

They fire. Another run in the same batch recorded 1,000 dataset items. At my highest per-item
price, one call like that is ten dollars.

The part that is my fault

Go back to that sentence in call-actor:

Prefer the Actor's own input fields to bound work.

Apify is right about this. A billing cap stops the invoice, not the work. The real bound
belongs in the
input schema,
where the author controls it.

So I checked whether my own Actors carry that bound.

All 23 have a field for it. maxUrls, maxPages, maxApps, maxPdfs, maxVehicles,
maxOrganizations, maxStories. Twenty-three out of twenty-three.

Not one of them declares a JSON Schema maximum.

pdf-inspector        maxPages(none), maxPdfs(none), maxFileMb(none), maxLinksToCheck(none)
sitemap-checker      maxUrlsToCheck(none), maxSitemaps(none), maxDepth(none)
http-status-checker  maxUrls(none), maxRedirects(none), maxConcurrency(none)

I named the fields as if they were limits. I never made them limits.

An agent can pass maxUrls: 1000000. The schema will accept it. I told the schema that any
integer is fine.

So the failure has two halves. Only one of them is the platform's.

Pinning removes the caller-side ceiling. That is the platform's shape. My Actors have no
author-side ceiling. That is mine, in 23 files I wrote myself.

An agent holding my pinned tools has no cap available from either direction.

What I got wrong on the way here

I started this measurement expecting a different problem.

My hypothesis was time. Actor runs are slow. Agents time out waiting. Authors ship tools an
agent cannot practically await.

I pulled the run history for all 23 and computed durations from the API.

	Value
Median of per-Actor medians	4.05 s
Slowest single run observed	58.0 s
Actors with a median above 60 s	0 of 23

There is no timeout problem. My hypothesis was wrong.

I dropped that half of the article rather than stretching the numbers to fit it. Only five of
the 23 had enough run history to produce a median, and that limit is worth stating. But nothing
in the data pointed toward latency. I would rather report a dead hypothesis than a decorated
one.

What to change, concretely

Three changes. The first two are the author's job. The third is a question for the platform.

1. Put a real ceiling in the input schema.

A field named maxPages should refuse an absurd value. The
maximum keyword does
this before your code runs:

"maxPages": {
  "title": "Max pages per PDF",
  "type": "integer",
  "default": 50,
  "minimum": 1,
  "maximum": 2000,
  "description": "Hard cap. Each page produces one billed result row."
}

default is not a limit. prefill is not a limit either, and the two are easy to confuse in
Console. That distinction bit me once already, from the correctness side. This is the same
lesson arriving from the money side.

You can verify the deployed schema rather than trusting the file:

curl -s https://api.apify.com/v2/acts/<user>~<actor> \
  | grep -o '"maximum":[0-9]*' | head

2. Say the price in the description the agent reads.

Once you pin, the agent has no other source. One clause is enough:

Each result row is one billed event ($0.002). A 500-URL input bills roughly $1.00.

That sentence costs you nothing. It is the only pricing signal a pinned agent will ever see.

3. The platform question.

call-actor exists in the default set and carries maxTotalChargeUsd. Pinned tools are
invoked directly, so no wrapper carries it.

A per-call spending cap that survived pinning would close the gap. It could be an optional
argument on every metered tool, or a connection-level parameter. Either way it would spare each
author from rediscovering this alone.

I do not know whether that is on anyone's roadmap. I am reporting the shape I measured.

Where this leaves an Actor author

Two connection modes, and a cost to each.

Leave the default. The agent can search, can read your pricing, can cap its spend. It will also
almost certainly never receive your Actor. The default set is 11 tools and none of them is
yours.

Pin your Actors. The agent receives them and calls them directly. It does so without a price
and without a ceiling.

I do not think either mode is wrong. I think the second one quietly moves a responsibility onto
the Actor author. The docs that recommend it do not mention the move.

Until they do, the fix is the schema you already control. Mine was missing it in all 23 files.
I only found out because I went looking for a different problem entirely.

Measurements were taken on 2026-08-12 against mcp.apify.com with a single account. Tool
listings came from tools/list over the MCP HTTP transport. Run durations and charge counts
came from
Apify API run records. No Actor was executed to
produce these numbers, so nothing here cost anything to measure.

I removed the under maintenance label from my Actor twice. It came back in four hours, then in one.

Ai-Q Labs — Fri, 18 Sep 2026 00:08:48 +0000

I removed the under maintenance label from my Actor twice. It came back in four hours, then in one.

I publish 23 Actors on Apify. Twenty-two of them take a public record and check whether what it
still claims is true.

The twenty-third is different. It reads an audit dataset and opens a GitHub issue for each finding.
It reaches GitHub through an Apify MCP connector,
so the Actor never handles my token.

That difference is the whole story. Two days after I published it, Apify flagged it as under
maintenance. I spent an afternoon removing the label by hand and learned that the label is not a
setting. It is a conclusion.

The email

The notification arrived at 10:08 on 8 August. My Actor had failed the automated tests.

Apify runs a daily test on every Actor.
The documentation is precise about what it does:

The test runs the Actor with its default input (defined by the prefill option in the input schema
file) and expects it to finish with a Succeeded status and non-empty default dataset within 5
minutes of the beginning of the run.

My Actor cannot do that. Not on a bad day — on any day.

Why this Actor and not the other 22

My other Actors take a URL, a domain, or a package name. Every one of those is a string. A string
goes in the prefill of the input schema, the test picks it up, and the run succeeds.

An authorized connector is not a string. It is a grant, tied to an account, held by the platform.
There is no value I can put in prefill that gives the test account access to my GitHub
installation. Nor should there be.

So the run fails at the first step, every day, forever.

The documentation says this outright, in a section I had not read:

Actors that require some sort of authentication will always fail the tests despite being fully
functional.

I want to be exact about what this means, because I got it wrong at first. I had assumed the cost
belonged to my design choice of failing fast on empty input. It does not. Twenty-two of my Actors
fail fast on empty input and none of them were flagged. The cost belongs to a narrower category:
Actors whose only successful path requires a credential the test account cannot hold.

If that describes your Actor, the daily test is not a quality signal about your code. It is a
structural mismatch, and it will not resolve itself.

What I did instead of reading the docs

I went to Console. Under Publication → Display information there is an Actor status control with an
"Under maintenance" switch. I turned it off and saved. The banner disappeared. The public page
stopped showing the badge.

I recorded it as fixed. It was not fixed. It was hidden.

At 17:08 the same day, about four hours after I cleared it, the second notification arrived. The
label was back.

I removed it again. At 18:08 — inside an hour — the third notification arrived.

Two removals, two returns, and the interval got shorter. That second number is what made me stop
and read.

The label is not a setting

Here is the mechanism, and it is stated plainly in the same documentation page I had skipped:

If the Actor fails to complete successful runs for three consecutive days, the developer will be
notified, and the Actor will be labeled under maintenance until it is fixed.

And the other direction:

The best course of action is to fix the Actor and rebuild it. The automatic testing system will
pick this up within 24 hours and mark it as healthy. In some cases, your Actor might break because
of issues with the target website. In such a case, if your Actor passes the majority of test runs
in the next 7 days, it will be marked as healthy automatically.

Read those two together. The label is derived from your run history, and the system recomputes
it. The switch in Console writes to a field that an evaluator overwrites on its own schedule.

For a normal broken Actor, the switch is honest and useful. You fix the bug, you rebuild, you clear
the label, and the next evaluation agrees with you. Your manual action and the computed state
converge.

My case had no convergence. The failures were still in the window and would be there tomorrow too.
Every removal I made was a claim the evidence contradicted, so the evaluator restored it.

I also stopped for a second reason, which has nothing to do with mechanics. Repeatedly toggling a
flag that an automated system keeps restoring is not a fix. It is an argument with a robot, in
public, on my own store listing. I did not want to find out how that reads from the other side.

The path that actually worked

The documentation tells you where to go:

If that's the case with your Actor, contact support and explain your specific use case that
justifies why the Actor should be excluded from the automated tests.

There is also a form. While my Actor was flagged, Console opened a skip-test request straight from a
query parameter:

https://console.apify.com/actors/<actorId>/publication?showSkipTestForm=true

One caveat I can only report, not explain. Two days later, with the exemption granted, that same URL
renders the ordinary publication page for me. So treat it as something that is there while you are
flagged, not as a permanent entry point. If it does not open, the documented route is support, and
that is the route the docs tell you to take anyway.

I filled it in with three sentences: what the Actor does, why the test account cannot authorize the
connector, and the sentence from the docs that describes exactly my situation. Quoting their own
documentation back to them felt lazy. It was the right call — it turned a request for special
treatment into a request to apply a rule that already existed.

The reply came in about 31 hours:

After reviewing your situation, we've approved your request. Your Actor will now not be checked by
our tests.

Then I cleared the label a third time. This one held, because there is no longer an evaluator
producing a verdict to overwrite it.

The order matters and I had it backwards. Get the exemption first. Clear the label second. Doing
it the other way round produces exactly what I got: a label that returns on a schedule you do not
control.

How to check the flag without lying to yourself

While I was doing this I needed a reliable way to answer "is it flagged right now?" I tried three
and only one of them is trustworthy.

The Console banner is fine, but it is one Actor at a time and it needs a browser.

The public Store page is not reliable. After my third removal I fetched the page and grepped for
the phrase. The badge was still in the HTML, on a page that was already correct in Console. When I
checked again later the same day, it was gone. So the page is not wrong — it lags, and on that day
the lag was hours. If you grep the page right after a change, you will read your own stale render
and conclude the change failed.

The API is the source of truth. The Actor object carries a top-level notice field:

curl -s "https://api.apify.com/v2/acts/aiqlabs~dataset-to-github-issues" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['notice'])"
# NONE

The values I have observed are UNDER_MAINTENANCE and NONE. This endpoint answers for public
Actors without a token, which means you can check anyone's — including, before you fork it, the one
you are about to depend on.

I did not find notice documented as the maintenance flag. I found it by reading the object.

A check you can run on everything you publish

After this, I stopped trusting my memory of which Actors were healthy. This prints the flag for a
list of slugs:

#!/usr/bin/env bash
# Print the maintenance flag for every Actor you name. No token needed for public Actors.
USER="aiqlabs"
for slug in dataset-to-github-issues github-repository-audit pdf-table-extractor; do
  notice=$(curl -s "https://api.apify.com/v2/acts/${USER}~${slug}" \
    | python3 -c "import json,sys; d=json.load(sys.stdin).get('data') or {}; print(d.get('notice','MISSING'))")
  printf '%-32s %s\n' "$slug" "$notice"
done

I ran it across the Actors I care about after the approval. All NONE, including the twenty-two
that were never at risk. That last part matters: it confirmed the flag had not spread from a shared
cause.

The trap next door

One warning that cost me three rounds of rework, because it sits on the same Console screen.

The Actor status switch lives under Publication → Display information. Saving that section dropped
one of my categories, three times in a row.

I had set three categories through the API. After each save, OPEN_SOURCE was gone, and I put it
back through the API.

Three for three is not an accident. But I want to be exact about the limit of what I know: when I
opened that same form today to write this, all three categories were sitting in the field. So I can
report the outcome I measured and not the cause.

So if you go anywhere near this screen, re-apply your categories through the API afterwards, and
read them back:

curl -s "https://api.apify.com/v2/acts/aiqlabs~dataset-to-github-issues" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['categories'])"
# ['DEVELOPER_TOOLS', 'AUTOMATION', 'OPEN_SOURCE']

Do not confirm this in the UI you just used. Confirm it in the API.

What it costs to ignore this

The label is not cosmetic, and the timeline is published:

After another 14 days of failing runs, you will receive another notification. Finally, if the runs
continue to fail after yet another 14 days, the Actor will be deprecated.

Three days of failures to get labeled. Roughly a month of them to get deprecated. For an Actor that
structurally cannot pass, that clock starts on the day you publish and never stops. Nothing in your
code will change it, which is precisely why manual removal feels like it works and is the most
expensive thing you can do with the time.

The rule I now apply

Before I publish an Actor that needs an authorized connector, I ask one question: is there any
input the test account can supply that produces a successful run?

If the answer is yes, I make sure that path is what prefill describes, and the daily test becomes
a free health check I get for nothing.

If the answer is no, I file the skip request in the same session as the publish, before the first
notification arrives. Then I never touch the status switch, because by then there is nothing left
for it to argue with.

The Actor in this story is dataset-to-github-issues.
Its source is public under ISC at
github.com/ai-q-labs/dataset-to-github-issues,
and the connector it uses speaks the Model Context Protocol.
The rest of what I publish is at apify.com/aiqlabs.

I gave my Actor a GitHub MCP connector. It could see 44 tools and use 4.

Ai-Q Labs — Fri, 18 Sep 2026 00:08:07 +0000

I gave my Actor a GitHub MCP connector. It could see 44 tools and use 4.

I publish 22 audit Actors on Apify. They all answer one question: is this public record still true?

The registry serves a package with no warning on it. Its repository has been archived. My Actors
find that gap and write it into a dataset.

Then nothing happens. A dataset is a place findings go to be correct in private.

So I built the missing half. It reads an audit dataset, decides what deserves attention, and opens
a GitHub issue for each finding.

It reaches GitHub through an
Apify MCP connector. That is a Model Context
Protocol (MCP) server, wired into the Actor's input by the platform. My Actor never touches my
token.

I expected the interesting part to be the writing. It was not. The interesting part was how little
the platform let my Actor do, and how much the connector appeared to offer.

At authorization, Console listed 44 tools. My run could see four. Both numbers are correct,
and the gap between them is the whole design.

Here is what I built, where in the run the connector fires, and the four things that surprised me.
One of them broke an assumption the Actor was designed around.

What I built, and why it is a separate Actor

The obvious move was to add issue-filing to
github-repository-audit directly. I did not,
for a boring reason: that Actor was under an unrelated measurement, and changing its build would have
destroyed the data.

The boring reason turned out to be the right architecture. Filing issues has nothing to do with
auditing. It needs a dataset, a tracker, and a policy. It does not need to know what npm is.

So dataset-to-github-issues takes a dataset ID and a connector, and nothing about my audit is
hard-coded into it.

The input schema is the interesting file:

{
    "datasetId": {
        "title": "Audit dataset",
        "type": "string",
        "editor": "resourcePicker",
        "resourceType": "dataset",
        "resourcePermissions": ["READ"]
    },
    "githubConnector": {
        "title": "GitHub connector",
        "type": "string",
        "resourceType": "mcpConnector",
        "mcpServers": [
            {
                "url": "https://api.githubcopilot.com/mcp/",
                "tools": {
                    "required": ["issue_read", "issue_write", "list_issues", "search_issues"]
                }
            }
        ]
    }
}

Two declarations sit side by side. One names what the run may read. The other names what it may call
through the connector. The platform enforces both. My code enforces neither.

That symmetry is easy to miss when you are writing it. It matters later.

Setting up the connector: the docs and Console disagree

The MCP connectors documentation says:

Supported services include Notion, Slack, GitHub, Sentry, and Supabase.

The Add new MCP connector dropdown in Console offers three presets. I scrolled to be sure. The
list does not grow.

Preset	URL
Sentry	`https://mcp.sentry.dev/mcp`
Notion	`https://mcp.notion.com/mcp`
Linear	`https://mcp.linear.app/sse`

Slack, GitHub, and Supabase are not presets. Linear is a preset and is not in that sentence.

This is not a bug. The URL field accepts free text, so the presets are a convenience, not an
allowlist. I typed the URL of
GitHub's own MCP server and waited.

https://api.githubcopilot.com/mcp/

About four seconds later the field turned green and an auth panel appeared. Apify had gone and asked
the server what it supports. It offered API key, selected by default. It rendered OAuth
grayed out, with this text:

This server doesn't support dynamic client registration. Your own OAuth client is recommended.

I like that the platform probes rather than assumes. It also told me exactly which path to take on a
free account: a personal access token used as a bearer credential.

The token I actually issued

I used a fine-grained personal access token,
scoped as tightly as GitHub allows:

Scope element	Value
Repository access	Only select repositories → one repo
Permissions	Issues: read and write
Permissions	Metadata: read-only (added automatically, marked Required)
Expiry	30 days

No contents write. No administration. No second repository. Remember this table; it is the point of
the next section.

The trap: the URL reverted while I was not looking

Between filling in the dialog and pasting the API key, the server URL changed under me. It went back
to https://mcp.sentry.dev/mcp, the first preset in the list. The field label also changed from
"MCP server URL" to "MCP server".

My API key field kept its contents. The URL showed a green validation check. Save was enabled.

If I had saved that, a GitHub token would have been registered as the bearer credential for
Sentry's MCP server. Nothing in the dialog would have complained.

Re-typing the GitHub URL and picking it from the suggestion fixed it, and the key survived the
change.

I have a rule now: read the server URL again immediately before you save. Typing it once is not
enough. A connector dialog holds a live credential, and it deserves the same paranoia as a payment
form.

44 tools discovered, 4 of them visible

Saving the connector produced an ID and a list. Console showed the tools it had discovered at
authorization time. All 44:

add_comment_to_pending_review  add_issue_comment  add_reply_to_pull_request_comment
create_branch  create_or_update_file  create_pull_request  create_repository  delete_file
fork_repository  get_commit  get_file_contents  get_label  get_latest_release  get_me
get_release_by_tag  get_tag  get_team_members  get_teams  issue_read  issue_write
list_branches  list_commits  list_issue_fields  list_issue_types  list_issues
list_pull_requests  list_releases  list_repository_collaborators  list_tags
merge_pull_request  pull_request_read  pull_request_review_write  push_files
request_copilot_review  run_secret_scanning  search_code  search_commits  search_issues
search_pull_requests  search_repositories  search_users  sub_issue_write
update_pull_request  update_pull_request_branch

Read that list against my token's permissions. create_repository is there. delete_file is there.
merge_pull_request, push_files, fork_repository, run_secret_scanning — all there, all far
outside a token that can only read and write issues in one repository.

The discovered tool list describes the server, not the token. It is a menu, not a grant. Every
one of those calls would die upstream, at GitHub itself.

That is a fine outer boundary. It is also the worst possible place to learn about it. So I looked at
what my run actually got. From the run log:

INFO  Read 12 row(s) from dataset iBbLEMf3NUSJcU0cZ.
INFO  The proxy exposes 4 tool(s) to this run: issue_read, issue_write, list_issues, search_issues

Four. Exactly the four names in mcpServers[0].tools.required, and nothing else. I changed nothing
about the connector between those two observations. The
input schema declaration
is what narrowed it.

The docs put it in one sentence, and now I have watched it happen:

The proxy enforces that an Actor can only call tools it explicitly declared in its input schema.

So there are three layers, and they do different jobs:

The token. What the upstream service will honor. Enforced by GitHub, discovered on failure.
The connector. What server this credential belongs to. Chosen once, in Console.
The input schema. What this Actor may call. Enforced by the proxy, before the request leaves Apify.

Layer 3 is the one that ties a tool to a specific Actor. It is also the only one a reader of my Actor
can see. Anyone can open my input schema and learn the worst thing my code can do to their GitHub
account.

I value that as much as the enforcement itself. A limit nobody can inspect is a promise.

Where the connector fires, and why not at the end

The easy design writes at the end. Audit, decide, file, done.

I fire the connector in the middle, twice, and the first call is a read. The Actor lists the open
issues in the tracker before it decides anything:

// This is the whole reason the connector fires here and not at the end: the decision of
// whether to open an issue depends on what is already open.
const openKeys = new Set();
for (let page = 1; page <= 5; page += 1) {
    const res = await call(client, readTool.name, argsFor(readTool, {
        method: methodValue(readTool, /list/),
        owner,
        repo,
        state: 'open',
        perPage: 100,
        page,
    }));
    if (!res.ok) { readError = res.error; break; }
    const list = Array.isArray(res.data) ? res.data : (res.data?.items ?? res.data?.issues ?? []);
    if (!Array.isArray(list) || list.length === 0) break;
    for (const issue of list) {
        // a title like "[dep-drift] npm:left-pad - deprecated..." gives back "npm:left-pad"
        const key = keyFromTitle(issue?.title ?? '');
        if (key) openKeys.add(key);
    }
    if (list.length < 100) break;
}

Identity lives in the issue title. Every issue my Actor opens is tagged [dep-drift], and the
finding key follows the tag. A later run recovers the key by parsing titles it wrote itself. No
state file, no external store, nothing to fall out of sync.

I proved it by running the same input twice, live, with the same settings.

First run:

INFO  0 finding(s) already have an open issue (read 0 title(s) over 1 page(s)).
INFO  Opened 3 issue(s).

Second run, identical input:

INFO  3 finding(s) already have an open issue (read 3 title(s) over 1 page(s)).
INFO  Opened 3 issue(s).
INFO  Done. skipped: 7, filed: 3, deferred: 2.

The Actor recognized its own three issues from the first run and skipped them. It filed the next
three. Two more hit the per-run ceiling, and it reported them as deferred instead of dropping
them.

Every row says why, in the dataset, whether or not it became an issue:

This is what a connector buys that a separate script does not. A write-only integration would
have re-filed the same three issues. Then it would do it again every scheduled run, and the tracker
becomes noise inside a week. Reading and writing in the same run, through the same authorization,
is what makes re-running safe.

The Actor also refuses to write blind. If the read fails, it stops:

if (readError) {
    await Actor.fail(
        `Reading the existing issues failed: ${readError}. Stopping before any write, because `
        + 'without that list this run cannot tell a new finding from one it filed last time.',
    );
}

Duplicates in someone's tracker cost more than a failed run.

The assumption that broke

My audit answers in three states. It found a defect. It found nothing. Or it could not check —
usually because GitHub's hourly allowance ran out mid-run.

The third state is the one I care about. Filing it turns "I don't know" into an alarm. Dropping it
silently turns an unanswered question into a clean bill of health.

So decide() gives it its own outcome:

export function decide(row, { minRiskLevel = 'high', openKeys = new Set() } = {}) {
    const key = findingKey(row);
    const real = realCodes(row);              // codes that mean "we found something"
    const unverified = unverifiedCodes(row);  // codes that mean "the check did not run"

    if (real.length === 0) {
        if (unverified.length) {
            return { decision: 'withheld', reason: `not verified (${unverified.join(', ')})`, key };
        }
        return { decision: 'skipped', reason: 'nothing to report', key };
    }
    if (openKeys.has(key)) {
        return { decision: 'skipped', reason: 'an open issue already covers this', key };
    }
    return { decision: 'file', reason: `${row.riskLevel}: ${real.join(', ')}`, key };
}

To exercise that path honestly, I needed an audit that really ran out of allowance. So I pointed the
audit Actor at gatsby's package.json — 166 dependencies. It duly ran out:

{
  "github":   { "requestsMade": 56, "distinctRepositories": 131,
                "repositoriesNotChecked": 75, "rateLimited": true },
  "findings": { "not_checked": 76, "repository_not_on_github": 76, "repo_moved": 7,
                "deprecated_on_registry": 4, "repo_archived": 2, "silent_abandonment": 1 }
}

Seventy-six unverified rows. I expected a dry run to report dozens of withheld findings. It reported:

INFO  Done. skipped: 161, would-file: 5.
0 finding(s) were withheld because the audit could not verify them

Zero.

I read the rows instead of guessing at the cause. Every single not_checked row also carried
repository_not_on_github. Not one row in 166 had an unverified code and nothing else.

That kills the assumption. I had built the decision around "a finding is either established or
unestablished". Real data says a row is usually both: something the audit established, plus a
caveat about a check that did not run. The whole-row case turns out to be the rare one.

The behavior that actually mattered was the other one. The caveat has to travel with the finding,
into the issue body. Here is issue #11, read back from the public GitHub API:

**npm:string-similarity** — risk: `high`

### What the audit found
- `deprecated_on_registry` (high) — The registry marks this deprecated: "Package no longer supported..."
- `repository_not_on_github` (low) — The registry links to git://github.com/aceakash/string-similarity.git,
  which is not a GitHub repository.

### What the audit could not check
- `not_checked` — aceakash/string-similarity was not checked: GitHub's hourly allowance ran out.

These are open questions, not clean results. Re-run the audit to close them.

A reader cannot mistake that last section for a clean result. The design goal held. The code path I
built to demonstrate it never fired.

I am leaving withheld in, with its unit tests, and saying plainly that no live audit has produced
one yet. It is designed behavior. I have never watched it happen.

The reply that looked complete and was not

Run 4 opened three issues and then crashed:

INFO  Opened 3 issue(s).
ApifyApiError: Schema validation failed   clientMethod: DatasetClient.pushItems
  instancePath: '/issueNumber'  message: 'must be integer'

The issues were real. I checked them from outside the run with an unauthenticated curl. The
report is what failed, because issueNumber came back null.

I had stored the first raw reply in the key-value store on purpose, so I could look instead of
guess. This is issue_write's answer, verbatim:

{"id":"5078084249","url":"https://github.com/ai-q-labs/github-repository-audit/issues/4"}

There is no number field. id is a string, and it holds GitHub's internal identifier. The
issue number — 4 — exists only inside the URL.

Reaching for id would have recorded issue "5078084249". Nothing would have errored. My dataset
would have looked complete and been wrong in a way no type check catches.

The fix reads the number where it actually lives, and refuses to invent one:

function findIssueRef(reply) {
    const out = { number: null, url: null };
    if (!reply) return out;
    if (typeof reply === 'object') {
        const o = reply.issue ?? reply.data ?? reply;
        const n = o?.number ?? o?.issue_number ?? o?.id;
        if (Number.isInteger(n)) out.number = n;    // note: a string id fails this on purpose
        const u = o?.html_url ?? o?.htmlUrl ?? o?.url;
        if (typeof u === 'string') out.url = u;
        if (out.number !== null && out.url !== null) return out;
    }
    const text = typeof reply === 'string' ? reply : JSON.stringify(reply);
    const m = /https?:\/\/github\.com\/[\w.-]+\/[\w.-]+\/issues\/(\d+)/.exec(text);
    if (m) {
        out.url ??= m[0];
        if (out.number === null) out.number = Number(m[1]);
    }
    return out;
}

Number.isInteger is doing real work there. It rejects the string id that would otherwise sail
through.

If you build against a connector, capture the first reply from every tool you call. A tool result is
whatever the upstream server decided to send. Assuming a field name and discovering the truth in
production is exactly the failure this prevents.

Two more failures worth designing for

A run cannot open an arbitrary dataset. My very first run died here:

ACTOR: Running under "LIMITED_PERMISSIONS".
ERROR Could not read dataset iBbLEMf3NUSJcU0cZ: Insufficient permissions for the dataset.

Same account, same owner, still refused. The fix is not a token — it is the resourceType and
resourcePermissions declaration from the schema at the top of this article. The schema names the
thing being read and the thing being written through. The platform grants exactly that, and no more.

A run that changed the outside world must not fail on its own bookkeeping. That crash after
"Opened 3 issue(s)" marked the whole run FAILED. FAILED reads as "nothing happened". That is the
opposite of the truth, and it sends the next run straight back at the same findings.

I now guard each dataset write on its own. Anything that fails lands in the summary instead of
killing the run:

for (const d of decisions) {
    try {
        await Actor.pushData(d);
    } catch (err) {
        reportProblems.push({ key: d.key, error: String(err?.message || err).slice(0, 300) });
    }
}

Order the work so the irreversible part happens last, and make everything after it non-fatal.

Seven runs got me here, and the first two are the ones worth reading:

The bug my new Actor found in my old one

Look again at issue #11. The registry link is git://github.com/aceakash/string-similarity.git.

That is on GitHub. The same row resolves aceakash/string-similarity and queues it for a GitHub
lookup. One row, two statements, contradicting each other.

My first explanation was that the URL parser rejects the git:// scheme. I ran the parser to check,
and it does not. It accepts git://, git+ssh://, git@host:path and github:owner/name alike.

The real cause is one line further on. The finding fires when the GitHub record is missing:

if (registryFound && !pkg.repoUrl) {
    add('no_repository_link', 'The registry record does not link to any source repository, so nothing can be verified against it.');
} else if (registryFound && pkg.repoUrl && !repo) {
    add('repository_not_on_github', `The registry links to ${String(pkg.repoUrl).slice(0, 120)}, which is not a GitHub repository.`);
}

The record is also missing when the lookup never ran. GitHub allows an unauthenticated caller 60
requests an hour. Past that, my Actor stops asking and stores null:

if (state.rateLimited) { state.skipped += 1; return null; }

So !repo means two different things, and the code reads only one of them. Every row cut off by the
rate limit is reported as a repository that is not on GitHub. That is why all 76 unverified rows were
paired. The pairing tracks the rate limit. The URL scheme has nothing to do with it.

It is the same mistake as the one in my first article, made a fifth time. A missing answer is not a
negative answer.

I have not fixed it. github-repository-audit is inside an unrelated observation window, and
changing its build would destroy that measurement. It is recorded, and it goes in next.

I did not expect the reader of an Actor's output to become the best test of that Actor. It is
obvious in hindsight. A dataset you only look at is a dataset nobody checks.

What it cost, and what I would do differently

Eleven issues, none duplicated, across four live runs and two audit datasets. All of it on the
free plan. When I stopped, my account usage read $0.61 against the $5 monthly credit. MCP
connectors are not a paid feature, which surprised me enough to check twice.

Three things I would tell myself before starting:

Declare the smallest tool set you can name. Not for safety theater — because the declaration is public. Anyone can read what my Actor is allowed to do without reading my code.
Fire the connector where the decision is, not where the output is. Read before you write, and stop if the read fails.
Store the first reply from every tool. The one that cost me a run was shaped nothing like I assumed. It failed quietly, in the one direction a type check cannot catch.

And one I got wrong: I designed a decision path around a data shape I had never seen. A branch, a
reason string, and unit tests, for a row that has not appeared once in 178 audited packages. Next
time I look at the data before I write the branch.

Run it yourself

The full source of the Actor in this article is at
ai-q-labs/dataset-to-github-issues —
the input schema, the decision function, the connector client, and the unit tests, including the one
covering the case that has never fired.

You will need three things to run it: an audit dataset with riskLevel and issueCodes on each
row, an MCP connector authorized against GitHub, and a repository you are willing to file issues in.
A free Apify account covers the rest.

Where the findings came from:
github-repository-audit is public and free.
Everything it produced through the connector is open in the
tracker repository, tagged
[dep-drift], including the one that exposed my own parser bug.

Full-size versions of every screenshot in this article sit in the source repository above, under
docs/screenshots.

The shortest version of the whole thing: your connector shows you a menu. Your Actor declares what it
eats. The proxy is what makes the difference real.

I connected my audit Actor to Claude, and it audited three packages nobody asked for

Ai-Q Labs — Fri, 18 Sep 2026 00:07:21 +0000

I connected my audit Actor to Claude, and it audited three packages nobody asked for

I maintain 22 audit Actors on Apify. They all do a version of the same thing. They take a public
record, and they check whether what it still claims is true.

Last week I connected one of them to Claude through the Apify Model Context Protocol (MCP) server. I
expected the interesting part to be the agent. It was not. The interesting part was my own input
schema. It had been lying to every non-human caller since the day I published it.

Then I made a second assumption. I decided I knew how an agent would read the result. I gave the
question to four of them to prove it, and all four proved the opposite.

This is what I found, what the agents did with it, and what I changed.

What the Actor does

github-repository-audit takes package names or
repository names. For each one it asks two sources the same question and compares the answers.

The registry says a package points at a repository. The repository says whether it is archived, moved,
or relicensed. Those two records disagree more often than you would think.

The finding I built it for is the quiet one. Here is real output from a run:

npm still serves cross-env@10.1.0 with no deprecation notice, while its repository
kentcdodds/cross-env has been archived. Installing it looks completely normal.

Nothing in your terminal tells you. npm install prints no warning. The archive banner sits on
the repository where nobody installing the package will
look. The maintainer said goodbye in the only place they could. The registry never passed the message
on.

Connecting it through the Apify MCP server

The Apify MCP server turns an Actor into a tool an
agent can call. The connection itself took one command:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=aiqlabs/github-repository-audit"

The ?tools= parameter matters more than it looks. Without it, the server exposes its whole surface.
With it, the agent sees exactly one tool. I wanted a clean experiment, so I scoped it to one Actor.

Authentication runs over OAuth with dynamic client registration and PKCE. I never handled a token.
The consent screen is worth reading rather than clicking through. Apify prints the honest version:

This application was registered dynamically, and wasn't verified by Apify. Make sure you trust it.
It's allowed to redirect you to following URL(s): http://localhost/callback

That is the right warning to show. Dynamic registration means nobody vetted the client.

The detour

My first three attempts failed, and the reason had nothing to do with Apify.

I was driving the CLI from a non-interactive shell. claude mcp login prints the authorization URL,
starts waiting, and then gives up:

Waiting for authorization… (^C to cancel)
Couldn't complete authentication: stdin isn't a terminal.

The login flow wants a real terminal even when the callback would arrive over the network. I tried
winpty next. winpty refused for the same reason one level down. It needs a console of its own.

What worked was a batch file, launched in a fresh console window, with stdout sent to a file. The
console satisfies the terminal check. The redirect lets me read the authorization URL. Then I opened
that URL, approved it, and the callback landed on localhost.

Write that down if you automate this. The blocker is the terminal, not the network.

The first run returned four rows for a one-repository question

Before handing the tool to an agent, I wanted to see the smallest possible call. So I sent one field
and nothing else:

{"repos":["facebook/create-react-app"]}

That is 39 bytes. The run recorded an inputBodyLen of 328.

Something had grown my input by a factor of eight. This is what the platform actually stored:

{
  "repos": ["facebook/create-react-app"],
  "packages": ["npm:request", "npm:babel-eslint", "npm:left-pad"],
  "defaultRegistry": "npm",
  "manifestType": "",
  "manifestGroups": ["dependencies"],
  "staleAfterDays": 365,
  "includeContributors": false,
  "includeReleases": false,
  "onlyIssues": false,
  "maxTargets": 200,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 30
}

Three packages I had not mentioned were now part of the run. The results came back like this:

`input`	`source`	`riskLevel`	`issueCodes`
`facebook/create-react-app`	`repos`	medium	`repo_moved`, `stale_no_push`
`npm:request`	`packages`	high	`deprecated_on_registry`, `stale_no_push`
`npm:babel-eslint`	`packages`	critical	`deprecated_on_registry`, `repo_archived`
`npm:left-pad`	`packages`	critical	`deprecated_on_registry`, `repo_moved`, `repo_archived`, `license_mismatch`

I asked about one repository. I got four rows. Every high and critical row belongs to something the
caller never mentioned.

request and
left-pad are not neutral filler either. They are two of the
most famously abandoned packages in the npm registry. They produce findings by design.

prefill and default are not the same word

The cause is one line in my own input schema, written months earlier.

{
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] },
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] }
}

I had used the two keys interchangeably. I thought both were examples for the form. The
input schema specification
is unambiguous, and I had simply never read this paragraph carefully:

Default — If the user omits the value when starting the Actor via any means (API, CLI,
scheduler, or user interface), the platform automatically passes the Actor this default value.

Prefill — this field is only used in the user interface but does not affect the Actor
functionality and API.

prefill is a suggestion. default is a promise the platform keeps on your behalf.

There is a second edge to this, and I only saw it once the Actor was a tool. The MCP server turns
the input schema into a JSON Schema and hands it to the model. Here is what arrives:

{
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] },
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] }
}

default is a JSON Schema keyword. It means "the value used when this is omitted", and a model
reading the tool definition can act on it. prefill is not a JSON Schema keyword at all. It passes
through as an unrecognised key.

So the field nobody asked about announces itself in the language of the specification. The field that
answers the question carries a word the model has no rule for. I had put the standard keyword on the
wrong field.

My Actor's own code is innocent here. It destructures with packages = []. The injection happens
above it, before Actor.getInput() ever returns.

What I assumed would happen next

Open the Actor in Apify Console and the bug is visible immediately. The Packages box has three values
sitting in it. You delete them, or you leave them, and either way you decided.

The form is doing something important. It shows you the whole input, including the parts you did not
supply.

An agent never sees a form. It sends the fields it decided to send. It receives rows.

My rows do carry a source field, so the information needed to separate them exists. An agent that
reads it can tell the four rows apart. I want to be exact about that. It is the part I got right by
accident, because I did not add source for this reason.

But an agent does not read fifty-four fields per row back to a user. It reads the summary. So I
opened the two records my Actor writes for exactly that purpose.

SUMMARY counts the run:

{
  "checked": 4,
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 },
  "licenseComparison": { "comparable": 3, "mismatchRate": 0.333, "notComparable": 1 }
}

Two critical and one high, for a question about one repository. No count in that block says which
findings belong to the caller's target. The mismatch rate is worse. It is a fraction whose
denominator the caller never chose.

ACTION_LIST is the record I named for what to do next. Its entries carry target, registry,
repo, riskLevel and issues. It has no source field at all. It is sorted by severity:

#	`target`	`riskLevel`	did the caller ask for it?
1	`babel-eslint`	critical	no
2	`left-pad`	critical	no
3	`request`	high	no
4	`facebook/create-react-app`	medium	yes

The one entry that answers the question sits at the bottom. Above it are three the caller never
mentioned. The record offers no field that would tell them apart.

That is the shape of the trap. Attribution survives in the raw rows, which an agent skims. It
disappears from both records built to be read instead.

So I had a prediction, and it was a tidy one. Ask an agent whether create-react-app is safe to depend
on. It will read critical: 2 and pass that on.

I was wrong four times in a row

I had already read the schema, so my own answer proved nothing. I gave the question to four fresh
agents instead. None had seen any of this. The Actor was still unfixed.

Is facebook/create-react-app safe to depend on?

All four got it right. Every one of them reported medium risk for create-react-app, which is the
correct answer. Not one passed on the critical count.

Three of the four went further and told the user my tool was broken.

One thing worth flagging about the tool: my first run asked about a single repository but produced
4 rows. The extra 3 are the Actor's built-in defaults for the packages field […] Those three
carry the alarming findings — 2 critical, 1 high […] Reading that summary at face value would
have produced a badly wrong answer.

Another said the same thing in different words, then added a line I did not want to read:

when you pass only repos to this Actor, you have to set packages to empty explicitly or
unrelated rows get mixed in.

That is a workaround for my Actor, written by an agent, addressed to my user.

The trap still cost something

Nobody was misled. I want to be careful not to soften that. But four correct answers is not the same
as no harm.

Two of the four ran the audit twice. Double the compute units, double the GitHub requests against
a 60-per-hour unauthenticated allowance, double the wait. The caller paid to undo my mistake.

One told the user the tool was defective. That paragraph is now part of what my Actor looks like
to somebody deciding whether to use it.

One said nothing at all. The fourth agent gave a clean, correct answer and never mentioned the
three rows it had discarded. The caller was billed for four rows and got the use of one. Nothing in
the exchange told them.

And the wrong answer was sitting there the whole time. byRiskLevel still reads critical: 2 with
nothing beside it to say whose. Four agents declined to take it. That is not the same as it not being
there.

One of them found a second bug I had missed

The agent that ran the audit twice explained why:

I ran the audit a second time because the repo-only run left registryDeprecated, licenseMatch
and silentAbandonment all null — it never consulted npm, so it couldn't have answered "safe
to depend on" as asked.

It is right. A repository has no registry side, so three checks cannot run on it. One of them is
silent_abandonment, the finding this Actor exists for. A caller who names a repository never gets
it, and nothing said so. The fields just came back null.

I had written the rule against this in my own README. Return "broken", "fine" and "could not check"
as three different values. Then I returned the third as a bare null and let it read like the second.

Two of the four agents worked around it the same way, without being asked. That is a design gap, not
a coincidence.

What I changed

Five changes, in order of how much they mattered.

I removed default from every field that names a target. Sample values moved to prefill, which
keeps the Console form useful and never reaches the API. A field that decides what to audit must
come from the caller. A field that decides how to audit can have a default, and staleAfterDays
still does.

I made the empty call fail loudly. With no targets the Actor now stops with a message that names
the three ways in. It used to be impossible to reach that path, because the default guaranteed there
was always something to audit. That was the bug hiding the bug.

I moved the relevance rule into the description. The source field is now documented where an
agent reads it, not only in the README.

That last one had a second layer I only found while writing this. source did not appear anywhere in
my dataset_schema.json either. So the default Console table did not show it, and neither did the
"Problems only" view.

Opening that table made it worse. My view leads with packageName, and the row I asked for came from
repos, so it has no package name. The Console printed it as null.

#	Package Name	Registry	Risk	Resolved Repo
1	null	null	medium	react/create-react-app
2	request	npm	high	request/request
3	babel-eslint	npm	critical	babel/babel-eslint
4	left-pad	npm	critical	left-pad/left-pad

The one row that answers the question looks like the broken one. The three rows nobody asked for look
authoritative. I had built a view that ranked my own output by how little the reader wanted it.

Both views now lead with the two columns that answer "did I ask for this row?": input and source.

I made the summaries carry attribution too. This is the change I would have missed if I had
stopped at the schema. Removing the default stops this particular injection. It does not make the
output attributable.

ACTION_LIST now begins each entry with input and source. SUMMARY now carries a bySource
block beside the flat counts:

{
  "checked": 4,
  "bySource": {
    "repos":    { "checked": 1, "critical": 0, "high": 0, "medium": 1, "low": 0, "ok": 0 },
    "packages": { "checked": 3, "critical": 2, "high": 1, "medium": 0, "low": 0, "ok": 0 }
  },
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 }
}

The flat line still says two critical. Now something next to it says whose.

That block is eight lines in src/audit.js, inside the function that builds SUMMARY:

// A single call can mix targets the caller typed with targets that arrived some
// other way - a manifest that expanded into forty dependencies, or a schema
// default. A flat count of "2 critical" cannot be acted on, because it does not
// say whose.
const bySource = {};
for (const row of rows) {
    const key = row.source ?? 'unknown';
    bySource[key] ??= { checked: 0, critical: 0, high: 0, medium: 0, low: 0, ok: 0 };
    bySource[key].checked += 1;
    if (row.riskLevel in bySource[key]) bySource[key][row.riskLevel] += 1;
}

ACTION_LIST took two lines: input and source moved to the front of each entry, above
target. That ordering is the whole change. A reader who cannot answer "did I ask for this?"
cannot act on the row.

This matters beyond the bug that started it. My Actor takes repositories, packages and a manifest in
one call. A manifest URL can expand into forty dependencies from a single field. Any of those
mixes produces a count the caller cannot take apart. The default was one way in. It was not the only
one.

Going through the rest of the output found one more. LICENSE_REPORT ends with a list of licences
that could not be resolved. That list is an instruction to open files by hand. It held bare names. So
I gave those entries input and source as well. Sending someone to read a licence for a dependency
they never named wastes the same afternoon a false finding does.

I made "could not check" say so out loud. This is the one an agent found for me. A
repository-only row now produces a note in SUMMARY:

1 target(s) were given as repositories, so no registry was consulted for them. registryDeprecated,
licenseMatch and silentAbandonment are null on those rows because they could not be checked, not
because they came back clean. Pass the package name to check them.

The same sentence went into the repos field description, where an agent reads it before calling.
null was already the honest value. It was not a legible one.

Four tests hold this. Three check the new fields on the three records. The fourth is a control. A run
from a single source must report one group, and its numbers must equal the flat totals. So the
breakdown cannot invent structure that is not there. The suite went from 48 to 52.

What actually changed, measured the same way

I pushed build 0.1.8 and sent the same one-field call again.

	before	after
rows returned	4	1
stored `INPUT`	328 bytes	267 bytes
`byRiskLevel`	`critical 2, high 1`	`critical 0, high 0`
`bySource`	absent	present
first `ACTION_LIST` entry	`babel-eslint`, critical	the repository I asked about

Then I ran the agent trial again, same question, same plain prompt, two fresh agents.

Both answered correctly, as before. Neither mentioned a defect in the tool. Before the change,
three of four had. That is the entire measurable result, and it is a modest one. Nobody was ever
given a wrong answer, so nothing about correctness improved.

What improved is that my Actor stopped asking its callers to compensate for it.

One thing did not change. Both agents still ran the audit twice, once by repository and once by
package. That is now the right behaviour, and the output asks for it in writing. I should say that my
new note may be causing the second call rather than merely permitting it. One agent made the same
second call before the note existed, which argues against that. With four trials I cannot separate
the two.

Then I checked the other twenty-one

One bad field is a typo. I wanted to know whether it was a habit. So I read every input schema I
have published and sorted the defaults into two piles.

A default is safe when it decides how the work is done. It is dangerous when it decides what
the work is done to.

Twenty-two published Actors. Every one of them sets a default somewhere. Ten of those defaults
still name a target, and I had just removed an eleventh.

Actor	field	what it audits when the caller says nothing
`seo-ai-visibility-auditor`	`startUrls`	`apify.com`
`bulk-domain-checker`	`domains`	`apify.com`
`domain-availability-checker`	`domains`	`apify.com`
`sitemap-checker`	`domains`	`apify.com`
`tech-stack-detector`	`domains`	`apify.com`
`dead-link-checker`	`domains`	`docs.apify.com`
`http-status-checker`	`urls`	`apify.com/store`
`pdf-inspector`	`pdfUrls`	a US tax form
`pdf-to-text-markdown`	`pdfUrls`	the same tax form
`hacker-news-link-rot`	`list`	the Hacker News `topstories` list

Call any of the first nine with an empty input and you get a finished report. It covers Apify's own
website, or a US tax form. Nothing in the response says it is a sample. An agent has a well-formed
answer to a question nobody asked.

Then I noticed the part that embarrassed me most. Nine of those ten fields also appear under
required in the same schema.

I had been reading required as a promise that the caller named the target. So I tested that
reading. I called seo-ai-visibility-auditor with an empty object.

echo '{}' > empty.json
apify call GuuMKUiWcaUUqGGhG --input-file=empty.json

The run succeeded and audited apify.com. Here is what the platform stored as my input:

{
  "startUrls": [{ "url": "https://apify.com" }],
  "crawlSite": false,
  "maxPages": 10,
  "checkBrokenLinks": false,
  "maxLinksToCheck": 50,
  "proxyConfiguration": { "useApifyProxy": false }
}

I sent {}. The Actor received a target, and nothing rejected the call.

The specification
treats the two settings as alternatives rather than as a pair. required is for fields that "don't
have a reasonable default". default is passed by the platform whenever the caller omits the field,
"via any means". Put both on one field and the second one decides. required survives as a note to
whoever is reading the form.

The other twelve Actors are fine. country, outputFormats, robotsAgent, manifestGroups —
omitting those does not invent work.

One row in the table made me think harder. hacker-news-link-rot defaults its list field to
topstories, and that does pick the target. But the Actor has no other way in. An empty call has to
mean something. It is also the one row I never marked required.

So the rule is not "never use default". It is narrower than that:

A caller who says nothing must not receive results they cannot tell apart from results they asked
for.

hacker-news-link-rot satisfies that with one sentence in its description. The other nine did not
satisfy it at all.

The same mistake, four times, before an agent was involved

Here is what stung. This was not a new class of error for me. It was the fourth time.

Every audit Actor I have written has produced a confident finding that was really a gap in my own
knowledge. Each time, the fix was the same shape: split one value into two.

A rate limit that looked like deletion. My Chrome extension auditor read 130 listings from
Google's own sitemap, one every 600 milliseconds. Ninety-three came back as "the store has never
heard of this ID". Every one of them was a healthy extension I had listed minutes earlier.

Google serves its rate-limit interstitial as a redirect away from the store. A reader that only asks
"did I land on a listing page?" sees exactly what a deleted extension looks like. The Actor now checks
for that first. It never treats the interstitial as a fact about the extension. It stops the run
instead of producing ninety-three more.

A 404 that was an advertising server refusing. My podcast auditor reported a 1.7% episode death
rate. Nine of those failures came from one host, and all nine shows were running fine.

A per-listener redirect service was serving that audio, and stitching in advertising as it went. It
would not build a redirect for an automated request. So it answered 404, with the body Missing redirect URL. I had a real death rate of 0.5% and a fake one three times larger.

Those cases became undetermined, not dead. The finding text says what I actually know:

Whether the audio is there cannot be established without behaving like a listener, which this
Actor does not do.

A licence warning on lodash. GitHub could not match lodash's LICENSE to a standard licence, so it
returned NOASSERTION. My first version called that license_non_standard and raised it.

lodash, jQuery UI and UglifyJS all land there. They are ordinary MIT and BSD projects whose LICENSE
carries an extra paragraph. A warning that fires on healthy rows does not add information. It buries
the rows that matter. The code comment I left says it better than I can paraphrase:

Raising this would put a warning on healthy rows and bury the ones that matter.

A severity I set before I measured anything. I once shipped "no update in three years" as a high
severity finding. Then I measured the base rate across 1,312 App Store apps. It is 18.4%.

A finding that fires on one row in five describes the ecosystem. It does not describe a problem with
your dependency. It sits at medium now.

For the Shopify auditor I dropped the check entirely. Shopify publishes a launch date and no update
date. Building the check anyway would have meant guessing.

The rule I ended up with

Every one of these is the same rule, arrived at four times the slow way.

Return "broken", "fine", and "I could not check" as three different values.

I used to justify that by saying an agent cannot tell the third from the second. My own trials say
otherwise. Four agents pulled the distinction out of a source field I had not documented. Two
worked out on their own that a repository-only row never touches the registry.

So the reason is not that agents cannot cope. It is what coping costs.

Every ambiguity you leave in your output is work you have handed to the caller. Sometimes they pay it
in a second run against a rate-limited API. Sometimes they pay it by writing a paragraph explaining
your tool's quirk to their user. Sometimes they pay it silently, by throwing away three quarters of
what you charged them for.

None of that shows up as an error. It shows up as your Actor being slightly more expensive and
slightly less trusted than the one next to it.

The input schema is the other half. If a field can change what gets audited, the caller must set it.
Silence has to mean silence.

What to check in your own schema

Five things, in the order that cost me the most time.

Search your schema for "default". For each hit, ask what happens when an API caller omits that field. If the answer changes what the Actor works on, move it to prefill. Listing the field under required will not do this for you.
Call your Actor with the minimum viable input, then read the stored INPUT record. Do not read the input you sent. Read what the platform saved. That is what your code receives.
Make sure an empty call fails. If your Actor can always find something to do, you cannot tell an empty call from a real one. I checked what that costs, because the platform runs published Actors on its own schedule. Twelve of my twenty-two already have nothing to do on an empty call, three of them because a required field carries no default at all. All twenty-two show zero failed runs across 159 platform runs in the last thirty days.
Write your limits into description, not the README. The description is what an agent reads. Mine now says why pyproject.toml is unsupported: half-parsing a manifest produces findings about dependencies you do not have.
Hand the tool to an agent and read what it says about you. This found more than my own review did. An agent that works around your quirk will usually explain the quirk to its user, in writing, in the answer. That paragraph is a free bug report. It is also what your Actor looks like to a prospective user.

Code

The full source for the Actor in this article is on GitHub:
ai-q-labs/github-repository-audit. That is
the code behind the published Actor at
apify.com/aiqlabs/github-repository-audit. It
includes the input schema this article is about, 52 unit tests, and a live check against the real
GitHub, npm and PyPI APIs.

Clone it, run npm install, then npm test for the unit tests or npm run test:live for the live
check. The live check needs no key, but GitHub allows unauthenticated callers 60 requests an hour, so
set GITHUB_TOKEN if you run it more than once.

Full-size versions of every screenshot in this article sit in the same repository, under
docs/screenshots.

My Actor ranked #1 in an agent's search. No one else's agent could see it at all.

Ai-Q Labs — Fri, 18 Sep 2026 00:07:12 +0000

My Actor ranked #1 in an agent's search. No one else's agent could see it at all.

Four days ago I published a number I was pleased with. I had measured how often an AI agent finds one of my 23 Actors when it searches the Apify Store through the Apify MCP server. The answer came out at roughly one query in five, and on sitemap checker my Actor came back first.

Both figures were wrong. On the queries I re-tested this morning, an agent that is not mine finds my Actors zero times out of six.

The arithmetic was fine. The problem was the connection I measured through, and it is a problem that any Actor author can have without noticing, because the platform gives you no obvious way to measure it any other way.

The measurement I trusted

My script for that article built one URL:

const url = `https://mcp.apify.com/?token=${encodeURIComponent(token)}`;

Then it called search-actors with a keyword, paged through the results and recorded where my Actor landed. sitemap checker put it at rank 1. PDF tables put it at rank 2. I wrote both numbers down and reasoned from them for a week.

I never asked whose search that was.

The same query, with the token and without

search-actors needs a token, so I could not run it anonymously. The Store also has a plain REST endpoint that serves the same shelf, and that one takes the Authorization header or does without it. So I ran the same query twice:

// store-auth-diff.mjs — is my Actor in the results, or only in *my* results?
const TOKEN = process.env.APIFY_TOKEN;

async function search(query, withAuth) {
    const url = `https://api.apify.com/v2/store?search=${encodeURIComponent(query)}&limit=100`;
    const res = await fetch(url, { headers: withAuth ? { authorization: `Bearer ${TOKEN}` } : {} });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const { data } = await res.json();
    const ids = data.items.map((a) => `${a.username}/${a.name}`);
    const at = ids.findIndex((id) => id.startsWith('aiqlabs/'));
    return { total: data.total, returned: ids.length, mineAt: at < 0 ? null : at + 1 };
}

for (const q of ['sitemap checker', 'pdf table extractor', 'github repository audit']) {
    console.log(q, '\n  auth:', await search(q, true), '\n  anon:', await search(q, false));
}

Its output on 16 August:

sitemap checker
  auth: { total: 339, returned: 78, mineAt: 1 }
  anon: { total: 339, returned: 77, mineAt: null }
pdf table extractor
  auth: { total: 757, returned: 86, mineAt: 2 }
  anon: { total: 757, returned: 84, mineAt: null }
github repository audit
  auth: { total: 417, returned: 71, mineAt: 1 }
  anon: { total: 417, returned: 69, mineAt: null }

Read the columns in order. total is the same number with and without the token — 339 stays 339. The count of items actually returned is one to three higher when the token is present. The extra items are mine.

I ran it over five queries. With the token my Actors sit at ranks 1, 2, 1, 1 and 5. Without it, none of them is anywhere in the first hundred results.

Same endpoint, same query, one HTTP header apart.

The MCP tool reads the same shelf

That only bears on the earlier article if search-actors ranks the same way the REST endpoint does. It does. Six queries through both, comparing the top five:

query	`search-actors` (token)	`/v2/store` (token)	`/v2/store` (no token)	top 5 identical
`sitemap checker`	#1	#1	absent	5/5
`PDF tables`	#2	#2	absent	5/5
`GitHub repository`	absent to 30	#40	absent	5/5
`broken links`	absent to 30	#29	absent	5/5
`RSS feed`	absent to 30	#47	absent	5/5
`tech stack`	absent	absent	absent	5/5

The top five agree on every query. Where my Actor appears in both channels, the rank is the same integer. So the first place I published — first place on sitemap checker, for an AI agent — was first place for an agent carrying my token. There is one such agent and I built it.

One query parameter says why

At this point I assumed the token was doing something to the ranking. It is not. The Store endpoint documents a parameter that tells you exactly what is happening, and you can use it without any credentials at all.

From the GET /v2/store reference, on includeUnrunnableActors, describing what the default excludes:

Actors from developers who haven't passed KYC, or full-permission Actors without a large user base

So I ran every query anonymously, twice — once plain, once with that flag on:

// unrunnable-probe.mjs — no credentials anywhere in this file
const q = 'github repository audit';
const base = `https://api.apify.com/v2/store?search=${encodeURIComponent(q)}&limit=100`;

for (const url of [base, `${base}&includeUnrunnableActors=true`]) {
    const { data } = await (await fetch(url)).json();
    const ids = data.items.map((a) => `${a.username}/${a.name}`);
    const at = ids.findIndex((id) => id.startsWith('aiqlabs/'));
    console.log(url.endsWith('true') ? 'unrunnable ON ' : 'default      ',
        'total:', data.total, 'mine at:', at < 0 ? '—' : at + 1);
}

default       total: 417 mine at: —
unrunnable ON total: 502 mine at: 1

Across five queries, unauthenticated both times:

query	default	`includeUnrunnableActors=true`
`sitemap checker`	absent, total 339	#3, total 352
`pdf table extractor`	absent, total 757	#2, total 877
`github repository audit`	absent, total 417	#1, total 502
`google play audit`	absent, total 5,102	#1, total 5,386
`http status checker`	absent, total 3,189	#6, total 3,379

Both calls are unauthenticated. The only difference is one documented query parameter.

The totals move. Between 13 and 284 more Actors enter each result set, and mine come back near the top of them. My Actors were never ranked badly. They were filtered out before ranking, for everyone except me.

Which of the two exclusions is mine

The documented filter covers two groups, and it is worth knowing which one you are in, because only one of them is fixed by paperwork. The permission level of any public Actor is readable without a token:

const r = await fetch('https://api.apify.com/v2/acts/aiqlabs~sitemap-checker');
const { data } = await r.json();
console.log(data.actorPermissionLevel, data.isPublic);   // LIMITED_PERMISSIONS true

All five Actors I checked come back LIMITED_PERMISSIONS, so the full-permission half of the filter does not apply to them. That leaves the other half, and Apify Console states it plainly:

Insights → Payouts, 16 August 2026. Billing details are registered on the same screen; the identity check is not.

The document I hold keeps being rejected by the verification vendor, which is an open thread with Apify support and not this article's subject.

What matters here is the shape of it. An incomplete account setting removed my Actors from every search except my own, and eleven days of work on titles, categories and output schemas went into Actors that were not in the result set while I was tuning them. Every one of those changes was aimed at Actor quality score, which Apify documents as the ranking input for both Store search and search-actors. Ranking inputs do nothing for a row that never reaches the ranking stage.

The Actor works. That was never the problem.

It is worth being clear about what is and is not broken here, because "invisible in search" and "unusable as a tool" are different failures and only one of them applies.

Name the Actor in the connection string and an agent picks it up as a tool immediately. Mine is pinned into my editor's MCP config, so I asked the agent to audit two well-known repositories. It called the Actor itself, waited for the run, and read the dataset back:

run m3PNCMD8HAAIGFpAq   SUCCEEDED in 2.402s   0.0027 compute units   2 items

requested	resolved to	archived	moved	last push	risk	issues
`facebook/create-react-app`	`react/create-react-app`	no	yes	547 days	medium	`repo_moved`, `stale_no_push`
`babel/babel-eslint`	`babel/babel-eslint`	yes	no	1,823 days	critical	`repo_archived`

Two and a half seconds, a fraction of a cent, and the agent had something it could act on: one dependency whose repository has quietly moved owner while the old URL still answers, and one that is archived outright. That is the whole point of exposing an Actor over MCP — the agent reaches for it mid-task instead of asking me to go and look.

So the tool definition, the input schema and the output fields all do their job. An agent that has this Actor uses it fine. The failure is one step earlier and entirely invisible from here: an agent that does not already have the name will not find it.

Why your own MCP session cannot tell you

The obvious control is to connect to the MCP server without a token and search again. That is not available:

$ curl -s -X POST https://mcp.apify.com/ -H 'content-type: application/json' \
    -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'
{"error":"invalid_token","error_description":"Missing or invalid access token. Pass an Apify API
token in the Authorization: Bearer <token> header. ..."}

HTTP 401, with no token and with an empty ?token= alike. The server speaks JSON-RPC over HTTP, as MCP requires, and every session belongs to somebody.

This is the part I would ask you to sit with, because it is structural and not personal. If you publish an Actor and want to know whether an agent can find it, the natural move is to connect your own client and search. That test is authenticated by construction. It will show you your Actor whether or not anyone else can see it. Your test passes, and it passes for the one reason that cannot generalise.

Nothing in the MCP documentation warned me about this, and I do not think it should have to. Returning an author their own Actors is a reasonable thing for a store to do — it is what makes testing an unpublished or unreviewed Actor possible at all. The gap is not in the behaviour. It is that the only search surface an Actor author naturally reaches for is the one that cannot be run anonymously.

The four-line check to run before you tune anything

No token, no dependencies, no cost:

// visible.mjs — run: node visible.mjs your-username sitemap checker
const [user, ...words] = process.argv.slice(2);
const url = (extra) =>
    `https://api.apify.com/v2/store?search=${encodeURIComponent(words.join(' '))}&limit=100${extra}`;

const seen = async (extra) => {
    const { data } = await (await fetch(url(extra))).json();
    const at = data.items.findIndex((a) => a.username === user);
    return `${at < 0 ? 'absent' : `#${at + 1}`}  (total ${data.total})`;
};

console.log('as everyone sees it :', await seen(''));
console.log('including filtered  :', await seen('&includeUnrunnableActors=true'));

Run against my own account and against a publisher who is not filtered, on the same query:

$ node visible.mjs aiqlabs sitemap checker
as everyone sees it : absent  (total 339)
including filtered  : #3  (total 352)

$ node visible.mjs automation-lab sitemap checker
as everyone sees it : #1  (total 339)
including filtered  : #6  (total 352)

Two different diagnoses from the same two lines. Mine is absent until the filter comes off. The other publisher is first without it, and slips to sixth with it — which is what a healthy result looks like, because turning the filter on adds thirteen competitors to that shelf.

So: if the first line says absent and the second gives you a rank, your Actor is being filtered out of search for every user but you, and no amount of ranking work will change that until the filter lifts. If both lines say absent, you have an ordinary ranking problem. If the first line gives you a rank, you are visible and can trust the numbers you measure.

I would rather have run those four lines on 5 August than on 16 August.

What I got wrong

I published a reachability figure and a rank without asking whose view produced them. The script that produced them had ?token= written into the URL, by me.

It is the second time in two weeks that I have measured my own visibility from inside my own session. The first was a rank I read off the Store web UI in a browser I was logged into. I caught that one, corrected it privately, and then made the same mistake again in a script — which tells me the lesson is not "be careful with browsers" but something duller: if a measurement can only be taken while authenticated, the number it returns is a statement about you, not about your users.

The earlier article's other findings survive. A default MCP connection really does hand an agent eleven tools with a single Actor among them; limit really does default to 5 and cap at 10; the result pages really are ragged, so a short page is not the end of the list; and broadening a keyword really does push my Actor down — that comparison sits inside one channel, so its direction holds even though both ranks are token-local.

What has to be withdrawn is the headline. One query in five is what I could see. For anyone else's agent, on the queries I re-tested, it was none of them.

I had never read my own Actors the way an agent reads them

Ai-Q Labs — Fri, 11 Sep 2026 05:18:24 +0000

I have 23 Actors on the Apify Store. I have read their Console input forms hundreds of times - every
field label, every hint, every checkbox, because I wrote them and then fixed them and then fixed them
again.

I had never once read the tool definition an AI agent receives when it calls them through the
Apify MCP server.

This is what I kept looking at. It is not what my callers see.

Last week I did. Of the 246 field descriptions I have written across those 23 Actors, 53 arrive at
the agent exactly as I typed them. That is 21.5%. The rest are altered on the way out.

Most of those alterations are documented, sensible, and probably improvements. That is not the point.
The point is that I shipped 23 tools without ever looking at what my callers actually receive, and
when I finally looked, I found three things the docs do not mention and one mistake that was entirely
mine.

Here is how to look, and what I found.

Reading your own tool definitions

The Apify MCP server speaks Streamable HTTP, the transport defined by the
Model Context Protocol. You post JSON-RPC to it, you get
tools/list back, and that response is the ground truth for what an agent sees.

The first trap costs a minute: the endpoint is /, not /mcp. Post to /mcp and the server
tells you off in plain English:

There is nothing at route POST /mcp?actors=... This Model Context Protocol (MCP) server supports the Streamable HTTP transport.

The second trap costs a little more: notifications/initialized answers with an empty body, so a
naive JSON.parse on every response throws. I wrote that bug while writing this article.

// Read your own Actors the way an AI agent reads them.
// Usage: APIFY_TOKEN=... node read-my-tools.mjs aiqlabs/sitemap-checker aiqlabs/pdf-inspector

const token = process.env.APIFY_TOKEN;
const actors = process.argv.slice(2).join(',');
const url = `https://mcp.apify.com/?token=${token}&actors=${encodeURIComponent(actors)}`;

async function rpc(body, sessionId) {
    const headers = { 'content-type': 'application/json', accept: 'application/json, text/event-stream' };
    if (sessionId) headers['mcp-session-id'] = sessionId;
    const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
    const sid = res.headers.get('mcp-session-id') ?? sessionId;
    const text = await res.text();
    if (!text.trim()) return { sid, payload: null }; // notifications answer 202 with no body
    const line = text.split('\n').find((l) => l.startsWith('data:'));
    return { sid, payload: JSON.parse(line ? line.slice(5) : text) };
}

const init = await rpc({
    jsonrpc: '2.0', id: 1, method: 'initialize',
    params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'read-my-tools', version: '1.0' } },
});
await rpc({ jsonrpc: '2.0', method: 'notifications/initialized' }, init.sid);
const { payload } = await rpc({ jsonrpc: '2.0', id: 2, method: 'tools/list' }, init.sid);

Point it at three of mine - say sitemap-checker and two
others - and it prints:

aiqlabs--github-repository-audit  (490 chars, 16 params)
  injected by the server : waitSecs
  prefill left in schema : repos, packages
  truncated at 500 chars : repos

aiqlabs--sitemap-checker  (421 chars, 12 params)
  injected by the server : waitSecs
  prefill left in schema : domains

Three lines of output, three things worth knowing. Take them in order.

The budget nobody tells you about

The description an agent reads is not your description. It is this:

This tool calls the Actor "<user>/<slug>" and retrieves its output results.
Use this tool instead of the "call-actor" if user requests this specific Actor.
Actor description: <your description>

The README states that descriptions are truncated at 500 characters (MAX_DESCRIPTION_LENGTH). That
limit applies to the whole string above, preamble included - so what you actually get is 500 minus
the preamble.

And the preamble is not a fixed cost. Your Actor's full name sits inside it, which means a longer
slug buys you a shorter description:

budget = 338 − length("<user>/<slug>")

Across my 23 that works out to 303 to 317 characters. domain-availability-checker - my longest
slug at 35 characters - gets 303. pdf-inspector gets 317. Nothing anywhere told me that naming an
Actor spends part of its description.

None of mine are over. But the margins are thinner than I would have guessed: my tightest is
github-repository-audit at 10 characters of headroom, then hacker-news-link-rot at 11 and
chrome-extension-audit at 12. Median description across the 23 is 278. Three of them are one
average sentence away from being delivered with the last clause replaced by ....

I found the real numbers late, and only because I checked. My first draft of this section said the
preamble was "174 characters, fixed" and the budget was 326 - measured off a single Actor, which
happened to be one of the longest-named ones, and then stated as a constant.

What the server changes, in numbers

Across 23 Actors and 246 author-written fields:

Change	Fields	Documented
`Example values: ...` appended	188	yes
`items.description` generated where I wrote none	36	no
`prefill` kept in the exposed JSON Schema	33	no
`waitSecs` parameter added	23	no
`REQUIRED` prefix added	17	yes
`Possible values: ...` appended	8	yes
description truncated at 500 chars	1	yes

Zero of my 246 fields had an empty description, so none of this is blanks being filled in. It is text
I wrote being changed.

The documented half is in the server's README and it is
worth reading once:

Descriptions are truncated to 500 characters (as defined in MAX_DESCRIPTION_LENGTH).

Required fields are explicitly marked with a REQUIRED prefix in their descriptions for compatibility with frameworks that may not handle the JSON schema properly.

I checked the second claim rather than trusting it: 17 fields are declared required in my schemas,
and all 17 carry the prefix in the tool definition. It does what it says.

The truncation that undoes itself

Exactly one of my 246 fields is longer than 500 characters: the repos field on
github-repository-audit, at 624. Its top-level description arrives cut, with the last 124 characters
replaced by ....

Those 124 characters read:

...they could not be checked, not because they came back clean. Give the package name in Packages instead if you need those three.

That is not decoration. That sentence exists to stop a caller reading a null as a clean result - the
single misreading this Actor was built to prevent. Truncation ate exactly the warning.

Except the agent still gets it. The server had also generated an items.description for that array
field - something I never wrote - containing the full 624 characters, byte for byte. A documented
truncation was quietly cancelled by an undocumented generation.

I want to be precise about how much this shows. One field in 246 was long enough to test it. I am
reporting a mechanism, not a rate. But the mechanism is worth knowing in both directions: if your long
description sits on an array, the full text survives in items; if it sits on a string, nothing
catches it.

Where does items.description come from? Of my 39 array fields, 36 got one. Twenty-four are exact
copies of the parent description, eleven are the parent plus the **REQUIRED** prefix, and one is a
structural expansion of editor: requestListSources into its url / method / payload / headers
shape. The three that got nothing all have an enum on their items.

That last case caught me out while I was measuring. I first recorded it as "my 138-character
description was replaced by a 19-character string" and nearly published that. It was wrong: the parent
description arrives intact, and the 19 characters are the title of the expanded object type. Nothing
was lost. I mention it because it is the kind of error that reads perfectly well in a draft.

The parameter you did not write

Every one of my 23 tools carries a parameter I have never declared:

{
  "type": "integer", "minimum": 0, "maximum": 45, "default": 30,
  "description": "Max seconds (0–45, default 30) to cap the wait for the Actor run to reach terminal state. For long-running Actors the response returns at the cap with the current run status; follow `nextStep` to poll via get-actor-run. Set to 0 to fire-and-forget."
}

It is on the server's own tools too - get-actor-run carries it - so this is a server-wide
convention rather than something done to Actors specifically.

I am not reporting this as a defect. The description tells the agent precisely how to recover, which
is more than most timeouts do. I am reporting it because an author who has never read their tool
definition does not know their Actor now has a 45-second contract with its callers. If your Actor
takes four minutes on a realistic input, every agent call returns before it finishes and the caller's
experience depends on whether their client follows nextStep. That is a design constraint on your
Actor, and it arrived without you.

One key, two layers

prefill survives into the exposed JSON Schema on 33 of my fields. It is not a
JSON Schema keyword; nothing in
the spec says what a consumer should do with it. An agent reading the schema sees a value
sitting next to my field and can reasonably read it as a suggestion.

I have met this key before. In an earlier article I traced a run that returned four rows when I had
asked about one repository: the platform was treating my prefill as a default and injecting it
into every call that omitted the field. Same key, different layer, different behaviour - and neither
behaviour is one I asked for when I filled in a Console form hint.

Then I let an agent choose between all 23

Reading the definitions tells you what arrives. It does not tell you whether the catalog works. So I
ran a second trial: give an agent all 23 descriptions and one realistic task, and see which tool it
reaches for.

I wrote seven tasks, fixed the intended answer for each one in a file before launching anything,
and wrote down what I expected to happen. Two independent agents per task, fourteen trials, no Actor
executed.

I expected failures. My catalog has obvious traps in it - three PDF tools, three tools that all touch
broken links, two that both check domains.

Fourteen out of fourteen picked the tool I intended. Three of my seven predictions were wrong, and
every one of them was wrong in the same direction: the agents were more careful than I gave them
credit for. Task 7 is a fair example. I expected http-status-checker to pull votes away from
dead-link-checker; instead both agents explained, unprompted, that a status checker can only test
URLs you already have and therefore cannot recover the URLs of a blog that moved two years ago.

So the selection layer was not the story. This was:

Confidence fell to medium in exactly 4 of the 14 trials, and a tie was declared in exactly those
same 4. The other ten were unanimous, high, and tie-free. Those four trials are the two pairs
where I had written overlapping descriptions.

Here is one of them, in my own words, from the live catalog:

seo-ai-visibility-auditor - "classic SEO health and for AI/LLM discoverability - AI crawler access in robots.txt, llms.txt, structured data, metadata"
seo-audit-tool - "on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag"

The phrase "AI crawler access in robots.txt, llms.txt" appears verbatim in both. I wrote both
descriptions, months apart, and had never once read them next to each other.

Asked "is my robots.txt blocking AI crawlers like GPTBot", both agents picked
seo-ai-visibility-auditor, and both told me why the pick was thin:

both entries explicitly list "AI crawler access in robots.txt" as a check, so either would answer
the question; I picked seo-ai-visibility-auditor because AI/LLM discoverability is its stated
primary purpose rather than one of four add-on checks. Neither description names GPTBot
specifically, so the match rests on the "AI crawler access in robots.txt" wording alone.

That is a correct answer arrived at by elimination, with the reasoning shown and the weakness
declared. The agent did the honest thing. The duplication it had to route around is mine.

What I wanted to conclude here was: the signal is the confidence, not the choice - every medium
landed on a real duplication, so watch where the agent hesitates. I had the sentence written. Then I
measured it and had to take it out. That is the next section.

Two caveats regardless. n is 2 per task, all Claude, all in one harness - this is a check on my
catalog, not evidence about agents in general. And the agents saw name and description only; a real
client also reads the input schema, which might break a tie my prose leaves open.

Fixing it, and measuring the fix the same way

The two SEO Actors were never duplicates in the code. One takes a starting URL and crawls
(startUrls, crawlSite, maxPages); the other takes a list and audits it page by page (urls,
maxUrls). They are genuinely different tools. My descriptions simply never said so - they both led
with the checks, and the checks overlap.

So I rewrote both to lead with the input shape, and to name each other:

seo-ai-visibility-auditor - "...Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead."
seo-audit-tool - "Give it a list of URLs... One row per URL - to crawl a whole site instead, use SEO & AI Visibility Auditor."

Then I pulled tools/list again to confirm the agent-facing text had actually changed - no tool in
the catalog still carries the shared phrase - and ran the same task past three fresh agents. All
three: high, no tie. One explained it without being asked:

aiqlabs/seo-audit-tool also covers "AI crawler rules" but is scoped to a supplied list of URLs, and
both descriptions explicitly cross-reference each other to resolve that split, so this is not a tie.

Clean result. I nearly stopped there.

The control that ruined it

I also re-ran task 4 - the domain pair - as a control. I had not touched either of those two
descriptions.

pair	edited?	before	after
SEO	✅ yes	medium ×2, tie ×2	high ×3, tie none
domain	❌ no	medium ×2, tie ×2	high ×2, tie none

The pair I did not fix improved exactly as much as the pair I did. So the confidence change cannot be
attributed to my rewrite, and the sentence I wanted to publish - watch where the agent hesitates -
does not survive its own control.

The cause is a flaw in my design, not a mystery. My before and after prompts were not identical. The
before agents were told "answer in exactly this structure and nothing else"; they produced answers and
then sat on them, and I had to chase them with a second message asking them to actually send it. The
after agents had the delivery instruction from the start. One changed variable, sitting right next to
the one I was trying to measure.

What survives is narrower and still worth having. The duplication was real: the same clause, verbatim,
in two live descriptions I wrote months apart. Four of fourteen before-trials declared a tie and all
four landed on the two pairs where my descriptions genuinely overlapped - no false alarms among the
other ten. The fix shipped, and the shared phrase is gone from what agents receive. What I cannot tell
you is whether fixing it changed anything measurable, because I broke my own instrument while using it.

If I had measured only the pair I edited - which was my plan until I added the control as an
afterthought - I would have published a causal claim resting on a changed prompt.

One more honest note on that edit. While writing the new description I listed "soft 404s answering 200" as
one of the four checks. The Actor does detect those - there is a soft_404 code in its source - but
my own README defines the four as AI crawler rules, /llms.txt, path-dropping redirects, and
X-Robots-Tag noindex. I had quietly swapped one out. I caught it by reading the README before
publishing rather than after, which is the only reason it is a footnote instead of a correction.

What I changed

Four things, none of them large:

I compute the budget per Actor - 338 − length("<user>/<slug>") - instead of assuming one number. Three of mine sit within 12 characters of losing a sentence.
No field description over 500 characters unless the field is an array. On an array, items catches the overflow. On a string, it does not.
tools/list before publish. It takes ten seconds and it is the only view of your Actor that your non-human callers actually have.
Read your descriptions as a set, not one at a time. This is the one that actually cost me something. I wrote two SEO Actors months apart, gave them the same clause, and never once put the two sentences side by side. A catalog is a document; mine was 23 documents that had never met.

The wider point is smaller than a bug and more annoying than one. I built 23 tools for AI agents to
call and spent all of my review time in a form built for humans. The one interface my actual callers
use, I had never opened.

prefill and default look the same in Console. Only one of them lets an agent skip your required field.

Ai-Q Labs — Fri, 11 Sep 2026 05:18:13 +0000

I have 23 audit Actors on the Apify Store. They all do a version of the same thing: take a public record, check whether what it still claims is true, and write the verdict into a dataset.

Last week I set out to prove they were unsafe for AI agents to chain together. I had a specific accusation in mind, I built the experiment to demonstrate it, and the experiment refused. What I found instead was a one-line schema fix that I have now applied, and a distinction between two input-schema settings that I had treated as interchangeable for months.

The accusation I was going to make

Chaining means running Actor A, taking something out of its output, and passing it into Actor B. For an agent to do that through the Apify MCP server, B has to be callable with a value that only A can supply. MCP is the Model Context Protocol, the interface that exposes your Actor to clients like Claude and Cursor as a tool they can call on their own.

So I read my own catalogue with a script instead of from memory. Of my 23 published Actors, 9 declare a URL or a domain in their input schema's required array. Counting every ordered pair where A emits the kind of identifier B requires, I get 61 possible chains: 36 joined by a URL, 25 by a domain.

A tenth Actor requires an identifier too, and it is the interesting one. dataset-to-github-issues takes a dataset ID and a repository name. Nothing else in my catalogue emits either, so no agent can ever reach it from another Actor's output - it is a second stage with no possible first stage. I had not noticed that until the script told me.

Then I looked at what those 9 do when the value never arrives.

I had swept my catalogue for target-naming defaults once before and found them, so the count itself was not the surprise. Restricting it to the Actors that can actually be a second stage sharpens it: seven of the nine carry a default that names a concrete target.

Actor	field	default
`bulk-domain-checker`	`domains`	`["apify.com"]`
`dead-link-checker`	`domains`	`["docs.apify.com"]`
`domain-availability-checker`	`domains`	`["apify.com"]`
`http-status-checker`	`urls`	`["https://apify.com/store"]`
`seo-ai-visibility-auditor`	`startUrls`	`[{"url":"https://apify.com"}]`
`sitemap-checker`	`domains`	`["apify.com"]`
`tech-stack-detector`	`domains`	`["apify.com"]`

All seven are also listed in required. I had assumed that word did some work.

My accusation was going to be this: in a two-stage workflow, an agent that drops the value between stages gets no error. The second Actor runs on apify.com, returns a well-formed dataset, and the agent reports a confident answer about a website the user never asked about. A wrong answer with no failure attached to it.

First, the part that turned out to be true

Before involving any agent, I called two of my own Actors with an empty input from the command line.

The first, seo-audit-tool, has urls in required and no default:

apify call aiqlabs/seo-audit-tool --input '{}'

Run: Calling Actor aiqlabs/seo-audit-tool (ZjU3YtyaWgqpNriNj)

Error: Input is not valid: Field input.urls is required

No run was created. No dataset, no compute, no partial result to misread. The platform named the missing field and stopped.

The second, http-status-checker, has urls in required and a default:

apify call aiqlabs/http-status-checker --input '{}'

INFO  Checking 1 URL(s) with 8 in flight. HTML analysis: on.
INFO  Done. 1 URL(s): 0 error, 0 warning, 0 info, 1 clean.

Apify call result: SUCCEEDED
Run ID: sMLok9a5KMSSdtnD1

I sent nothing. It audited something. Reading the input the platform stored for that run shows what:

{"urls":["https://apify.com/store"],"analyzeHtml":true,"respectRobotsTxt":true,
 "robotsAgent":"Googlebot","onlyIssues":false,"maxConcurrency":8,
 "requestTimeoutSecs":20,"maxRedirects":8,"slowResponseMs":3000,"maxUrls":10000}

Same platform, same account, same minute, same field name, both fields marked required. One call was refused and one was silently completed. The only difference is the default.

There is a second thing in that pair worth pulling out. seo-audit-tool does carry a prefill on urls - two example URLs that appear in the Console form. It was still rejected. prefill populates the form for a human; default is substituted for whoever omits the field.

The input schema specification is not vague about this. On prefill:

this field is only used in the user interface but does not affect the Actor functionality and API

and, in the same paragraph:

the Prefill value won't be used by existing integrations such as Actor tasks or API calls, but the Default will be if specified

On default, it names the callers explicitly: the platform passes the value when the user omits it "via any means (API, CLI, scheduler, or user interface)". An agent calling through the MCP server is one of those means.

I had read that page. I read it as a description of two features and used both, on the same field, for months.

There is a detail in http-status-checker that makes the gap concrete. The two settings on urls do not even hold the same value:

"urls": {
  "prefill": ["https://apify.com/store",
              "https://apify.com/this-page-does-not-exist",
              "http://apify.com/"],
  "default": ["https://apify.com/store"]
}

Three URLs for the form, one URL for anyone who omits the field. I wrote both, months apart, and never put them side by side. A caller that skips urls does not get the demo I built - it gets a shorter thing I stopped thinking about, and no part of the interface ever shows the two together.

Then the accusation collapsed

To test the chain I needed a question that one Actor cannot answer on its own. That took two attempts.

My first idea was sitemap-checker into http-status-checker: enumerate a site's sitemap, then check the URLs. It is a bad example, because sitemap-checker has checkUrlStatus set to true by default and already requests every URL. My second idea was hacker-news-link-rot into domain-availability-checker, and that fails the same way - the first Actor already emits domainRegistrationStatus.

That is worth pausing on. I had built every one of these Actors to answer a question completely, which is right for a standalone Store listing and is exactly what makes them poor chain stages. Composable and self-contained are not the same design, and I had only ever aimed at one of them.

The question I settled on works because it needs HTML analysis, which the sitemap Actor does not do:

Among the URLs listed in crawlee.dev's sitemap, which ones are marked noindex, and which ones are soft 404s?

A noindex page is one that tells search engines to keep it out of the index; a soft 404 is a missing page that returns 200 anyway. Both need the HTML of the page. sitemap-checker emits 11 fields and none of them is either. http-status-checker has both, behind an analyzeHtml switch that is on by default. The chain is necessary, and crawlee.dev is a real documentation site with a large sitemap, which makes it a fair test.

I gave that question, word for word, to four fresh agents with both Actors exposed through the Apify MCP server, and no other instructions. I did not tell them to be careful, and I did not mention inputs or defaults - in an earlier experiment I learned that asking an agent to count its rows is the same as telling it where to look.

Then I ignored what they told me and read what the platform stored.

stage-2 run	`urls` passed	first entry
`p5qwOESMQB0Vwj0ls`	500	`https://crawlee.dev/js/api/3.11/playwright-crawler/...`
`NPM7mW5oC848wn2ZL`	174	`https://crawlee.dev/js/api/3.16/puppeteer-crawler/...`
`3fO6vTbKnLlkDwGnw`	500	`https://crawlee.dev/blog`
`H0HkakmGE2bfkAdWU`	400	`https://crawlee.dev/js/api/3.15/core/interface/HttpResponse`
`2cJHKXtsrapw9aikV`	400	`https://crawlee.dev/js/api/3.14/browser-pool/class/PuppeteerPlugin`
`oGbGCrfPEM4uDuOBY`	400	`https://crawlee.dev/js/api/3.12/puppeteer-crawler`
`sDQnaUb7NxX9iz88B`	400	`https://crawlee.dev/js/api/3.11/core/class/SystemStatus`
`QDpb6PG685ptVthUN`	400	`https://crawlee.dev/blog`
`bB20ynAk9ymZafaiY`	724	`https://crawlee.dev/js/api/3.14/utils/function/downloadListOfUrls`

Nine stage-2 runs. Every one carries real crawlee.dev URLs. Not one carries https://apify.com/store. They also batched - 174 to 724 URLs per call - rather than making one call per URL.

The trap I had spent a day setting never closed.

Why it never closed

The mechanism is in the result once you stop arguing with it. A default stands in for a value the caller never had. Inside a chain, the agent produced that value itself one step earlier and is still holding it. The omission the default is waiting for does not occur.

That reconciles this with the case where the same defect does bite. When an agent calls one of these Actors cold - no preceding step, nothing in hand - the missing field is real, the platform fills it, and the run proceeds on a target nobody chose. The danger lives at the entry point of a workflow, not in the joins.

I would rather publish that than the article I planned, because it changes where you should spend attention. If you are worried about agents mangling your Actor, the risk is concentrated in the first call, not in the handoffs.

What is wrong with my own experiment

A reviewer would find this, so I will say it first.

The question I picked has a null answer. Here is the summary record from one 500-URL batch:

{"checked":500,"severity":{"ok":500,"info":0,"warning":0,"error":0},
 "headline":{"broken":0,"soft404s":0,"noindexOnLivePages":0,"metaRefreshRedirects":0}}

crawlee.dev has no soft 404s and no noindex pages in those URLs. "Which ones are noindex?" answers to "none". An agent that had quietly audited apify.com/store instead would probably also have answered "none".

So the agents' answers cannot tell a careful run from a careless one here. Only the stored inputs can. That is the one thing I got right by accident of habit: I built the measurement on records the platform keeps, not on what the agents said about themselves. It mattered more than I expected - all four agents finished without returning a report to me at all, and every number above survived that because none of them came from an agent.

A stronger version of this trial would use a site with known noindex pages, so a substituted target produces a different answer rather than the same one. If you repeat this, pick your target site for that property.

The fix, and what it costs

For any input field that names what to work on - a URL, a domain, a repository, an account - delete the default and keep the prefill.

"urls": {
  "title": "URLs",
  "type": "array",
  "editor": "stringList",
  "prefill": ["https://apify.com/store"]
}

with "required": ["urls"] alongside it. The Console form still opens with an example filled in, so nothing changes for a human clicking through your Actor. What changes is that a caller who omits the field now gets Field input.urls is required instead of a clean dataset about somebody else's website.

The specification anticipated this too. Its guidance for required reads:

Use for fields that don't have a reasonable default and MUST be entered by the user (e.g., API token, password)

Fields that don't have a reasonable default. Seven of mine had both, which is a combination the sentence quietly rules out. Marking a field required and then supplying the answer yourself is not a stricter setting; it is two instructions that cancel.

Keep default for settings that describe how to work - concurrency, timeouts, depth limits, feature switches. Those are genuinely safe to assume, and an empty call that inherits them is behaving correctly.

The distinction is worth stating as a rule: default on a method is a convenience; default on a subject is a substitution.

Checking your own Actors

I wrote the catalogue sweep as a script so I would stop trusting my memory of my own schemas. It reads every local .actor/ directory, classifies each input and output field into an identifier kind, and reports which of your Actors can be a chain stage and which of them will accept an empty call:

node composability.mjs           # summary and pair matrix
node composability.mjs --rules   # print the classifier so you can audit it

The --rules flag exists because the classifier is regular expressions over field names, and a regex table you cannot see is a number you should not trust. Mine misfired the first time I ran it: it reported that github-repository-audit requires no identifier, which is nonsense for an Actor that audits repositories. The cause was real and boring - that schema has no required key at all, so every field is optional and the defaults decide everything.

If you want a single check with no script, call your own Actor with an empty input and see what happens:

apify call <your-actor> --input '{}'

If it succeeds, look at what it audited. That is what an agent gets when it forgets.

What I changed and what I did not

I have not stripped the defaults from all seven Actors yet. The measurement says the harm is concentrated in cold calls, and I want to measure that path on these specific Actors before I change seven live listings - the last time I acted on a confident prediction here, the prediction was wrong twice.

What I have changed is the rule I apply to new fields, and the check I run before publishing. Both of them came out of an experiment that failed to prove its own thesis, which is becoming a habit I am not entirely unhappy about.

My screener recommended every job it could not read

Ai-Q Labs — Sat, 22 Aug 2026 08:05:41 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Today I wrote a screener for a freelance job board. One category alone holds ~800 open postings, most of which I can rule out from the text: the ones that require a video interview, the ones restricted by age or gender, the ones that want your personal anecdotes, and — the one that actually matters to me — the ones that state the work must be written without AI assistance.

The tool is two stages, run from the browser console on a listing page:

Parse the listing DOM for id, pay, applicant count, open slots, deadline. Drop anything with no slots left or a bad applicant-to-slot ratio.
fetch() each survivor's detail page and test the body text against nine disqualifier regexes. Whatever survives both stages is the shortlist.

It ran. It produced a shortlist. I was about to apply off that shortlist.

Bug Fix or Performance Improvement

Bug 1: a failed fetch was an endorsement

The whole second stage was this:

const h = await fetch(url, { credentials: 'include' })
                .then(r => r.text())
                .catch(() => '');                        // <- here
const t = new DOMParser().parseFromString(h, 'text/html')
                .body.innerText.replace(/\s+/g, ' ');
const hard = Object.entries(HARD).filter(([, r]) => r.test(t)).map(([k]) => k);

if (hard.length) { hard.forEach(k => ngc[k] = (ngc[k]||0) + 1); continue; }  // reject
ok.push(...);                                            // everything else: recommend

Follow the failure path. fetch rejects, t is the empty string, no regex matches an empty string, hard.length === 0, and the posting lands in ok.

The function had exactly two outcomes: found a disqualifier and recommend. There was no outcome for I could not look. So a network error, a 404, an expired session, a rate-limited response — each one silently converted into a positive recommendation. The worse the fetch went, the cleaner the posting looked.

Nothing throws. Nothing logs. The shortlist just quietly gets longer.

Here it is, measured, against a posting id that does not exist:

old implementation -> response body: 58 characters (a 404 page)
                   -> disqualifiers matched: none
                   -> verdict: RECOMMENDED

new implementation -> verdict: unreadable (HTTP 404)

The fix

Give "I could not look" its own outcome, and check twice:

let t = '';
try {
  const res = await fetch(url, { credentials: 'include' });
  if (!res.ok) throw new Error('HTTP ' + res.status);
  t = new DOMParser().parseFromString(await res.text(), 'text/html')
        .body.innerText.replace(/\s+/g, ' ');
} catch (e) {
  unreadable.push(`${id} (${e.message})`);   // not "clean" - unjudged
  continue;
}
// A 200 is not proof you got the page you asked for.
if (!/DETAIL_MARKER/.test(t)) { unreadable.push(`${id} (no body)`); continue; }

The second guard is the one I would have skipped a year ago. A redirect to a login wall is a perfectly successful HTTP response, and it contains none of my disqualifier phrases — which under the old code made it a great job.

Bug 2: zero results, silently

Same shape, different surface.

I also wanted a quick count per keyword, so I fetched the search URLs directly instead of navigating. Eight keywords, one pass:

{ "Claude": "0", "ChatGPT": "0", "Python automation": "0",
  "scraping": "0", "SEO writing": "0", ... }

Eight zeros. I nearly wrote this board has nothing in my areas into my notes and moved on.

Then I navigated to the first URL in a real tab. 99 results.

The listing pages are client-rendered; the detail pages are server-rendered. Same origin, same session, same cookies — different rendering, and only one of them survives fetch. fetch returned HTTP 200 and well-formed HTML every time. The HTML simply had no postings in it yet.

Fix — make the parser prove it looked:

const total = +((document.body.innerText.match(/([0-9,]+) of/) || [])[1] || '0')
                .replace(/,/g, '');
if (total > 0 && items.length === 0) {
  throw new Error(`parsed 0 items but the page reports ${total}`);
}

The page states its own row count in its header. If it says 804 and my parser extracted 0, the shelf is not empty — I am not looking at it. Throwing is right here. Returning [] is a lie with a plausible shape, and a plausible shape is what makes it survive review.

Why these are one bug

Both are the same failure: a check that can only answer "found" or "nothing" cannot tell you the difference between "nothing is there" and "I never looked." Absence of evidence gets recorded as evidence of absence, and it gets recorded in the format that means good news — a clean posting, an empty shelf.

Every silent-zero bug I have shipped has had this shape. A collector of mine once exited 0 holding 27% of its rows. A link checker of mine once reported 9 of 9 while three links in the file were already dead. Same skeleton, different clothes: the failure path produced a value indistinguishable from success.

The rule I now apply: if a function's failure path can produce a value, that value must be a third kind of answer, not a copy of the happy one.

Result

One page, re-run after the fix:

{ total: 804, listed: 54, cand: 7,
  ok: 3,
  ngc: { own-experience-required: 2, interview: 1, pay-unspecified: 1 },
  unreadable: [] }

unreadable: [] now carries information. Before the fix it carried none, because it did not exist — and its contents were being counted as job recommendations.

Two more regexes went in as well, both false negatives I caught by hand while spot-checking: one posting disqualified itself with the phrasing "without using AI tools" (my pattern only looked for the word prohibited), and another asked for "your own impressions" (my pattern only looked for personal experience). Those two are ordinary pattern gaps. The two above are the ones worth writing down, because no amount of adding patterns would have found them.

I paginated by 100 and lost 39 of 422 rows. At 99 and 101, nothing was missing.

Ai-Q Labs — Sat, 22 Aug 2026 02:15:13 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I wanted one number: what share of the entries in this challenge declare a particular prize category. It decides whether writing another entry is worth the evening.

I got that number wrong three times in one morning. Every wrong answer looked finished.

First wrong answer: 19. The listing endpoint returns titles and tags but not bodies, and the prize category is declared by a heading inside the body. So I counted titles containing "gemini" or "google". That found 19 of the 66 entries that actually carry the heading — and one of the 47 it missed was my own most recent entry, whose title mentions neither word. I had been quoting that proxy for days.

Second wrong answer: 23.9%. So I fetched all 412 bodies, six at a time, no delay. My loader was this:

arts = []
for f in dir.glob("*.json"):
    try:
        a = json.loads(f.read_text())
        if a.get("body_markdown"):
            arts.append(a)
    except Exception:
        bad += 1

277 of the 412 responses had the body Retry later. Two words, plain text, where JSON was expected. The except branch counted them and moved on, and I printed a clean table from 134 items:

                items   coverage   category share
naive             134      32.5%          23.9%
complete          411     100.0%          18.5%

5.4 points off, with nothing anomalous in it. The number was not noisy. It was wrong, and it was plausible — which is the only combination that actually costs you anything.

The third wrong answer is the interesting one, and I only found it because I asked something else to look at my fix. That's further down.

Bug Fix or Performance Improvement

Fix 1: "did not parse" is not "not there"

The loader has no bug in it. It does exactly what it says. The bug is in what it means: an exception during parse was being treated as evidence about the item, when it was evidence about the transport.

def collect_strict(server, ids, max_rounds=8):
    """Same fetch, two extra rules: unparsed is not absent, and slow down on refusal."""
    done, pending, rounds = {}, list(ids), 0
    while pending and rounds < max_rounds:
        rounds += 1
        still = []
        for i in pending:
            rec = parse(server.get(i))
            if rec is None:
                still.append(i)
            else:
                done[i] = rec
        pending = still
        server.cooldown //= 2          # back off: fewer requests per unit time
    missing = [i for i in ids if i not in done]
    if missing:
        raise RuntimeError(f"{len(missing)} of {len(ids)} ids never resolved")
    return [done[i] for i in ids], rounds

Against the real API — 0.45s between requests, doubling to a 2s ceiling — that recovered all 277 in four rounds: 134, then 77, then 63, then 3, then zero unresolved. One id stayed unresolved and turned out to be a genuine 404 (an entry deleted between the listing call and the fetch), so the assertion permits resolved-as-404 alongside parsed.

The offline reproducer at the end of this post runs the same two collectors against a fake server that refuses in bursts:

                       items   coverage   in category    share
ground truth             412     100.0%            97    23.5%
naive collector          157      38.1%            39    24.8%
strict collector         412     100.0%            97    23.5%

Note how boring the naive row is. 24.8% against a true 23.5%. Nobody looks twice at that.

Fix 2, which I would not have found on my own

With the numbers in hand, I handed the whole thing to Gemini and asked, among other things: what can still silently truncate the dataset underneath my completeness assertion?

Its first-ranked answer was that my assertion validates the fetched set against the listing, and says nothing about whether the listing is complete.

I went to check. This is the same endpoint, same tag, same minute, walked page by page until an empty page:

 per_page  pages  unique ids   note
       25     17         422   empty page 18
       50                      non-JSON at page 10: 'Retry later'
       75      6         422   empty page 7
       99      5         422   empty page 6
      100      4         383   empty page 5
      101      5         422   empty page 6
      125      4         422   empty page 5
      150      3         422   empty page 4
      200      3         422   empty page 4

Only per_page=100 loses data. 99 is fine. 101 is fine. 100 comes up 39 short of 422 and announces it by returning an empty page 5 — the universal signal for you have reached the end.

It gets better. Those four pages of 100:

rows returned : 400
unique ids    : 383
ids appearing on more than one page: 17
  id 4267457 appears 2x on pages [2, 3]
  id 4228478 appears 2x on pages [2, 3]
  ...

Four pages of 100 returned exactly 400 rows — the count you would sanity-check against — while containing 383 articles, 17 of them twice, and 39 not at all. The 39 are not random: the per_page=100 walk reaches back to 2026-07-14, the full listing reaches back to 2026-06-21. It is the oldest entries that vanish, which is precisely the population you would use to say anything about how the challenge has changed over time.

I want to be careful about what I am claiming here. I did not find the cause inside DEV's code; I have no access to it. What I measured is that the same query returns 422 or 383 items depending only on page size, and the short answer terminates cleanly. That is reproducible from any machine, in about sixty requests, with no credentials.

Fix 3: the check I now put in front of the other checks

def assert_listing_stable(fetch, tag, sizes=(99, 100, 101, 200)):
    """A listing you cannot reproduce at two page sizes is not a population."""
    counts = {}
    for pp in sizes:
        ids, page = set(), 1
        while True:
            rows = fetch(tag, pp, page)
            if not rows:
                break
            ids.update(r["id"] for r in rows)
            page += 1
        counts[pp] = ids
    best = max(counts.values(), key=len)
    for pp, ids in counts.items():
        if len(ids) != len(best):
            raise RuntimeError(
                f"per_page={pp} yields {len(ids)} ids, per_page="
                f"{max(counts, key=lambda k: len(counts[k]))} yields {len(best)}"
            )
    return best

Nine lines, and it fails the build on an API defect I could not have guessed at.

Code

The offline reproducer is self-contained and needs nothing but the standard library. It builds a corpus whose property-of-interest is correlated with position — which is the case that matters — and runs both collectors against a server that refuses in bursts.

"""A rate limiter that answers with the words "Retry later" is not an error your
parser will notice. It is a sampler you did not know you installed.

Runs offline against a fake server, so the numbers are reproducible without
touching anyone's API.

  collect_naive()   drops anything that fails to parse and returns what it got
  collect_strict()  treats "did not parse" as "not fetched yet", slows down,
                    retries, and refuses to return until every id is accounted for

Requires: nothing but the standard library.
"""
import sys

if hasattr(sys.stdout, "reconfigure"):
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")

import json

N = 412              # items in the listing
BURST = 8            # requests served before the limiter trips
COOLDOWN = 16        # requests refused before it serves again


def truth(i):
    """Ground truth: does item i belong to the category being counted?

    The first third of the listing is denser than the rest -- newer submissions
    mention the prize category more often than older ones. That is the part that
    matters: the property being counted is correlated with position.
    """
    threshold = 40 if i < N // 3 else 15
    return (i * 37) % 100 < threshold


class Server:
    """Serves JSON in bursts. When the limiter trips it answers 'Retry later'.

    The refusal is a 200 with a plain-text body. Nothing raises, nothing retries
    itself, and the caller gets a str where it expected JSON.
    """

    def __init__(self, cooldown=COOLDOWN):
        self.n = 0
        self.cooldown = cooldown
        self.served = 0
        self.refused = 0

    def get(self, i):
        self.n += 1
        if self.cooldown and (self.n % (BURST + self.cooldown)) > BURST:
            self.refused += 1
            return "Retry later\n"
        self.served += 1
        return json.dumps({"id": i, "in_category": truth(i)})


def parse(raw):
    try:
        return json.loads(raw)
    except Exception:
        return None


def collect_naive(server, ids):
    """What I actually wrote. There is no bug in it -- it does exactly what it says."""
    out = []
    for i in ids:
        rec = parse(server.get(i))
        if rec is not None:
            out.append(rec)
    return out


def collect_strict(server, ids, max_rounds=8):
    """Same fetch, two extra rules: unparsed is not absent, and slow down on refusal."""
    done, pending, rounds = {}, list(ids), 0
    while pending and rounds < max_rounds:
        rounds += 1
        still = []
        for i in pending:
            rec = parse(server.get(i))
            if rec is None:
                still.append(i)
            else:
                done[i] = rec
        pending = still
        server.cooldown //= 2          # back off: fewer requests per unit time
    missing = [i for i in ids if i not in done]
    if missing:
        raise RuntimeError(f"{len(missing)} of {len(ids)} ids never resolved")
    return [done[i] for i in ids], rounds


def share(records):
    n = len(records)
    hits = sum(1 for r in records if r["in_category"])
    return hits, n, (hits / n * 100 if n else 0.0)


def main():
    ids = list(range(N))
    actual = [{"id": i, "in_category": truth(i)} for i in ids]

    naive = collect_naive(Server(), ids)
    strict, rounds = collect_strict(Server(), ids)

    print(f"{'':<20}{'items':>8}{'coverage':>11}{'in category':>14}{'share':>9}")
    for label, recs in (("ground truth", actual),
                        ("naive collector", naive),
                        ("strict collector", strict)):
        hits, n, pct = share(recs)
        print(f"{label:<20}{n:>8}{n / N * 100:>10.1f}%{hits:>14}{pct:>8.1f}%")

    _, _, t = share(actual)
    _, _, g = share(naive)
    print(f"\nThe naive collector reported {g:.1f}% from {len(naive)}/{N} items. "
          f"The answer is {t:.1f}%.")
    print(f"It raised nothing, logged nothing, and its table looked complete.")
    print(f"The strict collector resolved every id in {rounds} rounds.")


if __name__ == "__main__":
    main()

The page-size sweep is fifteen lines of curl around the same idea and is quoted in full above.

My Improvements

Done:

The collector treats an unparsed response as unfetched, backs off, retries, and asserts coverage before anything computes a share. Recovered 277 of 277.
The completeness assertion permits resolved as 404 as well as parsed, because "deleted since the listing" is a real state and silently dropping it is the same bug one level down.
assert_listing_stable runs before the fetch. On this tag it fails, correctly.

Done because the review said so:

Everything in the previous section. The listing defect was Gemini's first-ranked answer and I had not considered it.

Not done, and I'd rather say so:

I still cannot tell you what HTTP status those 277 refusals carried. I wrote the bodies to disk with curl -o and never recorded the status codes, so the question of whether a plain resp.raise_for_status() would have caught the whole thing is one I destroyed the evidence for. Sequential requests do not reproduce it, and I am not going to hammer a free API until it stops talking to me just to find out. The lesson stands on its own: store the status next to the body, or you cannot debug the fetch afterwards.
The fix lives in the loader. It belongs in the fetch layer, raising on a transport failure before anything is written to disk. That is a rewrite, not an edit, and I have not done it.

Left as a guard, deliberately:

Recomputing the denominator at two page sizes every time, even when nothing has changed. It costs about sixty requests and it is the only reason I know the number at all.

Best Use of Google AI

I used Gemini (free tier, Flash) once: after the fix worked, with the numbers already in hand, to ask what my fix still could not see. I gave it both wrong answers, the loader, the strict collector, the assertion, and four questions. I did not adopt any of it — I measured each claim.

It named the class first:

This is a Silent Partial Sample (or Silent Truncation) resulting from Plausible Degradation. [...] The system experienced soft failures—surrogate metrics (Attempt 1) and swallowed rate-limit payloads (Attempt 2)—that degraded the data quality into a plausible subset rather than triggering an explicit system error.

That one sentence covers both of my wrong answers, which is what I had asked for and had not managed to write myself. Then four ranked failure modes, and a verdict on the fix.

Claim                                             Verdict        Measurement
Rank 1  the listing itself is truncated           held           per_page=100 -> 383 of 422,
                                                                 17 duplicated rows, empty page 5
Rank 2  valid JSON that lacks the key fields      held           the deleted entry parses fine and
                                                                 has no body_markdown
Rank 4  HTTP 200 carrying an error payload        not observed   the deleted entry returns a real 404
Q3      backoff+assert is a patch, not a fix      accepted       and see "Not done" above
Q4      I may not call the 32.5% sample biased    conceded       claim retracted, see below

Rank 1 is the reason this post exists. I asked what my completeness check could not see, and the answer was: the thing it checks against. Sixty requests later I had an API defect that is reproducible by anyone, in which the page size everybody reaches for first is the only one that loses data.

Question 4 cost me a claim I liked. I had written that the 32.5% sample was biased, not merely small, because rate-limit refusals arrive in bursts and bursts land on neighbours in a listing. Gemini pointed out that I overwrote the failed responses with the successful refetch, so I no longer have the evidence for that:

You cannot claim the 32.5% sample was definitively biased due to listing-order burst clustering. You cannot claim spatial, chronological, or network locality for the failed items, because you lost the exact temporal sequence and per-request metadata.

What it left me is narrower and I think stronger:

The 32.5% sample is unvalidated and methodologically unreliable because the sampling mechanism was governed by server load-shedding rather than random selection. [...] the sample cannot be assumed to be Missing Completely at Random (MCAR).

So: 23.9% versus 18.5% is an empirical divergence of 5.4 points, and the sample was drawn by a server deciding what it felt like answering. That is enough to throw the number away. It is not enough to say why it leaned the way it did, and I have edited that claim out of my notes.

One last thing, which happened while I was measuring the above. My sweep script crashed decoding curl's output — the console here is cp932 and one of the titles was not — and my error handler, which was written to notice Retry later, reported it as "rate-limited at page 1" for all nine page sizes. I spent a minute believing the API had cut me off, in the middle of writing a post about mistaking a local failure for an absent record.

That is the whole thing, really. Six of my submissions this month are the same shape: every check passed and the answer was still wrong. A ranking only I could see. A collector reporting success on 27% of the data. A rate limiter counting the retries. A verifier that read every character back while three links in the document pointed at nothing. And now a denominator that was wrong three ways, where the third way was invisible until I asked something outside my own head what my check was not looking at.

The checker said 9 of 9. Three links in the file were already 404.

Ai-Q Labs — Fri, 21 Aug 2026 09:05:14 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I generate a PDF from a plain-text source with reportlab, and I verify it before it leaves the machine. The verifier is the obvious one: open the PDF with PyMuPDF, pull the text back out, assert that every required item is in there.

Nine probes. Nine passes. I sent the file.

The file I sent was fine, but only because I had also looked at it. When I rendered the pages to PNG and put my eyes on them, two things were wrong that the checker had not mentioned: a section heading was set in the wrong face, and a long URL ran off the right-hand side of the paper.

That second one is the interesting one, because of what "off the paper" does to a text-extraction check.

The URL was 115 characters. The text column is 511.3pt wide. At 8.2pt Courier, indented, the line needed 575.6pt. The last few characters were drawn past x=595.3 — the edge of an A4 sheet — and they are not on the page in any sense: not in the print, not in the render, not in the extracted text.

So the extractor returned the URL without its tail. And a dev.to URL without its tail is a different URL:

https://dev.to/aiq_labs/...-only-one-of-them-was-real-1jh0   ->  200
https://dev.to/aiq_labs/...-only-one-of-them-was-real-       ->  404

My checker asserted the URL was present. It was present — four characters short, pointing at nothing. None of my nine probes reached far enough into the string to notice, because when you write a probe by hand you write the part that identifies the item, not the part that makes it resolve.

A text-extraction check cannot see the right-hand edge of the paper. It reads what the content stream says. It does not know where any of it landed.

Bug Fix or Performance Improvement

Measuring it instead of asserting it

The original naive generator was never committed, so I reconstructed it from the fixed one by removing exactly the two changes I had made: the shrink-to-fit loop and the structural heading test. Then I ran both versions of the real document through both kinds of check.

Same source text, A4, 42pt margins:

                            content check   spans off-column   URLs truncated   chars lost
  pre-fix (reconstructed)      9/9 PASS            4              3 of 10           23
  shipped                      9/9 PASS            0              0 of 10            0

Three of ten links silently truncated, twenty-three characters gone, and the content check reports 9/9 PASS on both. It is not that the check is weak. It is that the check is answering a different question than the one I thought I was asking.

Two ways to leave the column, and only one of them loses data

Building a minimal reproducer made the shape clearer. There are two regimes, and the difference matters:

v1, spans that left the text column:
  p1  + 44.7pt  past the paper   https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-sa
  p1  + 34.8pt  into the margin  Verification tooling: deterministic geometry checks

The first line ran past the sheet, so its tail is gone from the extracted text too — the content check could have caught it with a luckier probe. The second stopped inside the paper but outside the column. Nothing is lost there. The characters extract perfectly. The document just looks broken, and no content check of any kind will ever tell you, because from the extractor's point of view nothing happened.

In the reproducer, both versions score:

             content check   geometry check   URL fidelity
v1 naive       10/10 PASS     2 off-column    4 chars lost
v2 fitted      10/10 PASS     0 off-column    0 chars lost

The content column is constant. Every bit of signal is in the other two.

The fix

Two changes, on two different layers.

Generator: shrink the line until it fits the column, with a floor.

size = BODY_SIZE
while size > MIN_SIZE and stringWidth(line, "Courier", size) > COL:
    size -= 0.15
c.setFont("Courier", size)
c.drawString(LEFT, y, line)

Verifier: stop asking what the document says and start asking where it put it.

def check_geometry(pdf_path, slack=0.5):
    """Every drawn span must lie inside the text column."""
    offenders = []
    with fitz.open(pdf_path) as doc:
        for pno, page in enumerate(doc, 1):
            right_edge = page.rect.width - LEFT
            for block in page.get_text("dict")["blocks"]:
                for line in block.get("lines", []):
                    for span in line["spans"]:
                        x0, _, x1, _ = span["bbox"]
                        if x1 > right_edge + slack or x0 < LEFT - slack:
                            offenders.append({
                                "page": pno,
                                "overshoot_pt": round(x1 - right_edge, 1),
                                "past_paper_pt": round(x1 - page.rect.width, 1),
                                "text": span["text"].strip(),
                            })
    return offenders

That is a dozen lines and it turns "looks fine to me" into a number that fails a build. past_paper_pt is what separates the two regimes: positive means the glyphs are gone, zero or less means they printed somewhere they should not have.

I also added a fidelity check, because it is cheap and it catches the failure that actually costs something: pull every URL out of the source and every URL out of the PDF, compare them positionally, and report the character delta. Twenty-three, in the pre-fix document.

Code

Everything above is reproducible from one file with two dependencies (reportlab, pymupdf). Measured on Python 3.14.2, reportlab 4.5.1, PyMuPDF 1.27.2.2, Windows. It builds the same source document twice and runs all three checks over both — copy, run, and you get the table above.

"""A text-extraction check cannot see page geometry.

Builds one source document twice:
  v1  naive   - fixed font size (what a first draft does)
  v2  fitted  - shrink each line until it fits the text column

Then runs two verifiers over both PDFs:
  check_content()   - "is every field present?"      (PyMuPDF text extraction)
  check_geometry()  - "is every glyph on the paper?" (PyMuPDF span bboxes)
  check_fidelity()  - compares extracted URLs against the source, character for character

Requires: reportlab, pymupdf
"""
import sys

if hasattr(sys.stdout, "reconfigure"):
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")

import pathlib
import re

import fitz
from reportlab.lib.pagesizes import A4
from reportlab.pdfbase.pdfmetrics import stringWidth
from reportlab.pdfgen import canvas

HERE = pathlib.Path(__file__).resolve().parent
W, H = A4                       # 595.3 x 841.9 pt
LEFT, TOP, BOTTOM = 42, 52, 46
COL = W - 2 * LEFT              # the text column: 511.3 pt
BODY_SIZE, LEAD, MIN_SIZE = 8.2, 10.6, 5.4

SOURCE = """Ai-Q Labs
independent developer - remote

----------------------------------------------------------------
Selected writing
----------------------------------------------------------------
Race conditions in parallel agent sessions
  https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-same-race-condition-gemini-told-me-only-one-of-them-was-real-1jh0
A collector that reported success on 27% of the data
  https://dev.to/aiq_labs/my-collector-reported-success-it-had-27-of-the-data-2n0d

----------------------------------------------------------------
Tooling
----------------------------------------------------------------
Apify actors published            23
dev.to articles published          6
  Verification tooling: deterministic geometry checks for generated PDFs, so a broken layout cannot pass a test
"""

# What a person actually writes when asked "check the resume came out right".
# Nine probes, one per thing that must appear. Note how each one names the
# item -- none of them reaches for the tail end of a URL.
PROBES = [
    "Ai-Q Labs",
    "independent developer",
    "Selected writing",
    "Race conditions in parallel agent sessions",
    "https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-same-race-condition",
    "https://dev.to/aiq_labs/my-collector-reported-success",
    "Tooling",
    "Apify actors published",
    "dev.to articles published",
    "deterministic geometry checks for generated PDFs",
]


def is_rule(s):
    s = s.rstrip()
    return len(s) > 3 and set(s) == {"-"}


def build(out_path, fitted):
    lines = SOURCE.splitlines()
    c = canvas.Canvas(str(out_path), pagesize=A4)
    y = H - TOP
    for i, raw in enumerate(lines):
        line = raw.rstrip()
        prev = lines[i - 1] if i else ""
        nxt = lines[i + 1] if i + 1 < len(lines) else ""

        if y < BOTTOM + LEAD:
            c.showPage()
            y = H - TOP

        if is_rule(line):
            y -= 3
            c.setStrokeGray(0.75)
            c.setLineWidth(0.5)
            c.line(LEFT, y, W - LEFT, y)
            c.setStrokeGray(0)
            y -= LEAD
            continue

        if bool(line.strip()) and is_rule(prev) and is_rule(nxt):
            c.setFont("Helvetica-Bold", 9.6)
            c.drawString(LEFT, y, line.strip())
            y -= LEAD + 1
            continue

        if line:
            size = BODY_SIZE
            if fitted:
                while size > MIN_SIZE and stringWidth(line, "Courier", size) > COL:
                    size -= 0.15
            c.setFont("Courier", size)
            c.drawString(LEFT, y, line)
        y -= LEAD
    c.save()
    return out_path


def extract(pdf_path):
    with fitz.open(pdf_path) as doc:
        return "".join(page.get_text() for page in doc)


def check_content(pdf_path):
    """The check I actually ran: extract text, assert every field is present."""
    text = extract(pdf_path)
    missing = [p for p in PROBES if p not in text]
    return len(PROBES) - len(missing), len(PROBES), missing


def check_geometry(pdf_path, slack=0.5):
    """Every drawn span must lie inside the text column."""
    offenders = []
    with fitz.open(pdf_path) as doc:
        for pno, page in enumerate(doc, 1):
            right_edge = page.rect.width - LEFT
            for block in page.get_text("dict")["blocks"]:
                for line in block.get("lines", []):
                    for span in line["spans"]:
                        x0, _, x1, _ = span["bbox"]
                        if x1 > right_edge + slack or x0 < LEFT - slack:
                            offenders.append({
                                "page": pno,
                                "overshoot_pt": round(x1 - right_edge, 1),
                                "past_paper_pt": round(x1 - page.rect.width, 1),
                                "text": span["text"].strip(),
                            })
    return offenders


def check_fidelity(pdf_path):
    """Compare each URL in the PDF against the URL at the same position in the source.

    Positional, not prefix-matched: an earlier attempt paired them by prefix and
    happily matched a truncated URL against a shorter unrelated one.
    """
    url_re = re.compile(r"https://\S+")
    want = url_re.findall(SOURCE)
    got = url_re.findall(extract(pdf_path))
    return [(w, g, len(w) - len(g)) for w, g in zip(want, got)]

def main():
    v1 = build(HERE / "out_v1_naive.pdf", fitted=False)
    v2 = build(HERE / "out_v2_fitted.pdf", fitted=True)

    print(f"{'':<12}{'content check':>16}{'geometry check':>18}{'URL fidelity':>16}")
    for label, path in (("v1 naive", v1), ("v2 fitted", v2)):
        ok, total, missing = check_content(path)
        off = check_geometry(path)
        lost = sum(d for _, _, d in check_fidelity(path))
        print(
            f"{label:<12}"
            f"{(str(ok) + '/' + str(total) + (' PASS' if not missing else ' FAIL')):>16}"
            f"{('0 off-column' if not off else str(len(off)) + ' off-column'):>18}"
            f"{(str(lost) + ' chars lost'):>16}"
        )

    print("\nv1, spans that left the text column:")
    for d in check_geometry(v1):
        edge = "past the paper" if d["past_paper_pt"] > 0 else "into the margin"
        print(f"  p{d['page']}  +{d['overshoot_pt']:>5}pt  {edge:<16} {d['text'][:52]}")

    print("\nv1, URLs as extracted:")
    for want, got, delta in check_fidelity(v1):
        print(f"  -{delta} chars: ...{want[-34:]}")
        print(f"              ...{got[-34:] if got else '(nothing extracted)'}")


if __name__ == "__main__":
    main()

The claims in the last section are a second file, check_gemini_claims.py, built the same way: each claim becomes a deliberately broken PDF, then a measurement.

One caveat about the vertical bounds check, found while writing that file: as written it flags the first line of a page, because a baseline placed exactly at the top margin puts the glyph ascenders above it. That is arguably a real defect in my generator rather than a false positive, but it is a real thing you will hit on the first run.

My Improvements

Done:

Shrink-to-fit in the generator; 0 off-column spans in the shipped document.
check_geometry in the verifier, run on every build.
URL fidelity check, source vs. PDF, positional, character-exact.

Done because the review said so:

Vertical bounds and line-collision checks, both verified to catch cases the horizontal check reports clean (numbers below).

Not done, and I'd rather say so:

The generator fix is a patch, and I now have the measurement that proves it (below). The real fix is to stop drawing long strings as single un-wrapped drawString calls and let a layout engine break them. I have not done that yet, because I have exactly one document and the verifier now fails the build if the patch is ever insufficient. When there is a second document, the patch stops being defensible.

Left as a guard, deliberately:

The render-and-look step. It is the only reason I found any of this, and no check I have written since would have caught the wrong-typeface heading — that one has no numeric signature at all.

Best Use of Google AI

I used Gemini (free tier, Flash) at one point: after both fixes were written and the numbers were in, before I decided the job was done. I handed it the measurement table, both check functions, both fixes, and three questions — state the general class of defect, name what my new check still misses, and tell me for each of my two fixes whether it is a fix or a patch.

It gave the class cleanly:

A purely semantic text-extraction test [...] cannot detect geometric presentation defects [...] Text extractors read the stream's character data (the what); they do not simulate clipping paths, viewports, or bounding-box intersections with canvas boundaries (the where and how).

Then four ranked failure modes my check still misses, each with code, and a verdict on each of my two fixes: shrink-to-fit — PATCH. Geometry check — GENUINE FIX.

I did not adopt any of it. I built each claim as a deliberately broken PDF and measured.

Claim                                          Verdict        Measurement
shrink-to-fit still fails at the size floor    held           202-char line shrank to 5.35pt
                                                              (floor 5.4), still outside the column
Rank 1  bottom overflow                        held           horizontal check 0, vertical check 6
Rank 2  overlapping lines                      held           both bounds checks 0, collision check
                                                              5 pairs, overlap up to 7.0pt
stringWidth disagrees with PyMuPDF metrics     did not hold   Courier +0.00pt, embedded TTF -0.03pt
Rank 4  text hidden by a clip path             did not hold   and it inverted, see below

Three held. Two did not, and the two failures are the useful part.

The font-metrics claim was wrong on my setup. The argument was that stringWidth uses target font metrics while PyMuPDF recomputes bounding boxes from embedded metrics, so my shrink loop is built on a number that lies. Measured, they agree to 0.00pt for base-14 Courier and to 0.03pt for an embedded TrueType face. Plausible mechanism, real in principle, not happening here.

The clip-path claim inverted. The prediction was that get_text() returns text that graphics commands have clipped away, so my content check would pass while the page printed only part of the string. I built exactly that document — 74 characters drawn inside a 150pt-wide clip rectangle — and got the opposite:

extracted : 'PAYMENT TERMS: net 30 days from'
text check: FAIL  (the string is not there)
geometry  : PASS  (the clipped bbox is inside the margins)
rendered  : ink stops at 150pt of a 153pt span

PyMuPDF 1.27 honours the clip during extraction. So the check that got called weak is the one that catches this, and the check that got called a genuine fix is the one that reports the page clean. My geometry check does not compare the span bbox against the clipping path — it never sees one.

That is worth more to me than the three that held. The three that held told me where to add code. The one that inverted told me that my new check has the same shape of blind spot as the old one: it trusts a number the document hands it, and that number has already been through the machinery it is supposed to be auditing.

The thread running through all of my submissions this month is the same, and I keep walking into it: every check passed and the answer was still wrong. A ranking only I could see. A collector reporting success while holding 27% of the data. A rate limiter counting the retries. A guard that had stopped guarding. Two fixes where one was theatre. And now a verifier that read every character back correctly while three of the links in the document it had just approved pointed at nothing.

What breaks the pattern is never a stricter version of the same check. It is a second fact, derived a different way, that is allowed to disagree with the first. Here that was pixels versus characters — and, once I had run out of ways to doubt myself, a second reader willing to hand me four things to go break.

I wrote two fixes for the same race condition. Gemini told me only one of them was real.

Ai-Q Labs — Fri, 21 Aug 2026 05:56:51 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

This morning my conversation log was saved under a different project's name.

I run several AI coding-agent sessions in parallel on one machine — same OS user, sometimes the same repository. One of them has a skill that archives the current conversation into a notes vault. Because non-ASCII arguments get mangled on this shell, the metadata for that archive (title, destination project) is not passed on the command line. It is written to a file:

~/.claude/temp/obsidian-meta.json

and a separate script reads that file back a moment later.

Read that path again. There is no session id in it. There is one such file per machine, and I was running more than one session per machine.

Here is what the two sessions did, interleaved:

session A   write meta.json  {"title": "paid-writing pitches", "project": "self-catering"}
session B   write meta.json  {"title": "MC mail triage",       "project": "Rtoner-Google"}
session A   run archiver  ->  reads meta.json  ->  gets B's title, B's folder

Session A's conversation was written to disk with session B's title, in session B's project directory.

Nothing failed. The file existed. The JSON parsed. json.load() returned a dict with exactly the keys the archiver expected. Every check passed, and the answer was somebody else's.

I only caught it because the archiver stamps frontmatter into the file it writes — the real session_id, the real working directory — and those contradicted the filename it had just chosen. The body of the file said one thing and the name of the file said another. Without that redundancy I would still not know.

Then I remembered that four days earlier, the same thing had happened somewhere else entirely.

Bug Fix or Performance Improvement

The second surface: git's index

Four days before, I had run git add -- <my-paths>/ and then paused to ask the user whether to commit. While I waited, a different agent session in the same repository ran git commit.

My eleven staged files went into that session's commit, under a message about an unrelated subproject.

Nothing was lost. But my changes are now buried in a commit whose title does not describe them, which is its own kind of data loss — the kind you discover six months later while running git log and finding nothing.

What is worth noticing is which safeguard did not help. I had a rule, written after an earlier incident: always name the paths explicitly when you stage; never git add .. I followed it. I even machine-checked that nothing outside my scope was staged.

That rule protects the contents of my commit. It says nothing about another session's commit consuming my staged state. I had written a rule about the wrong half of the problem.

Making the window measurable

Two anecdotes are not a bug report, so I built the smallest thing that would show the window is real: two threads, each playing one agent session, each doing write-then-read-back 200 times. One run with a shared filename, one with a per-session filename. Count how often a session reads back data that isn't its own.

mode=fixed     handoffs=400  wrong-owner=219  rate=54.8%
  sess-A: 114/200 reads returned another session's data
  sess-B: 105/200 reads returned another session's data

mode=isolated  handoffs=400  wrong-owner=0    rate=0.0%
  sess-A: 0/200 reads returned another session's data
  sess-B: 0/200 reads returned another session's data

The shared-name number moves between runs, because thread scheduling decides it. Over 7 runs it landed between 54.8% and 62.3% (219–249 of 400). The per-session number did not move: 0 of 400, in all 6 runs.

Neither of those is the real-world rate, and I want to be explicit about it. Two threads in a tight loop maximise the overlap on purpose. In production this has bitten me exactly once, over weeks of parallel sessions. The percentage does not tell you how often it happens — it tells you the window is real, and that closing it takes the count to a hard zero rather than merely lowering it. The gap between "a number that wanders around 60" and "a number that is 0 every time" is the entire finding; the number itself is not evidence of anything.

The interesting column is the failure mode, not the rate. Look at what a wrong-owner read looks like from inside the process:

got = json.loads(path.read_text(encoding="utf-8"))
# no exception. valid JSON. all expected keys present.
if got["session"] != session_id:   # <- the only thing that would ever notice
    wrong += 1

Nothing in the normal path checks that line. A crash would have been a gift.

Two fixes, and the one I got wrong

I wrote both fixes before asking anyone:

Handoff file — use a per-session name, obsidian-meta-<session_id>.json.
Git — never leave anything staged across a pause. Emit git add -- <paths> && git commit -F - as a single command, and get approval before staging rather than between.

Both looked equally good to me. I gave the whole thing to Gemini — both incidents, the reproduction numbers, and both fixes — and asked five questions, one of which was "is the per-session filename a genuine fix or an accident-avoidance patch?"

I expected a yes on both. I got a split verdict:

Failure 1 (Per-session filename): Genuine Fix.
By scoping the path to <session_id>, you eliminate the shared state altogether.

Failure 2 (Atomicity via atomic chaining): A Patch, Not a Fix.
Getting approval first and running git add ... && git commit shrinks the window of vulnerability, but it does not fix the underlying lack of isolation. If Session B executes its git commit in the precise millisecond between Session A's git add and Session A's git commit, Session B will still sweep up Session A's staged files.

It is right, and I had not seen it. Fix 1 removes the shared resource. Fix 2 keeps the shared resource and runs past it faster. Those are not the same kind of thing, and I had filed them under the same heading because they made the same symptom stop appearing in my testing.

The generalisation is the part I'll actually keep:

The failure of Git's index highlights that reducing time-at-risk is merely a mitigation; physical or logical isolation is the only general solution. When a shared resource cannot be renamed or parameterised at the application level, you must isolate the environment itself. For Git, the true structural fix is to give each agent session its own isolated working tree and index (e.g., using git worktree or running sessions inside separate container/VM environments).

Renaming a temp file is available to me because I own the code that names it. .git/index is at a fixed path by design — I cannot parameterise my way out of it, so the fix has to move up a level, to the environment. Same root cause, two different fixes, and the reason they differ is not the bug but how much of the surface I control.

The annoying part: my agent runtime already supports per-session git worktrees. I had a working isolation primitive sitting in the toolbox and reached for a stopwatch instead.

Code

Full reproduction, dependency-free, stdlib only:

"""Minimal reproduction of the handoff-file race that mislabelled my conversation log."""

import json
import pathlib
import sys
import tempfile
import threading

TRIALS = 200
TMP = pathlib.Path(tempfile.gettempdir()) / "race_repro"
TMP.mkdir(exist_ok=True)


def meta_path(mode: str, session_id: str) -> pathlib.Path:
    """The only difference between the bug and the fix is this function."""
    if mode == "fixed":
        return TMP / "meta.json"                    # one file for everybody
    return TMP / f"meta-{session_id}.json"          # one file per session


def session(mode: str, session_id: str, project: str, results: dict) -> None:
    """One agent session: write my metadata, then read it back to use it."""
    wrong = 0
    for i in range(TRIALS):
        path = meta_path(mode, session_id)

        # step 1 - hand off my own metadata
        payload = {"session": session_id, "project": project, "n": i}
        path.write_text(json.dumps(payload), encoding="utf-8")

        # step 2 - the downstream script reads it back
        try:
            got = json.loads(path.read_text(encoding="utf-8"))
        except (json.JSONDecodeError, FileNotFoundError):
            # a torn read also counts as a loss, but it is the *loud* failure;
            # the dangerous one is the silent branch below
            wrong += 1
            continue

        # the read succeeded and the JSON parsed. Is it mine?
        if got["session"] != session_id:
            wrong += 1

    results[session_id] = wrong


def main() -> int:
    mode = sys.argv[1] if len(sys.argv) > 1 else "fixed"
    if mode not in ("fixed", "isolated"):
        print("usage: race_repro.py [fixed|isolated]")
        return 2

    results: dict[str, int] = {}
    threads = [
        threading.Thread(target=session, args=(mode, "sess-A", "self-catering", results)),
        threading.Thread(target=session, args=(mode, "sess-B", "Rtoner-Google", results)),
    ]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    total = sum(results.values())
    runs = TRIALS * len(threads)
    print(f"mode={mode:<9} handoffs={runs}  wrong-owner={total}  rate={total / runs:.1%}")
    for name in sorted(results):
        print(f"  {name}: {results[name]}/{TRIALS} reads returned another session's data")
    return 0


if __name__ == "__main__":
    sys.exit(main())

$ for i in 1 2 3 4 5; do python race_repro.py fixed | head -1; done
mode=fixed     handoffs=400  wrong-owner=243  rate=60.8%
mode=fixed     handoffs=400  wrong-owner=239  rate=59.8%
mode=fixed     handoffs=400  wrong-owner=232  rate=58.0%
mode=fixed     handoffs=400  wrong-owner=248  rate=62.0%
mode=fixed     handoffs=400  wrong-owner=249  rate=62.3%

$ for i in 1 2 3 4 5; do python race_repro.py isolated | head -1; done
mode=isolated  handoffs=400  wrong-owner=0  rate=0.0%
mode=isolated  handoffs=400  wrong-owner=0  rate=0.0%
mode=isolated  handoffs=400  wrong-owner=0  rate=0.0%
mode=isolated  handoffs=400  wrong-owner=0  rate=0.0%
mode=isolated  handoffs=400  wrong-owner=0  rate=0.0%

The whole diff between "broken" and "correct" is one f-string in meta_path(). That is what makes this class of bug worth writing about: the fix is trivial once you see it, and completely invisible until something contradicts itself in front of you.

My Improvements

Done:

Per-session handoff filenames (obsidian-meta-<session_id>.json). 0 of 400 in every one of the 6 harness runs.
The mislabelled conversation log was located and removed. I could identify it because the frontmatter inside disagreed with the filename — the redundancy that caught the bug in the first place.
Approval for a commit is now requested before staging, not between staging and committing.

Changed because of the review, not yet shipped:

Git isolation via per-session worktrees, instead of trying to be fast between add and commit. My runtime supports this natively; I had simply never turned it on for the parallel case. Shipping it means changing how sessions start, which is not a thing to rush at the end of a session that has already found two bugs.

Not done, and I'd rather say so:

The skill's own instruction file still documents the fixed path. The per-session name works, but the written procedure has not been updated to require it, so the next session that follows the documentation reintroduces the bug. Changing that file is a harness change and my own rules require reading the design docs before touching it — which is correct, and which I have not done yet. A fix that lives only in my head is not a fix.

Left as a guard, deliberately:

The archiver writes the true session_id and working directory into the output. That redundancy is the only reason I ever found out. When a handoff crosses a process boundary, having the payload carry its own identity — and having something downstream compare it — turns a silent wrong answer into a detectable one. Every check in the normal path passed. Only the disagreement between two independently-derived facts caught it.

Best Use of Google AI

I used Gemini (free tier, Flash) once, at one specific point: after I had reproduced the bug and written both fixes, and before I shipped either.

What I handed over was deliberately boring — the two incidents, the interleaving, the harness numbers, both proposed fixes, and one instruction: if my reasoning is wrong somewhere, say so directly. No code, no repo access. Five questions, of which the load-bearing one was whether my fixes were fixes.

Three things came back that I did not have:

A split verdict where I expected a pair. Fix 1 genuine, fix 2 a patch — with the exact interleaving that defeats fix 2 spelled out (commit landing in the millisecond between my add and my commit). I had tested fix 2 by not being able to make it fail, which is not the same as it not failing.
The category the two fixes actually belong to. "Reducing time-at-risk is merely a mitigation; physical or logical isolation is the only general solution." That sentence is why I stopped writing rules about how fast to move and started looking at worktrees — a primitive I already had.
The next five surfaces, concretely. Local TCP ports (EADDRINUSE, or worse, silently reaching the wrong session's dev server), shared build caches, global CLI config and credential files being rewritten mid-run by another session, local SQLite/Postgres and migrations, and lock/socket files in /tmp. Each one is a fixed path that exactly one process was ever expected to own.

It also gave me the vocabulary I had been talking around: TOCTOU, and state interleaving / cross-talk for the multi-tenant framing. The class is older than agents by decades. What is new is who is racing — not threads I wrote, but sessions I started, in tools that were designed when "one developer, one machine, one thing at a time" was too obvious to state.

Two of my earlier submissions this month were about measurements that were quietly wrong — a ranking only I could see, a collector that reported success holding 27% of the data. This one has the same shape and I keep walking into it: every check passed and the answer was still wrong. The thing that broke the pattern here was not a better check. It was a second, independently-derived fact that could disagree with the first — and, at the point where I'd stopped being able to see my own reasoning, a second opinion that was willing to tell me one of my two fixes was theatre.