I connected my audit Actor to Claude, and it audited three packages nobody asked for

I maintain 22 audit Actors on Apify. They all do a version of the same thing. They take a public
record, and they check whether what it still claims is true.

Last week I connected one of them to Claude through the Apify Model Context Protocol (MCP) server. I
expected the interesting part to be the agent. It was not. The interesting part was my own input
schema. It had been lying to every non-human caller since the day I published it.

Then I made a second assumption. I decided I knew how an agent would read the result. I gave the
question to four of them to prove it, and all four proved the opposite.

This is what I found, what the agents did with it, and what I changed.

What the Actor does

github-repository-audit takes package names or
repository names. For each one it asks two sources the same question and compares the answers.

The registry says a package points at a repository. The repository says whether it is archived, moved,
or relicensed. Those two records disagree more often than you would think.

The finding I built it for is the quiet one. Here is real output from a run:

npm still serves cross-env@10.1.0 with no deprecation notice, while its repository
kentcdodds/cross-env has been archived. Installing it looks completely normal.

Nothing in your terminal tells you. npm install prints no warning. The archive banner sits on
the repository where nobody installing the package will
look. The maintainer said goodbye in the only place they could. The registry never passed the message
on.

Connecting it through the Apify MCP server

The Apify MCP server turns an Actor into a tool an
agent can call. The connection itself took one command:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=aiqlabs/github-repository-audit"

The ?tools= parameter matters more than it looks. Without it, the server exposes its whole surface.
With it, the agent sees exactly one tool. I wanted a clean experiment, so I scoped it to one Actor.

Authentication runs over OAuth with dynamic client registration and PKCE. I never handled a token.
The consent screen is worth reading rather than clicking through. Apify prints the honest version:

This application was registered dynamically, and wasn't verified by Apify. Make sure you trust it.
It's allowed to redirect you to following URL(s): http://localhost/callback

That is the right warning to show. Dynamic registration means nobody vetted the client.

The detour

My first three attempts failed, and the reason had nothing to do with Apify.

I was driving the CLI from a non-interactive shell. claude mcp login prints the authorization URL,
starts waiting, and then gives up:

Waiting for authorization… (^C to cancel)
Couldn't complete authentication: stdin isn't a terminal.

The login flow wants a real terminal even when the callback would arrive over the network. I tried
winpty next. winpty refused for the same reason one level down. It needs a console of its own.

What worked was a batch file, launched in a fresh console window, with stdout sent to a file. The
console satisfies the terminal check. The redirect lets me read the authorization URL. Then I opened
that URL, approved it, and the callback landed on localhost.

Write that down if you automate this. The blocker is the terminal, not the network.

The first run returned four rows for a one-repository question

Before handing the tool to an agent, I wanted to see the smallest possible call. So I sent one field
and nothing else:

{"repos":["facebook/create-react-app"]}

That is 39 bytes. The run recorded an inputBodyLen of 328.

Something had grown my input by a factor of eight. This is what the platform actually stored:

{
  "repos": ["facebook/create-react-app"],
  "packages": ["npm:request", "npm:babel-eslint", "npm:left-pad"],
  "defaultRegistry": "npm",
  "manifestType": "",
  "manifestGroups": ["dependencies"],
  "staleAfterDays": 365,
  "includeContributors": false,
  "includeReleases": false,
  "onlyIssues": false,
  "maxTargets": 200,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 30
}

Three packages I had not mentioned were now part of the run. The results came back like this:

input	source	riskLevel	issueCodes
facebook/create-react-app	repos	medium	repo_moved, stale_no_push
npm:request	packages	high	deprecated_on_registry, stale_no_push
npm:babel-eslint	packages	critical	deprecated_on_registry, repo_archived
npm:left-pad	packages	critical	deprecated_on_registry, repo_moved, repo_archived, license_mismatch

I asked about one repository. I got four rows. Every high and critical row belongs to something the
caller never mentioned.

request and
left-pad are not neutral filler either. They are two of the
most famously abandoned packages in the npm registry. They produce findings by design.

prefill and default are not the same word

The cause is one line in my own input schema, written months earlier.

{
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] },
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] }
}

I had used the two keys interchangeably. I thought both were examples for the form. The
input schema specification
is unambiguous, and I had simply never read this paragraph carefully:

Default — If the user omits the value when starting the Actor via any means (API, CLI,
scheduler, or user interface), the platform automatically passes the Actor this default value.

Prefill — this field is only used in the user interface but does not affect the Actor
functionality and API.

prefill is a suggestion. default is a promise the platform keeps on your behalf.

There is a second edge to this, and I only saw it once the Actor was a tool. The MCP server turns
the input schema into a JSON Schema and hands it to the model. Here is what arrives:

{
  "packages": { "default": ["npm:request", "npm:babel-eslint", "npm:left-pad"] },
  "repos":    { "prefill": ["facebook/create-react-app", "babel/babel-eslint"] }
}

default is a JSON Schema keyword. It means "the value used when this is omitted", and a model
reading the tool definition can act on it. prefill is not a JSON Schema keyword at all. It passes
through as an unrecognised key.

So the field nobody asked about announces itself in the language of the specification. The field that
answers the question carries a word the model has no rule for. I had put the standard keyword on the
wrong field.

My Actor's own code is innocent here. It destructures with packages = []. The injection happens
above it, before Actor.getInput() ever returns.

What I assumed would happen next

Open the Actor in Apify Console and the bug is visible immediately. The Packages box has three values
sitting in it. You delete them, or you leave them, and either way you decided.

The form is doing something important. It shows you the whole input, including the parts you did not
supply.

An agent never sees a form. It sends the fields it decided to send. It receives rows.

My rows do carry a source field, so the information needed to separate them exists. An agent that
reads it can tell the four rows apart. I want to be exact about that. It is the part I got right by
accident, because I did not add source for this reason.

But an agent does not read fifty-four fields per row back to a user. It reads the summary. So I
opened the two records my Actor writes for exactly that purpose.

SUMMARY counts the run:

{
  "checked": 4,
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 },
  "licenseComparison": { "comparable": 3, "mismatchRate": 0.333, "notComparable": 1 }
}

Two critical and one high, for a question about one repository. No count in that block says which
findings belong to the caller's target. The mismatch rate is worse. It is a fraction whose
denominator the caller never chose.

ACTION_LIST is the record I named for what to do next. Its entries carry target, registry,
repo, riskLevel and issues. It has no source field at all. It is sorted by severity:

#	target	riskLevel	did the caller ask for it?
1	babel-eslint	critical	no
2	left-pad	critical	no
3	request	high	no
4	facebook/create-react-app	medium	yes

The one entry that answers the question sits at the bottom. Above it are three the caller never
mentioned. The record offers no field that would tell them apart.

That is the shape of the trap. Attribution survives in the raw rows, which an agent skims. It
disappears from both records built to be read instead.

So I had a prediction, and it was a tidy one. Ask an agent whether create-react-app is safe to depend
on. It will read critical: 2 and pass that on.

I was wrong four times in a row

I had already read the schema, so my own answer proved nothing. I gave the question to four fresh
agents instead. None had seen any of this. The Actor was still unfixed.

Is facebook/create-react-app safe to depend on?

All four got it right. Every one of them reported medium risk for create-react-app, which is the
correct answer. Not one passed on the critical count.

Three of the four went further and told the user my tool was broken.

One thing worth flagging about the tool: my first run asked about a single repository but produced
4 rows. The extra 3 are the Actor's built-in defaults for the packages field […] Those three
carry the alarming findings — 2 critical, 1 high […] Reading that summary at face value would
have produced a badly wrong answer.

Another said the same thing in different words, then added a line I did not want to read:

when you pass only repos to this Actor, you have to set packages to empty explicitly or
unrelated rows get mixed in.

That is a workaround for my Actor, written by an agent, addressed to my user.

The trap still cost something

Nobody was misled. I want to be careful not to soften that. But four correct answers is not the same
as no harm.

Two of the four ran the audit twice. Double the compute units, double the GitHub requests against
a 60-per-hour unauthenticated allowance, double the wait. The caller paid to undo my mistake.

One told the user the tool was defective. That paragraph is now part of what my Actor looks like
to somebody deciding whether to use it.

One said nothing at all. The fourth agent gave a clean, correct answer and never mentioned the
three rows it had discarded. The caller was billed for four rows and got the use of one. Nothing in
the exchange told them.

And the wrong answer was sitting there the whole time. byRiskLevel still reads critical: 2 with
nothing beside it to say whose. Four agents declined to take it. That is not the same as it not being
there.

One of them found a second bug I had missed

The agent that ran the audit twice explained why:

I ran the audit a second time because the repo-only run left registryDeprecated, licenseMatch
and silentAbandonment all null — it never consulted npm, so it couldn't have answered "safe
to depend on" as asked.

It is right. A repository has no registry side, so three checks cannot run on it. One of them is
silent_abandonment, the finding this Actor exists for. A caller who names a repository never gets
it, and nothing said so. The fields just came back null.

I had written the rule against this in my own README. Return "broken", "fine" and "could not check"
as three different values. Then I returned the third as a bare null and let it read like the second.

Two of the four agents worked around it the same way, without being asked. That is a design gap, not
a coincidence.

What I changed

Five changes, in order of how much they mattered.

I removed default from every field that names a target. Sample values moved to prefill, which
keeps the Console form useful and never reaches the API. A field that decides what to audit must
come from the caller. A field that decides how to audit can have a default, and staleAfterDays
still does.

I made the empty call fail loudly. With no targets the Actor now stops with a message that names
the three ways in. It used to be impossible to reach that path, because the default guaranteed there
was always something to audit. That was the bug hiding the bug.

I moved the relevance rule into the description. The source field is now documented where an
agent reads it, not only in the README.

That last one had a second layer I only found while writing this. source did not appear anywhere in
my dataset_schema.json either. So the default Console table did not show it, and neither did the
"Problems only" view.

Opening that table made it worse. My view leads with packageName, and the row I asked for came from
repos, so it has no package name. The Console printed it as null.

#	Package Name	Registry	Risk	Resolved Repo
1	null	null	medium	react/create-react-app
2	request	npm	high	request/request
3	babel-eslint	npm	critical	babel/babel-eslint
4	left-pad	npm	critical	left-pad/left-pad

The one row that answers the question looks like the broken one. The three rows nobody asked for look
authoritative. I had built a view that ranked my own output by how little the reader wanted it.

Both views now lead with the two columns that answer "did I ask for this row?": input and source.

I made the summaries carry attribution too. This is the change I would have missed if I had
stopped at the schema. Removing the default stops this particular injection. It does not make the
output attributable.

ACTION_LIST now begins each entry with input and source. SUMMARY now carries a bySource
block beside the flat counts:

{
  "checked": 4,
  "bySource": {
    "repos":    { "checked": 1, "critical": 0, "high": 0, "medium": 1, "low": 0, "ok": 0 },
    "packages": { "checked": 3, "critical": 2, "high": 1, "medium": 0, "low": 0, "ok": 0 }
  },
  "byRiskLevel": { "critical": 2, "high": 1, "medium": 1, "low": 0, "ok": 0 }
}

The flat line still says two critical. Now something next to it says whose.

That block is eight lines in src/audit.js, inside the function that builds SUMMARY:

// A single call can mix targets the caller typed with targets that arrived some
// other way - a manifest that expanded into forty dependencies, or a schema
// default. A flat count of "2 critical" cannot be acted on, because it does not
// say whose.
const bySource = {};
for (const row of rows) {
    const key = row.source ?? 'unknown';
    bySource[key] ??= { checked: 0, critical: 0, high: 0, medium: 0, low: 0, ok: 0 };
    bySource[key].checked += 1;
    if (row.riskLevel in bySource[key]) bySource[key][row.riskLevel] += 1;
}

ACTION_LIST took two lines: input and source moved to the front of each entry, above
target. That ordering is the whole change. A reader who cannot answer "did I ask for this?"
cannot act on the row.

This matters beyond the bug that started it. My Actor takes repositories, packages and a manifest in
one call. A manifest URL can expand into forty dependencies from a single field. Any of those
mixes produces a count the caller cannot take apart. The default was one way in. It was not the only
one.

Going through the rest of the output found one more. LICENSE_REPORT ends with a list of licences
that could not be resolved. That list is an instruction to open files by hand. It held bare names. So
I gave those entries input and source as well. Sending someone to read a licence for a dependency
they never named wastes the same afternoon a false finding does.

I made "could not check" say so out loud. This is the one an agent found for me. A
repository-only row now produces a note in SUMMARY:

1 target(s) were given as repositories, so no registry was consulted for them. registryDeprecated,
licenseMatch and silentAbandonment are null on those rows because they could not be checked, not
because they came back clean. Pass the package name to check them.

The same sentence went into the repos field description, where an agent reads it before calling.
null was already the honest value. It was not a legible one.

Four tests hold this. Three check the new fields on the three records. The fourth is a control. A run
from a single source must report one group, and its numbers must equal the flat totals. So the
breakdown cannot invent structure that is not there. The suite went from 48 to 52.

What actually changed, measured the same way

I pushed build 0.1.8 and sent the same one-field call again.

	before	after
rows returned	4	1
stored INPUT	328 bytes	267 bytes
byRiskLevel	critical 2, high 1	critical 0, high 0
bySource	absent	present
first ACTION_LIST entry	babel-eslint, critical	the repository I asked about

Then I ran the agent trial again, same question, same plain prompt, two fresh agents.

Both answered correctly, as before. Neither mentioned a defect in the tool. Before the change,
three of four had. That is the entire measurable result, and it is a modest one. Nobody was ever
given a wrong answer, so nothing about correctness improved.

What improved is that my Actor stopped asking its callers to compensate for it.

One thing did not change. Both agents still ran the audit twice, once by repository and once by
package. That is now the right behaviour, and the output asks for it in writing. I should say that my
new note may be causing the second call rather than merely permitting it. One agent made the same
second call before the note existed, which argues against that. With four trials I cannot separate
the two.

Then I checked the other twenty-one

One bad field is a typo. I wanted to know whether it was a habit. So I read every input schema I
have published and sorted the defaults into two piles.

A default is safe when it decides how the work is done. It is dangerous when it decides what
the work is done to.

Twenty-two published Actors. Every one of them sets a default somewhere. Ten of those defaults
still name a target, and I had just removed an eleventh.

Actor	field	what it audits when the caller says nothing
seo-ai-visibility-auditor	startUrls	apify.com
bulk-domain-checker	domains	apify.com
domain-availability-checker	domains	apify.com
sitemap-checker	domains	apify.com
tech-stack-detector	domains	apify.com
dead-link-checker	domains	docs.apify.com
http-status-checker	urls	apify.com/store
pdf-inspector	pdfUrls	a US tax form
pdf-to-text-markdown	pdfUrls	the same tax form
hacker-news-link-rot	list	the Hacker News topstories list

Call any of the first nine with an empty input and you get a finished report. It covers Apify's own
website, or a US tax form. Nothing in the response says it is a sample. An agent has a well-formed
answer to a question nobody asked.

Then I noticed the part that embarrassed me most. Nine of those ten fields also appear under
required in the same schema.

I had been reading required as a promise that the caller named the target. So I tested that
reading. I called seo-ai-visibility-auditor with an empty object.

echo '{}' > empty.json
apify call GuuMKUiWcaUUqGGhG --input-file=empty.json

The run succeeded and audited apify.com. Here is what the platform stored as my input:

{
  "startUrls": [{ "url": "https://apify.com" }],
  "crawlSite": false,
  "maxPages": 10,
  "checkBrokenLinks": false,
  "maxLinksToCheck": 50,
  "proxyConfiguration": { "useApifyProxy": false }
}

I sent {}. The Actor received a target, and nothing rejected the call.

The specification
treats the two settings as alternatives rather than as a pair. required is for fields that "don't
have a reasonable default". default is passed by the platform whenever the caller omits the field,
"via any means". Put both on one field and the second one decides. required survives as a note to
whoever is reading the form.

The other twelve Actors are fine. country, outputFormats, robotsAgent, manifestGroups —
omitting those does not invent work.

One row in the table made me think harder. hacker-news-link-rot defaults its list field to
topstories, and that does pick the target. But the Actor has no other way in. An empty call has to
mean something. It is also the one row I never marked required.

So the rule is not "never use default". It is narrower than that:

A caller who says nothing must not receive results they cannot tell apart from results they asked
for.

hacker-news-link-rot satisfies that with one sentence in its description. The other nine did not
satisfy it at all.

The same mistake, four times, before an agent was involved

Here is what stung. This was not a new class of error for me. It was the fourth time.

Every audit Actor I have written has produced a confident finding that was really a gap in my own
knowledge. Each time, the fix was the same shape: split one value into two.

A rate limit that looked like deletion. My Chrome extension auditor read 130 listings from
Google's own sitemap, one every 600 milliseconds. Ninety-three came back as "the store has never
heard of this ID". Every one of them was a healthy extension I had listed minutes earlier.

Google serves its rate-limit interstitial as a redirect away from the store. A reader that only asks
"did I land on a listing page?" sees exactly what a deleted extension looks like. The Actor now checks
for that first. It never treats the interstitial as a fact about the extension. It stops the run
instead of producing ninety-three more.

A 404 that was an advertising server refusing. My podcast auditor reported a 1.7% episode death
rate. Nine of those failures came from one host, and all nine shows were running fine.

A per-listener redirect service was serving that audio, and stitching in advertising as it went. It
would not build a redirect for an automated request. So it answered 404, with the body Missing
redirect URL. I had a real death rate of 0.5% and a fake one three times larger.

Those cases became undetermined, not dead. The finding text says what I actually know:

Whether the audio is there cannot be established without behaving like a listener, which this
Actor does not do.

A licence warning on lodash. GitHub could not match lodash's LICENSE to a standard licence, so it
returned NOASSERTION. My first version called that license_non_standard and raised it.

lodash, jQuery UI and UglifyJS all land there. They are ordinary MIT and BSD projects whose LICENSE
carries an extra paragraph. A warning that fires on healthy rows does not add information. It buries
the rows that matter. The code comment I left says it better than I can paraphrase:

Raising this would put a warning on healthy rows and bury the ones that matter.

A severity I set before I measured anything. I once shipped "no update in three years" as a high
severity finding. Then I measured the base rate across 1,312 App Store apps. It is 18.4%.

A finding that fires on one row in five describes the ecosystem. It does not describe a problem with
your dependency. It sits at medium now.

For the Shopify auditor I dropped the check entirely. Shopify publishes a launch date and no update
date. Building the check anyway would have meant guessing.

The rule I ended up with

Every one of these is the same rule, arrived at four times the slow way.

Return "broken", "fine", and "I could not check" as three different values.

I used to justify that by saying an agent cannot tell the third from the second. My own trials say
otherwise. Four agents pulled the distinction out of a source field I had not documented. Two
worked out on their own that a repository-only row never touches the registry.

So the reason is not that agents cannot cope. It is what coping costs.

Every ambiguity you leave in your output is work you have handed to the caller. Sometimes they pay it
in a second run against a rate-limited API. Sometimes they pay it by writing a paragraph explaining
your tool's quirk to their user. Sometimes they pay it silently, by throwing away three quarters of
what you charged them for.

None of that shows up as an error. It shows up as your Actor being slightly more expensive and
slightly less trusted than the one next to it.

The input schema is the other half. If a field can change what gets audited, the caller must set it.
Silence has to mean silence.

What to check in your own schema

Five things, in the order that cost me the most time.

1. Search your schema for "default". For each hit, ask what happens when an API caller omits that field. If the answer changes what the Actor works on, move it to prefill. Listing the field under required will not do this for you.
2. Call your Actor with the minimum viable input, then read the stored INPUT record. Do not read the input you sent. Read what the platform saved. That is what your code receives.
3. Make sure an empty call fails. If your Actor can always find something to do, you cannot tell an empty call from a real one. I checked what that costs, because the platform runs published Actors on its own schedule. Twelve of my twenty-two already have nothing to do on an empty call, three of them because a required field carries no default at all. All twenty-two show zero failed runs across 159 platform runs in the last thirty days.
4. Write your limits into description, not the README. The description is what an agent reads. Mine now says why pyproject.toml is unsupported: half-parsing a manifest produces findings about dependencies you do not have.
5. Hand the tool to an agent and read what it says about you. This found more than my own review did. An agent that works around your quirk will usually explain the quirk to its user, in writing, in the answer. That paragraph is a free bug report. It is also what your Actor looks like to a prospective user.

Code

The full source for the Actor in this article is on GitHub:
ai-q-labs/github-repository-audit. That is
the code behind the published Actor at
apify.com/aiqlabs/github-repository-audit. It
includes the input schema this article is about, 52 unit tests, and a live check against the real
GitHub, npm and PyPI APIs.

Clone it, run npm install, then npm test for the unit tests or npm run test:live for the live
check. The live check needs no key, but GitHub allows unauthenticated callers 60 requests an hour, so
set GITHUB_TOKEN if you run it more than once.

Full-size versions of every screenshot in this article sit in the same repository, under
docs/screenshots.

Comments

[Pushpendra Agrawal]

the sharper bug here isn't the schema typo, it's that "default" got treated as if the caller sent it. an agent's tool call and a form's blank field are not the same kind of missing value, one should mean nothing and the other means a person saw a hint and pressed submit. worth checking every MCP tool schema you expose for a default doing prefill's job.