One word in 29 config files: required. Every field it guarded already had a default.
Hi! I'm Tim, a self-taught developer in Nairobi. I build smal...
For further actions, you may consider blocking this person and/or reporting abuse
Your distinction between schema validation and the Actor’s defaulting is the key seam. I’d keep a tiny offline contract fixture for each schema with
{}, the prefilled example, and one invalid value, and run it through the same validation configuration astools/call; then assert both acceptance and the effective input that reaches the Actor. That catches the list/call mismatch without spending a live run.Btw that's a very good idea, I tried it. small offline script with ajv: for each schema it checks that {} and the prefilled example get accepted, that one wrong-typed value gets rejected, and what {} turns into once defaults are filled in. on my 31 public actors everything passed and none have a required field with a default anymore. the useful bit was that last check, 2 of them get {} with nothing filled in at all, so the fallback lives in code and only a local run proves it. do you pin the ajv options to whatever the mcp server uses, or just run default ajv?
Nice, that last check surfaced a different edge than validation: two actors accept
{}but get no effective input at all, so only the local path proves fallback behavior. I’d keep that case in the fixture as its own assertion. Does your check pin the Ajv version and options too?Not at first, it borrowed whatever ajv the apify cli ships. your question made me fix that: it’s pinned to 8.20.0 now, same options (strict off because Apify schemas have their own keywords like editor and prefill), and the version and options sit in a baseline file the check compares against. i also made the “gets nothing from {}” case its own assertion like you said, plus what {} becomes for every actor, so any default that changes fails it. still don’t know what the mcp server itself runs though
Pinning Ajv 8.20.0 and the options gives you a reproducible local baseline. The remaining gap is server parity: until its runtime is inspected, the check cannot establish that
tools/calluses the same version and options. Could the test output label that explicitly as “local baseline verified; server parity unknown”?good call, added it. a pass now says “local baseline verified” and every run ends with “server parity unknown”, since nothing on my side can see what apify’s mcp server actually runs
at this point your comments have basically written this script lol
The root of it is in JSON Schema itself: default is an annotation keyword in draft 2020-12, so a validator doesn't fill it in. tools/list treating it as optional is the generous reading; tools/call is the literal one. Dropping required on any field that has a default is the fix that matches the spec.
Good point, I was calling it a platform bug but the strict side is the one following the spec. got curious and tried it with ajv (8.20): {} against a required field with a default fails with “must have required property”, same wording as the MCP error, so I’m guessing that’s what the server uses. with useDefaults: true the same call passes and the default gets filled in, so one option would make both sides agree. i still dropped required on my end since i can’t change the server and Apify fills defaults when the run starts anyway. which fix would you rather see, defaults applied before validation or tools/list keeping required?
One catch if the server ever turns useDefaults on: Ajv writes the defaults into the data object in place, and it doesn't apply defaults inside anyOf, oneOf or not. So a schema that nests its options in a oneOf would still reject {}.
Really interesting read, Tim!
The part about daily tests passing while AI agents were hitting a completely different path caught my attention.
I've been running into similar questions while building MAGDC, especially around what it actually means for an AI-assisted task to be considered complete.
I also liked your point about zero rows needing an explanation rather than just returning SUCCEEDED.
I'm curious: did fixing these issues change how you plan to test your Actors going forward, especially when they're called by AI agents instead of people?
Thanks! yeah it did. {} is a real test case now, not an edge case. before i change any input schema i run a small offline check with a pinned ajv: every actor has to accept {} and its own example, reject a wrong type, and what {} turns into after defaults can’t change without me noticing. the 2 actors where {} becomes nothing at all get a local run, because only the code proves that fallback. and the ones i reworked this week say why when they return zero rows, or mark rows that came from the example instead of the caller’s input. what i still can’t do cheaply is run everything through the mcp server, my own runs count as users in my analytics, so most of it stays local. what counts as done in magdc, the task finishing or the output being checked?
I really like that you've made
{}a real test case rather than treating it as an edge case. And checking actual runtime behavior separately makes sense — schema validation alone can't prove what the code will do.In MAGDC, finishing the task and verifying the result are two different things.
What we've implemented: We have an Evidence Gate. An agent reporting "DONE" isn't enough to establish that a verification claim has passed. The evidence needs to support the actual claim, and missing or failed required evidence means it isn't complete.
We also distinguish technical verification from practical acceptance by the Product Owner when an increment requires it.
For example, an agent might successfully implement a feature and pass its local tests, but that doesn't automatically prove the feature works in the deployed environment or that I've accepted the result.
This is something I've learned from real development problems: good code and a successful task report don't always mean the intended behavior was delivered.
So my answer is: the checked result matters, not just the finished task.
Your
{}example is a great illustration of why that distinction matters.apify kinda forces that split on me lol. it runs every public actor once a day with its example input and flags it after 3 failed days, no matter what my own tests said. that’s how i found out a gov data site had started refusing my requests
the sneakier one was a schema default. my code had a “no query given” fallback that could never fire on apify, because the platform fills defaults in before the run even starts
This is a great example of why schema correctness and runtime behavior need to be tested together. A field having a default doesn't automatically mean a JSON Schema validator will apply it, so the mismatch between tool discovery and tool execution can be easy to miss. I especially like the point about testing empty inputs and making zero-result responses explain what actually happened. Would contract tests for both tools/list and tools/call help catch these inconsistencies earlier?
ran the tools/list half against apify’s mcp server for all my actors. fields, defaults and required lists matched everywhere, but the server drops minimum and maximum from every field, so an agent is told maxResults has no ceiling while the actor’s own schema caps it at 100
so the discovery side and the execution side disagree on 85 limits across 28 actors lol. tools/call is still the hard half since every call is a real run
Fix 3 got me. I've had an agent take an empty payload and write a confident summary over it, and a SUCCEEDED run with zero rows becoming "no leads found" is that same failure. Can a caller branch on the note key, or does a real lead row carry a note too?
yeah you can branch on note. real lead rows never have that key, only the empty-run row does, and when it shows up it’s the only row in the dataset
the one thing i wouldn’t branch on is a null opportunityScore. sites that block automated checks come back as real rows with a null score too, their “Couldn’t audit” reason lives in opportunityReasons instead
tbh a rowType field would be cleaner than checking whether a key exists lol, your question kinda makes the case for it
I would make the example fallback visible in the returned data, not only the log. Otherwise a caller could mistake results for “antitrust” for results from their own missing query.
Could each result include the effective query and whether a default was used?
Good catch, and it’s worse than the log thing. since the field has a default in the schema, Apify fills it in before the actor even starts, so the code can’t tell “nobody sent a query” from “someone searched antitrust”, and that log line never fires on the platform. the per diem and package ones already put the location and package in every row, but the court search doesn’t. going to drop the schema default on that field, keep the fallback in code, and add the query plus an exampleQuery flag to each row. thanks