DEV Community

Foxy_Grandpa
Foxy_Grandpa

Posted on

Your agent's output is invalid and you'll find out the expensive way

#ai #mcp #api #webdev

I build agent pipelines on the side. The failure mode that kept costing me actual money was never the dramatic one — it was always downstream and quiet: the agent generates JSON, an API response, or a SQL query; something subtly invalid slips through (a missing required field, a string where a number should be, a SELEKT typo three joins deep); and I pay for it later. A broken pipeline step, a corrupted record, or my personal favorite — an LLM retry loop that burns tokens re-generating the whole thing because one field was wrong.
Why "just ask the model to check" doesn't work
The obvious fix is self-review: feed the output back and ask the model to verify it. Two problems.
First, it's unreliable in exactly the wrong way. The model that made the mistake is drawing from the same distribution when it reviews — it misses its own errors at a rate that's hard to budget for. Validation is the one place where "usually right" is not a property you want.
Second, it's expensive. A full model call to check a JSON object against a schema is like hiring a lawyer to check whether a number is even. Schema conformance is a solved, deterministic problem — it should cost milliseconds and fractions of a cent, and give the same answer every time.
What I built
machinegrade validate is a small service that checks agent-generated artifacts against a contract before you act on them:

json_schema — artifact vs. a JSON Schema, all errors collected
openapi_response — response body vs. the response schema for a given path + method + status in an OpenAPI spec
sql — syntax check per dialect

The design decision I care most about: a verdict is not an error. An invalid artifact returns HTTP 200 with a structured verdict — because for an agent, "invalid" is a normal, expected outcome it needs to act on, not an exception:

{
  "valid": false,
  "errors": [
    {
      "path": "/age",
      "code": "type",
      "message": "must be number",
      "fix_hint": "Change the value at /age to a number (got string)."
    }
  ],
  "latency_ms": 2
}
Enter fullscreen mode Exit fullscreen mode

Every error has a path, a stable code, and a fix_hint written to be consumed by the agent on the retry — so the retry is targeted ("fix /age") instead of "regenerate everything and hope".
Actual HTTP errors (bad key, over limit, unsupported type) are typed JSON too — {code, message, hint, docs_url} — never free-form strings. If you've ever written if "rate limit" in error_message you know why.
Try it in 30 seconds

# Get a key (free tier: 500 calls/month)
curl -s -X POST https://api.machinegrade.dev/keys \
  -H 'content-type: application/json' \
  -d '{"email": "you@example.com"}'

# Validate
curl -s -X POST https://api.machinegrade.dev/v1/validate \
  -H 'content-type: application/json' -H 'X-Api-Key: sk_...' \
  -d '{"type": "json_schema",
       "artifact": {"name": "Ada", "age": "thirty"},
       "contract": {"schema": {"type": "object", "required": ["name", "age"],
         "properties": {"name": {"type": "string"}, "age": {"type": "number"}}}}'
Enter fullscreen mode Exit fullscreen mode

For MCP there are two ways in: a remote endpoint you can point a client at directly — https://api.machinegrade.dev/mcp (streamable HTTP; tools/list works anonymously, tools/call needs the key) — or a stdio adapter on npm (@machinegrade/validate). With Claude Code:

claude mcp add --transport http validate https://api.machinegrade.dev/mcp \
  --header "X-Api-Key: sk_..."
Enter fullscreen mode Exit fullscreen mode

It's in the official MCP registry as io.github.machinegrade/validate, and on Smithery.
The part where I'm honest about what this is
This is an early-stage demand test, and I'd rather say that than pretend otherwise. The API contract is stable (breaking changes only via /v2/, never silently), the free tier stays, and the code is MIT-licensed — if the hosted service ever sunsets, keys work for 90 days and you can self-host the same behavior.
The obvious objection is: "I'll just wire up ajv myself." Legitimate! That's precisely the assumption I'm testing — whether one HTTP/MCP call that covers multiple contract types, with agent-readable errors and zero infrastructure to maintain, is worth paying for versus N libraries you integrate and keep updated yourself. (Fun fact from building this: ajv itself doesn't run on Cloudflare Workers — it compiles schemas via new Function(), which the runtime forbids. The things you learn only by deploying.)
If you build agent pipelines: would you use this? What's missing? Open an issue, or just tell me here why you wouldn't — that's the most useful data of all.

Repo: https://github.com/machinegrade/validate
API docs: https://api.machinegrade.dev/openapi.yaml

Top comments (0)