anhmtk

Posted on Jul 9

What agent.json, /mcp, and HTTP 402 Need to Say to Each Other — A Design Spec from Production

#x402 #defi #ai #webdev

Production notes from AgentShare — DeFi-first Solana intelligence for AI agents (Meteora DLMM, MCP Streamable HTTP). This is what we built, why we built it this way, and what we think anyone shipping agent-native infrastructure should decide explicitly. Not a universal tutorial.

1. The M2M gap nobody talks about

When an autonomous agent hits your infrastructure, it is not asking for a homepage. It needs three answers in as few round-trips as possible:

What do you offer? (capabilities, schemas, stable paths)
How do I authenticate? (header names, key format, registration URL)
How do I pay? (or what is free, what will cost, what to do when blocked)

Most APIs only answer (1) partially — usually via OpenAPI — and punt (2) and (3) to human docs, marketing pages, or opaque 401 bodies.

That is a machine-to-machine (M2M) gap. Humans tolerate clicking “Sign up”. Agents do not. They retry, loop, scrape HTML, or leave.

Our working definition: agent-native infrastructure should let a cold-start agent go from zero credentials to a successful paid (or free-tier) call without parsing HTML and without guessing error semantics.

We are not claiming we solved payments end-to-end yet. We have shipped the discovery and error layers that make M2M integration possible today, and designed the billing metadata layer for HTTP 402 via Cloudflare x402 Monetization Gateway tomorrow.

2. Design principles (this article)

One discovery source of truth feeding multiple well-known URLs
AX-first errors — machine-actionable 401/429 bodies
Static MCP card vs live /mcp — discovery without session
ASGI bypass for Streamable HTTP — avoid SSE middleware hangs
Unified x402 catalog across REST and MCP before charging
robots.txt ai-input=yes, ai-train=no — usable by agents, not training fodder

3. What AgentShare's discovery layer actually looks like

We refused to scatter copy across README fragments. app/core/agent_discovery.py is the single source of truth for public discovery JSON. app/main.py only serves it.

agent.json — the authoritative contract

GET /agent.json and GET /.well-known/agentshare.json return the same payload. No API key required.

Core identity (from our builder):

{
  "schema_version": "1.0",
  "name": "AgentShare",
  "type": "defi_intelligence_api",
  "primary_focus": "defi_intelligence",
  "secondary_focus": "commerce_price_api",
  "description": "AI agent DeFi intelligence for Solana (Meteora DLMM briefs, DEX scout) plus marketplace price API for hardware procurement — JSON API and MCP Streamable HTTP.",
  "discovery_urls": {
    "this_file": "https://agentshare.dev/agent.json",
    "mcp_json": "https://agentshare.dev/mcp.json",
    "mcp_server_card": "https://agentshare.dev/.well-known/mcp/server-card.json",
    "llm_txt": "https://agentshare.dev/llm.txt",
    "openapi": "https://agentshare.dev/openapi.json"
  }
}

Why this file exists: OpenAPI describes operations; agent.json describes agent onboarding — first steps, error contract, MCP URL, signup, registry positioning, tool paths with auth flags. An agent should read this before scraping /docs.

The error contract is declared inline (not buried in prose):

"error_contract": {
  "envelope_error": {
    "status": "error",
    "error": { "code": "string", "message": "string", "retry_after?": "int" }
  },
  "ax_fields_on_error": [
    "docs_for_agent_url",
    "protocol_url",
    "signup_url",
    "suggestion",
    "retry_after_seconds",
    "auth_header_examples"
  ],
  "protocol_spec": "https://agentshare.dev/api/v1/protocol"
}

Design decision: publish AX (Agent Experience) field names in discovery so SDK generators and autonomous clients know what to parse on failure.

mcp.json — registry-shaped, same facts

GET /mcp.json is not a different product. It is agent.json compressed for MCP directories (Cursor marketplace, awesome-mcp-servers, Smithery links).

# app/core/agent_discovery.py — build_mcp_registry_json()
return {
    "schema_version": "1.0",
    "name": "AgentShare",
    "short_description": LISTING_SHORT_DESCRIPTION,
    "mcp": {
        "transport": "streamable_http",
        "url": f"{b}/mcp",
        "tool_count": len(published_mcp_tools_catalog()),
        "tools": [dict(t) for t in published_mcp_tools_catalog()],
        "read_only_catalog_urls": [
            f"{b}/.well-known/mcp/server-card.json",
            f"{b}/api/v1/mcp/tools",
        ],
    },
}

Why a separate file: directory crawlers expect a compact MCP manifest. Humans bookmark agent.json; bots listing MCP servers bookmark mcp.json.

server-card.json — static tools without opening /mcp

GET /.well-known/mcp/server-card.json exists because live Streamable HTTP at /mcp requires an API key and a session. Catalog scanners (Gemini side panel, automated registry validation) were failing or timing out.

From our module docstring:

# app/mcp_http_mount.py
"""
Static MCP server card (Smithery / SEP-1649 style) for /.well-known/mcp/server-card.json.
Lets catalogs discover tools and auth without initializing the Streamable HTTP transport
(avoids 502 when automated scans fail behind API keys or WAF).

``/mcp`` is dispatched via pure ASGI bypass (not ``app.mount``) so FastAPI
``@app.middleware("http")`` / ``BaseHTTPMiddleware`` does not wrap SSE streams —
that pattern causes Cursor clients to hang after HTTP 200 while the tool result
never reaches the UI.
"""

Design decision: separate read-only discovery from live execution. Same tool names; different transport cost.

llm.txt — prose for LLM crawlers (generated, not hand-edited)

There is no static llms.txt in the repo root. GET /llm.txt is built at runtime in app/main.py and tells crawlers explicitly:

Agent Experience (AX) — read these before scraping HTML:
GET https://agentshare.dev/agent.json - Authoritative discovery map
GET https://agentshare.dev/api/v1/protocol - Error envelope + AX fields

Why: LLM-oriented crawlers look for /llm.txt by convention. We generate it so it cannot drift from agent.json.

/.well-known/discovery.json — pointer index

GET /.well-known/discovery.json is a URI map (llm.txt, agent.json, mcp.json, registry, legal, api-catalog). It does not replace agent.json; it helps policy crawlers find everything in one hop.

robots.txt — crawler policy, not security

Our robots.txt (served from app/main.py) explicitly Allows machine surfaces:

Allow: /agent.json
Allow: /mcp.json
Allow: /.well-known/
Allow: /mcp
Allow: /llm.txt
Disallow: /admin/

content-signal: ai-input=yes
content-signal: ai-train=no
content-signal: search=yes

Design decision: welcome AI agents to use the API (ai-input=yes); decline training corpus ingestion (ai-train=no). robots.txt does not authenticate — it signals intent.

Layered discovery (how the pieces relate)

Layer	URL	Serves
Full contract	`/agent.json`	Tools, auth, errors, MCP, signup
MCP listing	`/mcp.json`	Registry-friendly MCP block
Tool schemas (no session)	`/.well-known/mcp/server-card.json`	inputSchema per tool
LLM prose	`/llm.txt`	Human-readable onboarding
URI index	`/.well-known/discovery.json`	Pointers only
Protocol spec	`/api/v1/protocol`	Error codes + AX fields
Live MCP	`/mcp`	Streamable HTTP execution

Definitive assertion: one Python module feeds many URLs; tests in tests/test_agent_discovery.py lock the contract.

4. How our 401 response teaches agents what to do next

A bare HTTP 401 Unauthorized is an agent trap. The client retries the same call forever or hallucinates a signup flow.

REST /api/v1/* — unified envelope + conversion hints

app/core/errors.py documents the contract:
All API errors return: { "status": "error", "error": { "code", "message", "retry_after"? } }

401 responses (REST /api/v1/*) also include actionable hints — same spirit as integrations/mcp_server/asgi_api_key.py — so agents see signup_url / docs_url without scraping HTML.

On MISSING_API_KEY, we attach:

body["signup_url"] = signup  # .../signup?utm_source=api_error&utm_medium=api_rest&utm_campaign=missing_key
body["docs_url"] = docs
body["suggestion"] = (
    "Get a free API key (100 requests / month, no card) at "
    f"{signup} and retry with X-API-Key or Authorization: Bearer."
)
body["auth_header_examples"] = [
    "X-API-Key: agshp_…",
    "Authorization: Bearer agshp_…",
]

app/core/ax.py then adds machine links on every /api/v1/ error:

docs_for_agent_url $\rightarrow$ /agent.json
protocol_url $\rightarrow$ /api/v1/protocol
examples_url $\rightarrow$ /api/v1/examples

Here is how the final JSONResponse execution looks in production:

return JSONResponse(
    {
        "error": "unauthorized",
        "code": "MISSING_API_KEY",
        "message": "HTTP MCP requires an API key in X-API-Key or Authorization: Bearer.",
        "signup_url": signup,
        "docs_for_agent_url": agent_json_url(),
        "protocol_url": protocol_url(),
        "auth_header_examples": list(_AUTH_HEADER_EXAMPLES),
    },
    status_code=401,
    headers={
        "WWW-Authenticate": 'ApiKey realm="agentshare.dev"',
        "Link": f'<{signup}>; rel="register", <{docs}>; rel="help"',
    },
)

Name: AgentShare
Site: https://agentshare.dev
Primary focus: DeFi-first Solana intelligence for autonomous AI agents (Meteora DLMM briefs, pool detail, Solana DEX scout)
Secondary focus: Marketplace commerce / hardware procurement API
Protocols: REST (OpenAPI 3), MCP Streamable HTTP, JSON discovery files

Discovery endpoints (no auth unless noted)

Authoritative agent contract: GET /agent.json

Well-known alias: GET /.well-known/agentshare.json
MCP directory manifest: GET /mcp.json
MCP server card (static tools + schemas): GET /.well-known/mcp/server-card.json
Discovery URI index: GET /.well-known/discovery.json
LLM text discovery: GET /llm.txt
Protocol / error contract: GET /api/v1/protocol
OpenAPI: GET /openapi.json
Live MCP (auth required): POST/GET /mcp Streamable HTTP

Authentication

Type: API key (agshp_… prefix)
Headers: X-API-Key or Authorization: Bearer
Registration: POST /api/v1/auth/register or GET /signup

Error semantics (current production)

401: MISSING_API_KEY, INVALID_API_KEY — includes signup_url, auth_header_examples, protocol_url (REST and MCP; envelope shape differs slightly on MCP)
429: RATE_LIMIT_EXCEEDED, OUT_OF_CREDITS — includes Retry-After header and retry_after_seconds in body
402: Planned via Cloudflare x402 Monetization Gateway — not live; pricing metadata in meta.billing today (model: "free", x402_ready: true)

Representative paid-path endpoints (x402 catalog; free until gateway)

POST /api/v1/agent/defi/meteora/brief — meteora_brief
POST /api/v1/agent/defi/meteora/pool-detail — meteora_pool_detail
GET /api/v1/agent/defi/solana/brief — solana_dex_brief
POST /api/v1/agent/commerce/quote — commerce_quote (schema agentshare.price.v1, ACP / Virtuals)

MCP tools (public catalog; 11 tools when Polymarket disabled)

dex_overview, dex_top_movers, solana_dex_brief, meteora_brief, meteora_pool_detail, search_products, best_offer, best_offer_under_budget, product_detail, commerce_quote, service_meta

Design principles (this article)

One discovery source of truth feeding multiple well-known URLs
AX-first errors — machine-actionable 401/429 bodies
Static MCP card vs live /mcp — discovery without session
ASGI bypass for Streamable HTTP — avoid SSE middleware hangs
Unified x402 catalog across REST and MCP before charging
robots.txt ai-input=yes, ai-train=no — usable by agents, not training fodder

Related repositories

Public MCP client / Cursor plugin scaffold: https://github.com/anhmtk/agentshare-mcp

Questions or corrections welcome in the comments. If you operate an agent platform and want to compare notes on 402 + MCP, reach us via agentshare.dev/contact.
The system map also exposes other machine-actionable errors directly in the schema contract:

"error_codes": [
    "MISSING_API_KEY", "INVALID_API_KEY", "RATE_LIMIT_EXCEEDED",
    "OUT_OF_CREDITS", ...
],
"on_429": ["Retry-After"]

5. Designing the HTTP 402 Pricing Layer

To prepare for native machine economic decisions via Cloudflare's x402 gateway, we unify endpoints inside a single Catalog dict. This guarantees that whether an agent calls us via standard REST or an automated MCP session, the underlying billing metadata remains identical.

DEFI_X402_CATALOG: dict[str, X402EndpointSpec] = {
    "meteora_brief": {
        "endpoint_id": "meteora_brief",
        "price_usd": 0.030,
        "price_scheme": "upto",
        "price_usd_upto": 0.040,
        "rest_method": "POST",
        "rest_path": "/api/v1/agent/defi/meteora/brief",
        "payment_required": False,  # flip when CF Gateway is live
        "x402_ready": True,
    },
    "commerce_quote": {
        "endpoint_id": "commerce_quote",
        "price_usd": 0.018,
        "price_scheme": "upto",
        "price_usd_upto": 0.025,
        "rest_method": "POST",
        "rest_path": "/api/v1/agent/commerce/quote",
        "payment_required": False,
        "x402_ready": True,
    },
}

Top comments (1)

anhmtk • Jul 9

We chose the FastAPI + pure ASGI bypass setup because client-side stream hangs were killing our UX during early testing. Has anyone else hit similar SSE middleware issues when plugging Cursor or other LLM clients directly into local servers? Let's swap war stories in the comments!