I Watched an AI Agent Make 30+ Failed Requests Trying to Find My API Docs

We run CryptoDataAPI, a market data API for crypto trading bots and AI agents. We had great documentation — a full /llms.txt file, OpenAPI spec, custom docs page. We thought we were set.

Then we watched a Claude-based AI agent try to use our API for the first time.

It made 30+ HTTP requests before finding a single working endpoint.

The Request Log (All 404s)

Here's what the agent actually tried, in order:

GET /api/v1                    → 404
GET /docs                      → 404
GET /redoc                     → 404
GET /api/v1/docs               → 404
GET /api/v1/openapi.json       → 404
GET /openapi.json              → 404
GET /documentation             → 404
GET /swagger                   → 404
GET /api-docs                  → 404
GET /schema                    → 404
GET /endpoints                 → 404

After exhausting standard paths, it started brute-force guessing:

GET /api/v1/derivatives        → 404
GET /api/v1/binance/ticker     → 404
GET /api/v1/funding-rates      → 404
# ...15 more 404s

It then web-searched "cryptodataapi.com API documentation" — no useful results. Eventually it found working endpoints through pure trial and error.

The irony? Our /llms.txt file — purpose-built for AI agent consumption — had everything the agent needed. It never found it.

Why This Happened

Our docs existed. They were just at non-standard paths:

What the agent tried	Where our docs actually were
/docs	/api/docs (custom path)
/openapi.json	/api/openapi.json
/api/v1	No endpoint (404)
/llms.txt	Existed, but nothing pointed to it

Every attempt was a near-miss. The agent was looking in the right places — we just weren't there.

What AI Agents Try First

Watching the agent revealed a clear priority order:

1. /docs — the FastAPI/Swagger UI default. First thing every agent tries.
2. /openapi.json — the OpenAPI spec. Agents parse this programmatically.
3. /api/v1 — the API root. Developers expect an index here.
4. The homepage — looking for an "API Docs" link.
5. Web search — last resort, usually fruitless for smaller APIs.

If you're not at these paths, AI agents won't find you.

5 Fixes That Took 30 Minutes

We fixed everything in a single commit. Here's what we changed:

1. Enable /docs at the standard path

# Before
app = FastAPI(docs_url=None, ...)

# After
app = FastAPI(docs_url="/docs", ...)

One line. Highest impact change. Every developer and AI agent tries /docs first.

2. Add breadcrumbs to 404 responses

@app.exception_handler(404)
async def not_found_handler(request, exc):
    if request.url.path.startswith("/api"):
        return JSONResponse(status_code=404, content={
            "detail": "Not Found",
            "docs": "https://example.com/llms.txt",
            "endpoints": "https://example.com/api/openapi.json",
        })
    return JSONResponse(status_code=404, content={"detail": "Not Found"})

Now every wrong guess is a signpost instead of a dead end.

3. Add an API root index

@router.get("")
async def api_v1_index():
    return {
        "version": "v1",
        "docs": "https://example.com/llms.txt",
        "openapi": "https://example.com/api/openapi.json",
        "endpoints": {
            "health": "/api/v1/health",
            "daily": "/api/v1/daily",
            # ...all your endpoint groups
        },
    }

First-time consumers now have an entry point.

4. Standard discovery paths

# /.well-known/ai-plugin.json — AI agent discovery manifest
@app.get("/.well-known/ai-plugin.json")
async def ai_plugin_manifest():
    return {
        "schema_version": "v1",
        "name_for_model": "your_api",
        "description_for_model": "What your API does...",
        "api": {"type": "openapi", "url": "https://example.com/api/openapi.json"},
        "llms_txt": "https://example.com/llms.txt",
    }

# /openapi.json → redirect to actual location
@app.get("/openapi.json")
async def openapi_redirect():
    return RedirectResponse(url="/api/openapi.json")

5. Link header on every API response

We added this to our ASGI middleware:

LINK_HEADER = (b"link", b'<https://example.com/llms.txt>; rel="service-desc"')

Every API response now includes a Link header. Agents that inspect headers find documentation from any endpoint — even ones they hit by accident.

The llms.txt Standard

If you haven't heard of it: llms.txt is a simple convention — a plain-text file at /llms.txt that describes your API for LLM consumption. Think robots.txt for AI agents.

Ours includes:

• API description and auth instructions
• Key endpoints with URLs
• Response format examples
• Tips for AI agents ("start with /daily for a complete overview")

But having a great llms.txt isn't enough — you need to make it findable. We now surface it through 5 different paths:

1. 404 response breadcrumbs
2. /.well-known/ai-plugin.json manifest
3. Link response header on all API responses
4. <link rel="service-desc"> in homepage HTML
5. API root index endpoint

The Checklist

Here's what we now use for every API we build:

Check	Why
/docs returns Swagger UI	First thing every agent tries
/openapi.json returns or redirects to spec	Agents parse this programmatically
API root (/api/v1) returns an index	Entry point for exploration
/llms.txt exists with endpoint docs	LLM-optimized documentation
/.well-known/ai-plugin.json exists	Emerging AI agent discovery standard
404 responses include docs links	Turns dead ends into signposts
Link header on API responses	Discovery from any endpoint
<link rel="service-desc"> in homepage	Machine-readable homepage discovery

None of these are hard. The entire set took us 30 minutes and ~100 lines of code.

Why This Matters Now

AI agents are becoming a real source of API traffic. They don't read marketing pages or Google "best APIs" — they probe standard paths and parse structured responses. If your API isn't discoverable at the paths agents check, you're invisible to an entire class of consumers.

The investment is trivial. The payoff is that every AI agent that touches your API finds documentation on first contact.

---

We built CryptoDataAPI for exactly this use case — a unified crypto market data API designed for AI agent consumption. If you're building crypto trading bots or research agents, check out our llms.txt to see what AI-optimized API docs look like.