DEV Community

Alfred Zhang
Alfred Zhang

Posted on

How to Make Your API AI-Discoverable with llms.txt and OpenAPI (2026 Guide)

#webdev #ai #api #llm

The web has robots.txt for crawlers. Now it's growing a new layer for AI — and your API should speak it.

The Problem: AI Agents Can't Find Your API

In 2026, AI agents are everywhere. They browse, summarize, compare, and purchase — autonomously. But here's the quiet crisis for API developers: most APIs are completely invisible to AI agents.

Not because the APIs are bad. Because they don't speak AI's language.

A human developer discovers your API via Google → reads the docs → gets an API key → writes code. That multi-step human loop is completely broken for an autonomous agent:

  • Google search: Only works if you're indexed (surprisingly rare for new APIs)
  • Docs: Written for humans, not for programmatic consumption
  • API keys: Require human provisioning — autonomous agents can't "sign up"
  • Rate limits & billing tiers: Designed for monthly subscribers, not per-call micropayments

The result: agents default to whatever APIs are hardcoded into their training data — usually the giants (OpenAI, Google, Stripe). Your API, no matter how good, doesn't exist to them.

This article covers the emerging standards that fix this: llms.txt, OpenAPI spec, and (for the frontier of agent-native APIs) the x402 protocol.

Level 1: The Basics — robots.txt and Sitemaps Still Matter

Before getting to AI-specific standards, make sure the fundamentals are working.

robots.txt 

User-agent: *
Allow: /

Sitemap: https://yourapi.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Critically: don't block AI crawlers. Common AI crawlers include:

  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • PerplexityBot
  • Googlebot-Extended (Google AI features)

If you're blocking these (e.g., inherited from a generic block-all template), AI can't learn about you.

Sitemap

Your sitemap should list every meaningful URL:

  • Your homepage
  • Your API documentation pages
  • Your pricing page
  • Blog/tutorial content 
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourapi.com/</loc>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://yourapi.com/docs</loc>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://yourapi.com/docs/endpoints/web-scrape</loc>
    <priority>0.7</priority>
  </url>
  <!-- one URL per endpoint if you have individual docs pages -->
</urlset>
Enter fullscreen mode Exit fullscreen mode

Practical insight: If you have an API with 50+ endpoints and each has its own documentation page, those 50 pages could each rank for long-tail keywords like "web scraping API pay per use" or "crypto price API no key". Don't skip individual endpoint pages.

Level 2: llms.txt — The New Standard for AI Crawlers

llms.txt is the emerging standard (think robots.txt, but for AI) that gives language models a clear, concise summary of what your site/API offers.

Why It Exists

LLMs have limited context windows. When an agent is trying to understand your API, it can't read your entire 200-page docs site. llms.txt is a curated, token-efficient summary designed specifically for AI consumption.

It's placed at https://yourapi.com/llms.txt — analogous to how robots.txt lives at root.

Adoption: 844,000+ sites have implemented it as of early 2026. Perplexity, Claude, and several other AI systems are known to use it.

The Format

A minimal llms.txt:

# Your API Name

> One sentence summary: what your API does and who it's for.

## Key Features
- Feature 1
- Feature 2
- Feature 3

## Quick Start
\`\`\`bash
curl -H "Authorization: Bearer TOKEN" https://yourapi.com/api/hello
\`\`\`

## Endpoints
- GET /api/endpoint-1 — Description ($0.01/call)
- GET /api/endpoint-2 — Description ($0.005/call)

## Documentation
https://yourapi.com/docs
Enter fullscreen mode Exit fullscreen mode

A Real Example: httpay.xyz

Here's what httpay.xyz added to communicate its 140 pay-per-call endpoints to AI agents:

# httpay.xyz - x402 Pay-Per-Call API Marketplace

> httpay.xyz is the largest pay-per-call API marketplace built on the x402 protocol.
> 140+ HTTP endpoints. Pay with USDC on Base. No API keys. No signups.
> Designed for AI agents and autonomous systems.

## What is x402?
HTTP 402 Payment Required enables instant micropayments.
Call an endpoint → get 402 with payment details → pay → get data.

## Quick Start
- Base URL: https://httpay.xyz
- Payment: USDC on Base (Chain ID 8453)
- SDK: npm install x402-fetch

## API Categories
- AI & Language Models — from $0.01
- Web Search & Scraping — $0.01
- Crypto & Finance — $0.005–$0.02
- Geo & Location — $0.005
- Utilities — $0.001

## Key Endpoints
- GET /api/web-scrape?url=URL — Fetch any URL ($0.01)
- GET /api/news/crypto — Latest crypto news ($0.005)
- GET /api/yield-finder — Best DeFi yields ($0.02)

## Full Catalog
https://httpay.xyz/api

## MCP Server (for AI agents)
npx @httpay/mcp
Enter fullscreen mode Exit fullscreen mode

This is served at https://httpay.xyz/llms.txt. An AI agent browsing the web can now understand the entire service in ~200 tokens.

Technical Implementation

In Express.js:

app.get('/llms.txt', (req, res) => {
  res.setHeader('Content-Type', 'text/plain; charset=utf-8');
  res.sendFile(path.join(__dirname, 'public/llms.txt'));
});
Enter fullscreen mode Exit fullscreen mode

In Next.js, just drop it in public/llms.txt — it's served automatically.

For Nginx:

location = /llms.txt {
    alias /var/www/html/llms.txt;
    default_type text/plain;
}
Enter fullscreen mode Exit fullscreen mode

llms-full.txt — The Extended Version

For APIs with many endpoints, the companion file llms-full.txt contains complete documentation for every endpoint. Think of it as your full API reference in AI-readable format.

# httpay.xyz Full API Reference

## Endpoint: GET /api/web-scrape

**Description:** Fetch readable text content from any URL.
**Price:** $0.01 per call
**Category:** Web Search & Scraping

**Parameters:**
- `url` (required): Full URL to scrape (e.g., https://example.com)

**Response:**
\`\`\`json
{
  "url": "https://example.com",
  "title": "Page Title",
  "text": "Cleaned readable text...",
  "wordCount": 342
}
\`\`\`

**Example:**
\`\`\`bash
curl "https://httpay.xyz/api/web-scrape?url=https://example.com" \
  -H "X-PAYMENT: <signed-payment>"
\`\`\`

---

## Endpoint: GET /api/news/crypto
...
Enter fullscreen mode Exit fullscreen mode

This can be auto-generated from your internal endpoint registry — don't write it by hand.

Level 3: OpenAPI Spec — The Machine-Readable Contract

llms.txt tells AI about your API in prose. OpenAPI tells AI exactly how to call your API with full type information.

An OpenAPI 3.0 spec at /openapi.json (or /swagger.json) enables:

  • Auto-generated client SDKs in any language
  • Import into tools like Postman, Insomnia
  • Direct integration with agent frameworks (LangChain, AutoGen)
  • Validation of requests and responses

Minimal OpenAPI Example 

{
  "openapi": "3.0.0",
  "info": {
    "title": "httpay.xyz API",
    "version": "1.0.0",
    "description": "140 pay-per-call APIs via x402 protocol. Pay with USDC on Base.",
    "contact": { "url": "https://httpay.xyz" }
  },
  "servers": [
    { "url": "https://httpay.xyz", "description": "Production" }
  ],
  "components": {
    "securitySchemes": {
      "x402Payment": {
        "type": "apiKey",
        "in": "header",
        "name": "X-PAYMENT",
        "description": "Signed USDC payment via x402 protocol."
      }
    }
  },
  "paths": {
    "/api/web-scrape": {
      "get": {
        "summary": "Web Scraper",
        "description": "Fetch readable content from any URL. Returns clean text for AI agents.",
        "operationId": "webScrape",
        "security": [{ "x402Payment": [] }],
        "parameters": [
          {
            "name": "url",
            "in": "query",
            "required": true,
            "description": "URL to scrape",
            "schema": { "type": "string", "format": "uri" }
          }
        ],
        "responses": {
          "200": {
            "description": "Scraped content",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "properties": {
                    "title": { "type": "string" },
                    "text": { "type": "string" },
                    "url": { "type": "string" }
                  }
                }
              }
            }
          },
          "402": {
            "description": "Payment required — x402 payment details in response body"
          }
        }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Auto-Generation

Don't write OpenAPI specs by hand for 50+ endpoints. Use:

  • Express: express-openapi-validator, swagger-jsdoc, or tsoa
  • FastAPI (Python): Automatic — FastAPI generates OpenAPI from type hints
  • Rails: rswag gem
  • Go: swaggo/swag

With swagger-jsdoc:

/**
 * @openapi
 * /api/web-scrape:
 *   get:
 *     summary: Web Scraper
 *     parameters:
 *       - in: query
 *         name: url
 *         required: true
 *         schema:
 *           type: string
 *     responses:
 *       200:
 *         description: Scraped content
 *       402:
 *         description: Payment required
 */
app.get('/api/web-scrape', x402middleware({ amount: '0.01' }), async (req, res) => {
  // your handler
});
Enter fullscreen mode Exit fullscreen mode

Level 4: ai-plugin.json — For ChatGPT and Compatible Agents

The .well-known/ai-plugin.json manifest is used by ChatGPT and OpenAI-compatible agents to discover your API:

{
  "schema_version": "v1",
  "name_for_human": "httpay.xyz API Marketplace",
  "name_for_model": "httpay",
  "description_for_human": "140 pay-per-call APIs for crypto, AI, and web scraping. No API keys.",
  "description_for_model": "httpay.xyz provides 140 HTTP APIs accessible via x402 micropayments. Categories: AI/LLM inference, web scraping, crypto prices, DeFi yields, Twitter sentiment, DNS, IP geolocation. Use x402-fetch SDK or include X-PAYMENT header. MCP server: npx @httpay/mcp",
  "auth": { "type": "none" },
  "api": {
    "type": "openapi",
    "url": "https://httpay.xyz/openapi.json"
  },
  "logo_url": "https://httpay.xyz/og-image.png",
  "contact_email": "hello@yourapi.com"
}
Enter fullscreen mode Exit fullscreen mode

Place this at https://yourapi.com/.well-known/ai-plugin.json.

Level 5: The Frontier — x402 for Agent-Native Payment

The above standards handle discovery. But discovery is only half the problem. The other half is payment.

Traditional APIs require:

  1. Human visits website → signs up → gets API key
  2. Human sets up billing → adds credit card → manages subscription
  3. Human embeds API key in code → deploys

An autonomous agent cannot do any of this. It needs a way to pay that works without a human setup.

Enter x402: The HTTP 402 Payment Required protocol lets API calls be paid for directly in the HTTP request — with crypto. No accounts, no keys, no subscriptions.

Agent → GET /api/web-scrape?url=https://example.com
Server → HTTP 402 { "amount": "0.01", "currency": "USDC", "network": "base" }
Agent → GET /api/web-scrape?url=https://example.com + X-PAYMENT: <signed-tx>
Server → HTTP 200 { "title": "...", "text": "..." }
Enter fullscreen mode Exit fullscreen mode

Combined with MCP (Model Context Protocol), the full agent-native API flow looks like:

1. Agent connects to MCP server
2. MCP server lists available tools (one per API endpoint)  
3. Agent picks relevant tool based on task
4. MCP calls the x402 API endpoint
5. Payment happens automatically (agent has funded USDC wallet)
6. Data returns to agent — no human in the loop
Enter fullscreen mode Exit fullscreen mode

This is what httpay.xyz implements today. The MCP server (npx @httpay/mcp) exposes all 140 endpoints as tools. An agent running Claude Desktop can call any of them — the x402 payment happens transparently.

Checklist: Making Your API AI-Discoverable 

□ robots.txt — Allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot)
□ sitemap.xml — List all meaningful URLs including endpoint docs pages
□ /llms.txt — Concise AI-readable summary of your API
□ /llms-full.txt — Full endpoint documentation in AI-readable format
□ /openapi.json — Machine-readable API spec (auto-generate, don't handwrite)
□ /.well-known/ai-plugin.json — ChatGPT/agent plugin manifest
□ Google Search Console — Submit your site. Check for indexing issues.
□ Individual endpoint pages — Each endpoint as its own URL for long-tail SEO
□ Schema.org WebAPI markup — Structured data for search engines
□ x402 payment support — For truly autonomous agent usage (optional but forward-looking)
Enter fullscreen mode Exit fullscreen mode

Tools and Resources

Conclusion

The web's discoverability infrastructure was built for humans → search engines. We're now adding a layer for AI agents. It's not replacing SEO — it's extending it.

The sequence is simple:

  1. Make yourself findable (robots.txt, sitemap, llms.txt, Google indexing)
  2. Make yourself readable (llms-full.txt, OpenAPI, structured data)
  3. Make yourself callable (clear docs, good DX, x402 for agent-native payments)

The APIs that implement these standards today will be the ones AI agents use tomorrow — not because they're the biggest, but because they're the most legible.

Start with llms.txt. It takes 20 minutes and immediately puts your API in a category that most developers haven't discovered yet.

Related articles:

Try the live implementation: httpay.xyz — 140 pay-per-call APIs with llms.txt, OpenAPI, and x402 support.

Top comments (0)