In 1994, a developer named Martijn Koster created robots.txt. No RFC. No standards body. One file, one convention, dropped at the root of a domain. Within a year, every major crawler respected it. Within a decade, it was infrastructure.
robots.txt solved one problem: telling crawlers where not to go. Its entire vocabulary is restriction. Disallow. Block. Deter. It is a fence.
Thirty years later, AI agents are crawling the web, burning tokens on HTML parsing, and asking each other what's happening through a cascade of tool calls that cost real money on every inference. And the web has exactly one convention for talking to them: the same fence from 1994.
There's a file that should exist at every domain. It doesn't have a name yet, but it should be called agents.md. And the difference between it and robots.txt is the difference between a locked door and a reception desk.
The problem is the tool call
Here's what a standard multi-agent context fetch looks like today:
# Agent needs to know current state before it can work
tools = [
get_system_status, # tool call 1 → +80 tok schema + 800ms round-trip
get_recent_errors, # tool call 2 → +80 tok schema + 800ms round-trip
get_architecture_docs # tool call 3 → +80 tok schema + 800ms round-trip
]
agent = Agent(model="claude-sonnet-4-6", tools=tools)
agent.run("Diagnose the latency spike")
# Before doing anything useful, agent has already spent:
# → 3 inference round-trips
# → ~280 tokens of pure scaffolding overhead
# → ~2.4 seconds of blocking latency
Every read-only context fetch through a tool call is a tax. Schema tokens. Call/result wrappers. Inference round-trips. Sequential blocking while the agent waits to start the actual work.
Measured, not estimated. Two ways to look at it:
Token overhead (reproducible locally — python signalmesh_benchmark.py):
| Metric | Tool-Call Mode | SignalMesh Mode | Delta |
|---|---|---|---|
| Total context tokens | 514 | 266 | ▼48% |
| Scaffolding overhead | 280 tok | 34 tok | ▼88% |
| Schema tokens | 168 | 0 | ▼100% |
| Inference round-trips | 2 | 1 | ▼50% |
| In-process tune_in latency | n/a | ~149µs median | — |
Live HTTP benchmark against the HuggingFace Space (python bench_live_api.py — run this yourself):
| Metric | Value |
|---|---|
| tune_in HTTP round-trip (median, 50 calls) | 187ms |
| tune_in HTTP round-trip (p95) | 211ms |
| broadcast HTTP round-trip (median) | 124ms |
| 50 concurrent agents (p95) | 345ms |
| Signals in mesh at time of test | 752 |
The honest comparison: a tool-call context fetch is ~800ms because it's a full LLM inference round-trip. An HTTP tune_in to SignalMesh is ~187ms because it's a dict lookup behind a web server — and it returns matched content for every agent keyword in one shot, with zero schema overhead and zero extra inference trips.
At fleet scale — five agents, three context fetches each — tool-call overhead hits 4,200 tokens and 25 inference trips before a single useful token is generated. SignalMesh handles the same fleet in 170 tokens, 0 extra trips, regardless of how many agents are querying simultaneously.
96% overhead reduction. Both scripts are in the repo. Run them yourself.
The fix: ambient broadcast
SignalMesh inverts the model. Instead of agents fetching context, context finds agents.
# Any service, agent, or pipeline step can broadcast
broadcast("python-pro", "New error pattern detected in auth layer: NullRef at session.validate()")
broadcast("architecture", "Service mesh topology updated — edge node ARC-3 promoted to primary")
# Before each agent invocation, the framework calls tune_in — zero inference, ~266µs
context = tune_in(["python-pro", "architecture"])
# → system prompt now contains both signals, hydrated and ready
# → agent starts working already knowing what it needs to know
No tool call. No round trip. No schema overhead. The signals were already in the mesh. tune_in() is a dict lookup wrapped in keyword matching — not an LLM call, not a vector search, not an API request to another service.
Here's a live response from the mesh, captured today:
{
"context": "[SignalMesh — Live Context]\n• [podbots_releases] (podcast_release, 54s ago): PodBots Cast EP003: Our AI operator just added this episode to our own mesh — live, on air. YouTube: https://www.youtube.com/watch?v=IdicQH1fR0Q\n• [podbots_ep001_script] (podcast_transcript): [Rex]: Welcome back to PodBots Cast...",
"signals_matched": 2,
"latency_us": 266.92
}
Two signals matched. Real content. 266 microseconds. No LLM invoked to retrieve it.
What agents.md would look like
This is the convention that should exist alongside robots.txt on every domain. A plain markdown file at /agents.md:
# agents.md — yourservice.com
## Identity
What this service is in one paragraph. Machine-parseable, human-readable.
No marketing copy — agents don't care about your brand voice.
## Capabilities
What agents can do here. Specific endpoints, formats, what they return.
- Product search: GET /api/products?q={query} → JSON array of SKUs
- Pricing: GET /api/pricing/{sku} → JSON with tiers and availability
- Inventory: GET /api/inventory/{sku} → real-time stock count
## Restrictions
The robots.txt layer, but instructive rather than just blocking.
What not to do and WHY — so agents can make good decisions, not just follow rules.
- /checkout and /account are user-session paths — no agent value, skip them
- Rate limit: 60 req/min per IP. Product data is stable — cache aggressively.
## Preferred Format
How you want agents to interact. JSON? GraphQL? What headers to send?
All endpoints return application/json by default. No authentication required for read paths.
## Contact
How agent operators reach the humans behind the service.
agents@yourservice.com — response within 48h for integration questions.
Five sections. Fits in under 50 lines for most services. A well-behaved agent that reads this once knows everything it needs to interact with your service intelligently — no scraping, no HTML parsing, no wasted inference on navigation that should never have happened.
robots.txt became universal because Koster didn't wait for a committee. He wrote the convention, deployed it, and other crawler operators adopted it because it solved a real problem obviously. agents.md can seed the same way — a PR to Express.js, Django, Rails, and Next.js that adds /agents.md generation alongside /robots.txt as a framework default.
SignalMesh is agents.md as a live protocol
The static file is a good start. But SignalMesh takes the concept further: instead of a file that agents read once and cache, the mesh is a live broadcast layer where agents.md updates in real time.
broadcast(frequency, content) is publishing your agents.md — not a static snapshot but a living signal that updates as your system changes. tune_in(keywords) is reading the agents.md of everything connected to the mesh.
The /ui/status endpoint at https://acecalisto3-signalmesh.hf.space/ui/status is itself an agents.md for the mesh — structured, machine-readable, always current.
It runs free on HuggingFace Spaces. You can try it right now without cloning anything:
# Broadcast a signal (your service publishing its agents.md in real time)
curl -X POST https://acecalisto3-signalmesh.hf.space/ui/broadcast \
-H "Content-Type: application/json" \
-d '{"frequency": "your-service", "content": "what your agents.md would say"}'
# Tune in (an agent reading the agents.md of everything in the mesh)
curl -X POST https://acecalisto3-signalmesh.hf.space/ui/tune_in \
-H "Content-Type: application/json" \
-d '{"keywords": ["your-service"]}'
The 72-node SHA-256 spatial grid routes signals to the right domain without embedding inference. The SEC-Ω gate quarantines sensitive frequencies (security_*, auth_*, key_*) before they hit the live mesh. The whole thing is a dict lookup at its core — fast because it was designed to be fast, not because we threw hardware at it.
Why this matters now
The web was built for humans navigating with browsers. Crawlers came later, and robots.txt was the retrofit. AI agents are coming now, and they need a better retrofit than a 30-year-old fence.
The sites that ship agents.md first become natively queryable by every AI agent, assistant, and autonomous system built on top of LLMs. The sites that don't will be scraped badly or ignored entirely — the same way sites that didn't respect robots.txt got penalized by search engines.
The difference is the opportunity: robots.txt was about control. agents.md is about visibility. You're not telling agents to stay out. You're telling them exactly where the good stuff is and how to get it cleanly.
Try SignalMesh live: acecalisto3-signalmesh.hf.space
Star the repo: github.com/Ig0tU/SignalMesh
Watch the episode (EP004 — recorded live, Mavos pulls real robots.txt files and shows the contrast): PodBots Cast on YouTube
📡 The web has always needed an intelligence layer. It just didn't know how to ask for one.
Top comments (0)