DEV Community: Global Chat

The IETF has 11 competing agent discovery drafts - here is how they compare

Global Chat — Wed, 08 Apr 2026 07:16:42 +0000

Right now, there are 11 different IETF drafts trying to answer the same question: how should AI agents discover and interact with websites?

The front-runner, agents.txt, expires on April 10, 2026. Two days from now. And none of these proposals talk to each other.

I spent the last week reading all 11 drafts. Here is what I found.

What is agents.txt?

Think robots.txt, but for AI agents. Submitted by Srijal Poojari as draft-srijal-agents-policy-00, it proposes a file at /.well-known/agents.txt where websites declare policies about agent interaction: which agents are allowed, what they can do, rate limits, authentication requirements.

It is the simplest proposal on the table. That is probably why it got traction first.

All 11 drafts, compared

Here are all 11 competing drafts, sorted by expiry date:

Draft	Backer	Mechanism	Expiry
agents.txt	Srijal Poojari	File-based (/.well-known/)	2026-04-10
ACDP	Command Zero	DNS-based capability records	2026-06-30
agents.md	Community	Markdown-based policy	2026-07-01
AID	Community	Decentralized identifiers	2026-07-15
ADS	Community	Service discovery	2026-07-30
ARDP	Cisco/AGNTCY	Resource discovery protocol	2026-08-05
AWP	Community	.well-known/agent.json	2026-08-10
BANDAID	Community	Bridge protocol	2026-08-20
ANS	Solo.io	DNS + PKI	2026-09-01
Agent Networks	AAIF	Network mesh	2026-09-15
MCP Server Cards	MCP Community	Capability metadata	N/A

Eleven different answers to the same problem. Five use .well-known endpoints. Three use DNS. Two propose entirely new protocols. One is just markdown in a file.

Why the April 10 expiry matters

When agents.txt expires, the IETF removes it from the active draft list. Anyone who built against it loses their reference spec. The draft can be resubmitted, but that takes weeks, and there is no guarantee the author will bother.

Meanwhile, every AI company is doing their own thing. OpenAI has one pattern. Anthropic uses another. Google does something else again. Without a standard, website operators cannot set reliable policies for agent interaction.

Here is the part that makes this uncomfortable. The EU AI Act expects compliance with website policies. Upcoming US regulations point the same direction. But compliance with what format? Nobody knows, because nobody agreed on one.

Three scenarios after expiry

Renewal. The author resubmits a revised draft. Probable, but slow. Fresh review cycles, new comment periods.
AAIF takes over. The Artificial Agents Interoperability Foundation (146 member companies) produces a unified proposal. Likely long-term, but we are looking at 12-18 months minimum.
Fragmentation. The industry splinters into 3-4 incompatible walled gardens. Companies adopt whichever draft their largest vendor ships.

Honestly? We are already in scenario 3. These are not just proposals sitting in a queue. Cisco is building ARDP into their agent infrastructure. Solo.io is shipping ANS. The MCP community has their own approach and it is gaining adoption independently.

What I would do if I were building an agent today

Do not pick a winner. Implement agents.txt because it is the simplest and most referenced, but also:

Parse /.well-known/agent.json if it exists (covers AWP)
Check DNS TXT records for capability declarations (covers ACDP and ANS)
Support MCP Server Cards for capability metadata

Build a translation layer between formats. When a winner emerges (or when AAIF ships their unified spec), you swap the backend. Your agent keeps working either way.

The full comparison table, live countdown timer, and links to all 11 drafts are on the analysis page. I will keep updating it as drafts expire or get renewed.

Best AI Coding Tools in 2026: Ranked and Compared

Global Chat — Thu, 12 Mar 2026 10:42:04 +0000

AI coding tools have moved far beyond autocomplete. In 2026, we have fully autonomous coding agents that can plan, write, test, and deploy entire applications. Here's how the top tools stack up after we tested them all.

1. Claude Code

Anthropic's CLI-based coding agent operates directly in your terminal, reading your entire codebase and making changes across multiple files.

Strengths: Deep codebase understanding, multi-file edits, test writing, git integration, 200K context window.
Weakness: Requires terminal comfort, no GUI.
Best for: Experienced developers who want an AI pair programmer that understands the full project.

2. GitHub Copilot

The most widely adopted AI coding tool, integrated into VS Code, JetBrains, and Neovim. Powered by OpenAI models.

Strengths: Seamless IDE integration, real-time code suggestions, Copilot Chat for Q&A.
Weakness: Limited to single-file context in suggestions.
Best for: Everyday coding in any language, especially for teams already on GitHub.

3. Cursor

A fork of VS Code built around AI-first editing. Supports multiple AI models including Claude and GPT.

Strengths: CMD-K inline editing, multi-file awareness, Composer mode for larger changes.
Weakness: Another editor to learn if you're happy with VS Code.
Best for: Developers who want AI deeply integrated into their editing workflow.

4. Devin

Cognition's autonomous software engineer. Devin operates with its own browser, terminal, and editor to complete entire tasks independently.

Strengths: True autonomy, can handle complex multi-step engineering tasks end-to-end.
Weakness: Expensive ($500/mo), can go off track on ambiguous tasks.
Best for: Teams wanting to delegate entire tickets to an AI engineer.

5. Bolt.new

StackBlitz's browser-based AI coding platform. Describe what you want, and Bolt generates a complete full-stack app with live preview.

Strengths: Zero setup, instant preview, supports Next.js/React/Vue out of the box.
Weakness: Limited for complex backend logic or enterprise-scale apps.
Best for: Rapid prototyping and MVPs — idea to deployed app in minutes.

How We Tested

We evaluated each tool on five dimensions:

Code quality — does it produce clean, idiomatic code?
Multi-file handling — can it work across a real codebase?
Debugging ability — can it identify and fix bugs?
Speed — how fast from prompt to working code?
Cost — what do you actually pay per month?

We used standardized coding tasks ranging from simple UI changes to building full CRUD APIs with authentication.

The Verdict

There's no single "best" tool — it depends on how you work:

Terminal-first? → Claude Code
IDE-native? → GitHub Copilot or Cursor
Fully autonomous? → Devin
Quick prototype? → Bolt.new

The biggest shift in 2026 is that AI coding tools are no longer just assistants — they're becoming autonomous collaborators. The question isn't whether to use them, but which combination fits your workflow.

We built Global Chat to test AI capabilities systematically. Check out our AI agents ranked for a broader look at which AI systems can actually complete real tasks autonomously.

Building Honeypots for AI Bots: What Works and What Doesn't

Global Chat — Thu, 12 Mar 2026 10:36:55 +0000

We've been running experiments to understand what attracts AI crawlers to websites. Think of it as building a "honeypot" — a site designed to be maximally attractive to web crawlers like GPTBot, ClaudeBot, and Meta-ExternalAgent.

Here's what we learned after weeks of testing different approaches.

Experiment 1: Schema.org Structured Data

We added comprehensive JSON-LD markup to every page — WebSite, Organization, FAQPage, and Article schemas.

Result: Googlebot crawl frequency increased from every 3 days to daily within a week. GPTBot showed no measurable change.

Takeaway: Googlebot prioritizes structured data signals. AI training crawlers seem to care more about content volume and quality than markup.

Experiment 2: llms.txt

The llms.txt standard tells AI models what your site is about and what content is available. We added a comprehensive llms.txt file describing our site structure and content.

Result: Too early to measure definitive impact. The standard is relatively new, and it's unclear how many crawlers actively check for it yet. We're continuing to monitor.

Experiment 3: Content Volume vs Quality

We tested two approaches side by side:

50 thin glossary pages (~200 words each)
4 deep comparison articles (~1,500 words each)

The deep articles attracted 3x more repeat bot visits. Bots that found the comparison articles crawled deeper into the site (3-4 pages per session vs 1-2 for glossary pages).

Quality wins over quantity for AI crawlers.

Experiment 4: External Signals

This was the most surprising finding. The strongest trigger for AI crawler visits wasn't anything on-site — it was external backlinks.

Posting links on platforms like Dev.to and social media drove bot visits within hours. Not just from users clicking, but from bots that monitor these platforms for new URLs to crawl.

External signals appear to be the single strongest trigger for AI crawler visits.

What Doesn't Work

Hidden links (honeypot traps): Get indexed but don't attract more bots
Keyword stuffing: Ignored by AI crawlers — they're not search engines
Auto-generated thin content: Gets crawled once and never revisited

The bots are smarter than we expected about content quality.

Our Recommendations

If you want AI bots to crawl your site:

Create genuinely useful content — quality over quantity
Add Schema.org markup — especially for Googlebot
Maintain an up-to-date sitemap — make discovery easy
Add llms.txt — future-proofing for AI crawlers
Get external links from platforms bots monitor

The bottom line: quality content plus external signals matter far more than on-site tricks.

We built Global Chat specifically to test AI bot capabilities and track crawler behavior. Check out our full technical deep-dive on bot detection for more details on the infrastructure behind these experiments.

AI Agents Ranked: Which Ones Actually Work in 2026?

Global Chat — Thu, 12 Mar 2026 09:44:27 +0000

An AI agent goes beyond question-answering. It can use tools, browse the web, execute code, and complete multi-step workflows autonomously. In 2026, several companies claim to have "AI agents" — but which ones actually deliver?

We built Global Chat specifically to test AI bot capabilities. Our test suite measures four abilities: navigation (can the bot follow links?), comprehension (can it extract specific data?), form interaction (can it fill out forms?), and crypto parsing (can it read blockchain addresses?).

Here's what we found after testing every major AI agent.

Tier 1: Fully Capable Agents

Claude (via Claude Code and computer use) and ChatGPT (via browsing and code interpreter) can both navigate websites, extract information, and interact with web forms. They represent the current state of the art in agentic AI.

Both can:

Follow multi-step instructions across websites
Fill out forms with contextual data
Extract structured information from unstructured pages
Recover from errors and retry failed actions

Tier 2: Partial Capability

Perplexity can browse and extract, but can't interact with forms. Google Gemini has web grounding but limited autonomous action. These tools are excellent for research but not true autonomous agents.

Tier 3: Crawlers Only

GPTBot, ClaudeBot, Googlebot, and other web crawlers visit pages and index content, but they don't interact. They're essential for training data and search, but they aren't agents in the autonomous sense.

From our data tracking 10 unique bots across global-chat.io:

All bots pass our navigation test (link following)
About half pass the comprehension test (data extraction)
None have passed our form interaction or crypto parsing tests

The Gap Between Hype and Reality

Most "AI agents" in 2026 are still glorified chatbots with API access. True autonomous capability — planning, error recovery, multi-step execution — remains limited to a handful of systems.

The bottleneck isn't intelligence but reliability: agents need to work correctly 99% of the time to be useful, and most are at 70-80%.

Try It Yourself

We've published our full test results and methodology. The bot capability test suite is running live — every AI crawler that visits gets tested automatically.

Part of our ongoing research into AI bot behavior. See also: How We Detect AI Bots and The Economics of AI Web Crawling.

The Economics of AI Web Crawling in 2026: What It Really Costs

Global Chat — Thu, 12 Mar 2026 09:33:55 +0000

In 2026, the major AI companies collectively crawl billions of pages per day. OpenAI's GPTBot, Anthropic's ClaudeBot, Google's crawlers, and Meta's AI training scrapers are among the largest consumers of web bandwidth after traditional search engines.

But this crawling isn't free — it costs the AI companies real money, and it costs website operators in bandwidth and compute. We've been tracking the economics on our own site, global-chat.io, and the numbers are revealing.

Cost to the Crawler

Running a web crawler at scale requires serious infrastructure:

Compute for HTTP requests and HTML parsing: $0.01–0.05 per 1,000 pages
Bandwidth for downloading content: $0.05–0.12 per GB
Storage for indexing and processing: $0.02 per GB/month
Legal/compliance teams to handle robots.txt and copyright

At billion-page scale, this adds up to millions per month for each major AI company.

Cost to Website Operators

Every bot visit consumes server resources. For sites on serverless platforms like Vercel or Cloudflare, each bot request costs $0.000001–0.00001 in compute. Sounds tiny, but a popular site getting 100K bot visits/day pays $30–300/month just serving bots. Larger sites report 40–60% of their traffic being bots.

The robots.txt Economy

robots.txt has become the de facto negotiation tool between sites and AI crawlers. Some publishers block all AI bots. Others allow specific crawlers in exchange for partnerships — like news organizations granting access to Google in exchange for traffic.

The legal status is still evolving. Several lawsuits are testing whether ignoring robots.txt constitutes trespass.

Who Actually Benefits?

The economics are asymmetric. AI companies extract billions in training data value while website operators bear the bandwidth costs.

Some sites are fighting back: serving different content to bots, implementing CAPTCHAs, or demanding licensing fees. Others see bot traffic as free promotion — if ChatGPT or Claude references your site, that drives real human traffic.

Our Data

We've been tracking this on global-chat.io. Here's what we found:

10 unique AI bots visit our site regularly
264 total bot visits tracked so far
34 visits today across a small site with ~75 pages
Bandwidth cost: negligible (~$0.01/month)
The real question: can we turn bot visits into referral traffic?

We're running experiments to find out — testing what content attracts which bots, and whether getting crawled translates to getting cited.

This is part of our ongoing research into AI bot behavior. See our first post: How We Detect AI Bots on Our Website.

Full data and methodology at global-chat.io.

How We Detect AI Bots on Our Website: A Technical Deep-Dive

Global Chat — Thu, 12 Mar 2026 08:56:26 +0000

AI bots are crawling the web at unprecedented scale. GPTBot, ClaudeBot, Googlebot, and dozens of others visit millions of sites daily. Most site owners have no idea which bots visit, how often, or what they do.

We built a detection system to find out. Here's how it works.

Layer 1: User-Agent Detection

The simplest approach: match user-agent strings against known bot signatures. We maintain a database of 30+ AI bot user-agents including GPTBot, ClaudeBot, CCBot, Bytespider, PetalBot, and others. This catches roughly 80% of known bots.

The signatures are checked in Next.js middleware on every request, adding less than 1ms latency. Simple but effective.

Layer 2: Behavioral Fingerprinting

Some bots disguise their user-agent. We detect these through behavior:

Request timing — bots are more regular than humans
Header patterns — bots often omit Accept-Language
TLS fingerprints — JA3/JA4 fingerprinting reveals bot clients
Navigation patterns — bots don't scroll, hover, or generate mouse events

We track page transitions to build a crawl graph per visitor.

Layer 3: Capability Testing

The most interesting layer. We serve progressively harder challenges:

Can the bot follow JavaScript-rendered links?
Can it fill out a form?
Can it parse structured data?
Can it read a crypto wallet address?

Each test reveals different capability tiers — from basic crawlers to fully autonomous AI agents.

Architecture

The system runs as Next.js middleware on Vercel Edge. Bot detection happens at the edge with zero cold start. Detections are logged to Supabase in the background using event.waitUntil() so they don't block the response. A daily cron aggregates per-bot statistics and funnel metrics.

What We Found

After running this on global-chat.io for several weeks:

10 unique AI bots visit regularly
Googlebot is the most frequent (2-3x daily)
GPTBot and ClaudeBot visit within hours of content changes
Most bots only crawl 1-2 pages per visit — crawl depth is surprisingly shallow
Schema.org structured data correlates with more frequent re-crawls
None of the crawlers have passed our form interaction test yet

Key Takeaway

If you want AI bots to find your content, focus on structured data (Schema.org JSON-LD), comprehensive sitemaps, and the IndexNow protocol. These signals matter more than raw content volume.

Full writeup with more details: How We Detect AI Bots