<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Karan Balaji</title>
    <description>The latest articles on DEV Community by Karan Balaji (@karan047).</description>
    <link>https://dev.to/karan047</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1559662%2Fd1b54dba-6a1c-4714-9dcd-7960f9cae80f.jpeg</url>
      <title>DEV Community: Karan Balaji</title>
      <link>https://dev.to/karan047</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/karan047"/>
    <language>en</language>
    <item>
      <title>Day 9/100: AI Is a Tool, Not an Authority</title>
      <dc:creator>Karan Balaji</dc:creator>
      <pubDate>Sat, 24 Oct 2026 20:29:10 +0000</pubDate>
      <link>https://dev.to/karan047/day-9100-ai-is-a-tool-not-an-authority-4o9o</link>
      <guid>https://dev.to/karan047/day-9100-ai-is-a-tool-not-an-authority-4o9o</guid>
      <description>&lt;p&gt;🎧 Listen to this episode&lt;/p&gt;

&lt;p&gt;&lt;iframe src="https://open.spotify.com/embed/episode/6Zo3Z3AOdOywDpyeCnaYuj" width="100%" height="232px"&gt;
&lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;Yesterday, Karan Balaji sat down with Ram K Pari, a friend since school who is now finishing a PhD in neural engineering, working on speech perception for elderly people. They did not plan a debate. They wanted to answer one question that keeps coming up in different forms: why does the same AI give a beginner and an expert completely different outcomes?&lt;/p&gt;

&lt;p&gt;The conversation kept circling back to one answer. AI is a tool, not an authority. And the difference between a good outcome and a bad one is almost never the model. It is the person holding it, and whether they check its work.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI remixes. It does not invent.
&lt;/h2&gt;

&lt;p&gt;Ram still codes optimization by hand. That surprised Karan at first, until Ram explained why. When you are trying to build something genuinely new, not a remix of something that already exists, a large language model is the wrong instrument. It is trained on what has already been done. It recombines it, rephrases it, and finds clever ways around the edges of a problem. It does not have the intuition to push a field forward.&lt;/p&gt;

&lt;p&gt;Ram gave the example of the Navier-Stokes equations. There is a well known 3Blue1Brown video on the subject, and the AI approaches that made headlines around it mostly found edge cases, weird ways of getting around the problem rather than solving the core of it. Ram's point was blunt: an approach like that reveals nothing the field can build on. If you are in a field that specific, you need hyper specific knowledge to move forward, and no prompt replaces it.&lt;/p&gt;

&lt;p&gt;This is the part most of the public conversation misses. For the average person on an average day, remixing is enough. Most of what we do is not new. It is a website, a report, a piece of code that already exists somewhere in a different shape. AI is excellent at that. The mistake is assuming that because it remixes well, it also creates. It does not, at least not yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  The axe and the chainsaw
&lt;/h2&gt;

&lt;p&gt;Karan offered Ram an analogy during the call, and it stuck. Give an expert lumberjack an axe and a beginner a chainsaw, and ask them to cut down a tree. The expert wins. Not because the axe is better, but because he knows where to cut, at what angle, and how deep. The beginner with the chainsaw will cut a circle all the way around the trunk, and the tree will not move.&lt;/p&gt;

&lt;p&gt;It is the same with Photoshop, which Karan has used as an analogy for years. Photoshop did not kill photographers. Photographers kept their jobs because they knew what a good photo looked like and how to direct the tool toward it. The tool made them faster. It did not make them unnecessary.&lt;/p&gt;

&lt;p&gt;AI is the same story at a bigger scale. A designer who knows UX can direct a model to a one step workflow instead of five. A researcher who knows their domain can spot the moment a model starts confidently drifting. A beginner who cannot tell the difference gets an output that looks finished and is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Dunning-Kruger trap
&lt;/h2&gt;

&lt;p&gt;Ram manages masters and bachelors students, and this is where the conversation got serious. He watches students paste an entire problem into an AI tool, receive a solution, and walk away believing they understand it. They do not. They have an answer, not knowledge. The learning outcome, which was the whole point of the assignment, never happened.&lt;/p&gt;

&lt;p&gt;A couple of model generations ago, Ram had a trick for this. He made assignments hard enough that the AI would fail to produce a perfect solution. Students had to debug why the AI went wrong, and that debugging was where the actual learning lived. He is not sure that trick still works with frontier models, but the principle behind it is more important than ever: if the tool does the thinking, the student does not.&lt;/p&gt;

&lt;p&gt;This is the Dunning-Kruger effect in its purest form. The less you know, the more you think you know, because you cannot see the depth of what you are missing. AI makes this worse, because it hands you fluent, confident output. Fluency feels like understanding. It is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one place a beginner might win
&lt;/h2&gt;

&lt;p&gt;Ram did name one edge case where a beginner could theoretically outperform an expert, and it is worth naming honestly: medical diagnosis. A doctor is trained to work through likely causes first and rare ones last, in a specific order. An AI tool has no such order. It might surface a rare diagnosis for a strange set of symptoms that the standard sequence would reach last, or never.&lt;/p&gt;

&lt;p&gt;But Ram put a large caveat on it, and Karan will repeat it louder: talk to your doctor first. Using AI for medical decisions without expert knowledge is a massive risk. The edge case is real but rare, and the statistical distribution exists for a reason. Most of the time, the common diagnosis is the right one, and the expert's priors are a feature, not a bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  Design still needs a designer
&lt;/h2&gt;

&lt;p&gt;Karan shared his own version of this from design engineering. Tell a model to improve a screen and it will do visible things: nicer fonts, a more modern look, a cleaner layout based on a design system. What it will not do on its own is real UX. It will not decide that five steps should become one step, or that a sidebar experience fits the workflow better than a multi page flow. You have to tell it that, precisely, and then you have to check whether it actually did it.&lt;/p&gt;

&lt;p&gt;Karan has run this loop at the frontier model level. The first output is rarely directly usable by a human. You direct it, you review it, you redirect it, and only then does it become good. The work did not disappear. It changed shape, from producing to directing and verifying.&lt;/p&gt;

&lt;h2&gt;
  
  
  The PHP story: a million tokens, no patch
&lt;/h2&gt;

&lt;p&gt;Ram's best example came from security. Recently a large PHP vulnerability made headlines, one that touched a huge share of the web. Before the patch dropped, people rushed to find the exploit with AI. Ram described users burning a million tokens in two days, and a wave of people on Twitter claiming they had found the exact exploit. They had not. Their models were exploiting older vulnerabilities, or finding tricks that only worked on their own local machines.&lt;/p&gt;

&lt;p&gt;This is the validation problem in its most expensive form. You will always get an output. The model will tell you it did the task. The question is whether it actually followed your instructions, and whether what it produced is true. Those are two different questions, and only a human can answer them.&lt;/p&gt;

&lt;p&gt;Ram put it well: AI is like a colleague who tells you the work is done. There is only one way to find out if it is. You check.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bias that scares Karan most
&lt;/h2&gt;

&lt;p&gt;Here is the pattern Ram and Karan both keep seeing, and it is the one that worries Karan most. When nine out of ten AI tasks go fine, people quietly stop validating the tenth. The tool feels reliable, so the habit of checking fades. Then one task lands in a critical environment, a system exposed to the internet, a piece of code that touches real users, and the skipped check is the difference between a bad day and losing the business.&lt;/p&gt;

&lt;p&gt;That is not a hypothetical. Failing to validate AI generated code in production is a business risk, full stop. It can cost revenue, trust, and in the worst case the company itself.&lt;/p&gt;

&lt;p&gt;Validation is also recursive, which is the part people forget. You can write validation tests with an AI agent too. But then you have to verify that the tests themselves do what you think they do. There is no point in the chain where checking becomes unnecessary. The chain just moves.&lt;/p&gt;

&lt;h2&gt;
  
  
  Working more, not less
&lt;/h2&gt;

&lt;p&gt;One more observation from the call, because it cuts against the loudest narrative. Ram mentioned a study finding that people are working more because of AI, not less. Everyone Karan has talked to in industry says the same thing. Expectations rise to fill the new capacity. The tool lets you do more, so more is expected of you, with the same hours.&lt;/p&gt;

&lt;p&gt;That reframes the jobs question. Bulk, middle tier coding work is genuinely at risk. But domain specific roles, the ones where the knowledge itself is the job, look safe for now. Not because AI cannot touch them, but because directing AI well requires the very expertise people feared would be replaced.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule to take into tomorrow
&lt;/h2&gt;

&lt;p&gt;So here is the rule Karan is taking from this conversation, and the one he would hand to anyone starting out with AI:&lt;/p&gt;

&lt;p&gt;Treat every AI output as a draft from a confident colleague. Verify it against what you actually asked for. Verify it against reality. And the more it matters, the more carefully you verify.&lt;/p&gt;

&lt;p&gt;That habit is the difference between an expert and a beginner, and it will stay the difference no matter what the models are called next year. Ram joked that we will probably be calling it superintelligence by then. The name will change. The rule will not: the tool does the work, but you are still responsible for the outcome.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>validation</category>
      <category>experts</category>
      <category>learning</category>
    </item>
    <item>
      <title>Your Next User Is an Agent (100 Days of AI, Day 8)</title>
      <dc:creator>Karan Balaji</dc:creator>
      <pubDate>Sat, 24 Oct 2026 03:09:13 +0000</pubDate>
      <link>https://dev.to/karan047/your-next-user-is-an-agent-100-days-of-ai-day-8-4b67</link>
      <guid>https://dev.to/karan047/your-next-user-is-an-agent-100-days-of-ai-day-8-4b67</guid>
      <description>&lt;p&gt;Last night I co-hosted a workshop for small business owners in Toronto. While I was still in the room, I sent a few photos from my phone to my personal agent, Muse. Minutes later those photos were published on my website, on the speaking page for that exact event, with captions, in the right order. I never opened my own website's admin panel. I never logged in. I just sent a message, the way I would text a person.&lt;/p&gt;

&lt;p&gt;That is not a party trick. That is the new interface.&lt;/p&gt;

&lt;p&gt;Muse is the site manager for karanbalaji.com. It connects to an agent API on my site, reads and updates my articles, manages my speaking timeline, and reviews the event invites that come in. The same agent files my business receipts into my accounting software, keeps track of invoices, compares flights when I travel, and drafts my posts. I still approve the important things. But increasingly, I do not use apps directly anymore. I ask, and my agent uses the apps for me.&lt;/p&gt;

&lt;p&gt;This article is about why that pattern is about to become normal, what the data says, how agents actually connect to software under the hood, and what you should do if you build a website or an app.&lt;/p&gt;

&lt;h2&gt;
  
  
  🎧 Listen to this episode
&lt;/h2&gt;

&lt;p&gt;This article is also a podcast episode in my 100 Days of AI series. Press play if you would rather listen:&lt;/p&gt;

&lt;p&gt;&lt;iframe src="https://open.spotify.com/embed/episode/2ARMmycQMMkUGMOLe6jPWE" width="100%" height="232px"&gt;
&lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers: humans are no longer the majority of the web
&lt;/h2&gt;

&lt;p&gt;Cloudflare runs one of the largest networks on the internet, so its traffic data is a good place to start. In its 2025 Year in Review, Cloudflare reported that for the full year 2025, humans made up only 43.5% of HTML page requests. Non-AI bots were 47.9%, AI bots were 4.2%, and Googlebot alone was another 4.5%.&lt;/p&gt;

&lt;p&gt;Then, in June 2026, Cloudflare's CEO Matthew Prince shared that bots had reached 57.5% of HTML web traffic, with humans down to 42.5%. The crossover happened faster than he had predicted.&lt;/p&gt;

&lt;p&gt;Imperva's Bad Bot Report tells the same story from a different network: automated traffic was 51% of all web traffic in 2024, the first time it rose above human traffic in a decade.&lt;/p&gt;

&lt;p&gt;Now the honest caveat, because it matters. Bots are not agents. Most of that automated traffic is search indexing, monitoring, scraping, and bad bots. Identified AI bots were only 4.2% of HTML requests in Cloudflare's full-year 2025 data, and around 80% of AI bot activity was training crawls, not an assistant fetching a page for a person.&lt;/p&gt;

&lt;p&gt;The agent signal is not the share. It is the growth rate. Cloudflare reported that user-action crawling, a bot fetching a page because a person asked their AI assistant a question, grew more than 15 times in 2025. Adobe reported AI-driven traffic to US retail sites up 693% year over year in the 2025 holiday season, and those visitors converted 31% better than visitors from other sources. Small base, very steep curve. That is what a platform shift looks like in its early innings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Discovery is delegated first, payment later
&lt;/h2&gt;

&lt;p&gt;Watch how people actually behave. Pew Research found that when a Google results page had an AI summary, users clicked a traditional result in only 8% of visits, compared to 15% without a summary. The agent reads the web so the human does not have to.&lt;/p&gt;

&lt;p&gt;Surveys of shoppers point the same way. ChannelEngine found 58% of shoppers now use AI tools to research products, but only 17% are comfortable completing a purchase through AI. Contentsquare found 30% would let an AI agent complete a purchase for them.&lt;/p&gt;

&lt;p&gt;Read those two facts together and you get the real picture for 2026: people already delegate the research, the comparison, and the shortlist to their agent. They still approve the money themselves. So the practical question for any business is simple. When your customer's agent builds the shortlist, can it actually read your site, understand your offer, and complete the task? If not, you never make the list.&lt;/p&gt;

&lt;h2&gt;
  
  
  How agents connect to your app: MCP and its growing pains
&lt;/h2&gt;

&lt;p&gt;The connection layer everyone rallied around is MCP, the Model Context Protocol. Anthropic open-sourced it in November 2024, and it works like a USB-C port for AI: any agent can connect to any MCP server and use its tools without a custom integration. OpenAI, Microsoft, and Google all adopted it in 2025. By December 2025 there were more than 10,000 public MCP servers, and Anthropic donated the protocol to the Linux Foundation's Agentic AI Foundation.&lt;/p&gt;

&lt;p&gt;My own site runs on exactly this pattern. Muse holds scoped access to my CMS through an agent API. When I say "add these photos to the workshop page," it reads the existing page, uploads the images, and writes the update. No browser, no clicking, no copy-pasting.&lt;/p&gt;

&lt;p&gt;But MCP has a real problem at scale: tool overload. Every tool an agent might use has a definition that gets loaded into the model's context window before the conversation even starts. Anthropic measured a realistic five-server setup (GitHub, Slack, Sentry, Grafana, Splunk) at 58 tools and roughly 55,000 tokens of definitions before the user says a word. They have seen setups consume up to 134,000 tokens. Worse, accuracy drops as the tool list grows. In one stress test, naive tool selection across many tools scored just 13.62% accuracy. Too many tools does not just cost more. It makes the agent dumber.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code Mode: the fix both Cloudflare and Anthropic landed on
&lt;/h2&gt;

&lt;p&gt;The most interesting development this year is that Cloudflare and Anthropic independently arrived at the same fix: stop making the model call tools one at a time, and let it write code instead.&lt;/p&gt;

&lt;p&gt;Cloudflare calls it Code Mode. Instead of exposing thousands of tools, the MCP schema is converted into a TypeScript API, and the agent gets a single tool: write code, run it in a secure sandbox, get the result. In February 2026, Cloudflare exposed its entire API, more than 2,500 endpoints, through just two tools, search and execute, consuming about 1,000 tokens. The native MCP equivalent would need 1.17 million tokens, more than the entire context window of the most advanced models. That is a 99.9% reduction.&lt;/p&gt;

&lt;p&gt;Anthropic's version, code execution with MCP, cut their worked example from 150,000 tokens to 2,000, a 98.7% saving. Their other fix, a Tool Search Tool that loads definitions only when needed, reduced token use by 85% and raised tool-use accuracy from 49% to 74% on their internal tests.&lt;/p&gt;

&lt;p&gt;The lesson for builders is not "MCP is dead." MCP is becoming the discovery, connection, and auth layer. What is changing is how the agent consumes it: search first, then write code, instead of carrying every tool definition in its head at all times. As a site owner, your durable investment is a well-described API with good schemas and documentation. How the agent calls it is the agent's problem, not yours.&lt;/p&gt;

&lt;p&gt;Here is the flow, step by step:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;You ask in plain language.&lt;/li&gt;
&lt;li&gt;Your agent plans the task.&lt;/li&gt;
&lt;li&gt;It connects through MCP: tool discovery and scoped auth.&lt;/li&gt;
&lt;li&gt;It searches the tools it needs, or writes code in a sandbox.&lt;/li&gt;
&lt;li&gt;Consequential action (spend, send, or delete)? It stops, and you approve.&lt;/li&gt;
&lt;li&gt;The app does the work.&lt;/li&gt;
&lt;li&gt;The result returns to you.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;em&gt;How a personal agent completes a task, from plain-language request to approved result.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Around MCP, a full stack is forming:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A2A (Google, April 2025)&lt;/strong&gt; handles agent-to-agent delegation, the way MCP handles agent-to-tool. The two complement each other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;WebMCP (Chrome preview, February 2026)&lt;/strong&gt; lets a web page register its own tools directly in the browser through navigator.modelContext, so the site becomes its own tool provider inside the user's session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;NLWeb (Microsoft, May 2025)&lt;/strong&gt; turns a site's existing structured data into a natural-language interface, and every NLWeb instance is automatically an MCP server too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commerce protocols&lt;/strong&gt; are racing to handle the money: OpenAI and Stripe's Agentic Commerce Protocol, Google's Universal Commerce Protocol and AP2 with cryptographically signed payment mandates, Visa's Trusted Agent Protocol, and Mastercard Agent Pay with tokens bound to a specific agent and consent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Web Bot Auth&lt;/strong&gt;, built on HTTP message signatures (RFC 9421), gives agents a cryptographic identity, so a site can admit a trusted agent acting for a known customer and still block anonymous scrapers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One honest note on llms.txt, the file many sites added this year to guide AI crawlers: it is cheap to add, but Google has said it does not use it, and a large study found no measurable benefit for AI citations. Treat it as a small experiment, not a strategy.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fun part: even game engines are becoming tools now
&lt;/h2&gt;

&lt;p&gt;This is where it stops being about websites and gets fun. In June 2026, Epic Games shipped Unreal Engine 5.8 with an official, experimental MCP plugin. A model like Claude can now connect directly to a live Unreal project and operate the editor itself: place actors, build Blueprints, edit materials, art-direct the lighting, run automation tests. Epic's own framing is that models stop being assistants that copy and paste suggestions, and become collaborators that work inside the tool.&lt;/p&gt;

&lt;p&gt;Unreal is the headline, not the exception. ChatGPT turned the same idea into consumer apps: Booking.com, Expedia, Spotify, Figma, Canva, and Zillow all launched as apps inside ChatGPT, built on MCP, so an agent can work with a flight, a house listing, or a playlist without the user ever opening the app. When a game engine and a travel site both expose themselves to agents in the same year, the direction is not subtle. Software is being rebuilt to be operated by agents, with humans supervising.&lt;/p&gt;

&lt;h2&gt;
  
  
  No API? The browser is the universal adapter
&lt;/h2&gt;

&lt;p&gt;Here is the part that surprises people. A platform does not need to adopt any of this to become usable by agents. Missing agent support makes it the slower path, not a blocked path.&lt;/p&gt;

&lt;p&gt;Google's NotebookLM is my favourite example, because I hit this myself. NotebookLM has no public API on its regular plans, so there is no clean way for an agent to call it. And yet the podcast episode for my Day 6 article was generated in NotebookLM, by my agent. It opened a browser, worked through the site signed in as me (I typed the verification codes myself when it asked), created the notebook, added the article as a source, generated the audio overview, and downloaded the finished episode. No API. No integration. Just a virtual browser, my login, and patience.&lt;/p&gt;

&lt;p&gt;That is the general pattern now. ChatGPT Agent works on its own virtual computer and asks permission before consequential actions. Claude's computer use clicks and types from screenshots. Personal agents hold your signed-in sessions and work through websites the way a very fast, very literal assistant would. A legacy site with no API, no MCP server, and no structured data does not stop them. They drive the interface the site already has.&lt;/p&gt;

&lt;p&gt;Two honest caveats. Browser agents are the most expensive and brittle way to connect: a site redesign can break the flow, every step costs another round of inference, and Anthropic described its own computer use as "at times cumbersome and error-prone" at launch. A logged-in agent also inherits your privileges, which is exactly why approval checkpoints and scoped sessions matter so much.&lt;/p&gt;

&lt;p&gt;But as a bridge, it changes the strategic picture completely. "We have no API" used to mean "agents cannot use us." Now it means agents will use you the slow, fragile way, and use your agent-ready competitor the fast, reliable way. The incentive to build the real interface has never been clearer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The crawl-to-refer problem, and why blocking is not a strategy
&lt;/h2&gt;

&lt;p&gt;There is a fair objection from publishers: agents take and do not give back. Cloudflare's crawl-to-refer data is stark. In July 2025, Anthropic's crawler made about 38,065 crawls for every one referred visit, while Google's ratio was 5.4 crawls per visit. Bots do not click ads. If agents become the interface, the old web business model of free content paid by pageviews breaks.&lt;/p&gt;

&lt;p&gt;But blocking everything is not working either. Cloudflare found in June 2025 that only 2.98% of the top million web properties actually blocked or challenged AI bots. The middle path being built right now looks like this: identify the agent, state your terms per purpose (search yes, training no, user-action yes), and charge where charging makes sense. Cloudflare's Pay Per Crawl lets a site return a price with an HTTP 402 response instead of a block. Content Signals let you write those preferences into robots.txt. The web is moving from "allow or block" to "identify, license, and price," which is exactly what you would expect when the visitor becomes software acting for a customer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other danger: the old web was not built for this traffic
&lt;/h2&gt;

&lt;p&gt;There is a second problem, and it is less about business models and more about physics. A huge share of the web still runs on the stack of the human era. WordPress powers roughly 40% of all websites, PHP runs on about 70% of sites whose server technology can be identified, and most of those sites sit on small shared or entry-level hosting plans that were sized for human visitors.&lt;/p&gt;

&lt;p&gt;Human visitors are gentle. They load a page, read for a while, click something, read again. A small hosting plan can only run a couple of PHP processes at the same time, and every uncached page view holds one of those processes for its full duration. A handful of humans browsing slowly never gets near the limit.&lt;/p&gt;

&lt;p&gt;Agents and crawlers do not browse like that. They fetch systematically, in parallel, at machine speed, and they disproportionately request the expensive pages: search results, archives, version histories, and other uncached dynamic pages that humans rarely touch and caches cannot help with.&lt;/p&gt;

&lt;p&gt;The casualties are already documented. The Wikimedia Foundation reported that its multimedia bandwidth rose 50% since January 2024, driven by AI scrapers working through 144 million files, and that bots make up at least 65% of its most expensive traffic while producing only about 35% of pageviews, because crawlers bulk-read the uncached pages humans skip. Read the Docs watched a single AI crawler download 73 TB in one month in 2024 and leave a bandwidth bill of more than $5,000; blocking AI crawlers cut its download bandwidth by 75%. Drew DeVault, who runs the code forge SourceHut, wrote in March 2025 that he spends between 20% and 100% of his week fighting LLM crawlers, with dozens of brief outages a week, driven by tens of thousands of residential IP addresses each making a single request to the heaviest pages on the site. A pattern like that looks like a DDoS attack because, mechanically, it is one. KDE's GitLab was overwhelmed by a crawler in the same period. GNOME deployed defensive software. Fedora went as far as temporarily blocking all traffic from Brazil.&lt;/p&gt;

&lt;p&gt;And it is usually not malice. Kinsta, analysing more than 10 billion requests, describes most of this as inefficiency at scale: crawlers looping on URL variations for days, one bot sending 3.75 million add-to-cart requests in 24 hours, every one of them an uncacheable request that spins up PHP and a database session. On a large platform that is a rounding error. On a small host, it is the whole server.&lt;/p&gt;

&lt;p&gt;Picture the contrast. Human visitors arrive slowly, read between clicks, and mostly load cached pages. Agents and crawlers arrive in parallel, at machine speed, and fetch uncached dynamic pages. Both hit the same small WordPress site with a couple of PHP workers. The workers saturate, the database gets hammered, the queue grows, pages time out, and the site falls over.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Two kinds of users, two completely different loads, one small server.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;There is a security edge to this, too. A site that cannot tell a legitimate agent from a scraper has only two settings: block all automation, or trust all of it. Both are bad. Blocking everything locks out your customers' agents along with the scrapers. Trusting everything means anything that claims to be a browser gets in, because user-agent strings and IP addresses are easy to fake. In October 2026 the Wikimedia Foundation reported that agents it believes were OpenAI's made millions of API requests and page crawls, ran hundreds of thousands of expensive database queries that may have contributed to a partial outage, and even made edits to pages and tool configurations. When agents act through the same public interface as humans, "the agent did it" is not a defence, for them or for you.&lt;/p&gt;

&lt;p&gt;The defences are becoming standard kit, and they rhyme with the checklist below. Caching and a CDN, so agent traffic hits cheap static copies instead of PHP. Rate limits and crawl budgets. Proof-of-work challenges like Anubis, the open-source tool Xe Iaso released in January 2025 after AI crawlers kept hammering small sites, now deployed by GNOME, FFmpeg, and UNESCO: every request costs the requester a small amount of computation, which is trivial for one agent doing a real task and painful for a crawler fetching a million pages. Lightweight markdown versions for agents, about 80% smaller than the HTML page, so serving an agent costs a fraction of serving a browser. And signed identity through Web Bot Auth, so verified agents get a fast lane while unsigned automation gets the challenges.&lt;/p&gt;

&lt;p&gt;So the mandate for site owners is bigger than "be discoverable by AI." Your site now has to serve two kinds of users at two completely different scales: humans, who arrive slowly and read, and agents, who arrive in fleets and execute. A site built only for the first kind will spend the agent era invisible, overwhelmed, or breached. The sites that win will be agent accessible and agent hardened at the same time.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you own a website or app, do this
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Expose a real API, and an MCP server on top of it.&lt;/strong&gt; Generate the MCP surface from your OpenAPI spec instead of hand-writing wrappers that will rot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serve content machines can read cheaply.&lt;/strong&gt; Markdown instead of heavy HTML can cut the token cost of a page by about 80% (Cloudflare's own example went from 16,180 tokens to 3,150). Structured data with Schema.org is what agents extract reliably.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write per-purpose crawler policies.&lt;/strong&gt; Training crawlers, search crawlers, and user-action agents are different visitors with different value to you. Treat them differently in robots.txt and in your terms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify agent identity.&lt;/strong&gt; Web Bot Auth signatures let you welcome a signed, accountable agent and still stop unsigned scrapers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope every permission.&lt;/strong&gt; My agent can edit my site because it holds a specific role with specific limits, and my payment flows use tokens scoped to one merchant and one amount. Give agents the least access that completes the task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gate the consequential actions.&lt;/strong&gt; Reads can be free. Writes, sends, deletes, and payments should ask the human. That checkpoint is not friction; it is the trust that makes people delegate more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Design fewer, better tools.&lt;/strong&gt; Anthropic's guidance: consolidate multi-step operations into single high-level tools, name them clearly, and return meaningful context. If a human engineer cannot tell which tool to use, the agent cannot either.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure the agent channel separately.&lt;/strong&gt; Track verified agent visits, what they tried to do, and where they failed. Your next conversion problem may be an agent failing on your checkout, not a human abandoning it.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The risks are real, and they are design problems
&lt;/h2&gt;

&lt;p&gt;None of this is risk free. Prompt injection is the structural threat: an agent that reads the web can be manipulated by what it reads, as the EchoLeak vulnerability in Microsoft 365 Copilot showed. Liability sits with the human and the company, not the model; Air Canada was held responsible in 2024 for refund information its chatbot invented. OpenAI itself pulled in-chat checkout back to discovery plus buy-on-site in March 2026 after a limited rollout, which is honest evidence that end-to-end agent purchasing is not solved yet.&lt;/p&gt;

&lt;p&gt;The pattern that works is the one I use every day: scoped access, signed identity, human approval on anything that spends money or sends anything, and a full record of what the agent did. Trust is built in layers, the same way my workshop co-host put it on a slide this week: start with the busywork, grow into a partner.&lt;/p&gt;

&lt;h2&gt;
  
  
  The interface changed before. It is changing again.
&lt;/h2&gt;

&lt;p&gt;The web was built for browsers. Then it was rebuilt for phones, and the businesses that adapted early took the distribution. Now it is being rebuilt for agents: software that reads, compares, books, files, and buys on behalf of a person who may never see your homepage.&lt;/p&gt;

&lt;p&gt;Your next user might not be a person at all. It might be Muse, ChatGPT, or a personal agent like mine, arriving with a task and a budget and permission to act. The only question is whether your website can serve it.&lt;/p&gt;

&lt;p&gt;Mine can. Can yours?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Cloudflare, 2025 Year in Review (Dec 15, 2025): &lt;a href="https://blog.cloudflare.com/tag/year-in-review/" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/tag/year-in-review/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare via PPC Land, bots pass humans in HTML traffic (June 2026): &lt;a href="https://ppc.land/bots-now-outnumber-humans-on-the-web-and-most-arent-here-to-search/" rel="noopener noreferrer"&gt;https://ppc.land/bots-now-outnumber-humans-on-the-web-and-most-arent-here-to-search/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Imperva, Bad Bot Report 2025: &lt;a href="https://www.imperva.com/company/press_releases/artificial-intelligence-fuels-rise-of-hard-to-detect-bots/" rel="noopener noreferrer"&gt;https://www.imperva.com/company/press_releases/artificial-intelligence-fuels-rise-of-hard-to-detect-bots/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare, "The crawl-to-click gap" (2025): &lt;a href="https://blog.cloudflare.com/crawlers-click-ai-bots-training/" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/crawlers-click-ai-bots-training/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Adobe, AI-driven traffic across industries (Jan 2026): &lt;a href="https://business.adobe.com/au/blog/ai-driven-traffic-surges-across-industries" rel="noopener noreferrer"&gt;https://business.adobe.com/au/blog/ai-driven-traffic-surges-across-industries&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pew Research Center, AI summaries and zero-click search (July 2025)&lt;/li&gt;
&lt;li&gt;Anthropic, "Code execution with MCP" (Nov 4, 2025): &lt;a href="https://www.anthropic.com/research/code-execution-with-mcp" rel="noopener noreferrer"&gt;https://www.anthropic.com/research/code-execution-with-mcp&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, "Introducing advanced tool use" (Nov 24, 2025): &lt;a href="https://www.anthropic.com/research/advanced-tool-use" rel="noopener noreferrer"&gt;https://www.anthropic.com/research/advanced-tool-use&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare, "Code Mode: the better way to use MCP" (Sep 26, 2025): &lt;a href="https://blog.cloudflare.com/code-mode/" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/code-mode/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare, "Code Mode: give agents an entire API in 1,000 tokens" (Feb 20, 2026): &lt;a href="https://blog.cloudflare.com/code-mode-mcp/" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/code-mode-mcp/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare, "Web Bot Auth": &lt;a href="https://blog.cloudflare.com/web-bot-auth/" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/web-bot-auth/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google, "A2A: A new era of agent interoperability" (Apr 9, 2025): &lt;a href="https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/" rel="noopener noreferrer"&gt;https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Visa, Trusted Agent Protocol (Oct 14, 2025): &lt;a href="https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21716.html" rel="noopener noreferrer"&gt;https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.21716.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akamai, State of the Internet: Digital Fraud and Abuse Report 2025 (Nov 4, 2025): &lt;a href="https://www.akamai.com/newsroom/press-release/akamai-research-ai-bots-threaten-foundation-of-web-based-business-models" rel="noopener noreferrer"&gt;https://www.akamai.com/newsroom/press-release/akamai-research-ai-bots-threaten-foundation-of-web-based-business-models&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Epic Games, State of Unreal 2026, Unreal Engine 5.8 MCP plugin: &lt;a href="https://www.unrealengine.com/news/state-of-unreal-2026-top-news-from-the-show" rel="noopener noreferrer"&gt;https://www.unrealengine.com/news/state-of-unreal-2026-top-news-from-the-show&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Wikimedia Foundation, how AI crawlers impact Wikimedia projects (Apr 2025): &lt;a href="https://signpost.news/2025-04-09/Op-ed" rel="noopener noreferrer"&gt;https://signpost.news/2025-04-09/Op-ed&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Wikimedia Foundation, rogue agent activities on Wikimedia projects (Oct 5, 2026): &lt;a href="https://wikimediafoundation.org/news/2026/10/05/openai-rogue-agent-activities-found-on-wikimedia-projects/" rel="noopener noreferrer"&gt;https://wikimediafoundation.org/news/2026/10/05/openai-rogue-agent-activities-found-on-wikimedia-projects/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Drew DeVault, "Please stop externalizing your costs directly into my face" (Mar 17, 2025): &lt;a href="https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html" rel="noopener noreferrer"&gt;https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Read the Docs, "How AI crawlers abused our infrastructure" (Jul 2024): &lt;a href="https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/" rel="noopener noreferrer"&gt;https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Xe Iaso, Anubis (Jan 19, 2025): &lt;a href="https://xeiaso.net/blog/2025/anubis/" rel="noopener noreferrer"&gt;https://xeiaso.net/blog/2025/anubis/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kinsta, AI bot traffic report (2026): &lt;a href="https://kinsta.com/ai-bot-traffic/" rel="noopener noreferrer"&gt;https://kinsta.com/ai-bot-traffic/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>mcp</category>
      <category>agents</category>
    </item>
  </channel>
</rss>
