DEV Community

Cover image for Why your AI agent keeps quoting cookie banners (and how to fix it)
Tom Caulfield for Serpbit

Posted on

Why your AI agent keeps quoting cookie banners (and how to fix it)

I'm part of the team behind Serpbit. This post is about a problem we kept running into, and how we ended up solving it in our own API.

If you've ever wired an AI agent up to the web, you've probably had this moment. You ask it something, it says "according to the source...", and then confidently summarizes a cookie banner. Or a "please enable JavaScript" page. Or the first paragraph of an article that sits behind a paywall.

Search wasn't the problem. The agent found the right links. It just couldn't read them.

Search is the easy half

The usual setup has three steps:

  1. Ask a search API for links.
  2. Download each page.
  3. Strip the menus, ads and banners, and turn what's left into text the model can use.

Step 1 is a solved problem. Steps 2 and 3 are where the weekends go. Some pages are built entirely in JavaScript. Some sites block anything that looks like a script. Some "articles" are 90% navigation. And every one of those steps tends to be a separate service, with its own key, its own bill and its own way of failing.

We build a Google search API, so our users hit this all the time. They'd get great results from us and then spend days on the part after that.

So we moved the reading into the search

Now you can ask for the results and the text of the top pages in the same request. In Python it's one extra argument:

from serpbit import Serpbit

answer = Serpbit().search("how does a burr coffee grinder work", content=3)
print(answer["organic"][0]["content"]["text"])
Enter fullscreen mode Exit fullscreen mode

You get Google's results as usual, and the first three pages come back as clean Markdown: the main text, the title, the site name and the date when the page states one. If you'd rather paste everything straight into a prompt, there's an option that returns the whole thing as one Markdown document.

If your agent lives in Claude, Cursor or another MCP client, you don't need code at all. You add our MCP server as a connector, and the search tool gets the same "also read the top pages" switch. The model can decide for itself when reading is worth it.

Things we learned along the way

Reading is slow, and that's fine. In a quick test today, a search on its own took 1.4 seconds at the median. With three pages read it was 7.4 seconds, because those pages really are fetched and cleaned. That's why reading is opt-in, and why we like letting the model choose: plenty of questions are answered by the results alone.

Some pages will never be readable. Paywalls, sites that block readers, pages that time out. We decided early that a page we couldn't read shouldn't cost anything. It comes back with a short reason (blocked, timeout and so on) instead of text, so your agent knows to move on rather than guess.

Long pages eat context. One long article can fill a model's whole window. So each page's text has a size cap you can set, and the cut lands at the end of a paragraph when there's one nearby.

The source matters. When your agent reads the pages people actually see first on Google, its answers line up with what a person would find. That was the point of building on Google in the first place.

What it costs

One credit for the search and one for each page read. Pages that couldn't be read are free. The first 1,000 credits every month are free, no card needed, which is plenty to see whether it fits your agent.

We'd like to hear from you

We're still figuring out what agents need from the web beyond results and text. Tables? Comments? Just the paragraph that answers the question? If you're building something that reads the web, tell me in the comments where it breaks for you. That's what we'll work on next.

Top comments (0)