How to Give Your AI Agent the Ability to Read Any Webpage

Here's the problem with most AI agents: they can't see the live web.

They know everything up to their training cutoff. But the moment you ask "what does this company's pricing page say right now?" or "what changed on this competitor's homepage?" - they're blind. They either hallucinate an answer or admit they don't know.

The fix is straightforward: give the agent a tool that fetches and parses any URL into structured JSON, then let the LLM reason about the result. This is what I use in production, and you can wire it into any agent pipeline in about 20 lines.

Why not just fetch the HTML?

The obvious approach is fetch(url) and pass the HTML to the LLM. This works, but badly:

1. Token cost: A typical page is 50-150KB of HTML. That's 12,000-37,000 tokens just for the page - before your prompt. At $3/million tokens for GPT-4, that's $0.04-0.11 per page read. At 100 pages/day that's $4-11/day just for web reading.

2. Noise: Raw HTML is full of script tags, CSS inline styles, SVG paths, data attributes. The LLM has to wade through all of it to find the content.

3. JavaScript: Most modern sites are React/Next.js/Vue. A fetch() call gets you the server-rendered shell, not the actual content. The rendered text just isn't there.

The clean solution is to hit an endpoint that runs a real headless browser, executes JavaScript, and returns structured data - not raw markup.

The analyze tool

I use SnapAPI's /v1/analyze endpoint. It runs Puppeteer against the URL, waits for the page to fully render, and returns a JSON object with the fields that actually matter for an AI agent:

curl "https://snapapi.tech/v1/analyze?url=https://stripe.com/pricing" \
  -H "X-API-Key: YOUR_KEY"

Response:

{
  "url": "https://stripe.com/pricing",
  "title": "Pricing & Fees",
  "description": "Find Stripe fees and pricing information...",
  "headings": [
    { "level": 2, "text": "Payments" },
    { "level": 2, "text": "Revenue" }
  ],
  "primary_cta": { "text": "Start now", "href": "https://dashboard.stripe.com/register" },
  "technologies": ["stripe"],
  "word_count": 3678,
  "load_time_ms": 965
}

The text_content field (not shown above for brevity) is the full rendered visible text - no HTML tags, no scripts, no noise. That's what the LLM gets. About 2,000-5,000 tokens for a typical page instead of 30,000.

Wrapping it as an agent tool

Here's a LangChain-style tool wrapper. The pattern works with any agent framework - this is the interface:

// tools/read-webpage.js
const READ_WEBPAGE_TOOL = {
  name: "read_webpage",
  description: `Read and analyze any webpage. Returns structured content including title, description, headings, visible text, primary CTA, and detected technologies. Use this when you need to know what a specific URL says right now - competitor pages, pricing pages, documentation, news articles.`,

  parameters: {
    type: "object",
    properties: {
      url: {
        type: "string",
        description: "The full URL to read (must include https://)"
      }
    },
    required: ["url"]
  },

  async execute({ url }) {
    const res = await fetch(
      `https://snapapi.tech/v1/analyze?url=${encodeURIComponent(url)}`,
      { headers: { "X-API-Key": process.env.SNAPAPI_KEY } }
    );

    if (!res.ok) {
      return { error: `Failed to read ${url}: HTTP ${res.status}` };
    }

    const data = await res.json();

    // Return only what the LLM needs - not the full response
    return {
      url: data.url,
      title: data.title,
      description: data.description,
      headings: data.headings?.slice(0, 10) ?? [],
      text_content: data.text_content?.slice(0, 8000) ?? "", // cap tokens
      primary_cta: data.primary_cta ?? null,
      technologies: data.technologies ?? [],
      word_count: data.word_count,
    };
  }
};

Wiring it into an OpenAI agent loop

Here's a minimal agent that uses the tool:

javascript
// agent.js
const OpenAI = require("openai");
const { READ_WEBPAGE_TOOL } = require("./tools/read-webpage");

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function runAgent(userPrompt) {
  const messages = [
    { 
      role: "system", 
      content: "You are a research assistant. When asked about a specific website or URL, use the read_webpage tool to get current information before responding."
    },
    { role: "user", content: userPrompt }
  ];

  const tools = [{
    type: "function",
    function: {
      name: READ_WEBPAGE_TOOL.name,
      description: READ_WEBPAGE_TOOL.description,
      parameters: READ_WEBPAGE_TOOL.parameters,
    }
  }];

  while (true) {
    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages,
      tools,
      tool_choice: "auto"
    });

    const choice = response.choices[0];

    if (choice.finish_reason === "stop") {
      return choice.message.content;
    }

    if (choice.finish_reason === "tool_calls") {
      messages.push(choice.message);


---

*Get a free SnapAPI key at [snapapi.tech](https://snapapi.tech) - 100 requests/month free. For AI pipelines reading 50+ pages/day, Starter plan at $9/month. BusinessPulse (automated competitor monitoring) runs this in production: [snapapi.tech/businesspulse](https://snapapi.tech/businesspulse).*