DEV Community

XSron Hou
XSron Hou

Posted on • Originally published at scrapio.dev

How to Feed Live Website Data into an LLM Agent

#webscraping #api #ai #showdev

Originally posted on the Scrapio blog — sharing here too.

LLM agents are only as good as the context you give them. Static knowledge cuts off at a training date — but most interesting tasks require knowing what a page says right now: a competitor's pricing, a news article, a product listing.

The bottleneck is usually getting clean text into the prompt. Raw HTML is noisy. Playwright is complex. Here's the simple path.

What you'll need

  • A Scrapio API key (free tier covers this example)
  • An LLM with a large enough context window (GPT-4o, Claude 3.5, Gemini 1.5 work well)
  • Python 3.10+

Fetch a page as markdown

Scrapio returns clean markdown by default. One API call strips navigation, ads, and boilerplate — leaving just the readable content.

import httpx

API_KEY = "sk-..."

def get_page_markdown(url: str) -> str:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"url": url, "output": ["markdown"]},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["outputs"]["markdown"]
Enter fullscreen mode Exit fullscreen mode

Pass it to your agent 

from openai import OpenAI

client = OpenAI()

def answer_about_page(url: str, question: str) -> str:
    page_content = get_page_markdown(url)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are a research assistant. Answer based only on the provided web page content.",
            },
            {
                "role": "user",
                "content": f"Page content:\n\n{page_content}\n\nQuestion: {question}",
            },
        ],
    )
    return response.choices[0].message.content

print(answer_about_page(
    "https://example.com/pricing",
    "What is the cheapest paid plan and what does it include?"
))
Enter fullscreen mode Exit fullscreen mode

Why markdown beats raw HTML

Format Tokens per page (avg) Noise
Raw HTML 8,000–40,000 High (tags, scripts, styles)
Markdown 800–4,000 Low

Markdown is 5–10× smaller, which means lower cost and less context pressure on your model.

Handling JavaScript-heavy pages

If the target page loads content via JavaScript (SPAs, dashboards), set render_js to true:

json={"url": url, "output": ["markdown"], "render_js": True}
Enter fullscreen mode Exit fullscreen mode

Scrapio spins up a headless browser, waits for the page to settle, and returns the rendered content.

Next steps

Top comments (0)