How to Give Your AI Agent Live Web Access (Free URL Markdown API)

AnyCrawler — Thu, 05 Nov 2026 10:52:22 +0000

If you are building an AI agent or RAG pipeline and need live web access—a public URL turned into clean Markdown your model can actually use—you do not have to stand up a scraper, headless browser farm, or HTML-cleaning stack first. One free GET request can return readable page text as Markdown, ready to drop into a tool call or retrieval step.

This tutorial solves that single need: convert one public URL into Markdown for agents and RAG, with copy-paste examples in curl, Python, and JavaScript. The endpoint we use is AnyCrawler’s free crawl API (no API key). I work on AnyCrawler; that is disclosed again at the end.

The problem: agents need context, not raw HTML

Agent and RAG systems fail in predictable ways when you feed them the open web:

Raw HTML is noisy. Scripts, nav chrome, cookie banners, and layout markup burn tokens and confuse the model.
Homegrown fetch is fragile. One requests.get (or fetch) plus BeautifulSoup works for a demo page and breaks on the next docs site.
Browser automation is heavy. Playwright or Puppeteer is the right hammer for JS-heavy apps—not for “summarize this public article” in a tool loop.

What you usually want for a tool or retrieval step is narrower: one public page → title + description + Markdown body, in a stable JSON shape, without signing up just to try it.

The minimal solution: free URL → Markdown GET

AnyCrawler exposes a public free endpoint:

GET https://api.anycrawler.com/free/v1/crawl?url={url}

Constraints worth knowing up front:

No API key for this free path
One public page per request
Response JSON includes results.markdown (plus title / description / status when available)
Successful results may be cached up to 24 hours

You can try the same flow in the browser at the free URL→Markdown playground before wiring it into code.

Step 1 — Call it with curl

Pick any publicly reachable docs or example URL. Here we use https://example.com:

curl -sS "https://api.anycrawler.com/free/v1/crawl?url=https://example.com" \
  | python3 -m json.tool

A successful response looks like:

{
  "ok": true,
  "requested_url": "https://example.com/",
  "results": {
    "description": null,
    "markdown": "This domain is for use in documentation examples...",
    "title": "Example Domain"
  },
  "status_code": 200,
  "total_ms": 519,
  "timestamp": "2026-11-05T10:38:26.962Z"
}

For an agent tool, the field that matters most is results.markdown. Keep status_code and ok for error handling.

Step 2 — Tiny Python example (`requests`)

import requests

FREE_ENDPOINT = "https://api.anycrawler.com/free/v1/crawl"
url = "https://example.com"

resp = requests.get(FREE_ENDPOINT, params={"url": url}, timeout=60)
resp.raise_for_status()
data = resp.json()

if not data.get("ok"):
    raise RuntimeError(f"crawl failed: {data}")

title = (data.get("results") or {}).get("title")
markdown = (data.get("results") or {}).get("markdown") or ""

print(title)
print(markdown[:500])

Hook this into a LangChain / LlamaIndex / custom tool the same way: accept a URL string, return Markdown (and optionally title) as the tool result.

Step 3 — JavaScript / Node example (`fetch`)

const FREE_ENDPOINT = "https://api.anycrawler.com/free/v1/crawl";

async function urlToMarkdown(pageUrl) {
  const endpoint = new URL(FREE_ENDPOINT);
  endpoint.searchParams.set("url", pageUrl);

  const res = await fetch(endpoint, { method: "GET" });
  if (!res.ok) {
    throw new Error(`HTTP ${res.status}`);
  }

  const data = await res.json();
  if (!data.ok) {
    throw new Error(`crawl failed: ${JSON.stringify(data)}`);
  }

  return {
    title: data.results?.title ?? null,
    markdown: data.results?.markdown ?? "",
    statusCode: data.status_code,
  };
}

const page = await urlToMarkdown("https://example.com");
console.log(page.title);
console.log(page.markdown.slice(0, 500));

In an agent loop, expose urlToMarkdown as a single tool with a clear description: “Fetch a public web page and return clean Markdown for reading.”

Step 4 — Drop Markdown into an agent or RAG step

A practical pattern:

User (or planner) supplies a URL.
Tool calls the free crawl endpoint.
You pass results.markdown into the model context—or chunk/embed it for retrieval.
Optionally prepend title / description so citations stay grounded.

Because the body is already Markdown, you avoid a second “strip HTML / readability” stage for many public pages. Keep tool descriptions honest: this is read one public URL, not full-site crawl or authenticated browsing.

For RAG specifically, treat each crawl as a document source: chunk results.markdown, store the requested_url as metadata for citations, and decide whether your index should refresh on every query or reuse embeddings until the free cache window (up to 24h) is no longer acceptable. Cache awareness matters for freshness: if you need a reliable recrawl sooner than that window, move to the authenticated production API (next section).

When to graduate to the production crawl API

Stay on the free endpoint while you are:

Prototyping a tool
Spot-checking whether a page yields useful Markdown
Running low-volume demos

Move to AnyCrawler’s authenticated production crawl API when you need:

Explicit fetch vs browser render modes
Configurable output fields and usage records
Higher operational limits and repeatable production workflows
Control beyond one anonymous GET per public URL

The free path is intentionally small. Production is where you add modes, auth, and operational controls—not where you should start if you only need “URL → Markdown” for a first agent tool. Compare both on the free tool page.

Short FAQ (related searches)

How do I convert a URL to Markdown for RAG without building a scraper?

Call a crawl API that returns cleaned Markdown in JSON. The free AnyCrawler endpoint above does that for one public URL per request with no API key. For many RAG demos, results.markdown is enough to chunk and embed.

Can an AI agent browse the live web with one HTTP call?

Not full interactive browsing—but for “read this public page,” yes: a tool that GETs the crawl endpoint and returns Markdown gives the agent live (or recently cached) page text without shipping a browser inside the agent process. Pair it with search (or a user-supplied URL) so the agent chooses which page to read, then uses Markdown as the observation.

Is free URL→Markdown cached?

Yes. Successful free results may be served from cache for up to 24 hours. Treat that as fine for prototypes; for stricter freshness or render modes, use the production API.

What URLs work?

Submit a complete, publicly reachable http or https URL. Pages behind login, private networks, or otherwise inaccessible content are not suitable for the free tool.

Try it and build the tool

Paste a public docs URL into the AnyCrawler free URL→Markdown tool.
Copy the curl / Python / JS snippets above into a minimal agent tool.
Graduate to the authenticated crawl API when you need fetch/render modes and production limits.

Disclosure: I work on the AnyCrawler team. Feedback and questions: marketing@anycrawler.com.

DEV Community: AnyCrawler