DEV Community

Cover image for I got tired of writing scrapers for my AI agents, so I built Haunt
Daniel Cox
Daniel Cox

Posted on

I got tired of writing scrapers for my AI agents, so I built Haunt

#ai #api #webdev #showdev

Every time I wanted an AI agent to read a web page, I ended up in the same hole: writing a scraper, picking selectors, spinning up a headless browser for the JavaScript-heavy ones, then babysitting the whole thing as sites changed. Worse, many scraping tools "succeed" on a blocked page by handing you garbage, which quietly poisons whatever you feed it.

So I built Haunt. You give it a URL and a plain-English prompt, and it hands back clean structured JSON or Markdown. No selectors, no parsing code.

The basic call 

curl -sS https://hauntapi.com/v1/extract \
  -H "Content-Type: application/json" \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"url": "https://example.com/pricing", "prompt": "every plan with its price and features"}'
Enter fullscreen mode Exit fullscreen mode

You get back something like:

{
  "success": true,
  "data": { "plans": [ { "name": "Starter", "price": "$19/mo", "features": ["..."] } ] },
  "mode": "full_structured",
  "confidence": 0.9,
  "credits_used": 2,
  "credits_remaining": 998
}
Enter fullscreen mode Exit fullscreen mode

Ask for Markdown instead of JSON, and you get clean page text, which is what most people actually want for a RAG pipeline or an agent's context.

It's built for agents

If your agent speaks MCP, wiring it up is one command:

npx -y --package @hauntapi/cli@latest haunt-cli init
Enter fullscreen mode Exit fullscreen mode

After that, your agent has a tool it can call to read any public page, with no glue code.

The parts I actually spent my time on

The "LLM turns a page into JSON" bit is the easy part. The parts that took the real work:

  • Honest failures. When a page is blocked, behind a login, or bot-walled, Haunt tells you that. It does not fake a result. And it won't invent fields that aren't on the page: it grounds the output against the real page text and drops anything it can't find there.
  • Zero retention by default. It does not keep your page content, prompts, or results. It holds only the metadata it needs for billing and abuse prevention.
  • Failed calls don't burn credits. You pay for extractions that actually worked, not for the ones that didn't.

What it is not

Being straight, because this crowd will rightly ask:

  • The free and fast path is a direct fetch plus extraction. It does not run the heavy anti-bot stack (proxy rotation, full browser recovery) on the free tier.
  • It does not crack CAPTCHAs or bypass human verification. If you need to defeat every wall on earth, this is not that tool.

Try it

There is a live demo on the homepage with no signup: paste any public URL and watch it extract. Free tier is 1,000 credits a month, no card.

I built this solo and launched it today. I would genuinely love feedback, especially where it breaks or where it is weaker than you would need. If you try it and it falls over, tell me. That is the most useful thing I can get right now.

Top comments (0)