DEV Community

NexusCore
NexusCore

Posted on

I built an API that turns any URL into structured JSON for AI agents

#ai #agents #api #tutorial

I built an API that turns any URL into structured JSON for AI agents

If you've ever babysat a scraper to pull clean fields — price, title, specs, description — out of a product page, you know the pain. Fragile CSS selectors break on every redesign. Anti-bot measures block you. And half the time you're just re-extracting data that should've been an API call.

I built Agent API Gateway to fix that. Send a URL and a schema type. Get validated JSON back. No browser farm, no selector maintenance, no proxy rotation.

The problem

Every AI agent that needs to understand web content faces the same bottleneck: turning unstructured HTML into structured data. LLMs can read raw HTML, but it's wasteful (thousands of tokens for noise), slow, and you can't trust the output for anything that needs to be machine-accurate.

Traditional scrapers solve this for known sites. But agents need to handle any URL — a product link a user drops in chat, a research article, a company page. Writing a custom scraper for each is not feasible.

What it does 

POST /v1/extract
{
  "url": "https://example.com/product/widget-1000",
  "schema": "product"
}
Enter fullscreen mode Exit fullscreen mode

Returns:

{
  "name": "Widget 1000 Pro",
  "price": 49.99,
  "currency": "USD",
  "description": "Professional-grade widget...",
  "brand": "Acme Corp",
  "sku": "W1000-PRO",
  "availability": "in_stock",
  "image_url": "https://example.com/images/w1000.jpg"
}
Enter fullscreen mode Exit fullscreen mode

Three schema types available today:

  • product — name, price, brand, specs, availability, images
  • article — title, author, published date, body text, word count
  • company — name, description, team size, funding, social links

What makes it different from just asking an LLM to parse HTML

Speed. The gateway extracts and validates in under 2 seconds. Asking GPT-4 to parse a product page takes 5-15 seconds and costs 10x more in tokens.

Cost. Starting at $1 for 1,000 queries. A single GPT-4 call with raw HTML costs more than that.

Reliability. Schema-locked output means you always get the same fields. No "the LLM decided to return a slightly different JSON structure this time."

Safety. SSRF-protected fetches (the gateway won't hit internal/localhost URLs), rate limiting, and API key management built in.

Real use cases I've seen

  1. Price monitoring agents — track product prices across retailers without maintaining per-site scrapers
  2. Research assistants — extract structured data from academic papers and articles
  3. Market intelligence — automatically catalog competitor products from their websites
  4. Content pipelines — pull article metadata for curation and summarization

How to try it

Free tier: 100 queries/month, no credit card required.

# Sign up
curl -X POST https://agentapigw.dpdns.org/auth/signup \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com"}'

# Extract (after getting your API key from the dashboard)
curl -X POST https://agentapigw.dpdns.org/v1/extract \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com", "schema":"product"}'
Enter fullscreen mode Exit fullscreen mode

Credit packs (no subscription needed):

  • $1 → 1,000 queries
  • $4 → 5,000 queries
  • $15 → 25,000 queries

Subscriptions for heavy users:

  • Hobby: $29/mo → 5,000 queries + all schemas + priority
  • Pro: $99/mo → 25,000 queries + team + dedicated support

Built for agents, not just humans

The API has /agent.json and /llms.txt endpoints so AI agents can discover and use it programmatically. Your agent can:

  1. Fetch https://agentapigw.dpdns.org/llms.txt to learn what the API does
  2. Read the schema types and endpoints
  3. Start extracting data — all without human intervention

Tech stack

  • Python (FastAPI) on Render
  • Polar.sh for payments (instant checkout, credit packs)
  • Schema validation + SSRF protection on every request
  • OAuth (GitHub, Google) for zero-friction signup

What's next

  • More schema types (job listings, events, recipes)
  • Batch extraction (multiple URLs in one call)
  • Webhook callbacks for async extraction
  • Self-hosted option for enterprise

Try it free: agentapigw.dpdns.org

Buy credits: $1 starter — 1,000 queries, no subscription.

GitHub: Issues and feature requests welcome.

Built by NexusCore. If you're building agents that need structured web data, this is for you.

Top comments (0)