DEV Community

Cover image for How to Turn Any URL into Clean Markdown for AI Agents & Run Instant SEO Audits (WebPulse API)
Juan Lorenzo Ayala
Juan Lorenzo Ayala

Posted on

How to Turn Any URL into Clean Markdown for AI Agents & Run Instant SEO Audits (WebPulse API)

If you are building AI agents, RAG (Retrieval-Augmented Generation) pipelines, or automated scraping workflows, you already know the painful truth: raw web HTML is an absolute token burner.

Feeding a full raw webpage into GPT-4, Claude, or Gemini wastes up to 80% of your context window on navigation bars, cookie banners, tracking scripts, and inline SVGs.

To solve this, I built WebPulse AI & SEO Suite API: a lightweight, ultra-fast 5-in-1 web extraction API that converts noisy URLs into clean, token-optimized Markdown, extracts rich metadata, audits technical SEO, and discovers B2B contact info in milliseconds.

Here is how it works and how you can test it for free today.


⚡ 1. The Core Problem: LLM Token Bloat

When you fetch a typical blog post or documentation page:

  • Raw HTML size: ~250 KB (approx. 60,000 tokens)
  • Actual readable content: ~15 KB (approx. 3,500 tokens)

Sending raw HTML to your LLM costs real money, slows down responses, and degrades reasoning accuracy.

With the /api/v1/markdown endpoint, WebPulse strips away all boilerplate and returns:

  1. Pure GitHub-flavored Markdown.
  2. Formatted plain text.
  3. Pre-split semantic rag_chunks ready to embed into vector databases (Pinecone, Chroma, Supabase).
  4. An automated calculation of your exact token savings percentage (llm_token_savings_percent).

Quick Python Example:

import requests

url = "https://webpulse-ai-markdown-seo-audit-and-contacts.p.rapidapi.com/api/v1/markdown"
querystring = {"url": "https://stripe.com"}

headers = {
"x-rapidapi-key": "YOUR_RAPIDAPI_KEY",
"x-rapidapi-host": "webpulse-ai-markdown-seo-audit-and-contacts.p.rapidapi.com"
}

response = requests.get(url, headers=headers, params=querystring)
data = response.json()

print(f"Token savings: {data['data']['llm_token_savings_percent']}%")
print(f"Clean Markdown sample:\n{data['data']['markdown'][:300]}...")


🔍 2. Instant 25-Point Technical SEO Audit

Most SEO tools are heavy, expensive, or require full browser emulation.

The /api/v1/seo-audit endpoint runs a 25-checkpoint audit directly at the HTTP and DOM level in under 150ms:

  • SEO Score (0–100) & Grade (A+ to F)
  • Meta title, description, and canonical tag validation
  • Heading structure hierarchy (H1 to H6 check)
  • Missing image alt attributes count
  • Internal vs external link balance
  • Security & indexability verification (robots, HTTPS)
  • Structured lists of critical_issues, warnings, and passed_checks

Perfect for agencies offering automated client onboarding reports or developers building automated website monitors.


💼 3. B2B Contact & Social Profile Discovery

Need to enrich company profiles automatically?

The /api/v1/contacts endpoint crawls the target page and detects:

  • Corporate & support emails (categorized by role vs direct address)
  • Phone numbers
  • 10 major social networks: LinkedIn, X (Twitter), GitHub, YouTube, Instagram, Facebook, TikTok, Discord, Telegram, WhatsApp
  • Key corporate pages: Pricing, Contact Us, About, Privacy Policy

🚀 4. How to Test It in 60 Seconds

You can test WebPulse immediately in two ways:

  1. Interactive Web Playground (No login required):
    Try live URLs directly in your browser:
    👉 https://webpulseapi.vercel.app

  2. RapidAPI Marketplace (50 Free Requests / Month):
    Subscribe to the Free Tier ($0/month) to get your API key:
    👉 https://rapidapi.com/Juanamigazo/api/webpulse-ai-markdown-seo-audit-and-contacts


🛠️ Built for Developers, AI Builders & Automation Enthusiasts

Whether you are chaining nodes in n8n / Make / Zapier, fine-tuning local LLMs, or aggregating market intelligence, WebPulse is built to be fast, reliable, and cost-effective.

Feedback is deeply appreciated! What other endpoints or data extraction features would make your AI or scraping pipelines smoother? Drop a comment below!

Top comments (1)

Collapse
 
amigazofiel profile image
Juan Lorenzo Ayala •

Thanks for taking the time to read everyone! 🚀

I'm actively refining WebPulse to make it as lightweight and practical as possible for AI builders and automation workflows (n8n, Make, LangChain).

If you test the live playground or run into any specific scraping bottlenecks in your daily work, please drop a comment below. I'd love to hear what endpoints or data fields would be most useful to add next!