Building an Enterprise-Grade Web Metadata & Contact Extractor API (<200ms) with Python, FastAPI & C-Lexbor

Juanjo — Sun, 09 Aug 2026 15:03:15 +0000

Hello DEV Community! 👋

When building link preview cards (like WhatsApp or Slack), B2B lead enrichment pipelines, or web scrapers for AI/LLM RAG agents, fetching webpage metadata reliably and fast is surprisingly difficult.

Traditional scrapers suffer from:

🐢 High Latency: Heavy DOM parsers taking 1-3 seconds per request.
🚨 SSRF Vulnerabilities: DNS Rebinding & redirect attacks accessing internal cloud metadata.
📦 Broken SPA Rendering: Downloading tiny empty HTML shells from React/Next.js apps.

To solve this, I built the Web Metadata, OpenGraph & Contact Extractor API (v2.6.0) — an ultra-fast REST API designed for high-concurrency production workloads.

⚡ Key Tech Stack & Performance Architecture

C-Lexbor HTML Engine (selectolax): Parses HTML DOM tree 10x faster than BeautifulSoup.
Rust ORJSON Engine: Ultra-fast JSON response serialization.
Non-Blocking Async DNS Resolution: Uses asyncio.to_thread with an in-memory 5-minute DNS TTL cache to eliminate event-loop blocking.
Adaptive SPA Byte Limit (64 KB ➔ 256 KB): Automatically detects React, Next.js, Vue, Angular, Svelte, or Astro and dynamically expands the streaming buffer to capture server-side rendered data.
IP-Pinned Anti-SSRF Shield: Resolves DNS once, validates target IP against private/loopback/cloud metadata ranges (169.254.169.254), and pins the HTTP connection directly with TLS SNI headers.
Normalized 15-Min TTL In-Memory Cache: Repeat URL extractions complete in < 0.01 ms internal server time.

💎 What Data Does It Extract?

In a single GET request, the API returns a structured JSON payload containing:

SEO & OpenGraph Metadata: Title, Description, OG Image, OG Type, OG URL, OG Video, Favicon, Canonical URL, Language, Author, Theme Color, Robots directive (index, follow), and hreflang internationalization tags.
🛍️ E-Commerce Product Data: Automatically extracts Product Schema.org JSON-LD (Name, Price, Currency, Availability, Brand, Ratings & Reviews).
📧 Public Contacts: Public emails and telephone numbers with smart DOM script/style cleaning.
📲 Social Profiles: Twitter/X, LinkedIn, Facebook, Instagram, GitHub, YouTube, Telegram, TikTok.
🛠️ 100+ Tech Stack Detector: Identifies WordPress, Shopify, WooCommerce, Webflow, React, Next.js, Vue, Angular, TailwindCSS, Stripe, GA4, etc.
🤖 AI & LLM Clean Markdown: Converts web articles into clean Markdown for ChatGPT, Claude, and RAG pipelines (includes word count & reading time).
📊 SEO Audit Score: 8-point automated diagnostic score (0-100%) with actionable warnings.
🔒 Security Headers Audit: HSTS, CSP, X-Frame-Options, Referrer Policy score.

🚀 Quick Code Examples

JavaScript / Node.js

const url = 'https://web-metadata-and-contact-extractor-p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com';
const response = await fetch(url, {
  headers: {
    'X-RapidAPI-Key': 'YOUR_RAPIDAPI_KEY',
    'X-RapidAPI-Host': 'web-metadata-and-contact-extractor-p.rapidapi.com'
  }
});

const data = await response.json();
console.log('Title:', data.metadata.title);
console.log('Product:', data.product_data);
console.log('Tech Stack:', data.detected_technologies);
console.log('Execution Time:', data.execution_time_ms, 'ms');

Python

import requests

url = "https://web-metadata-and-contact-extractor-p.rapidapi.com/api/v1/extract"
headers = {
    "X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY",
    "X-RapidAPI-Host": "web-metadata-and-contact-extractor-p.rapidapi.com"
}
response = requests.get(url, headers=headers, params={"url": "https://stripe.com"})
print(response.json())

🔑 Try It Free on RapidAPI

The API is live and available on RapidAPI with 500 free requests/month:

👉 Get your free API Key on RapidAPI
👉 GitHub Repository

I'd love to hear your feedback, thoughts on the architecture, or feature requests!