DEV Community

Cover image for Building an Enterprise-Grade Web Metadata & Contact Extractor API (<200ms) with Python, FastAPI & C-Lexbor
Juanjo
Juanjo

Posted on Edited on

Building an Enterprise-Grade Web Metadata & Contact Extractor API (<200ms) with Python, FastAPI & C-Lexbor

#python #ai #webdev #showdev

Hello DEV Community! 👋

When building link preview cards (like WhatsApp or Slack), public contact-discovery pipelines, or web scrapers for AI/LLM RAG agents, fetching webpage metadata reliably and fast is surprisingly difficult.

Traditional scrapers suffer from:

  • 🐢 High Latency: Heavy DOM parsers taking 1-3 seconds per request.
  • 🚨 SSRF Vulnerabilities: DNS Rebinding & redirect attacks accessing internal cloud metadata.
  • 📦 Broken SPA Rendering: Downloading tiny empty HTML shells from React/Next.js apps.

To solve this, I built the Web Metadata, OpenGraph & Contact Extractor API (v4.1.0) — an ultra-fast REST API designed for high-concurrency production workloads.

⚡ Key Tech Stack & Performance Architecture

  • C-Lexbor HTML Engine (selectolax): Parses HTML DOM tree 10x faster than BeautifulSoup.
  • Rust ORJSON Engine: Ultra-fast JSON response serialization.
  • Non-Blocking Async DNS Resolution: Uses asyncio.to_thread with an in-memory 5-minute DNS TTL cache to eliminate event-loop blocking.
  • Adaptive SPA Byte Limit (64 KB ➔ 256 KB): Automatically detects React, Next.js, Vue, Angular, Svelte, or Astro and dynamically expands the streaming buffer to capture server-side rendered data.
  • IP-Pinned Anti-SSRF Shield: Resolves DNS once, validates target IP against private/loopback/cloud metadata ranges (169.254.169.254), and pins the HTTP connection directly with TLS SNI headers.
  • Normalized 15-Min TTL In-Memory Cache: Repeat URL extractions complete in < 0.01 ms internal server time.

💎 What Data Does It Extract?

In a single GET request, the API returns a structured JSON payload containing:

  1. SEO & OpenGraph Metadata: Title, Description, OG Image, OG Type, OG URL, OG Video, Favicon, Canonical URL, Language, Author, Theme Color, Robots directive (index, follow), and hreflang internationalization tags.
  2. 🛍️ E-Commerce Product Data: Automatically extracts Product Schema.org JSON-LD (Name, Price, Currency, Availability, Brand, Ratings & Reviews).
  3. 📧 Public Contacts: Public emails and telephone numbers with smart DOM script/style cleaning.
  4. 📲 Social Profiles: Twitter/X, LinkedIn, Facebook, Instagram, GitHub, YouTube, Telegram, TikTok.
  5. 🛠️ 40+ Tech Stack Detector: Identifies WordPress, Shopify, WooCommerce, Webflow, React, Next.js, Vue, Angular, TailwindCSS, Stripe, GA4, etc.
  6. 🤖 AI & LLM Clean Markdown: Converts web articles into clean Markdown for ChatGPT, Claude, and RAG pipelines (includes word count & reading time).
  7. 📊 SEO Audit Score: 14-point automated diagnostic score (0-100%) with actionable warnings.
  8. 🔒 Security Headers Audit: HSTS, CSP, X-Frame-Options, Referrer Policy score.

🚀 Quick Code Examples

JavaScript / Node.js 

const url = 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com';
const response = await fetch(url, {
  headers: {
    'X-RapidAPI-Key': 'YOUR_RAPIDAPI_KEY',
    'X-RapidAPI-Host': 'web-metadata-and-contact-extractor.p.rapidapi.com'
  }
});

const data = await response.json();
console.log('Title:', data.metadata.title);
console.log('Product:', data.product_data);
console.log('Tech Stack:', data.detected_technologies);
console.log('Execution Time:', data.execution_time_ms, 'ms');
Enter fullscreen mode Exit fullscreen mode

Python 

import requests

url = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"
headers = {
    "X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY",
    "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"
}
response = requests.get(url, headers=headers, params={"url": "https://stripe.com"})
print(response.json())
Enter fullscreen mode Exit fullscreen mode

🔑 Try It Free on RapidAPI

The API is live and available on RapidAPI with 1,000 free requests/month:

👉 Get your free API Key on RapidAPI
👉 GitHub Repository

I'd love to hear your feedback, thoughts on the architecture, or feature requests!

This piece is also cross-posted on Hashnode and, as a shorter companion version, on Medium.

Top comments (0)