DEV Community: Juanjo

I scanned the top 1,000 websites for security headers. The median score is 37/100.

Juanjo — Sun, 30 Aug 2026 06:55:21 +0000

Security-header advice has been the same for a decade: set HSTS, set a CSP, add X-Frame-Options and nosniff, done. I wanted to know how many sites that actually reach the biggest audiences on the web bother. So I took the Tranco top 1,000, fetched every homepage once, and graded six response headers.

Short version: the median security-header score is 37 out of 100. Fewer than 1 in 10 of the top sites score 70 or higher, and 13% send no security headers at all. The full dataset and scripts are on GitHub, there is an interactive version with a searchable table of every domain, and this post is the walk-through.

How the scan works

One GET https://<domain> per site, redirects followed, then six headers each graded missing / weak / reasonable / strong:

Strict-Transport-Security
Content-Security-Policy
X-Frame-Options
X-Content-Type-Options
Referrer-Policy
Permissions-Policy

The score is the mean of the six grades as a percentage. The one nuance worth stating: a CSP containing unsafe-inline, unsafe-eval or a wildcard source is graded weak, not strong. It parses, but it does not stop injected script from running.

I did the fetching and grading with an open-source API I maintain (the Web Metadata & Contact Extractor), because one call to its /api/v1/extract endpoint returns the header grades, an overall security score, an SEO score and a detected tech stack for a URL. That last part matters for the third finding below.

About the denominator

Of the 1,000 domains, only 489 are real, reachable homepages. 225 are infrastructure (DNS-only CDN and cloud domains that serve no site), 209 were unreachable from where I ran the scan (timeouts, geo-blocks, paywalls, apex redirect quirks), 43 returned a bot-challenge or WAF page, and 34 served a placeholder. Every number below is over those 489. The sites I could not read lean toward the heavily bot-protected, which are on average the more security-conscious ones, so the real picture is probably a little worse than this.

Finding 1: most of the top sites are missing most of the headers

Strict-Transport-Security is the only header on a majority of sites (71%). X-Frame-Options and X-Content-Type-Options - each a single static line with no downside - are missing on roughly 45%. Referrer-Policy is on 29%. Permissions-Policy is on 16%.

Finding 2: the score distribution is bottom-heavy

Median 37, mean 36, and a long tail into the low scores rather than the high ones. A quarter of the top 1,000 score 17 or below.

Content-Security-Policy is where it breaks down hardest. 46% of sites have no CSP at all. Of the 54% that do, all but 90 sites carry unsafe-inline or a wildcard. Put together, about 82% of the top 1,000 have no browser-enforced CSP that would actually stop an XSS. Three sites in the whole set have a CSP I would call strong.

That is not all negligence. A strict CSP on a large existing site is genuinely hard - third-party tags, inline handlers, A/B tools all fight it, and a bad rollout breaks the page. But "hard" is the explanation for the weak-CSP half, not for the 46% with nothing and the 84% with no Permissions-Policy.

Finding 3: the tech stack predicts the posture

Bucketing the 489 sites by detected platform, WordPress (n=44) sits about 10 points below the field: median score 27, and 61% ship no CSP versus 46% overall. Next.js and similar framework/edge stacks land near or just above the median, mostly because their hosting (Vercel, Netlify, Cloudflare Pages) sets a decent HSTS by default. The only groups with a median above 42 were Astro (n=8, too small to lean on) and Drupal (n=10).

The interesting part is what happens inside the WordPress bucket. Those 44 sites have a median SEO score of 83, well above the overall 75. They are not neglected sites. They are sites where SEO got attention and headers did not.

Finding 4: SEO effort and security effort barely correlate

Across all 489 sites, Pearson r between the SEO score and the security-header score is 0.17. The top of the web has largely solved SEO (median 75) and largely skipped security headers (median 37), and knowing how much a team invested in one tells you almost nothing about the other.

My read: SEO has a tight, visible feedback loop (rankings, traffic, revenue). Security headers have none until an incident. So on a busy roadmap the headers lose, every sprint, at almost every company - including ones with the resources to know better.

Try it on your own site

pip install httpx matplotlib
curl -sL -o tranco.zip https://tranco-list.eu/top-1m.csv.zip
python scan.py 1000    # -> results.jsonl
python analyze.py      # -> summary.json + top1000-web-report.csv
python make_charts.py

Everything - the scan script, the analysis, the raw JSONL, the flattened CSV - is in the repo under MIT. If you just want your own site graded, the API has a free tier and curl ".../api/v1/security?url=https://yoursite.com" gives you the same per-header grades used here.

If you find a bug in the grading or want a rank band or a platform I did not break out, open an issue with the dataset row and I will take a look.

This API vs ScrapingBee vs Firecrawl vs BuiltWith: an honest comparison

Juanjo — Tue, 25 Aug 2026 18:09:23 +0000

Hello DEV Community! 👋

"Just use ScrapingBee" or "just use Firecrawl" comes up a lot when someone asks how to pull structured data from a URL — but those are full scraping platforms with headless-browser pricing to match, and sometimes what you actually need is a lot simpler: SEO tags, detected tech stack, public contact signals, and a security-header grade for a normal, server-rendered page.

I maintain the Web Metadata & Contact Extractor API, and I kept getting asked "how is this different from ScrapingBee / Firecrawl / BuiltWith?" — so instead of a vague answer, I actually checked pricing pages and wrote down the honest comparison. No "we're better at everything" spin: these four tools solve overlapping but genuinely different problems.

Prices/limits below checked live in August 2026. Re-verify before relying on exact numbers — pricing pages change.

The one-line version

Tool	What it actually is	JS rendering?	Crawls a whole site?
This API	Fast metadata/SEO/contact/tech-stack extractor for one URL at a time	❌ No — reads server HTML only	❌ No — one URL in, one JSON out
ScrapingBee	General-purpose scraping API with a real headless browser	✅ Yes — full Chrome, clicks/scrolls/forms	❌ No built-in crawler, but handles single complex pages
Firecrawl	Scraping + crawling API built for feeding LLMs/RAG pipelines	✅ Yes	✅ Yes — follows links, crawls domains
BuiltWith	Technology-profiling database + lead/contact export tool	N/A — it's a lookup database, not a live fetcher	N/A — queries its own crawled index

If you need to click through a login form, scroll infinite-scroll content, or get past a JS challenge: this API is the wrong tool — go use ScrapingBee or Firecrawl. My own docs say so directly, see "Honest Limitations" in the README.

If you need one fast, structured answer for an ordinary page — without paying for a headless browser you don't need — that's the actual use case here.

Pricing, side by side

	Free tier	Cheapest paid plan	What it buys
This API	1,000 requests/mo	$5.99/mo (Pro)	50,000 requests/mo + email support
ScrapingBee	1,000 credits (trial, no card)	$49/mo (Freelance)	250,000 credits/mo, 50 concurrent requests
Firecrawl	1,000 credits/mo (forever)	$16/mo (Hobby, billed yearly)	5,000 credits/mo
BuiltWith	None	$295/mo (Basic)	2 technology/keyword/retail lookups, 1 login

Two takeaways, not marketing spin:

This API is dramatically cheaper per request than ScrapingBee or BuiltWith — because it does less. No JS execution, no headless browser, no crawler. Comparing $5.99/mo to $49/mo only makes sense once you've confirmed your target pages don't need JS rendering.
BuiltWith isn't really a live-extraction competitor at all — it's a historical technology database you query ($295/mo minimum, no free tier), not a "fetch this URL right now" tool.

When each one actually wins

This API — normal server-rendered pages (blogs, marketing sites, product pages, GitHub, Wikipedia-style sites), when you want SEO metadata + tech stack + public contacts + security headers + SEO score in one ~150-300ms call. Free tier, 1,000 req/mo, no card required.
ScrapingBee — pages that need real browser interaction: logins, infinite scroll, clicks, screenshots, JS-scenario scripting. $49/mo floor.
Firecrawl — crawling a whole site (not just one URL) into clean Markdown for an LLM/RAG pipeline, with a real always-free tier for testing.
BuiltWith — historical tech-stack data across large domain lists, plus exportable contact/lead lists, if $295/mo+ fits the budget for that specific use case.

The gap none of us solve for free

None of these turn "public email/phone found on a page" into verified company/people intelligence. My /api/v1/contacts endpoint is deliberately named "Contact Discovery," not "Lead Enrichment" — it returns raw signals, you still build the verification layer on top. That's true regardless of which extraction tool you start from.

Full write-up with sources lives in the repo: COMPETITOR_COMPARISON.md

👉 Try the free tier on RapidAPI
👉 GitHub Repository

Curious what you all think — especially if you've hit the JS-rendering wall with a similar "lightweight extractor" approach. Happy to be told I'm missing an angle here.

Building a B2B Prospecting Agent in 15 Minutes with CrewAI and a Single API Call

Juanjo — Mon, 24 Aug 2026 14:59:38 +0000

The problem

A B2B prospecting workflow usually looks the same regardless of industry: you have a list of target company domains, and for each one you need three things before you can write a useful outreach email — a way to contact them, a read on what they're running (so you can tailor the pitch), and a quick judgment call on whether they're even worth reaching out to.

Doing this by hand doesn't scale past a handful of leads. Doing it with a scraper you wrote yourself means dealing with SSRF-safe fetching, HTML parsing, and rate limits before you've written a single line of actual agent logic.

This walks through building a two-agent CrewAI crew that takes a list of company URLs and produces a prioritized, reasoned shortlist — using crewai-webmetadata-extractor to handle the extraction side, so the agents can focus on judgment instead of parsing.

Setup

pip install crewai crewai-webmetadata-extractor

You'll need two keys:

A free RapidAPI key for the Web Metadata & Contact Extractor API (1,000 requests/month, no card required)
An API key for whatever LLM provider you're running CrewAI's agents on (OpenAI, Anthropic, etc. — CrewAI handles the provider abstraction, not covered here)

export WEBMETADATA_API_KEY=your-rapidapi-key

The tools

crewai-webmetadata-extractor ships four ready-made BaseTool subclasses. For a prospecting crew, two matter most:

Tool	What it returns
`WebContactsTool`	Emails, phone numbers, and social links found on the page
`WebMetadataExtractTool`	Full picture: SEO/OpenGraph data, detected tech stack, security-headers grade, Schema.org data, links

Every tool returns a JSON string, and API errors come back as {"error": true, ...} instead of raising — so a bad URL in your lead list doesn't crash the whole crew run.

Building the crew

Two agents: one that gathers raw signal per company, one that turns that signal into a ranked, reasoned shortlist.

from crewai import Agent, Task, Crew, Process
from crewai_webmetadata_extractor import WebContactsTool, WebMetadataExtractTool

researcher = Agent(
    role="B2B Lead Researcher",
    goal="Extract contact information and technical footprint for a list of company websites",
    backstory=(
        "You investigate company websites to surface contact points and technology "
        "signals that a sales team can act on. You report facts, not conclusions."
    ),
    tools=[WebContactsTool(), WebMetadataExtractTool()],
    verbose=True,
)

qualifier = Agent(
    role="Sales Development Rep",
    goal="Turn raw research into a prioritized, reasoned outreach shortlist",
    backstory=(
        "You review research on prospective companies and decide who's worth "
        "contacting first, based on how good a fit their tech stack and public "
        "presence make them for our product."
    ),
    verbose=True,
)

target_urls = [
    "https://example-company-one.com",
    "https://example-company-two.com",
    "https://example-company-three.com",
]

research_task = Task(
    description=(
        f"For each of these URLs, extract public contact info and the detected tech "
        f"stack: {', '.join(target_urls)}. List what you found per company, including "
        f"any URL that returned no usable contact info."
    ),
    expected_output="A per-company breakdown of contacts found and tech stack detected.",
    agent=researcher,
)

qualify_task = Task(
    description=(
        "Using the research above, rank the companies from most to least worth "
        "contacting. Justify each ranking with a specific signal from the research "
        "(a detected technology, a missing security header, presence or absence of "
        "a direct contact channel) — not a generic guess."
    ),
    expected_output="A ranked list of companies with a one-line justification each.",
    agent=qualifier,
    context=[research_task],
)

crew = Crew(
    agents=[researcher, qualifier],
    tasks=[research_task, qualify_task],
    process=Process.sequential,
)

result = crew.kickoff()
print(result)

What the qualifier actually has to work with

The researcher's tool calls return structured JSON, not free text — which is what lets the qualifier reason about specific fields instead of vibes. A WebMetadataExtractTool call includes a tech_stack array (CMS, analytics, frameworks detected) and a graded security_headers object, so "rank by fit" can turn into something like "runs WordPress with no CMS-specific caching layer detected and a missing CSP header — a plausible fit for a dev/security retainer" instead of a made-up reason.

WebContactsTool's output is deliberately scoped to what's actually on the page (emails, phones, social links) — it's a raw signal for the qualifier to weigh, not a claim about verified company or people data.

Where this goes from here

The same two tools compose into other shapes: a WebSEOAuditTool pass turns this into an SEO-focused sales angle ("your homepage is missing H1 structure, here's a report"); dropping in WebMarkdownTool on the same URL list turns the researcher into the ingestion step for a RAG pipeline over your leads' own public content, instead of a prospecting crew.

The point of separating "extraction" from "judgment" into two agents (rather than one agent doing both) is that the researcher's output stays inspectable — you can log or cache the raw JSON independent of whatever the qualifier concludes from it, and swap the qualifier's prompt without re-running extraction.

Firecrawl vs. Jina Reader vs. Tavily: Picking the Right Web-to-LLM Tool (and When You Need None of Them)

Juanjo — Sun, 23 Aug 2026 15:26:36 +0000

If you're building a RAG pipeline or an autonomous agent, at some point you need to turn a URL into something an LLM can actually use. Over the last year a handful of APIs have grown up specifically to solve this — Firecrawl, Jina Reader, Tavily, and Crawl4AI are the four that come up constantly in agent-framework docs and Show HN threads. They get lumped together a lot, but they're not interchangeable, and none of them is really built for the narrower job of "get me this page's SEO metadata, contact info, and security posture." Here's an honest breakdown of what each one is actually for, what it costs, and where a purpose-built API still makes more sense.

What each one is actually optimized for

Firecrawl is a crawler first, a scraper second. Its core primitives are Scrape (one page), Crawl (a whole site, following links), and Map (site structure discovery), all with JS rendering. If your job is "ingest this entire documentation site into my vector store," Firecrawl's Crawl endpoint is the right shape of tool — nothing else here does multi-page traversal out of the box.

Jina Reader (r.jina.ai) is the opposite end of the spectrum: the simplest possible interface. Prepend https://r.jina.ai/ to any URL and you get clean Markdown back, no request body, no SDK required. It's genuinely the fastest way to paste "the content of this page" into a prompt. It doesn't crawl, and it doesn't extract structured fields beyond the content itself.

Tavily is built around search, not single-URL extraction. Its bread and butter is the search endpoint (query in, ranked results with summaries out) for research-style agent loops; it has an extract endpoint too, but the product is designed around "search the web and synthesize an answer," not "give me every structured field on this specific page."

Crawl4AI is a different kind of answer entirely: it's not a hosted API, it's an open-source (Apache 2.0), self-hosted Python library and Docker service — you run it yourself instead of calling someone else's endpoint. It gives you full Playwright/Chromium control for JS-heavy pages, LLM-ready Markdown output, and both LLM-driven and CSS-selector-based structured extraction. The tradeoff for that control is that you own the infrastructure — proxies, browser pool, scaling. A commercial "Crawl4AI Cloud API" is in closed beta as of this writing, with no public pricing yet.

None of the four is trying to be an SEO auditor, a security-headers grader, or a contact-discovery tool — that's a different job, and it's the one I've been building an open-source API for: https://github.com/JosejuX/rapidapi-metadata-extractor

Pricing and limits, compared

This is where the practical differences show up fastest. Three of the four meter usage through a credit or token system; Crawl4AI's core is free because you're paying for your own compute instead of theirs, and the metadata-focused API I maintain doesn't meter by credits either — flat per-request.

	Free tier	Paid entry point	Metering
Firecrawl	1,000 credits/mo (10 scrapes/min, 1 crawl/min)	$16/mo (Hobby, 5,000 credits)	Credits: 1/page (scrape/crawl/map), 2/10 results (search), 5/page (stealth mode)
Jina Reader	~20 req/min with no key; 100 RPM / 2 concurrent with a free key	Paid tier: 500 RPM, $0.02 per 1M output tokens	Token-based billing
Tavily	1,000 credits/mo, no card	$30/mo (4,000 credits) or $0.008/credit PAYG	Credits: 1 (basic search) to 2 (advanced search)
Crawl4AI	Unlimited — self-hosted, Apache 2.0, you cover your own compute	Cloud API in closed beta, pricing not yet public	No metering; you run it, so cost is your infra bill
Web Metadata & Contact Extractor	1,000 requests/mo, no card	$5.99/mo (Pro)	Flat per-request count, no credit/token math

The practical difference isn't "cheaper" so much as "simpler to reason about." With a flat request count, if you're doing 1,000 lookups a month you know exactly where you stand — there's no separate accounting for how many credits a "stealth" scrape burned versus a plain one, or converting a token bill back into "how many pages was that."

Where the narrower tool wins

If what you actually need is: "pull the SEO metadata, OpenGraph tags, public contact signals, tech stack, and a security-headers grade from this one URL," none of the three general-purpose tools return that as structured, named fields — you'd get raw Markdown or HTML back and have to parse it yourself. That's the gap this API fills:

from webmetadata_extractor import WebMetadataClient

client = WebMetadataClient(api_key="YOUR_RAPIDAPI_KEY")

# Structured fields, not raw markdown you have to re-parse
seo = client.seo_audit("https://example.com")
print(seo["seo_score_percentage"], seo["warnings"])

contacts = client.contacts("https://example.com")
print(contacts["emails"], contacts["social_links"])

security = client.security("https://example.com")
print(security["security_headers"])

Compare that to getting Markdown back from Jina Reader, Firecrawl's Scrape, or your own Crawl4AI instance, and then writing your own regex/BeautifulSoup pass to pull out emails, security headers, or a 14-point SEO checklist — doable, but it's work these tools weren't designed to save you.

Where it genuinely doesn't compete

To be fair about the other side: this API has no crawler — one URL in, one page's data out (see the "Honest Limitations" section in the README: https://github.com/JosejuX/rapidapi-metadata-extractor#-honest-limitations). If you need to crawl an entire site, Firecrawl is the right tool, full stop. If you just want the fastest possible "read this page for me" for a prompt and don't care about structured fields, Jina Reader's one-line interface is hard to beat. If your agent's job is genuinely "search the web and answer a question," Tavily's search-plus-synthesis loop is built for exactly that in a way this API isn't trying to be. And if you need full rendering control and don't mind operating your own crawler infrastructure, Crawl4AI's self-hosted flexibility is a real advantage — this API is a managed service, not a framework you deploy and own.

The actual takeaway

These aren't five competing products so much as five answers to five different questions:

"Crawl this whole site" → Firecrawl
"Give me this page as clean text, fastest way possible" → Jina Reader
"Search the web and summarize" → Tavily
"Full control, self-hosted, don't mind running infrastructure" → Crawl4AI
"Give me structured SEO/contact/security/tech-stack data on this specific URL, without token or credit accounting" → this API: https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor

Pick based on the shape of the job, not the hype cycle. If you want to try the structured-extraction angle, there's a free interactive demo (no signup) at https://rapidapi-metadata-extractor.onrender.com, a Python SDK at https://pypi.org/project/webmetadata-extractor/, and now LangChain (https://pypi.org/project/langchain-webmetadata-extractor/) and CrewAI (https://pypi.org/project/crewai-webmetadata-extractor/) tool packages if you want to hand it directly to an agent.

Grading Security Headers Isn't the Full Picture

Juanjo — Wed, 19 Aug 2026 21:01:39 +0000

A few days ago someone left a comment on my last post about grading security headers automatically:

"Security-header grading is most useful when it explains tradeoffs, not only pass/fail. CSP especially needs context because a stricter policy can be correct technically and still break the product if rollout is blind."

Fair hit. My original grader read Content-Security-Policy and Content-Security-Policy-Report-Only as the same thing — a strict policy still in report-only mode (i.e., not actually blocking anything yet) scored identically to one fully enforced. That's not a rounding error, it's a real blind spot: a team mid-rollout with a report-only CSP gets the same green checkmark as a team that shipped a broken policy and never noticed.

Fixing that made me look harder at what grading security from headers alone actually misses — and the biggest gap isn't in the headers at all. It's in the TLS layer underneath them.

Headers tell you intent. TLS tells you reality.

Strict-Transport-Security says "browsers should only ever load me over HTTPS." It says nothing about whether the certificate serving that HTTPS connection is valid right now, how many days until it expires, which TLS version actually got negotiated (a server can advertise HSTS and still fall back to TLS 1.0 for older clients), or who issued it. A site can have a perfect header score and be running on a cert that expires in 6 days. Headers audit configuration; a live handshake audits ground truth.

Doing the handshake for real

import ssl, socket
from datetime import datetime

def get_cert_info(hostname: str, port: int = 443, timeout: float = 3.0):
    ctx = ssl.create_default_context()
    with socket.create_connection((hostname, port), timeout=timeout) as sock:
        with ctx.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert = ssock.getpeercert()
            version = ssock.version()
    not_after = datetime.strptime(cert["notAfter"], "%b %d %H:%M:%S %Y %Z")
    return {
        "issuer": dict(x[0] for x in cert["issuer"]),
        "valid_until": not_after.isoformat(),
        "tls_version": version,
        "is_expired": datetime.utcnow() > not_after,
    }

Three things matter beyond the happy path:

It's blocking I/O. In an async service, wrap it: await loop.run_in_executor(None, get_cert_info, hostname).
It has to fail soft. Self-signed certs, non-TLS ports, and slow/unreachable hosts are all normal inputs here, not exceptions to crash on — catch broadly and return None, don't let a cert-inspection add-on take down a response that would otherwise be fine.
It's a second round-trip you didn't need before. If your header audit only reads response headers you already have in memory, bolting on a live handshake by default silently doubles your latency for every caller, most of whom never asked for it.

Make it opt-in, not default

That third point is why I didn't fold this into the existing security-audit endpoint's default behavior — I gated it behind a query flag (include_tls_details=true) that's off unless explicitly requested. The base audit stays header-only and fast; anyone who wants the deeper cert check asks for it and pays the extra round-trip knowingly.

I ended up shipping both fixes — Report-Only-aware header context and opt-in TLS inspection — in the API from my last post:

GitHub: https://github.com/JosejuX/rapidapi-metadata-extractor

Try it: https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor

Thanks again to the commenter who pushed on this — pass/fail without the why is exactly the kind of audit tool that trains people to chase a green checkmark instead of understanding the header.

Grading a Website's Security Headers Automatically (HSTS, CSP, X-Frame-Options) in One API Call

Juanjo — Sun, 16 Aug 2026 09:50:11 +0000

If you've ever run a site through Mozilla Observatory or securityheaders.com and gotten a mediocre grade with no idea what to actually change, this post walks through what each header does, why the common advice ("just add CSP") is more nuanced than it sounds, and how to check this programmatically instead of one URL at a time in a browser tool.

Why security headers matter more than they look like they should

Security headers don't stop a determined attacker with a zero-day. What they do is close off entire classes of cheap, automatable attacks: clickjacking, MIME-sniffing exploits, protocol-downgrade attacks, and a large chunk of XSS. A site with good headers isn't unhackable — it's just not hanging fruit for the scanners and bots that make up the overwhelming majority of real-world probing traffic.

The headers that actually matter, and what "correct" looks like

Strict-Transport-Security (HSTS) — tells the browser to never downgrade to plain HTTP for this domain, even if a link or bookmark points there. The common mistake: setting a short max-age (or none at all), which means the protection lapses the moment a user hasn't visited in a while. max-age=31536000; includeSubDomains is the baseline serious sites use.

Content-Security-Policy (CSP) — the header with the worst reputation, because a naive CSP breaks half the page (inline scripts, third-party widgets, analytics). The honest advice: don't reach for unsafe-inline and unsafe-eval as your first move just to make errors go away — that defeats most of what CSP protects against. Start with default-src 'self' and add specific sources as things break, rather than starting permissive and never tightening it.

X-Frame-Options / frame-ancestors — prevents clickjacking by controlling whether your page can be embedded in an <iframe> on someone else's site. SAMEORIGIN is right for almost everyone; DENY if you never embed your own pages either.

X-Content-Type-Options: nosniff — stops the browser from guessing a file's type based on content instead of the declared Content-Type. Without it, a file uploaded as an "image" that's actually JavaScript can get executed in some edge cases. One line, no downside, frequently missing anyway.

Referrer-Policy — controls how much of your URL gets leaked to third parties (analytics scripts, external links) via the Referer header. strict-origin-when-cross-origin is a reasonable default: full URL on same-origin requests, just the origin cross-origin, nothing over plain HTTP.

What a grading system should actually check

A binary "present/absent" check per header is a start, but it misses the cases that matter most: an HSTS header with max-age=0 (present, but functionally disabled), a CSP that's technically set but allows unsafe-inline for scripts (present, but not doing its job), or a Referrer-Policy set to unsafe-url (present, actively leaking more than having no policy at all in some browsers' default behavior). Grading each header's value, not just its presence, is what separates a useful audit from a checkbox exercise.

Doing this programmatically instead of one tab at a time

If you're auditing more than a handful of pages — a competitor teardown, a pre-launch checklist across a multi-page site, or a recurring monitor — checking headers by hand in browser dev tools doesn't scale. A single API call can fetch the page, read the response headers, grade each one, and return a percentage score plus per-header grades:

{
  "url": "https://example.com",
  "security_score_percentage": 61.7,
  "security_headers": {
    "strict_transport_security": "max-age=300;includeSubdomains",
    "content_security_policy": "default-src 'self'; script-src 'self' 'unsafe-inline'...",
    "x_frame_options": "SAMEORIGIN",
    "x_content_type_options": "nosniff",
    "referrer_policy": "no-referrer-when-downgrade",
    "permissions_policy": "autoplay=(), camera=(), ..."
  },
  "security_header_grades": {
    "strict_transport_security": "weak",
    "content_security_policy": "weak",
    "x_frame_options": "strong"
  }
}

Note the max-age=300 on HSTS in that example — technically present, graded "weak" because 5 minutes is far too short to matter in practice. That's the kind of nuance a presence-only check would miss entirely.

A real comparison: 61.7% vs. 93.3%

To see the scoring actually differentiate between sites, here's the same audit run against two real, live pages.

techcrunch.com — 61.7%: strong on X-Frame-Options and X-Content-Type-Options, reasonable on Referrer-Policy and Permissions-Policy, but weak on both Strict-Transport-Security (max-age=300 — five minutes) and Content-Security-Policy (contains unsafe-inline). Two present-but-weak headers are enough to drag an otherwise decent setup down by nearly 40 points.

This API's own demo site — 93.3%: strong across Strict-Transport-Security (max-age=63072000; includeSubDomains), a nonce-based Content-Security-Policy (no unsafe-inline, no wildcard sources), X-Frame-Options: DENY, X-Content-Type-Options: nosniff, and Referrer-Policy: strict-origin-when-cross-origin — only Permissions-Policy lands on "reasonable" instead of "strong", because the grading logic currently treats any non-empty policy as reasonable and doesn't have a "strong" tier for it. That's effectively the ceiling under the current heuristic, not a header that's actually missing anything meaningful.

The gap between those two scores is the entire point of grading values, not just presence — both sites have most of these headers, but one of them actually enforces what the header is supposed to do.

This is what the Web Metadata & Contact Extractor API's /api/v1/security endpoint returns — one call, real per-header grading instead of a pass/fail, and it composes with the same anti-SSRF-protected fetch layer as the rest of the API, so it's safe to point at user-submitted URLs. It's MIT-licensed on GitHub if you'd rather read the grading logic or self-host it, and there's a live demo if you just want to check a URL right now.

Building a Daily Competitor SEO & Tech-Stack Monitor with n8n and a Single API Call

Juanjo — Sat, 15 Aug 2026 18:08:17 +0000

Manually checking a handful of competitor sites every week for SEO regressions or a tech-stack change (a new CMS, a new analytics tool, a new payment provider showing up) doesn't scale, and it's exactly the kind of repetitive check that should just run itself. Here's how to automate it end-to-end with n8n, without writing a scraper.

What "monitoring a competitor" actually requires

Two signals matter most for competitive/SEO monitoring, and they're both annoying to get reliably on your own:

An SEO health score — are they missing meta descriptions, is their robots.txt blocking something it shouldn't, do they have broken canonical tags. This normally means building (or paying for) an auditing tool.
Detected tech stack — did they switch from Shopify to a custom storefront, did they add a new analytics or A/B testing tool, did they start using a new CMS. This normally means either manually inspecting page source/headers, or maintaining your own library of technology fingerprints (script src patterns, meta generator tags, cookie names) — which is a genuinely large, constantly-changing dataset to keep current.

Both of these are solved problems as external APIs, which means the actual automation work is just: read a list of URLs, call an API, compare to yesterday's result, alert on meaningful changes.

The n8n workflow

The shape of this workflow is simple enough to build in an afternoon:

Schedule Trigger — once a day.
Google Sheets (Read) — a Competitors tab with columns url, last_score, last_stack.
Loop over each row.
HTTP Request (SEO audit) and HTTP Request (tech stack) — two parallel calls per URL.
Merge the two responses.
Compare the new score/stack against last_score/last_stack from the sheet — flag if the score dropped more than a threshold (5 points is a reasonable default) or the detected stack changed at all.
IF node — only continue if something actually changed.
Slack/email alert (optional) — post the diff.
Google Sheets (Update) — write the new score/stack back for tomorrow's comparison.

That's the entire scenario. No scraper code, no maintaining your own tech-stack fingerprint database, no parsing HTML yourself.

The part people usually get wrong: comparing state correctly

The easy mistake is comparing the current result to some fixed baseline instead of the previous day's result — which either misses gradual regressions (a score that drops 2 points a day for a week is a 14-point drop nobody notices) or floods you with noise on volatile pages. Reading last_score/last_stack from the same row you're about to update, comparing, then overwriting, is what makes this a rolling day-over-day diff instead of a one-time snapshot.

For the tech-stack comparison specifically, sort the detected technology list before comparing (sorted(stack).join(',') or equivalent) — an API returning the same technologies in a different order between runs will otherwise look like a "change" when nothing actually changed.

Where the API calls come from

For the two HTTP Request nodes above, you need something that returns both an SEO audit score and a tech-stack fingerprint from a single URL — the Web Metadata & Contact Extractor API covers both endpoints (/api/v1/seo-audit and /api/v1/tech-stack) as part of its free tier (1,000 requests/month), which is enough headroom to monitor a meaningful watchlist of competitors daily without hitting limits.

I've published this exact workflow — Google Sheets read, parallel API calls, day-over-day comparison, Slack alert, sheet update — as a ready-to-import n8n template, so you don't have to wire the nodes up from the description above. Drop in your own RapidAPI key and Google Sheet, point it at a Competitors tab with a url column, and it runs daily on its own.

The API itself is open source (MIT license) if you'd rather self-host the extraction logic instead of using the hosted version — the n8n workflow works the same either way, you'd just point the HTTP Request nodes at your own instance.

Turning Any Webpage into Clean, LLM-Ready Markdown for RAG Pipelines

Juanjo — Sat, 15 Aug 2026 18:01:19 +0000

If you're feeding web content into an LLM — for RAG, for an agent's context window, for a summarization pipeline — raw HTML is the wrong input. This post covers why, and the specific extraction problems you'll hit once you try to fix it yourself.

Why raw HTML wastes your context window (and your accuracy)

A typical webpage's HTML is 70-90% markup, navigation, ads, cookie banners, and scripts that have nothing to do with the actual content. Dump that into an LLM prompt and you're paying token cost for <nav> boilerplate, burning context budget that could hold more actual retrieved content, and — worse — sometimes confusing the model with off-topic sidebar links and "related articles" widgets that get treated as part of the main content.

The fix is converting the page to clean Markdown: headings, paragraphs, lists, and links, with the chrome stripped out. This is a solved problem in principle (Readability-style content extraction has existed for years), but there are a few specific failure modes worth knowing about before you build it yourself.

Problem 1: content-extraction heuristics fail on modern SPA sites

Libraries like readability-lxml or newspaper3k work by looking at the static HTML and guessing which <div> holds the "main content" based on text density heuristics. That works fine for a WordPress blog. It fails completely on a React/Next.js/Vue app where the initial HTML response is a nearly empty shell (<div id="root"></div>) and the actual content only exists after client-side JavaScript runs.

If you're scraping server-side (no headless browser), you need to either detect this case and expand your fetch — reading more bytes to catch server-side-rendered (SSR) content that a naive byte-limited fetch would truncate — or fall back to a headless browser, which is much slower and heavier. A reasonable middle ground: detect known SPA framework signatures (React/Next.js/Vue/Angular/Svelte hydration markers) in the first chunk of HTML and adaptively increase how much you read before giving up, rather than always paying headless-browser cost.

Problem 2: "clean" shouldn't mean "lossy"

A lot of Markdown converters strip links entirely, or collapse them to plain text — which is bad for RAG, because the source URL of a claim is often exactly the citation info you want preserved for the agent to reference later. You want a converter that keeps [text](url) link structure intact, keeps heading hierarchy (so a model can tell "this was an H2 under this H1"), and drops navigation/footer/ad content without dropping legitimate body links.

Problem 3: word count and reading time aren't cosmetic — they're routing signals

If you're chunking content for a RAG pipeline, knowing the Markdown's word count before you decide how to chunk it (single chunk vs. split by heading vs. split by fixed token windows) saves a wasted round trip. Most raw-HTML-to-Markdown tools don't return this — you end up computing it yourself downstream, which is fine, but it's one more thing to build.

A minimal example of what "good" output looks like

{
  "markdown_content": "# Article Title\n\nActual paragraph content with [preserved links](https://example.com) and structure...\n\n## A subheading\n\nMore content...",
  "word_count": 842,
  "reading_time_minutes": 4
}

Versus what you get from naive HTML stripping:

Home About Contact Subscribe Article Title Actual paragraph content with preserved
links and structure More content Copyright 2026 Privacy Policy Terms

The difference matters more than it looks like on paper — the second version has already lost the information a downstream LLM needs to understand document structure, and it's mixed navigation text directly into what looks like body content.

Combining this with metadata extraction in one call

If your pipeline needs more than just the Markdown — OpenGraph title/description for a citation card, the page's detected tech stack for competitive research, contact info for lead enrichment — doing all of that as separate scraping steps means separate fetches, separate parsing logic, and separate places for SSRF bugs to creep in (see my post on SSRF-safe URL fetching for why that matters).

This is the exact gap the Web Metadata & Contact Extractor API is built for: one GET request returns clean, link-preserving Markdown (with word count and reading time) alongside SEO/OpenGraph metadata, contacts, social links, and a 40+ signature tech-stack detector — so a RAG ingestion step doesn't need five different tools glued together. It handles the SPA-detection problem with an adaptive byte-limit fetch, and every fetch goes through the same SSRF/DNS-rebinding-safe layer regardless of what data you're asking for.

It's MIT-licensed and open source if you want to see exactly how the Markdown conversion and SPA detection work, or try the live demo with any URL, no signup required.

How to Safely Fetch Metadata from User-Submitted URLs (Without Getting Hit by SSRF or DNS Rebinding)

Juanjo — Sat, 15 Aug 2026 17:53:33 +0000

If your app lets a user paste in a URL — a link preview, a "import from website" button, an AI agent that browses the web on someone's behalf — you have a Server-Side Request Forgery (SSRF) problem whether you've thought about it or not. This post walks through exactly why naive URL fetching is dangerous, why the obvious fixes don't work, and how to actually close the hole.

The naive approach (and why it's broken)

Most "fetch a URL and extract data from it" code looks like this:

import requests

def fetch(url: str):
    response = requests.get(url, timeout=5)
    return response.text

The first instinct to secure this is a blocklist check on the hostname or IP before fetching:

import socket
import ipaddress

def is_safe(url: str) -> bool:
    host = extract_host(url)
    ip = socket.gethostbyname(host)
    addr = ipaddress.ip_address(ip)
    return not (addr.is_private or addr.is_loopback or addr.is_link_local)

This looks reasonable and will pass casual testing. It is still exploitable.

DNS rebinding: the gap the blocklist misses

The check above resolves the hostname, validates the IP, and then — separately — requests.get() resolves the hostname again when it actually opens the connection. Those are two different DNS lookups, at two different points in time, and nothing guarantees they return the same IP.

An attacker controlling the DNS for evil.example.com can configure a very short TTL and serve a legitimate public IP on the first lookup (the one your safety check sees) and then flip the DNS record to 169.254.169.254 (the AWS/GCP/Azure cloud metadata endpoint) or 127.0.0.1 for the second lookup (the one that actually happens when the HTTP client connects). Your check passes. Your fetch hits internal infrastructure anyway.

This is DNS rebinding, and it's not a theoretical attack — it's been used in real SSRF exploits against exactly this pattern: "validate URL, then fetch URL" as two separate steps.

Redirects are the second gap

Even if you solve DNS rebinding, a URL that resolves cleanly to a public IP can still respond with an HTTP redirect to http://169.254.169.254/latest/meta-data/. If your HTTP client follows redirects automatically (most do, by default) and you only validated the original URL, the redirect target never gets checked at all.

The actual fix: resolve once, pin the connection, recheck every hop

The only approach that closes both gaps is to make the IP validation and the connection use the same resolved address, and to repeat that validation on every redirect hop, not just the first request:

Resolve the hostname's DNS once.
Validate that specific IP against private/loopback/link-local/cloud-metadata ranges.
Open the HTTP connection directly to that IP (not the hostname) — while still sending the correct Host header and TLS SNI so the target server routes and certificate-validates correctly.
If the response is a redirect, repeat steps 1–3 for the new location before following it. Never let a redirect chain sneak past validation.

In Python, this means using requests' HTTPAdapter with a custom connection pool (or httpx's transport hooks) to force the socket connection to the pre-validated IP instead of letting the library re-resolve the hostname at connect time. It's more code than a hostname blocklist, and it's the only version that's actually correct.

# Simplified sketch — the real implementation needs to handle
# IPv6, connection pooling, and TLS SNI carefully.
import socket, ipaddress
import httpx

BLOCKED_RANGES = [
    "127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
    "169.254.0.0/16",  # link-local + cloud metadata (169.254.169.254)
]

def resolve_and_validate(host: str) -> str:
    ip = socket.gethostbyname(host)
    addr = ipaddress.ip_address(ip)
    if any(addr in ipaddress.ip_network(r) for r in BLOCKED_RANGES):
        raise ValueError(f"Blocked IP range: {ip}")
    return ip

def safe_fetch(url: str, max_redirects: int = 5):
    for _ in range(max_redirects):
        host = extract_host(url)
        ip = resolve_and_validate(host)  # re-validated on every hop
        response = httpx.get(url, headers={"Host": host}, extensions={"sni_hostname": host})
        # pin the socket to `ip` via a custom transport — omitted here for brevity
        if response.is_redirect:
            url = response.headers["location"]
            continue
        return response
    raise ValueError("Too many redirects")

When this matters most: AI agents fetching arbitrary URLs

This whole class of bug gets a lot more dangerous the moment an LLM agent is the one deciding what URL to fetch — from a prompt, from a tool call, from a webpage it just scraped. A user (or a malicious webpage the agent visited earlier) can steer the agent toward fetching http://169.254.169.254/latest/meta-data/iam/security-credentials/ and exfiltrate your cloud provider's IAM credentials through the agent's own output. If you're building a RAG pipeline, a browsing tool, or any MCP server that accepts a URL parameter from a model, this isn't optional hardening — treat every URL as attacker-controlled input.

If you'd rather not build this yourself

This exact resolve-once/pin-connection/recheck-every-hop pattern is what powers the URL-fetching layer of the Web Metadata & Contact Extractor API — it's a small REST API (and MCP server) that takes any URL and returns metadata, contacts, tech stack, and clean Markdown, with this SSRF/DNS-rebinding protection built in by default. The code is MIT-licensed on GitHub if you want to read the real implementation or self-host it — there's also a live demo you can try with no signup.

Either way — build it yourself with the pattern above, or use something that already has it — the important part is not shipping the naive version.

The Two Security Gaps Every RapidAPI Provider Forgets to Fix

Juanjo — Tue, 11 Aug 2026 20:02:16 +0000

If you've published an API on RapidAPI, there's a good chance it has one (or both) of these two holes.

1. Your real backend URL isn't actually secret

RapidAPI's gateway is supposed to be the only way to reach your API. In practice, your real URL leaks constantly — a log line, a Host header, a curious subscriber poking around. Once someone has it, they call your backend directly and skip RapidAPI's billing entirely.

The fix is one dependency:

from fastapi import Header, HTTPException

def verify_rapidapi_secret(x_rapidapi_proxy_secret: str | None = Header(None)):
    if x_rapidapi_proxy_secret != RAPIDAPI_PROXY_SECRET:
        raise HTTPException(status_code=403, detail="Forbidden")

Add Depends(verify_rapidapi_secret) to a route, configure the same secret in RapidAPI Studio's header-injection settings, and that route only answers to traffic that actually came through the gateway.

2. If your API fetches a URL, it's an SSRF vector

Any endpoint that takes a URL and fetches it server-side (link previews, "extract data from this page," scrapers) is a pivot point into your own private network or your cloud provider's metadata endpoint (169.254.169.254). This is the single most common way a "fetch this for me" API turns into a real security incident.

The naive fix — checking the hostname before fetching — doesn't work, because of DNS rebinding: the name resolves to something safe at check time and something private at connect time. The fix has to resolve the DNS once, pin the IP, and connect to that exact IP:

ip = await resolve_and_validate(hostname)  # blocks private/loopback/reserved/metadata ranges
async with httpx.AsyncClient() as client:
    resp = await client.get(url, extensions={"sni_hostname": hostname})

And every redirect hop needs the same check re-run from scratch — a public URL that redirects to http://169.254.169.254/ is a very old trick.

Why I'm writing this

I built both of these (plus rate limiting, a circuit breaker, caching, and observability) for a RapidAPI-published API I run in production. After doing it once, I didn't want to rebuild the same plumbing for the next one, so I extracted it into a standalone FastAPI starter kit — tested (39 tests, including a full SSRF attack-matrix suite), with CI, and a hardened Docker image.

If you're publishing on RapidAPI and don't want to rebuild this yourself: FastAPI + RapidAPI Starter Kit

Happy to answer questions about the SSRF/DNS-rebinding part in the comments — it's a more interesting problem than it looks.

Building an Enterprise-Grade Web Metadata & Contact Extractor API (<200ms) with Python, FastAPI & C-Lexbor

Juanjo — Sun, 09 Aug 2026 15:03:15 +0000

Hello DEV Community! 👋

When building link preview cards (like WhatsApp or Slack), public contact-discovery pipelines, or web scrapers for AI/LLM RAG agents, fetching webpage metadata reliably and fast is surprisingly difficult.

Traditional scrapers suffer from:

🐢 High Latency: Heavy DOM parsers taking 1-3 seconds per request.
🚨 SSRF Vulnerabilities: DNS Rebinding & redirect attacks accessing internal cloud metadata.
📦 Broken SPA Rendering: Downloading tiny empty HTML shells from React/Next.js apps.

To solve this, I built the Web Metadata, OpenGraph & Contact Extractor API (v4.1.0) — an ultra-fast REST API designed for high-concurrency production workloads.

⚡ Key Tech Stack & Performance Architecture

C-Lexbor HTML Engine (selectolax): Parses HTML DOM tree 10x faster than BeautifulSoup.
Rust ORJSON Engine: Ultra-fast JSON response serialization.
Non-Blocking Async DNS Resolution: Uses asyncio.to_thread with an in-memory 5-minute DNS TTL cache to eliminate event-loop blocking.
Adaptive SPA Byte Limit (64 KB ➔ 256 KB): Automatically detects React, Next.js, Vue, Angular, Svelte, or Astro and dynamically expands the streaming buffer to capture server-side rendered data.
IP-Pinned Anti-SSRF Shield: Resolves DNS once, validates target IP against private/loopback/cloud metadata ranges (169.254.169.254), and pins the HTTP connection directly with TLS SNI headers.
Normalized 15-Min TTL In-Memory Cache: Repeat URL extractions complete in < 0.01 ms internal server time.

💎 What Data Does It Extract?

In a single GET request, the API returns a structured JSON payload containing:

SEO & OpenGraph Metadata: Title, Description, OG Image, OG Type, OG URL, OG Video, Favicon, Canonical URL, Language, Author, Theme Color, Robots directive (index, follow), and hreflang internationalization tags.
🛍️ E-Commerce Product Data: Automatically extracts Product Schema.org JSON-LD (Name, Price, Currency, Availability, Brand, Ratings & Reviews).
📧 Public Contacts: Public emails and telephone numbers with smart DOM script/style cleaning.
📲 Social Profiles: Twitter/X, LinkedIn, Facebook, Instagram, GitHub, YouTube, Telegram, TikTok.
🛠️ 40+ Tech Stack Detector: Identifies WordPress, Shopify, WooCommerce, Webflow, React, Next.js, Vue, Angular, TailwindCSS, Stripe, GA4, etc.
🤖 AI & LLM Clean Markdown: Converts web articles into clean Markdown for ChatGPT, Claude, and RAG pipelines (includes word count & reading time).
📊 SEO Audit Score: 14-point automated diagnostic score (0-100%) with actionable warnings.
🔒 Security Headers Audit: HSTS, CSP, X-Frame-Options, Referrer Policy score.

🚀 Quick Code Examples

JavaScript / Node.js

const url = 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com';
const response = await fetch(url, {
  headers: {
    'X-RapidAPI-Key': 'YOUR_RAPIDAPI_KEY',
    'X-RapidAPI-Host': 'web-metadata-and-contact-extractor.p.rapidapi.com'
  }
});

const data = await response.json();
console.log('Title:', data.metadata.title);
console.log('Product:', data.product_data);
console.log('Tech Stack:', data.detected_technologies);
console.log('Execution Time:', data.execution_time_ms, 'ms');

Python

import requests

url = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"
headers = {
    "X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY",
    "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"
}
response = requests.get(url, headers=headers, params={"url": "https://stripe.com"})
print(response.json())

🔑 Try It Free on RapidAPI

The API is live and available on RapidAPI with 1,000 free requests/month:

👉 Get your free API Key on RapidAPI
👉 GitHub Repository

I'd love to hear your feedback, thoughts on the architecture, or feature requests!

This piece is also cross-posted on Hashnode and, as a shorter companion version, on Medium.

DEV Community: Juanjo

I scanned the top 1,000 websites for security headers. The median score is 37/100.

How the scan works

About the denominator

Finding 1: most of the top sites are missing most of the headers

Finding 2: the score distribution is bottom-heavy

Finding 3: the tech stack predicts the posture

Finding 4: SEO effort and security effort barely correlate

Try it on your own site

This API vs ScrapingBee vs Firecrawl vs BuiltWith: an honest comparison

The one-line version

Pricing, side by side

When each one actually wins

The gap none of us solve for free

Building a B2B Prospecting Agent in 15 Minutes with CrewAI and a Single API Call

The problem

Setup

The tools

Building the crew

What the qualifier actually has to work with

Where this goes from here

Links

Firecrawl vs. Jina Reader vs. Tavily: Picking the Right Web-to-LLM Tool (and When You Need None of Them)

What each one is actually optimized for

Pricing and limits, compared

Where the narrower tool wins

Where it genuinely doesn't compete

The actual takeaway

Grading Security Headers Isn't the Full Picture

Headers tell you intent. TLS tells you reality.

Doing the handshake for real

Make it opt-in, not default

Grading a Website's Security Headers Automatically (HSTS, CSP, X-Frame-Options) in One API Call

Why security headers matter more than they look like they should

The headers that actually matter, and what "correct" looks like

What a grading system should actually check

Doing this programmatically instead of one tab at a time

A real comparison: 61.7% vs. 93.3%

Building a Daily Competitor SEO & Tech-Stack Monitor with n8n and a Single API Call

What "monitoring a competitor" actually requires

The n8n workflow

The part people usually get wrong: comparing state correctly

Where the API calls come from

Turning Any Webpage into Clean, LLM-Ready Markdown for RAG Pipelines

Why raw HTML wastes your context window (and your accuracy)

Problem 1: content-extraction heuristics fail on modern SPA sites

Problem 2: "clean" shouldn't mean "lossy"

Problem 3: word count and reading time aren't cosmetic — they're routing signals

A minimal example of what "good" output looks like

Combining this with metadata extraction in one call

How to Safely Fetch Metadata from User-Submitted URLs (Without Getting Hit by SSRF or DNS Rebinding)

The naive approach (and why it's broken)

DNS rebinding: the gap the blocklist misses

Redirects are the second gap

The actual fix: resolve once, pin the connection, recheck every hop

When this matters most: AI agents fetching arbitrary URLs

If you'd rather not build this yourself

The Two Security Gaps Every RapidAPI Provider Forgets to Fix

1. Your real backend URL isn't actually secret

2. If your API fetches a URL, it's an SSRF vector

Why I'm writing this

Building an Enterprise-Grade Web Metadata & Contact Extractor API (<200ms) with Python, FastAPI & C-Lexbor

⚡ Key Tech Stack & Performance Architecture

💎 What Data Does It Extract?

🚀 Quick Code Examples

JavaScript / Node.js

Python

🔑 Try It Free on RapidAPI