DEV Community: Daniel Igel

Scrape any URL with one POST — Cloudflare bypass included

Daniel Igel — Mon, 06 Jul 2026 14:18:44 +0000

Most scraping libraries fall apart the moment they hit a Cloudflare-protected page — you get a 403 or an infinite challenge loop. Running a headless browser locally isn't an option in a serverless or edge environment, and maintaining your own bypass infrastructure is a project in itself.

One POST fetches any URL — Cloudflare-protected or not — and returns clean HTML, stripped text, or both in a single response:

curl --request POST \
  --url 'https://web-scraping-api.p.rapidapi.com/api/v1/scrape' \
  --header 'Content-Type: application/json' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: web-scraping-api.p.rapidapi.com' \
  --data '{"url":"https://example.com","output":"text"}'

Set output to html for the raw markup, text for the stripped content (scripts and styles removed), or omit it to get both fields in one response alongside the final URL and HTTP status code.

When you only need specific data points, use the /extract endpoint with a CSS selector map:

const res = await fetch(
  'https://web-scraping-api.p.rapidapi.com/api/v1/extract',
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-rapidapi-key': process.env.RAPIDAPI_KEY,
      'x-rapidapi-host': 'web-scraping-api.p.rapidapi.com',
    },
    body: JSON.stringify({
      url: 'https://example.com/products',
      selectors: {
        title: 'h1',
        prices: '.price[]',
        thumbnail: 'img.hero@src',
      },
    }),
  }
);
const { data } = await res.json();
// { title: "Widget Pro", prices: ["$9.99", "$19.99"], thumbnail: "https://..." }

Append [] to a selector to collect all matching elements as an array; add @attr to pull an attribute value instead of text content. No headless browser on your end — the bypass runs server-side.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/web-scraping-api

What's the site type that's been hardest to scrape reliably in your projects — Cloudflare, heavy SPAs, or login-gated content?

Why your robots.txt is blocking AI crawlers (and how to fix it)

Daniel Igel — Sun, 05 Jul 2026 06:27:29 +0000

A User-agent: * rule in robots.txt blocks every bot that doesn't have its own entry — including the AI crawlers that power ChatGPT Search, Perplexity, and Google AI Overviews.

The bots that feed live AI answers have their own user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, and Google-Extended. If your robots.txt has a wildcard Disallow: without explicit rules for these seven, they inherit the restriction — and your content won't appear in AI citations.

The fix

Add explicit Allow: / entries before your wildcard group:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /private/

Specific rules always win over the wildcard (RFC 9309), so Googlebot and Bingbot are unaffected.

Diagnose your robots.txt in one call

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://yoursite.com"}'

The AI Crawler Access category reports a pass, warn, or fail for each bot individually — including which user-agents are blocked and the exact fix to apply.

const res = await fetch('https://citeready-api.sprytools.com/v1/audit', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ url: 'https://yoursite.com' }),
});
const { categories } = await res.json();
const aiAccess = categories.find(c => c.id === 'ai_access');
console.log(aiAccess.score, aiAccess.checks);

Free at https://citeready.sprytools.com — 3 checks/day, no signup.

Which AI crawler did you find blocked after auditing your site?

10-point AI citation readiness checklist for developers

Daniel Igel — Thu, 02 Jul 2026 21:12:08 +0000

Getting cited by ChatGPT Search, Perplexity, or Google AI Overviews isn't random — there's a specific set of technical signals those systems check. Here are the 10 that matter most.

1. robots.txt is present. Without it, crawlers assume everything is allowed, but you lose explicit AI-bot policy and the Sitemap: hint.

2. AI search bots are not blocked. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, and Google-Extended feed live AI answers. A Disallow: / that catches any of them removes your site from AI citations.

3. Snippets are not restricted. <meta name="robots" content="nosnippet">, noindex, or max-snippet:0 — in either a meta tag or an X-Robots-Tag response header — tell AI engines to skip quoting your content.

4. JSON-LD is present and valid. Invalid JSON-LD blocks are silently ignored. Test after every template change.

5. Organization or WebSite schema is present. Identity markup lets AI engines attribute your content to your brand rather than treating the page as anonymous text.

6. Main content is 600+ words. Strip scripts, nav, and footer before counting. Thin pages rarely become AI sources.

7. Paragraphs are 20–120 words each. That's the range AI engines extract quotes from. Break up walls of text; expand one-liners into full thoughts.

8. Page serves over HTTPS with HTTP 200. Insecure pages and error statuses are dropped from AI indexes.

9. Canonical URL is declared. Without it, duplicate variants of the same page split citation signals instead of pooling them.

10. llms.txt is present at /llms.txt. A plain Markdown index of your key pages — AI crawlers increasingly use it to prioritize what to read.

Score all 10 in one call

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://yoursite.com"}'

The response gives a 0–100 score across four categories — AI Crawler Access, Structured Data, Content Citability, Technical Foundation — with a pass/warn/fail on every check and a prioritized fix list.

Free at https://citeready.sprytools.com — 3 checks/day, no signup.

How many of these 10 does your main site currently pass? Curious where developers most often lose points.

Build link previews without parsing HTML: OG metadata in one API call

Daniel Igel — Wed, 01 Jul 2026 13:00:01 +0000

Building link unfurl panels — the preview cards you see in Slack, Notion, or iMessage — means fetching the raw page, parsing og: meta tags, falling back to <title> and <meta name="description"> when OG is missing, and resolving relative favicon paths. It's tedious boilerplate that rarely fits cleanly into whatever HTTP client you're already using.

One GET returns OG tags, title, description, canonical URL, and favicon for any URL:

curl --request GET \
  --url 'https://url-metadata-open-graph-api.p.rapidapi.com/api/v1/meta?url=https://github.com' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: url-metadata-open-graph-api.p.rapidapi.com'

You get back a normalized object — og.title, og.image, og.description, favicon, canonical — regardless of whether the page has OG tags or falls back to standard meta. The response is cached for 10 minutes, so repeated calls to the same URL skip the origin fetch.

const meta = await fetch(
  `https://url-metadata-open-graph-api.p.rapidapi.com/api/v1/meta?url=${encodeURIComponent(url)}`,
  { headers: { 'x-rapidapi-key': process.env.RAPIDAPI_KEY, 'x-rapidapi-host': 'url-metadata-open-graph-api.p.rapidapi.com' } }
).then(r => r.json());

Need to prefetch multiple links at once? POST /api/v1/meta/batch with { urls: [...] } handles up to 5 URLs concurrently — one round trip instead of five.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/url-metadata-open-graph-api

When building link previews, do you parse OG tags yourself or reach for a dedicated service?

Add llms.txt to your site in 5 minutes and get cited by ChatGPT, Perplexity & Co.

Daniel Igel — Thu, 25 Jun 2026 13:00:03 +0000

Most AI search tools — ChatGPT Search, Perplexity, Bing Copilot — crawl the web to find pages they can cite. The problem: they can't always tell which parts of your site are worth reading, or what your site is actually about. llms.txt fixes that.

It's a plain-text Markdown file you put at https://yoursite.com/llms.txt. Three lines are enough to get started.

The format

# Your Site Name

> One-sentence summary of what your site is and who it's for.

## Important pages

- [Page Title](https://yoursite.com/page): What this page covers.
- [Another Page](https://yoursite.com/other): Short description.

The first # heading is your site name. The > blockquote is the summary AI crawlers use as context when deciding whether your site is relevant to a query. Each ## Section lists the pages you want prioritized, with an optional short description per link.

That's the entire format. No build step, no framework, no schema validation — just a file at the right path.

Why it matters

Without llms.txt, an AI crawler has to infer what your site is about from the content of individual pages. It might index your blog posts but miss your documentation. It might understand what your tool does but not know which page to cite for a specific question.

With llms.txt, you give it a map. You're saying: this site is for developers who need X, and the pages most worth reading are these.

Perplexity, ChatGPT Search, and a growing number of AI crawlers actively look for this file. It's not mandatory, but skipping it is a missed signal that costs nothing to add.

A real example

Here's a llms.txt for a developer API tool:

# Sprytools

> Developer APIs for common backend tasks — email validation, IP geolocation, IBAN parsing, currency conversion — accessible behind one key.

## Key pages

- [API Hub & Pricing](https://sprytools.com/api): All APIs, pricing tiers, self-serve key management.
- [Email Validation](https://sprytools.com/tools/email-validation): Syntax + MX + disposable detection.
- [IP Geolocation](https://sprytools.com/tools/ip-geolocation): IPv4/IPv6, country, city, ASN.
- [IBAN Validator](https://sprytools.com/tools/iban-validator): MOD-97, 80+ countries, parsed output.

## About

All APIs share one key. No per-service sign-up. Free tier on each endpoint.

The description is factual, not aspirational. "Developer APIs for common backend tasks" tells a model what this site is in one phrase. "Empowering developers to build better APIs faster" does not.

Where to put it

The file goes at the root: https://yourdomain.com/llms.txt. Subdirectory paths won't be auto-discovered.

Static site / Next.js: drop it in public/:

public/
  llms.txt

Express or Hono: serve your public/ directory as static files, or add a route:

app.get('/llms.txt', (c) => {
  return c.text(`# My Site\n\n> Description.\n\n## Key pages\n\n- [Home](https://example.com/)\n`);
});

Astro: same as Next.js — put it in public/llms.txt.

After deploying, confirm it's reachable:

curl -I https://yoursite.com/llms.txt
# Should return HTTP/2 200, content-type: text/plain

Point to it from robots.txt

Add a pointer in your robots.txt file alongside the Sitemap directive:

Sitemap: https://yoursite.com/sitemap.xml
LLMs: https://yoursite.com/llms.txt

The LLMs: line isn't part of the original robots.txt spec, but it mirrors the established Sitemap: convention. Some parsers already read it, and it serves as documentation for anyone reading the file manually.

Common mistakes

Wrong path — must be /llms.txt at the root, not /docs/llms.txt or /blog/llms.txt. AI crawlers look for it at the root; subdirectory paths won't be found automatically.

Blocking AI crawlers in robots.txt while adding llms.txt — the crawlers still need Allow in robots.txt to fetch either file. A Disallow: / wildcard blocks llms.txt too.

Generic description — the > blockquote is the most important line. "Platform for developers" tells a model nothing. "REST API for email validation, DNS lookup, and IBAN parsing" tells it exactly what kind of queries your site can answer.

Missing UTF-8 — save as UTF-8 without BOM. Non-ASCII characters in site names or descriptions can cause parse errors in some crawlers.

Verify with CiteReady

After deploying, you can verify the full picture — llms.txt plus the four other AI readiness signals — in one call:

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://yoursite.com"}'

The llms_txt check in the response tells you whether the file was found and parseable:

"llms_txt": {
  "status": "pass",
  "score": 20,
  "details": "llms.txt found at /llms.txt"
}

A missing or unreachable file returns fail. A file with wrong content-type or a 404 after deploy is the most common issue — worth verifying with a curl -I before assuming it's working.

Free tier at https://citeready.sprytools.com — 3 checks/day, no signup.

Have you added llms.txt to any of your projects? Curious whether anyone has noticed a measurable change in Perplexity or ChatGPT citations after adding it.

How to audit your site for AI search readiness (step-by-step with CiteReady)

Daniel Igel — Wed, 24 Jun 2026 14:33:49 +0000

Your site ranks well — but AI search engines like ChatGPT, Perplexity, and Google's AI Overviews still won't cite it. Classic SEO tools measure rankings; they don't tell you whether GPTBot can crawl your page or whether your content is structured enough for a model to lift a clean passage.

Here's how to audit and fix that, category by category.

Step 1: Run your first audit (free, no signup)

Paste your URL into https://citeready.sprytools.com or call the API directly:

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://yoursite.com"}'

You get a score 0–100, a letter grade (A–F), and a per-category breakdown with pass/warn/fail on each check. The response looks like this:

{
  "score": 54,
  "grade": "D",
  "categories": {
    "ai_crawlers": { "status": "fail", "score": 0, "recommendation": "Unblock PerplexityBot, GPTBot in robots.txt" },
    "llms_txt": { "status": "fail", "score": 0, "recommendation": "Add llms.txt to improve LLM discoverability" },
    "structured_data": { "status": "warn", "score": 10 },
    "citability": { "status": "pass", "score": 24 },
    "technical": { "status": "pass", "score": 20 }
  }
}

That score of 54 is surprisingly common. Let's walk through each category.

Step 2: Understand the five categories

AI Crawler access

AI systems need to crawl your pages before they can cite them. The problem is that many sites block them accidentally. A common robots.txt pattern:

User-agent: *
Disallow: /

This blocks every crawler — including GPTBot (ChatGPT), OAI-SearchBot (OpenAI's separate indexing crawler), PerplexityBot, ClaudeBot, and Google-Extended (used for AI Overviews). You won't see warnings in Google Search Console because these aren't the traditional Googlebot.

Even a partial block causes problems. Disallow: /blog/ hides your best content from AI citation. Disallow: /api/ is usually fine, but check that you're not also blocking content pages by proximity.

llms.txt

llms.txt is a plain Markdown file at yourdomain.com/llms.txt. It tells language models what your site contains and where your important content is — similar in spirit to robots.txt, but written for models to read, not crawlers to obey.

It's optional, but it's a cheap signal. A three-line file can move you from fail to pass on this category.

JSON-LD structured data

Schema.org markup gives AI systems a machine-readable description of your content. Without it, a model has to infer your page's topic, author, and date from prose. With it, that context is unambiguous.

The most relevant types for AI citation: Article, WebPage, FAQPage, HowTo, WebApplication. A noindex page with perfect JSON-LD still scores zero — structured data amplifies citability, it doesn't create it.

Content citability

This is the hardest category to improve quickly because it depends on how you write. AI engines prefer passages that are self-contained — a sentence the model can quote without the surrounding paragraphs to make sense of it.

Low-citability: "As mentioned earlier, the tool integrates with all major platforms seamlessly."

High-citability: "CiteReady checks five signals: AI crawler access, llms.txt, JSON-LD, passage citability, and technical hygiene. Each category returns a pass/warn/fail with a specific fix."

The second version can be lifted from the article and quoted accurately. The first can't be cited without context and doesn't contain a quotable claim.

Technical signals

A few checks that either block AI content extraction entirely or reduce confidence:

HTTPS — most AI crawlers skip HTTP-only pages
nosnippet meta tag — this explicitly prevents excerpt extraction; if it's set, your citability score is zero regardless of content quality
Canonical tags — the canonical should point to the URL you want cited; a chain of redirects adds ambiguity
Sitemap — a sitemap at /sitemap.xml helps crawlers discover content; the robots.txt Sitemap: directive points them to it
lang attribute — language identification helps models pick the right passages for a given query

Step 3: Close the gaps

Fix AI crawler blocks first

If the audit returns fail on AI crawlers, this is your highest-leverage fix. Add explicit allow rules before any blocking rules:

# Allow AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# Your existing rules below
User-agent: *
Disallow: /private/
Disallow: /admin/

If you have a wildcard Disallow: /, you'll need to restructure the file. Crawler-specific rules listed before the wildcard take precedence in most parsers.

Remove nosnippet

If you have <meta name="robots" content="nosnippet"> anywhere, remove it unless you have a specific reason for it (legal content you don't want excerpted, for example). This one tag blocks all AI passage extraction.

Add JSON-LD

A minimal Article schema takes 10 minutes and typically improves your score by 10–15 points:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Your page title",
  "description": "One-sentence summary",
  "author": {
    "@type": "Person",
    "name": "Your Name"
  },
  "datePublished": "2026-06-23",
  "url": "https://yoursite.com/this-page"
}
</script>

For tool or SaaS pages, use WebApplication or SoftwareApplication instead of Article.

Create llms.txt

Drop a file at yourdomain.com/llms.txt:

# Site Name

> One-sentence description of what this site is and who it's for.

## Key pages

- [Home](https://example.com/)
- [Docs](https://example.com/docs/)
- [About](https://example.com/about/)

That's it. The description line (the > blockquote) is the most important part — write it as a factual summary, not a marketing tagline.

Improve prose citability (longer-term)

The structural fixes above move most sites from the 40–60 range to 65–80. Getting above 80 requires improving the prose itself:

Open each section with the conclusion, not the setup
Keep paragraphs to 3–4 sentences with a single claim each
Replace vague language with specific numbers, names, and outcomes
Avoid forward/backward references ("as mentioned above", "we'll cover this later")

Run the audit again after each change to measure the effect. The per-category breakdown tells you which fix had the most impact.

What to expect after fixing

A typical site that fixes crawler blocks + removes nosnippet + adds basic JSON-LD goes from ~50 to ~75 in one afternoon. The citability score moves more slowly because it depends on content rewriting, but it's the signal with the highest ceiling.

Free tier: 3 audits/day, no signup → https://citeready.sprytools.com

What's the most surprising thing the audit flags on your site? I'm curious whether structured data gaps or crawler blocks are more common in the dev.to audience.

What is GEO and why your site needs it in 2026

Daniel Igel — Tue, 23 Jun 2026 11:45:50 +0000

Traditional SEO optimizes for ranking. GEO (Generative Engine Optimization) optimizes for being cited. ChatGPT Search, Perplexity, and Google's AI Overviews don't send users down a list of blue links — they pick a handful of sources to quote inline. If your site can't be crawled and understood by AI systems, you're invisible in that layer no matter how well you rank.

The signals are different from classic SEO. There are four categories that matter:

AI crawler access — does your robots.txt allow GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended? A surprising number of sites accidentally block them with Disallow: / or a wildcard User-agent: * rule.
Structured data — JSON-LD (Article, WebPage, Organization) gives AI engines a machine-readable handle on what your page is about.
Content citability — can a model lift a clean, self-contained passage? Dense prose with a clear claim scores better than generic marketing copy.
Technical foundation — HTTPS, canonical tags, and no noindex/nosnippet directives that prevent excerpt extraction.

You can audit any public URL in 30 seconds — no API key, no signup:

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://yoursite.com"}'

The response is a score 0–100 with a letter grade (A–F), a per-category breakdown, and concrete next steps — e.g. "Unblock PerplexityBot in robots.txt — blocking it removes your site from Perplexity citations."

Prefer a UI? https://citeready.sprytools.com — free tier is 3 audits/day.

What's the GEO gap you keep seeing that most developers overlook?

Validate EU VAT numbers against VIES (with company lookup) in one GET

Daniel Igel — Thu, 18 Jun 2026 13:00:02 +0000

If you sell B2B in the EU, you need valid VAT numbers for reverse-charge invoicing. The official source is the EU's VIES service — but its SOAP endpoint is clunky, rate-limited, and occasionally down per member state, so you end up writing retry + parsing glue.

This wraps VIES in a clean REST call across all 27 member states (+ XI for Northern Ireland), with a 1-hour cache:

curl --request GET \
  --url 'https://eu-vat-number-validator-api.p.rapidapi.com/api/v1/validate?vat=DE811569869' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: eu-vat-number-validator-api.p.rapidapi.com'

GET /api/v1/lookup?vat=... additionally returns the registered company name and address (where the member state exposes it — useful to prefill an invoice). POST /api/v1/validate/batch validates several numbers at once.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/eu-vat-number-validator-api

Built this to gate EU B2B signups for reverse-charge billing. How does your stack handle VAT validation — VIES directly, or a provider?

Validate & parse IBANs (ISO 13616, 80+ countries) without a paid banking API

Daniel Igel — Wed, 17 Jun 2026 13:00:01 +0000

A "valid" IBAN isn't just a regex — it's the MOD-97 checksum over a country-specific length and structure. Get that wrong and you accept payment details that bounce later. Most devs either copy a sketchy gist or pay for a banking API they don't otherwise need.

This validates the checksum and structure for 80+ countries, and can parse an IBAN into its parts:

curl --request GET \
  --url 'https://iban-validator-parser-api.p.rapidapi.com/api/v1/validate?iban=DE89370400440532013000' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: iban-validator-parser-api.p.rapidapi.com'

GET /api/v1/parse?iban=... returns the country, bank code, account number and a pretty-printed form (DE89 3704 0044 0532 0130 00). There's POST /api/v1/validate/batch for up to 100 IBANs, and GET /api/v1/countries for the supported list.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/iban-validator-parser-api

Built this for a checkout form so I didn't have to vendor MOD-97 logic. What are you using for IBAN checks today?

Block abusive IPs before they hit your app: one call against 7 DNSBLs

Daniel Igel — Tue, 16 Jun 2026 13:00:01 +0000

Querying DNSBLs by hand means reversing the IP, appending each blacklist's zone, doing a DNS query per list, and interpreting the return codes — times seven if you want decent coverage. It's a lot of boilerplate for "is this IP a known spam/abuse source?".

One GET checks an IP against Spamhaus, Barracuda, SORBS, SpamCop, UCEPROTECT and more, and tells you which lists flagged it:

curl --request GET \
  --url 'https://ip-reputation-blacklist-checker-api.p.rapidapi.com/api/v1/check?ip=1.2.3.4' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: ip-reputation-blacklist-checker-api.p.rapidapi.com'

You get a per-list breakdown plus an overall verdict. GET /api/v1/lists shows the active blacklists, and POST /api/v1/check/batch checks up to 10 IPs at once — good for scoring signups or filtering a request log.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/ip-reputation-blacklist-checker-api

I built this to gate form submissions without standing up my own DNSBL resolver. How do you currently screen inbound IPs?

Is your site visible to AI search? Check it with one API call

Daniel Igel — Mon, 15 Jun 2026 13:00:02 +0000

AI assistants — ChatGPT Search, Perplexity, Google's AI Overviews — increasingly answer questions by citing web pages instead of sending a click. Whether your site can be crawled, parsed, and quoted by them comes down to a handful of signals most SEO tools never check:

Do your robots.txt rules actually let AI crawlers in (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot)? A surprising number of sites block them by accident.
Is there an llms.txt pointing models at your key content?
Is your content marked up with JSON-LD structured data?
Are your passages written so a model can lift a clean, self-contained citation?

I built a small audit engine that scores a URL 0–100 across exactly these dimensions and tells you what's failing. It's keyless to try:

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com"}'

You get a per-category breakdown — AI crawler access, llms.txt, structured data, citability, technical — with a pass/warn/fail on each check and a concrete recommendation, e.g. "No robots.txt found — you lose the Sitemap hint and explicit AI-bot policy."

Prefer pasting a URL? There's a web version: https://citeready.sprytools.com — free tier is 3 audits/day, no signup.

Curious what scores people get here — what's the most surprising thing it flags on your site?

DNS lookups (A/MX/TXT/reverse) as a single JSON endpoint — no resolver wrangling

Daniel Igel — Sun, 14 Jun 2026 09:23:34 +0000

Doing DNS lookups from app code is more annoying than it should be: dig isn't available everywhere, Node's dns/promises returns a different shape per record type, and you end up writing a switch statement for A vs MX vs TXT vs SOA every single time.

This wraps all of that into one GET. Ask for a single type, or pass type=ALL to get A, AAAA, MX, TXT, NS, CNAME and SOA in one round-trip.

curl --request GET \
  --url 'https://dns-lookup-api3.p.rapidapi.com/api/v1/lookup?domain=example.com&type=ALL' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: dns-lookup-api3.p.rapidapi.com'

There's also /api/v1/mx?domain=gmail.com (MX sorted by priority — handy for mail-deliverability checks), /api/v1/reverse?ip=8.8.8.8 for reverse DNS / PTR, and POST /api/v1/lookup/batch for up to 20 domain/type pairs in one call (useful for auditing a list of domains).

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/dns-lookup-api3

Built this because every project re-implemented the same record-type plumbing. What do you reach for today — shelling out to dig, or a library?