DEV Community: Evan

Turn any web page into LLM-ready Markdown for RAG

Evan — Fri, 30 Oct 2026 08:57:53 +0000

Turn any web page into LLM-ready Markdown for RAG

Retrieval-augmented generation (RAG) works better when documents are clean prose instead of raw HTML. Navigation bars, scripts, and layout noise waste tokens and confuse chunkers.

This tutorial shows a practical pattern: fetch a public URL, convert the main content to Markdown, chunk it, and (optionally) embed the chunks. We’ll call the public URL to Markdown (LLM-ready) API on RapidAPI:

https://rapidapi.com/evan-DewNPsuYp/api/url-to-markdown-llm-ready

You’ll need a RapidAPI key. Start on the free plan while you prototype.

Why Markdown?

Headings become natural chunk boundaries (#, ##, …).
Lists and tables survive in a form LLMs already understand.
Token counts are closer to what you send the model than raw HTML is.

This API is static HTML only — it does not run JavaScript. Sites that render all content client-side may return little text. It also rejects oversized HTML (about 200 KB body cap) with HTTP 413. Those limits are fine for many blogs and docs pages; know them before you design a crawler.

1. Subscribe and set the key

export RAPIDAPI_KEY='your-key-here'

Host header (from the public listing): url-to-markdown-llm-ready.p.rapidapi.com

2. Fetch Markdown

curl --request GET \
  --url 'https://url-to-markdown-llm-ready.p.rapidapi.com/v1/markdown?url=https%3A%2F%2Fexample.com&include_links=true&include_images=false' \
  --header "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  --header 'X-RapidAPI-Host: url-to-markdown-llm-ready.p.rapidapi.com'

Successful responses look like:

{
  "ok": true,
  "title": "Example Domain",
  "markdown": "This domain is for use in documentation examples...",
  "chars": 156,
  "tokensEstimate": 39,
  "truncated": false,
  "contentSource": "body",
  "finalUrl": "https://example.com/"
}

tokensEstimate is a rough chars / 4 heuristic — useful for budgeting, not a substitute for your model’s tokenizer.

Useful query flags:

Param	Default	Meaning
`include_links`	true	Keep `[text](url)` links
`include_images`	false	Emit `![alt](src)` when true
`include_tables`	true	Keep GitHub-style tables
`max_chars`	200000	Soft-truncate long Markdown

3. Node loader for a RAG pipeline

// markdownLoader.mjs
const HOST = "url-to-markdown-llm-ready.p.rapidapi.com";

export async function urlToMarkdown(targetUrl, opts = {}) {
  const endpoint = new URL(`https://${HOST}/v1/markdown`);
  endpoint.searchParams.set("url", targetUrl);
  endpoint.searchParams.set("include_links", String(opts.includeLinks ?? true));
  endpoint.searchParams.set("include_images", String(opts.includeImages ?? false));
  endpoint.searchParams.set("include_tables", String(opts.includeTables ?? true));
  if (opts.maxChars) endpoint.searchParams.set("max_chars", String(opts.maxChars));

  const res = await fetch(endpoint, {
    headers: {
      "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
      "X-RapidAPI-Host": HOST,
    },
  });
  const data = await res.json();
  if (!res.ok || !data.ok) {
    const err = new Error(data.error || `HTTP ${res.status}`);
    err.code = data.code;
    throw err;
  }
  return data;
}

/** Naive heading-aware splitter — replace with your preferred chunker. */
export function chunkMarkdown(md, { maxChars = 1200 } = {}) {
  const parts = md.split(/\n(?=#{1,6} )/g);
  const chunks = [];
  let buf = "";
  for (const part of parts) {
    if ((buf + "\n" + part).length > maxChars && buf) {
      chunks.push(buf.trim());
      buf = part;
    } else {
      buf = buf ? buf + "\n" + part : part;
    }
  }
  if (buf.trim()) chunks.push(buf.trim());
  return chunks;
}

const page = await urlToMarkdown("https://example.com", { maxChars: 20000 });
const chunks = chunkMarkdown(page.markdown);
console.log(page.title, chunks.length, page.tokensEstimate);

4. Plug into embeddings (sketch)

Pseudocode only — swap in your embedding provider:

for (const [i, text] of chunks.entries()) {
  // const vector = await embed(text);
  // await vectorDb.upsert({ id: `${page.finalUrl}#${i}`, text, vector, meta: { title: page.title } });
  console.log(`chunk ${i}:`, text.slice(0, 80));
}

At query time, embed the user question, retrieve top-k chunks, and pass them as context to your LLM.

Failure modes worth handling

Symptom	Likely cause
HTTP 413 / `too_large`	HTML over ~200 KB — try a cleaner URL or a different extractor
Sparse Markdown	JS-rendered SPA — need a browser-based tool instead
`ssrf_blocked`	Private/localhost URL — expected safety behavior
`upstream_error`	Target site 4xx/5xx

Need a link-preview card (title + image) instead of full Markdown? Use URL Metadata & Open Graph Extractor.

Keep keys on the server, cache by URL, and stay inside your RapidAPI quota while you validate retrieval quality. Once chunk quality looks good on a handful of real pages, wire the same loader into LangChain / LlamaIndex custom document loaders the same way.

Build link previews in 5 minutes

Evan — Fri, 30 Oct 2026 08:57:52 +0000

Build link previews in 5 minutes

When someone pastes a URL into your chat app, bookmark tool, or CMS, you usually want a small card: title, description, and an image. Parsing Open Graph tags yourself is tedious and easy to get wrong (redirects, relative image URLs, favicon fallbacks).

This walkthrough builds a minimal link-preview helper that calls a metadata API over HTTPS and renders a card. We’ll use the public URL Metadata & Open Graph Extractor on RapidAPI:

https://rapidapi.com/evan-DewNPsuYp/api/url-metadata-open-graph-extractor

You need a free RapidAPI account and an API key from the dashboard. The listing’s free tier is enough to try this locally.

What you get back

One GET request with a url query parameter returns JSON roughly like:

title, description, image, favicon
openGraph / twitter objects
canonical, siteName, jsonLd when present

The service fetches public HTML only (no login walls, no headless browser). SPA shells that don’t put meta tags in the initial HTML may return sparse results — that’s expected for v1.

1. Grab a key

Open the listing above and subscribe to a plan (start with Basic/Free if available).
Copy your X-RapidAPI-Key from https://rapidapi.com/developer/dashboard .
Host header for this API (from the public page): url-metadata-open-graph-extractor.p.rapidapi.com

Store the key in an environment variable — never commit it.

export RAPIDAPI_KEY='your-key-here'

2. One request with curl

curl --request GET \
  --url 'https://url-metadata-open-graph-extractor.p.rapidapi.com/v1/metadata?url=https%3A%2F%2Fwww.wikipedia.org%2F' \
  --header "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  --header 'X-RapidAPI-Host: url-metadata-open-graph-extractor.p.rapidapi.com'

You should see JSON with "ok": true, a title like "Wikipedia", a description, and an image URL pointing at the Wikipedia logo.

3. Tiny Node helper

// preview.mjs
const HOST = "url-metadata-open-graph-extractor.p.rapidapi.com";

export async function fetchLinkPreview(targetUrl) {
  const endpoint = new URL(`https://${HOST}/v1/metadata`);
  endpoint.searchParams.set("url", targetUrl);

  const res = await fetch(endpoint, {
    headers: {
      "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
      "X-RapidAPI-Host": HOST,
    },
  });

  const data = await res.json();
  if (!res.ok || !data.ok) {
    const err = new Error(data.error || `HTTP ${res.status}`);
    err.code = data.code;
    throw err;
  }

  return {
    title: data.title,
    description: data.description,
    image: data.image,
    favicon: data.favicon,
    url: data.finalUrl || data.url,
  };
}

// demo
const preview = await fetchLinkPreview("https://www.wikipedia.org/");
console.log(preview);

Run with:

node preview.mjs

4. Render a card (plain HTML)

function renderCard(p) {
  const img = p.image
    ? `<img src="${p.image}" alt="" width="120" height="120" />`
    : "";
  return `
<article class="link-card">
  ${img}
  <div>
    <a href="${p.url}" rel="noopener noreferrer">${p.title || p.url}</a>
    <p>${p.description || ""}</p>
  </div>
</article>`;
}

Wire fetchLinkPreview into your backend route so the RapidAPI key never ships to the browser.

Practical tips

Encode the URL. Always encodeURIComponent / searchParams.set the target.
Timeouts. Upstream sites can be slow; set a client timeout (e.g. 15–30s) and show a fallback card.
Errors. Watch for ssrf_blocked (private IPs), unsupported_media, and upstream_error. Surface a generic “preview unavailable” UI.
Caching. Cache successful previews by normalized URL for a few hours to stay inside your monthly quota.

What’s next

Add a small in-memory or Redis cache keyed by URL.
Fall back to the hostname when title is null.
If you also need the page body as Markdown for RAG, see the companion API: URL to Markdown (LLM-ready).

That’s it — a working link preview path in a few minutes without maintaining your own HTML meta parser.

Scrape all Trustpilot reviews with pagination (Apify Actor)

Evan — Fri, 30 Oct 2026 08:52:52 +0000

The problem

Several Trustpilot scrapers stop after the first page (~20 reviews). If you need a fuller history for reputation monitoring, reply analysis or competitor research, you need an Actor that keeps paging.

The Actor

evanatlasx/trustpilot-reviews-scraper

It reads Trustpilot's public review pages page by page until maxReviews or the last page, with star/language filters and dedupe. Pricing at time of writing: $0.0005 per review, plus a tiny Actor Start fee.

Example input

{
    "companyUrls": [
        "https://www.trustpilot.com/review/www.apple.com"
    ],
    "maxReviews": 50,
    "sortBy": "recency"
}

You can also pass a bare domain (booking.com); the Actor follows Trustpilot's redirect. To keep only complaints:

{
    "companyUrls": [
        "apple.com"
    ],
    "maxReviews": 30,
    "stars": [
        "1"
    ],
    "sortBy": "recency"
}

Example output

Real sample for Apple (maxReviews: 50):

{
    "companyUrl": "https://www.trustpilot.com/review/www.apple.com",
    "companyDomain": "www.apple.com",
    "reviewId": "6ae3a327e957768c0d64e03c",
    "rating": 1,
    "title": "Worst customer service experience.",
    "text": "Worst customer service experience.\n\nRuined my excitement of gettting a new phone.\n\n",
    "date": "2026-10-29T21:48:55.000Z",
    "experiencedDate": "2026-10-29T00:00:00.000Z",
    "language": "en",
    "countryCode": "GB",
    "isVerified": false,
    "authorName": "Anri Shengelia",
    "authorId": "6a913e8f174b66af9d05d699",
    "replyText": null,
    "replyDate": null,
    "reviewUrl": "https://www.trustpilot.com/reviews/6ae3a327e957768c0d64e03c",
    "scrapedAt": "2026-10-30T06:49:37.513Z"
}

How to export

Run the Actor with your company list.
Open the dataset → Export CSV/JSON.
Optional: schedule weekly and filter client-side on date / scrapedAt.

Cost sketch

100 reviews ≈ $0.05 in review-scraped events (plus Actor Start). Raise maxReviews only as far as you need — Trustpilot may still cap public listing depth.

Caveats (honest)

Public data only; respect Trustpilot's terms and privacy rules (author names are personal data).
WAF behaviour can change; the Actor prefers plain HTTP and falls back to headed Chrome when challenged.
Site redesigns of Trustpilot's Next.js data routes can require an Actor update.

Export mobile app reviews the same way with the App Store & Google Play Reviews Scraper ($0.00015/review).
For Shopify catalog / price monitoring, see the Shopify Products Scraper ($0.001/row).

Export App Store & Google Play reviews to CSV (with Apify)

Evan — Fri, 30 Oct 2026 08:52:51 +0000

What you will get

A CSV of public app reviews from the Apple App Store and Google Play, same columns for both stores: rating, text, date, app version, helpful votes and (on Google Play) developer replies. Useful for ASO, release post-mortems and sentiment notebooks.

Actor: evanatlasx/app-store-google-play-reviews-scraper

Pricing at time of writing: $0.00015 per review ($0.15 per 1,000), plus a tiny Actor Start fee.

Step 1 — Pick the apps

You can mix:

App Store numeric ids (310633997 for WhatsApp)
Google Play package names (com.whatsapp, com.spotify.music)
Full store URLs

Step 2 — Example input

{
    "store": "both",
    "apps": [
        "310633997",
        "com.spotify.music"
    ],
    "country": "us",
    "language": "en",
    "maxReviews": 500,
    "sort": "newest"
}

Optional: set "sinceDate": "2026-09-30" to keep only recent reviews.

Step 3 — Run and export CSV

Open the Actor on Apify and paste the input.
Click Start.
When the run finishes, open the Dataset tab → Export → CSV (JSON/Excel also work).

Each dataset item is one review. A Google Play example (live fetch of com.spotify.music, US, newest):

{
    "store": "googleplay",
    "appId": "com.spotify.music",
    "reviewId": "1d32707c-a28b-47d1-b9f7-0ae5ac14c8f4",
    "userName": "Nir Zala",
    "rating": 1,
    "title": null,
    "text": "Extremely Poor Music Recommendations and Discovery Spotify’s song recommendation algorithm is extremely disappointing. Despite paying for a subscription, the music discovery experience is poor, repetitive, and frustrating. It keeps recommending similar songs instead of understanding my actual taste and helping me discover new music. Learn something from YouTube’s recommendation algorithm. YouTube does a much better job of understanding user preferences and recommending relevant content.",
    "date": "2026-10-29T06:58:07.414Z",
    "appVersion": "9.1.94.1929",
    "developerReply": "Hi. We appreciate hearing your thoughts. We'd recommend restarting your device to see if that makes any difference. If that doesn't help, you can get in touch with our Customer Service by heading to support.spotify.com and searching \"Contact us\".",
    "developerReplyDate": "2026-10-29T17:01:08.418Z",
    "thumbsUp": 0,
    "country": "us",
    "lang": "en",
    "url": "https://play.google.com/store/apps/details?id=com.spotify.music&reviewId=1d32707c-a28b-47d1-b9f7-0ae5ac14c8f4"
}

App Store rows use the same schema with "store": "appstore"; developerReply is always null on App Store.

Limits to remember

App Store public feed ≈1,000 unique reviews per app per country after merging sort feeds.
Google Play can go much higher via continuation tokens; cap with maxReviews.
Only public data — no login bypass.

Cost example

500 reviews → about $0.075 in review events (plus Actor Start). Scale with maxReviews and how many apps you list.

Next ideas

Schedule a daily run with sinceDate = yesterday for a monitoring feed.
Pipe the CSV into a spreadsheet or a small sentiment script.
Compare the same app id/package across several country values.

DEV Community: Evan

Turn any web page into LLM-ready Markdown for RAG

Turn any web page into LLM-ready Markdown for RAG

Why Markdown?

1. Subscribe and set the key

2. Fetch Markdown

3. Node loader for a RAG pipeline

4. Plug into embeddings (sketch)

Failure modes worth handling

Related

Build link previews in 5 minutes

Build link previews in 5 minutes

What you get back

1. Grab a key

2. One request with curl

3. Tiny Node helper

4. Render a card (plain HTML)

Practical tips

What’s next

Scrape all Trustpilot reviews with pagination (Apify Actor)

The problem

The Actor

Example input

Example output

How to export

Cost sketch

Caveats (honest)

Related

Export App Store & Google Play reviews to CSV (with Apify)

What you will get

Step 1 — Pick the apps

Step 2 — Example input

Step 3 — Run and export CSV

Limits to remember

Cost example

Next ideas