DEV Community

Owen King
Owen King

Posted on

HTML to Markdown to embeddings: a RAG ingestion pipeline in ~50 lines of JavaScript

#api #ai #javascript #tutorial

Every RAG system has the same boring first mile: fetch a page, strip the HTML noise, split the text, embed the chunks, put the vectors somewhere useful. Most tutorials spend 300 words on BeautifulSoup selectors. Here is the version where the first mile is one API call, in about 50 lines.

Step 1: URL to clean Markdown + chunks + embeddings, in one call

const API = "https://rag-scrape-api.owerryking.workers.dev";

const res = await fetch(API + "/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example.com/pricing",
    chunk: true,
    chunkSize: 1200,
    embed: true,
  }),
});

const { markdown, chunks } = await res.json();
// chunks: [{ text, embedding }] - heading-aligned, bge-small (384-dim)
Enter fullscreen mode Exit fullscreen mode

No Cheerio, no readability heuristics, no "how do I split on token boundaries" - the chunks are aligned to heading boundaries (which matters: a chunk that starts mid-section retrieves badly), and each one arrives with its embedding already computed.

Step 2: upsert into your vector store

for (const c of chunks) {
  await store.upsert({
    id: hash(c.text),
    vector: c.embedding,
    text: c.text,
    source: "https://example.com/pricing",
  });
}
Enter fullscreen mode Exit fullscreen mode

That loop looks identical for Chroma, Qdrant, LanceDB, Pinecone or a local array. Because chunks are heading-aligned, your citations point at coherent sections instead of arbitrary 500-token windows.

Step 3: change-aware re-indexing (the part everyone skips)

Re-embedding an unchanged page wastes money and time. Pass a content hash and skip work you have already done:

const r2 = await fetch(API + "/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ url, chunk: true, embed: true, ifNoneHash: lastHash }),
});
const { notModified } = await r2.json();
if (notModified) return; // nothing changed, zero embedding cost
Enter fullscreen mode Exit fullscreen mode

Run that nightly over your sources and only genuinely updated pages cost you anything.

Step 4: whole documentation sites

const crawl = await fetch(API + "/crawl", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://docs.example.com", maxPages: 50 }),
});
const { pages, llmsTxt } = await crawl.json();
// pages: same shape as /scrape, already chunked if you ask
// llmsTxt: bonus - a ready-to-serve llms.txt for the site
Enter fullscreen mode Exit fullscreen mode

Same-host BFS, capped at 100 pages per call, one quota unit per successfully crawled page.

The whole pipeline - fetch, clean, chunk, embed, refresh - collapses to four fetch calls. The interesting work (retrieval quality, prompts, evals) starts after the first mile ends.

Free tier: 50 URLs/month, no card. OpenAPI spec: https://rag-scrape-api.owerryking.workers.dev/openapi.json

Top comments (0)