Web pages contain valuable information, but extracting that information for an LLM or a Retrieval-Augmented Generation (RAG) pipeline is not always straightforward.
A typical website contains navigation menus, scripts, styles, advertisements, tracking elements, and other content that your AI application probably doesn't need.
Sending all that HTML directly to an LLM can waste tokens, increase processing costs, and make your retrieval pipeline unnecessarily noisy.
In this tutorial, we'll use MarkRAG to extract clean Markdown from a website and optionally split it into semantic chunks ready for a RAG workflow.
1. What we'll build
We'll explore two endpoints:
-
POST /v1/extract: Extract clean Markdown from a URL. -
POST /v1/extract-and-chunk: Extract content, clean it, and generate chunks with contextual metadata.
We'll use Node.js 18+ and its native fetch API.
2. Get your RapidAPI credentials
Subscribe to MarkRAG on RapidAPI and retrieve your API key.
Set these environment variables:
RAPIDAPI_KEY=your_rapidapi_key
The API host is:
markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com
Never commit your API key to GitHub or expose it in frontend code.
3. Extract clean Markdown
Let's start with a simple request.
Create extract.js:
const HOST =
"markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com";
async function extractMarkdown(url) {
const response = await fetch(
`https://${HOST}/v1/extract`,
{
method: "POST",
headers: {
"Content-Type": "application/json",
"x-rapidapi-host": HOST,
"x-rapidapi-key": process.env.RAPIDAPI_KEY
},
body: JSON.stringify({
url,
options: {
strip_links: true,
cache: true
}
})
}
);
const result = await response.json();
if (!response.ok) {
throw new Error(
`MarkRAG returned HTTP ${response.status}: ${
JSON.stringify(result)
}`
);
}
return result;
}
const result = await extractMarkdown(
"https://www.timetoast.com/"
);
console.log("Extracted Markdown:");
console.log(result.data.markdown);
console.log("\nToken savings:");
console.log(result.data.metadata.tokensSaved);
console.log(
"Savings percentage:",
`${result.data.metadata.savingsPercentage}%`
);
Run it:
node --env-file=.env extract.js
The response includes the cleaned Markdown and metadata such as estimated tokens, reading time, and token savings.
4. Prepare content for RAG with semantic chunks
Now let's take it one step further.
Instead of returning only cleaned Markdown, MarkRAG can split the extracted content into chunks and provide additional context for each one.
Create extract-and-chunk.js:
const HOST =
"markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com";
async function extractAndChunk(url) {
const response = await fetch(
`https://${HOST}/v1/extract-and-chunk`,
{
method: "POST",
headers: {
"Content-Type": "application/json",
"x-rapidapi-host": HOST,
"x-rapidapi-key": process.env.RAPIDAPI_KEY
},
body: JSON.stringify({
url,
options: {
chunkSizeTokens: 512,
chunkOverlapTokens: 50,
includeLinks: true,
includeImages: false,
stripBoilerplate: true
}
})
}
);
const result = await response.json();
if (!response.ok) {
throw new Error(
`MarkRAG returned HTTP ${response.status}: ${
JSON.stringify(result)
}`
);
}
const data = result.data;
console.log("Success:", result.success);
console.log("RAG diagnostics:", data.qualityDiagnostics);
console.log("Cleaning statistics:", data.cleaningStats);
console.log(`\nGenerated ${data.chunks.length} chunks`);
for (const chunk of data.chunks) {
console.log(`\nChunk ${chunk.index}`);
console.log("ID:", chunk.chunkId);
console.log("Tokens:", chunk.tokenCount);
console.log("Context:", chunk.contextHeader);
console.log("Content:", chunk.content);
}
return data;
}
await extractAndChunk("https://www.timetoast.com/");
Run:
node --env-file=.env extract-and-chunk.js
Each chunk includes useful information:
-
chunkId: A unique identifier for the chunk. -
index: Its position in the generated sequence. -
headingPath: The heading hierarchy associated with the content. -
contextHeader: Additional section context. -
content: The extracted text. -
tokenCount: The estimated number of tokens in that chunk.
This structure gives your application more than a collection of arbitrary text fragments: it provides context that can be useful when storing and retrieving information.
5. A real-world extraction example
We tested MarkRAG using:
https://www.timetoast.com/
The extraction endpoint returned:
- HTTP status: 200 OK
- Estimated original tokens: 38,295
- Estimated Markdown tokens: 1,008
- Estimated tokens saved: 37,287
- Reported savings: 97.4%
The extract-and-chunk endpoint returned:
- HTTP status: 200 OK
- Estimated original tokens: 33,356
- Estimated cleaned tokens: 1,430
- Estimated tokens saved: 31,926
- Reported compression: 95.7%
- Generated chunks: 3
- RAG readiness score: 100
- Quality warnings: None
These are observed results from our Timetoast test, not guaranteed performance figures for every website.
6. Where can you use this?
This workflow can be useful for:
- Building knowledge bases from public websites.
- Preparing documentation for AI assistants.
- Extracting information from product pages.
- Processing educational resources.
- Creating website-powered RAG applications.
- Reducing unnecessary HTML before downstream LLM processing.
A possible architecture looks like this:
Website URL
|
v
MarkRAG
|
v
Clean Markdown
|
v
Semantic Chunks
|
v
Embeddings
|
v
Vector Database
|
v
RAG Application
|
v
LLM
MarkRAG handles the web extraction and content preparation stages. Your application can then take those chunks and connect them to your preferred embedding model and vector database.
7. Try MarkRAG
If you're building an AI application that needs information from websites, you can explore MarkRAG on RapidAPI:
Have you built a RAG pipeline that extracts information from websites? I'd love to hear how you handle HTML cleanup, chunking, and token optimization in your projects.
Disclosure: This tutorial demonstrates MarkRAG, an API published by the author on RapidAPI.
Top comments (0)