Why Every RAG Application Needs a Web Scraping API

Aakash R — Wed, 22 Jul 2026 02:45:03 +0000

I have been exploring RAG applications recently. The more I worked with them, the more I realized that building a RAG pipeline feels more approachable than many people think.

The bigger challenge came from the knowledge base. The quality of a RAG application depends on the content you feed into it. Getting content from websites is not always simple. Modern websites rely on JavaScript and anti bot protection. Inconsistent output adds another layer of complexity. I also needed clean Markdown that an LLM could process directly.

After trying a few options, I found the Geekflare Web Scraping API. It checked the boxes I was looking for, so I decided to build my RAG application with it. In this tutorial,

I will walk through the steps I followed to scrape website content, build a ChromaDB knowledge base, and create a simple RAG application.

Understanding the Role of Web Scraping in RAG

Every RAG application follows a simple workflow. It starts with content collection, followed by embedding generation and vector storage. User queries retrieve the most relevant content before the LLM generates a response.

Content collection becomes the foundation of the entire pipeline. Missing sections, incomplete pages, or noisy text reduce retrieval quality. A better knowledge base usually produces better answers.

For this project, website content comes from the Geekflare Web Scraping API. The API returns Markdown, the content moves into ChromaDB for embedding storage, and the retrieved documents provide context for the LLM.

Project Overview

In this tutorial, we will build a simple RAG application that answers questions using website content. .

The workflow looks like this:

Scrape website content
Generate embeddings
Store embeddings in ChromaDB
Retrieve relevant content
Generate a response with the LLM

Prerequisites

Before you begin, make sure you have:

Python 3.10 or later
A Geekflare API key
An API key for your preferred LLM provider

If you don't have a Geekflare API key yet, visit Geekflare, sign in to your account, open the dashboard, and copy your API key.

Install the Required Libraries

Install the libraries required for web scraping, vector storage, embedding generation, environment variable management, and LLM integration.

pip install geekflare-api chromadb sentence-transformers python-dotenv openai

Configure Environment Variables

Create a .env file in the project root and add your API keys.

GEEKFLARE_API_KEY=your_geekflare_api_key
LLM_API_KEY=your_llm_api_key
LLM_BASE_URL=your_llm_base_url
LLM_MODEL=your_model_name

Building the RAG Application

Now that the project is set up, let's build the RAG pipeline. We'll start by importing the required libraries, initializing the embedding model, vector database, and LLM client.

We'll also create helper functions for scraping website content, generating embeddings, retrieving relevant documents, and generating responses.

import os
import socket
from dotenv import load_dotenv

import chromadb
from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto
from sentence_transformers import SentenceTransformer
from openai import OpenAI

# Load environment variables
load_dotenv()

# Initialize the embedding model
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# Initialize ChromaDB
chroma_client = chromadb.Client()
collection = chroma_client.get_or_create_collection(
    name="website_knowledge"
)

# Initialize the LLM client
llm = OpenAI(
    api_key=os.getenv("LLM_API_KEY"),
    base_url=os.getenv("LLM_BASE_URL")
)

LLM_MODEL = os.getenv("LLM_MODEL")
Scrape Website Content

The first step is to collect the website content. The function below sends the target URL to the Geekflare Web Scraping API and returns the page in Markdown format.


def scrape_website(url: str):
    """Scrape a website and return Markdown content."""

    with GeekflareClient(api_key=os.getenv("GEEKFLARE_API_KEY")) as client:
        result = client.web_scrape(
            WebScrapeDto(
                url=url,
                format=["markdown"],
                render_js=False,
                block_ads=True
            )
        )

    if "data" not in result:
        raise RuntimeError(f"Scrape failed: {result}")

    return result["data"]["markdown"]

Split the Content into Chunks

Embedding an entire webpage at once isn't practical. Instead, split the content into smaller chunks before generating embeddings.

def split_text(text, chunk_size=1000):
    """Split text into chunks."""

    return [
        text[i:i + chunk_size]
        for i in range(0, len(text), chunk_size)
    ]
Generate Embeddings and Store Them in ChromaDB

Next, convert each chunk into an embedding and store it in ChromaDB. These embeddings will be used later to retrieve the most relevant content for a user query.

def store_documents(chunks):
    """Generate embeddings and store them in ChromaDB."""

    embeddings = embedding_model.encode(chunks).tolist()

    collection.add(
        documents=chunks,
        embeddings=embeddings,
        ids=[f"doc_{i}" for i in range(len(chunks))]
    )

Retrieve Relevant Context

When a user asks a question, convert it into an embedding and search ChromaDB for the most relevant chunks.

def retrieve_context(query):
    """Retrieve the most relevant chunks."""

    query_embedding = embedding_model.encode(query).tolist()

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=3
    )

    return "\n\n".join(results["documents"][0])

Generate the Response

Finally, combine the retrieved context with the user's question and send it to your preferred LLM.

def ask_llm(question, context):
    """Generate a response using the retrieved context."""

    prompt = f"""
Answer the question using only the context below.

Context:
{context}

Question:
{question}
"""

    response = llm.chat.completions.create(
        model=LLM_MODEL,
        messages=[
            {
                "role": "user",
                "content": prompt
            }
        ]
    )

    return response.choices[0].message.content

Run the Application

The main() function ties everything together. It scrapes the website, builds the knowledge base, and starts an interactive question answering session.

def main():

    url = input("Website URL: ")

    print("Scraping website...")
    markdown = scrape_website(url)

    print("Creating embeddings...")
    chunks = split_text(markdown)
    store_documents(chunks)

    print("Knowledge base ready.\n")

    while True:

        question = input("Ask a question (or 'exit'): ")

        if question.lower() == "exit":
            break

        context = retrieve_context(question)
        answer = ask_llm(question, context)

        print("\nAnswer:\n")
        print(answer)
        print()


if __name__ == "__main__":
    main()

You can get the entire code here: https://github.com/geekflare/geekflare-snippets/blob/stable/rag-with-web-scrapping.py

Running the Application

Run the script and enter the URL of the website you want to use as the knowledge base. You can use any publicly accessible website, such as documentation, blogs, news articles, or Wikipedia pages.

I used a Wikipedia page about Tropical Storm Brenda (1960) for this example.

You can also use any OpenAI-compatible LLM provider. In this tutorial, I used Gemini 2.5 Flash, but you can switch to OpenAI, Grok, or any other compatible model by updating the values in your .env file.

Finally, type exit to end the application.

Conclusion

That's it! You now have a simple RAG application that scrapes website content with the Geekflare Web Scraping API, stores embeddings in ChromaDB, and answers questions using an LLM.

This example uses a single webpage, but you can extend it to index multiple pages, documentation sites, blogs, or your own knowledge base.

Since the LLM provider is configurable through environment variables, you can also switch between Gemini, OpenAI, Grok, or any other OpenAI-compatible model without changing the application code.

DEV Community: Aakash R