How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

How do you make sure the documents entering your VectorStoreIndex contain actual content when building a LlamaIndex RAG (retrieval augmented generation) pipeline? This question comes up because the built-in SimpleWebPageReader struggles with many of the dynamic web pages it encounters.

Instead of valid page content, the reader can return Cloudflare challenge pages, empty React shells, or content with a poor signal-to-noise ratio relative to the tokens spent. This tutorial directly addresses this issue. It shows how replacing the built-in web reader with one that returns clean LlamaIndex documents from live URLs improves the quality of data available to agents, query engines, and vector indexes in RAG systems.

Why LlamaIndex web scraping loaders fail on real websites

There are three major failure modes that LlamaIndex's built-in web loaders show against modern web infrastructure.

The first is the Cloudflare/WAF (Web Application Firewall) block. When the SimpleWebPageReader makes a standard HTTP request using its underlying libraries like requests or urllib, it lacks capabilities such as browser fingerprints, TLS fingerprints, and human-like behavioral patterns. Security layers on websites protected by WAFs use these signals to identify automated requests and cause the reader to receive HTTP error pages or challenge screens instead.

The second failure mode is SPA (Single Page Application) empty shells. Many modern websites use frameworks like React, Angular, or Vue. When they are scraped, the initial HTML response is usually just an empty template shell containing an initialization script block. Since the reader does not execute JavaScript, it only sees the initial HTML shell.

The third is token bloat. Even for modern unprotected static sites, built-in readers pull raw HTML or raw text, which usually include navigation bars, headers, footers, sidebars, cookie banners, and tracking scripts. This unnecessary data, when indexed, costs tokens that would have been better spent pulling in more relevant content.

Using the SimpleWebPageReader to load data from an anti-bot challenge page shows exactly how output failures can occur.

from llama_index.readers.web import SimpleWebPageReader
documents = SimpleWebPageReader(html_to_text=True).load_data(
    ["https://www.scrapingcourse.com/cloudflare-challenge"]
)

print(documents[0].text[:200])

The reader was only able to pull the challenge page.

If a document like this is added to the VectorStoreIndex, LLM (Large Language Model) outputs would have no information related to the content that was meant to be pulled. At the same time, tokens are still consumed during both retrieval and generation. To avoid these failure modes, use a reader designed to access protected websites, execute JavaScript, and return cleaner content for indexing. The ZenRowsWebReader is one such reader.

Setting up ZenRowsWebReader

The ZenRowsWebReader is supported in LlamaIndex and can return valid Document objects from live pages that are ingestible by AI applications. It can access websites protected by WAF challenges, render JavaScript, and output markdown, all of which improve document quality. Here is how you can set it up.

To follow along, first generate a ZenRows API key from the ZenRows Dashboard and store it as the ZENROWS_API_KEY environment variable.

Next, import the reader and configure it with the following parameters to handle live web pages.

When reading sites protected by advanced anti-bot mechanisms like Cloudflare, enable the premium_proxy parameter to use residential IPs. Residential IPs are less likely to be identified as automated traffic by anti-bot systems.

from llama_index.readers.web import ZenRowsWebReader

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

Lastly, use the ZenRowsWebReader against the same website that the SimpleWebPageReader failed to scrape.

from llama_index.readers.web import ZenRowsWebReader
from dotenv import load_dotenv
import os

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge"
])

print(documents[0].text[:500])

Running it, the output shows that the reader successfully accessed the website, scraped its content, and returned it in Markdown.

Building a RAG pipeline with live web data

Having resolved the data quality issue with scraping web content from live pages using ZenRowsWebReader, the next step is to include the reader in a RAG pipeline. While the complete code is a single file, the implementation consists of four stages: configuring the pipeline to use OpenAI models, ingesting live web data with the reader, creating the vector index, and querying it to generate responses.

Step 1: Configure the OpenAI model and embedding model

Import all dependencies, load the API keys, and configure the OpenAI model and embedding model.

from dotenv import load_dotenv
import os

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

Settings.llm = OpenAI(
    model="gpt-4o-mini",
    api_key=OPENAI_API_KEY
)

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=OPENAI_API_KEY
)

This configuration uses gpt-4o-mini for fast, cost-effective response generation and text-embedding-3-small to create semantic embeddings that provide a good balance of retrieval quality, latency, and cost. Both models can be replaced with other supported OpenAI models depending on your specific requirements.

Step 2: Load live web data into LlamaIndex documents

Just like in the previous section, import the ZenRowsWebReader and configure it to load data from the target URLs into LlamaIndex Document objects.

from llama_index.readers.web import ZenRowsWebReader

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

print(f"Loaded {len(documents)} document(s)")

Step 3: Create the vector index

Import VectorStoreIndex and pass the Document objects into it to create the index.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

print("Index created successfully")

This example creates an in-memory VectorStoreIndex for simplicity. LlamaIndex also supports persisting indexes to disk or external vector stores.

Step 4: Query the index

Use the query engine to retrieve relevant content from the index and pass it to the OpenAI model for response generation.

query_engine = index.as_query_engine()

response = query_engine.query(
    "Summarize the information available across the indexed pages."
)

print("\nAnswer:\n")
print(response.response)

The result is that the LLM retrieved valid indexed data from the documents and generated output that matched the scraped content.

While this pipeline delivers valid data, websites frequently change layouts and content. So it’s important to have a layer that updates the index with fresh Document objects.

Keeping your knowledge base current

One-off loads deliver valid content, but they can quickly become outdated. Periodic refreshes allow current web content to be re-scraped from live pages as source content changes. But if the logic isn't right, it can cause the same content to be re-ingested repeatedly, increasing storage costs and reducing retrieval quality through duplicate content.

You can achieve efficient refreshes by generating hashes from the scraped page content and comparing them against previously stored values. This logic runs after loading documents with the ZenRowsWebReader and before passing the updated documents to the VectorStoreIndex.from_documents().

In addition to the existing reader setup, this implementation uses the json and hashlib modules to persist and compare content hashes.

import json
import hashlib

To set up the logic, create a hash store. In production, this is usually a database rather than a local file. Then create a set of functions that store hashes of scraped documents and compare them against newly scraped content.

HASH_STORE = "content_hashes.json"

def load_hashes():
    if os.path.exists(HASH_STORE):
        with open(HASH_STORE, "r") as f:
            return json.load(f)
    return {}

def save_hashes(hashes):
    with open(HASH_STORE, "w") as f:
        json.dump(hashes, f, indent=4)

def get_updated_documents(urls):
    documents = reader.load_data(urls)

    if not documents:
        print("No documents returned")
        return []

    stored_hashes = load_hashes()
    updated_documents = []

    for doc in documents:
        if not doc.text.strip():
            source_url = doc.metadata.get("source_url", "unknown")
            print(f"Skipping empty content: {source_url}")
            continue

        source_url = doc.metadata.get("source_url", "unknown")

        content_hash = hashlib.md5(
            doc.text.encode("utf-8")
        ).hexdigest()

        previous_hash = stored_hashes.get(source_url)

        if previous_hash == content_hash:
            print(f"No changes detected: {source_url}")
            continue

        print(f"Updated content detected: {source_url}")

        stored_hashes[source_url] = content_hash
        updated_documents.append(doc)

    save_hashes(stored_hashes)
    return updated_documents

documents = get_updated_documents([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

During each refresh cycle, the function compares the latest scraped content against the previously stored hashes. Unchanged pages are skipped, while pages with updated content are returned for re-indexing.

At this point, the pipeline is fully functional. But even with clean web data and a working retrieval pipeline, you can still encounter issues with answer quality when querying.

Troubleshooting incomplete answers

Incomplete answers can result from retrieval configuration in LlamaIndex or from incomplete page content returned by the web reader. Here is how to troubleshoot both.

Insufficient retrieval context

During retrieval, LlamaIndex returns only a subset of the highest-scoring chunks from the vector index to provide relevant context for the LLM. By default, this naive top-k retrieval matches query embeddings with chunk embeddings. Since live webpages are broken down into multiple markdown chunks, important context can end up distributed across multiple document chunks.

The fix

Increase the top-k parameter to 10 so the query engine pulls more relevant chunks into the retrieval context, improving the likelihood of retrieving related information.

query_engine = index.as_query_engine(
    similarity_top_k=10
)

Missing dynamic JavaScript content

If your final output indicates missing information or returns placeholder strings such as "Enable JavaScript to see products," the scraper may be capturing the page before JavaScript-rendered elements have finished loading.

The fix

In the ZenRows reader parameters, verify that js_render=True is enabled and that a wait=2000 configuration has been added. These settings allow dynamic elements time to load so the target elements can fully render before the page is converted into a LlamaIndex document.

Conclusion

You have now built a RAG pipeline that uses the ZenRowsWebReader to ingest live web data into LlamaIndex. It can retrieve content from protected and JavaScript-driven websites, keep indexed content up to date through periodic refreshes, and generate responses from documents that standard web loaders often fail to retrieve.

Comments

[Luis Cruzy]

I found the explanation of the three major failure modes of LlamaIndex's built-in web loaders to be particularly insightful, especially the issue with SPA empty shells. I've experienced similar problems when scraping modern websites and I'm interested in learning more about how the ZenRowsWebReader handles JavaScript rendering. Can you elaborate on how the ZenRowsWebReader executes JavaScript and whether it supports rendering complex web pages with multiple scripts? Additionally, I'd like to know if there are any limitations or potential workarounds when dealing with websites that use heavy JavaScript rendering.