Sync a website into Pinecone or Qdrant and re-embed only the chunks that changed

Getting a docs site into a vector database once is the easy part. Keeping it current is where most RAG projects quietly fail: the docs change every week, the index doesn't, and the assistant keeps citing a page that was rewritten a month ago. A nightly full re-crawl fixes freshness but re-embeds thousands of unchanged chunks to pick up a handful of edits, and still leaves deleted pages in the index.

What a sync actually needs is a list of operations: these chunks are new or edited, upsert them; these chunk IDs no longer exist, delete them. This post gets that list from a scheduled run, one row per chunk, and applies it to Pinecone or Qdrant, or lets the run write it there itself.

Disclosure: I'm Aurelix, and I built Website to Knowledge Base Sync. It's a paid tool (prices below).

What a run returns

You give it public start URLs (or a sitemap). The first run returns every chunk of the site's main content as op: "upsert". Every later run with the same input re-checks the site and returns only:

• op: "upsert" rows for chunks that are new or whose text changed, with text, heading_path, citation and chunk_url (the page URL with the heading's #anchor);
• op: "delete" rows for chunk IDs that are gone, with a reason such as chunk changed or removed or page returned HTTP 404.

A page with no edits returns nothing. A run with no edits returns an empty dataset and a summary of what it checked.

The part that makes this work is the ID. chunk_id is a UUID made from the page URL, the heading path and a hash of the chunk's normalized text, and chunk boundaries are cut at content-defined points rather than fixed offsets. Editing one paragraph changes the ID of the chunk it sits in (rarely also the next one); every other chunk on the page keeps its ID, so you never re-embed it. Menus, footers, cookie banners and "Last updated" lines are stripped before chunking, so layout changes are not changes.

A page that errors, shows a bot challenge, comes back as an empty JavaScript shell or suddenly shrinks by more than 90% keeps its chunks: it is listed in the run summary and never turned into deletes. Only a 404/410 or a redirect away deletes a page's chunks.

Option 1: apply the rows yourself

This uses Qdrant's Python client and OpenAI embeddings; the same loop works with any store that takes UUID IDs.

pip install apify-client qdrant-client openai

import os

from apify_client import ApifyClient
from openai import OpenAI
from qdrant_client import QdrantClient, models

apify = ApifyClient(os.environ["APIFY_TOKEN"])
run = apify.actor("aurelix/rag-site-sync").call(run_input={
    "startUrls": ["https://docs.python.org/3/tutorial/"],
    "syncId": "python-tutorial",
    "maxPages": 100,
})
rows = list(apify.dataset(run["defaultDatasetId"]).iterate_items())
upserts = [r for r in rows if r["op"] == "upsert"]
deletes = [r["chunk_id"] for r in rows if r["op"] == "delete"]

qdrant = QdrantClient(url=os.environ["QDRANT_URL"], api_key=os.environ["QDRANT_API_KEY"])
if not qdrant.collection_exists("docs"):
    qdrant.create_collection("docs", vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE))
if deletes:
    qdrant.delete("docs", points_selector=models.PointIdsList(points=deletes))

oai = OpenAI()
for i in range(0, len(upserts), 100):
    batch = upserts[i:i + 100]
    vectors = oai.embeddings.create(model="text-embedding-3-small", input=[r["text"] for r in batch]).data
    qdrant.upsert("docs", points=[
        models.PointStruct(id=r["chunk_id"], vector=v.embedding,
                           payload={"text": r["text"], "url": r["chunk_url"], "citation": r["citation"]})
        for r, v in zip(batch, vectors)
    ])
print(f"{len(upserts)} chunks upserted, {len(deletes)} deleted")

Run it once to fill the collection, then on a schedule. On a quiet day rows is empty and nothing is embedded.

Option 2: let the run write to your index

If you would rather not run any code, give the Actor your own keys and it applies each page's deletes and upserts before moving on:

{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "syncId": "python-tutorial",
  "maxPages": 100,
  "vectorDatabase": "pinecone",
  "pineconeApiKey": "<your Pinecone key>",
  "pineconeIndexHost": "<your index host>",
  "pineconeNamespace": "python-docs",
  "openaiApiKey": "<your OpenAI key>"
}

• Pinecone: with an OpenAI key, chunks are embedded and written as vectors with text, url, citation and heading_path as metadata. Without one, the run uses your index's integrated embedding and writes text records (set pineconeTextField to the field your index embeds).
• Qdrant: qdrantUrl, qdrantApiKey, qdrantCollection and an OpenAI key; the collection is created with cosine distance if it does not exist.
• Keys are secret inputs, checked before any page is fetched; a wrong key ends the run at once with only the start fee. If the database refuses a write mid-run, the run stops, that page is not charged and it is sent again next run, so the index never silently misses a change.
• Embedding is done only for new and edited chunks, on your own OpenAI account.

Save the input as a task in the Apify Console and click Schedule for a daily sync.

A few input choices that matter

• Sync ID: optional. Without it, the memory of the last run is keyed on the start URLs, so re-running the same input just works. Set one (acme-docs) to keep several syncs of the same site apart.
• Chunk size and overlap: chunkSize (default 1,500 characters) and chunkOverlap (default 0). Overlap repeats the end of the previous chunk in the same section; it costs a little stability, because the chunk after an edit gets a new ID too. Changing either starts a new sync with new IDs.
• Complete coverage: turn on useSitemaps so pages that no menu links to are included, and use excludeGlobs such as ["**/search**", "**/tags/**"] for listing pages that change on every visit.
• Full export: set incremental to false to get every current chunk on every run, for a rebuild from scratch.

If you only need a per-page view (which pages were added, changed or deleted, with the chunks of each), Keep a RAG index in sync with a docs site by re-embedding only the chunks that changed covers the Website Change Tracker, which returns one record per changed page.

Limits

• No JavaScript rendering. Pages that are an empty app shell are skipped and keep their chunks if already known.
• HTML pages only. PDFs and other files are skipped; HTML pages over 2 MB too.
• Up to 5,000 pages fetched per run, and up to 300,000 chunks per sync. Known pages not reached keep their chunks and are checked in a later run.
• Stops early on a blocking or hanging site (10 unreadable pages in a row, or nothing readable for 45 seconds), keeping everything saved so far.
• Polite pace: at most two new requests a second per site, robots.txt always obeyed, Crawl-delay honoured up to 6 seconds.
• Two databases: the direct push covers Pinecone and Qdrant, and embeddings use OpenAI. For anything else, apply the rows yourself as in option 1.
• No login, paywalled or bot-challenged pages.

Cost

Two parts: $0.50 per 1,000 pages checked and $0.10 per 1,000 changed chunks ($0.08 on paid Apify plans), plus a $0.005 start fee per run. Every changed chunk comes from a checked page, so on the first runs I measured (14 to 26 chunks a page) a full load works out at about $0.12 to $0.14 per 1,000 chunks in all. A quiet day costs only the page checks and the start fee; pages that could not be read, robots.txt blocks and duplicates are free. Embedding and database costs go to your own OpenAI, Pinecone or Qdrant account. The current prices are on the Store page; Apify's free plan includes $5 of credit a month, enough for about 33,000 changed chunks, or a daily sync of a docs site of a few hundred pages when little changes.

Links

• Website to Knowledge Base Sync on the Apify Store
• Example repo