Keeping a Whoosh index in sync: updates, deletes, and folder sync in pure Python

Full-text search tutorials love the first commit: build a schema, add a few
documents, run a query. Real apps have a harder problem — your data keeps
changing. Files get edited, rows get deleted, records get re-imported. If your
search index drifts from your source of truth, users get stale or missing
results.

Whoosh (the maintained
whoosh3 fork) makes this a first-class operation. No external service, no
daemon — just Python. Here are the three patterns you actually need.

1. Upsert with update_document

If your schema has a unique field, update_document is an upsert: it
deletes any existing document with that key and adds the new one. Same call for
"insert" and "update" — you don't have to check first.

from whoosh.fields import Schema, ID, TEXT
from whoosh.index import create_in

schema = Schema(path=ID(unique=True, stored=True), content=TEXT(stored=True))
ix = create_in("idx", schema)

w = ix.writer()
w.add_document(path="a.txt", content="the quick brown fox")
w.add_document(path="b.txt", content="lazy dog sleeps")
w.commit()

w = ix.writer()
w.update_document(path="a.txt", content="the quick brown fox jumps")  # replaces a.txt
w.update_document(path="c.txt", content="new document about cats")     # inserts c.txt
w.commit()

The unique=True on path is what makes this work — without it,
update_document behaves like add_document. After the second commit the index
holds three documents, and a search for jumps returns only a.txt.

2. Deleting documents

Two ways to remove documents:

w = ix.writer()
w.delete_by_term("path", "b.txt")   # by an indexed term (returns count deleted)
# w.delete_document(docnum)         # by internal doc number, if you have it
w.commit()

delete_by_term is the one you'll reach for 99% of the time — delete by the
same unique key you index on.

3. Syncing a whole folder (add / change / delete detection)

The classic real-world job: keep an index matching a directory of documents.
Store each file's modification time, then on each run compare what's indexed
against what's on disk. This is the recipe Whoosh has shipped for years, and it
still works unchanged:

import os
from whoosh.fields import Schema, ID, TEXT, STORED
from whoosh.index import create_in, open_dir, exists_in

schema = Schema(path=ID(unique=True, stored=True), time=STORED,
                content=TEXT(stored=True))

def sync(docs_dir, idx_dir):
    ix = open_dir(idx_dir) if exists_in(idx_dir) else create_in(idx_dir, schema)

    indexed = {}
    with ix.searcher() as s:
        for fields in s.all_stored_fields():
            indexed[fields["path"]] = fields["time"]

    on_disk = {n: os.path.getmtime(os.path.join(docs_dir, n))
               for n in os.listdir(docs_dir)}

    w = ix.writer()
    for path in set(indexed) - set(on_disk):        # gone from disk
        w.delete_by_term("path", path)
    for path, mtime in on_disk.items():
        if path not in indexed:                     # new file
            w.add_document(path=path, time=mtime,
                           content=open(os.path.join(docs_dir, path)).read())
        elif mtime != indexed[path]:                # changed file
            w.update_document(path=path, time=mtime,
                              content=open(os.path.join(docs_dir, path)).read())
    w.commit()

Run it once and everything gets indexed. Edit a file, add one, delete one, run
it again — only the deltas are touched. Run it a third time with no changes and
it does nothing. That's the whole point: cheap, incremental, idempotent.

A few things worth knowing

• One writer at a time. A Whoosh index allows a single writer; commits are atomic. Batch your changes into one writer()/commit() per sync pass rather than committing per document — it's much faster and produces fewer segments.
• Searchers see a snapshot. An open searcher() reads a point-in-time view; new commits don't disturb readers already in flight. Reopen (or use searcher.refresh()) to see fresh data.
• Deletes are logical first. Deleted documents are marked and removed from results immediately; physical space is reclaimed when segments merge on later writes. writer.commit(optimize=True) forces a full merge if you want it now.

Why this matters

The reason people reach for a search server is often just "my index needs to
stay in sync with my data." For a single app — a docs site, a notes tool, an
internal knowledge base — you can do all of it in-process with Whoosh: upsert
with update_document, delete with delete_by_term, and sync a folder with a
stored mtime. Pure Python, no Elasticsearch, no daemon to babysit.

Whoosh is being actively maintained again as whoosh3 on PyPI. If this kind of
"just add search to my Python app" story is useful to you, a ⭐ on
the repo genuinely helps others
find the maintained fork.

(Every snippet here was verified against whoosh3 3.52.0 on 2026-09-14.)