DEV Community

Priya Sundaram
Priya Sundaram

Posted on

A 25k-star project swapped Whoosh for a Rust search engine — and kept Whoosh's query language

#python #opensource #abotwrotethis #programming

Tagged #python #search #abotwrotethis. I'm Priya Sundaram, the current maintainer of whoosh3 — the actively-maintained fork of the pure-Python Whoosh full-text search library. This post is about a design decision I found genuinely interesting.

The story

paperless-ngx is a hugely popular self-hosted document management system. For years its full-text search was powered by Whoosh, the pure-Python search library. In its 3.0 release, the team replaced the Whoosh backend with tantivy — a Rust search engine with Python bindings — for indexing speed and a smaller on-disk index.

Here's the part worth stopping on: they swapped the engine, but they went out of their way to keep Whoosh's query syntax as the language their users type into the search box. A companion library was built specifically to parse the Whoosh query language and translate it into tantivy queries, so that every saved search and every muscle-memory query users had written over the years kept working.

You migrate the thing that's expensive to run. You preserve the thing your users have in their fingers. And what paperless users had in their fingers was Whoosh's query language.

What's actually in that query language

If you've only ever done LIKE '%term%' or a bare MATCH, it's easy to underrate how much a real query language gives your users. Here's a tour of what Whoosh parses out of the box — the same surface paperless decided was worth preserving.

Fielded terms. Restrict a term to a field with field:value:

title:python author:sundaram
Enter fullscreen mode Exit fullscreen mode

Boolean logic and grouping:

python AND (search OR indexing) NOT deprecated
Enter fullscreen mode Exit fullscreen mode

Phrases with slop. Words in order, allowing a few words in between:

"full text search"~2
Enter fullscreen mode Exit fullscreen mode

Ranges — numeric, and with TO between endpoints (great for dates):

date:[20240101 TO 20241231]
price:{10 TO 100}
Enter fullscreen mode Exit fullscreen mode

Curly braces make an endpoint exclusive; square brackets make it inclusive. Open-ended ranges (date:[20240101 TO]) work too.

Wildcards and prefixes:

sear*   te?t
Enter fullscreen mode Exit fullscreen mode

Fuzzy terms — match within an edit distance, for typo tolerance. These aren't on by default (fuzzy matching is expensive, so you opt in), but enabling them is one line:

from whoosh.qparser import QueryParser, FuzzyTermPlugin
qp = QueryParser("content", ix.schema)
qp.add_plugin(FuzzyTermPlugin())
# now: python~  (edit distance 1)   python~2  (edit distance 2)
Enter fullscreen mode Exit fullscreen mode

Boosts — weight part of the query higher when ranking:

title:python^2 body:python
Enter fullscreen mode Exit fullscreen mode

The reason this is nice is that it's readable. A user (or an admin writing a saved filter) can express "documents from 2024, mentioning invoices, but not drafts" without learning your ORM or your storage engine. That legibility is exactly why a downstream project treated the syntax as an asset to carry forward rather than baggage to drop.

The pure-Python angle

The paperless move is also an honest signal about the trade-off space, and I don't want to spin it: if you have a large corpus and the operational room to ship a compiled dependency, a Rust-backed engine will index faster and use less disk. That's real.

But a lot of projects are on the other side of that trade-off:

  • You want pip install to just work — no Rust toolchain, no wheels-for-every-platform matrix, no compiled extension to debug in someone's locked-down environment.
  • Your corpus is thousands to low-millions of documents, not hundreds of millions.
  • You'd rather read and patch your search layer in the same language as the rest of your app.

That's the niche pure-Python Whoosh has always served, and it's why I picked the project up. whoosh3 is the maintained fork: it runs on modern Python (3.9–3.13), the query language above works exactly as shown, and it installs with zero compiled dependencies.

pip install whoosh3
Enter fullscreen mode Exit fullscreen mode 
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser
import os, tempfile

schema = Schema(title=TEXT(stored=True), path=ID(stored=True), content=TEXT)
d = tempfile.mkdtemp()
ix = create_in(d, schema)

w = ix.writer()
w.add_document(title="First", path="/a", content="pure python full text search")
w.add_document(title="Second", path="/b", content="ranking and query parsing")
w.commit()

with ix.searcher() as s:
    q = QueryParser("content", ix.schema).parse("python AND search")
    for hit in s.search(q):
        print(hit["title"], hit["path"])
Enter fullscreen mode Exit fullscreen mode

The takeaway

The interesting lesson from the paperless-ngx migration isn't "Whoosh lost." It's that a query language people can actually read and type is a durable piece of UX — durable enough that a team re-plumbing their entire search stack chose to reimplement the parser rather than retrain their users.

If that legible, batteries-included query language is what you want, and pure-Python-no-compile is the trade-off you'd pick, whoosh3 is alive and maintained. Issues and PRs are welcome:

Credit to the original Whoosh author Matt Chaput, the whoosh-community maintainers, and the whoosh-reloaded effort that kept it going.

Top comments (0)