DEV Community

Priya Sundaram
Priya Sundaram

Posted on

Whoosh's query language is a stack of plugins — and you can rewrite it

Whoosh is a fast, featureful, pure-Python full-text search library. I maintain the actively-kept fork (priya-sundaram-dev/whoosh, published to PyPI as whoosh3). This is part of a series on getting real work done with it.

Most people meet Whoosh's QueryParser as a black box: you hand it a string like python +search -django, and out comes a query tree. What surprised me the first time I read the source is that the whole query language is assembled from small, swappable plugins. The default parser is just a list of them. Once you see that, you can add operators, alias field names, or turn features off — without writing a parser.

The parser is a plugin list

Every syntactic feature — whitespace handling, field:value prefixes, grouping with parentheses, boosts like term^2, ranges — is a Plugin. QueryParser holds an ordered list and you can add, remove, or replace entries:

from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser, PlusMinusPlugin

schema = Schema(title=TEXT, tags=TEXT, path=ID)
qp = QueryParser("title", schema)
qp.add_plugin(PlusMinusPlugin())

print(qp.parse("python +search -django"))
# ((title:search ANDMAYBE title:python) ANDNOT title:django)
Enter fullscreen mode Exit fullscreen mode

+ means "must", - means "must not", and a bare term becomes a "nice to have". That's Google-style syntax in one line of setup.

Fuzzy matching for typo tolerance

Add FuzzyTermPlugin and users get edit-distance matching with the ~ suffix:

from whoosh.qparser import FuzzyTermPlugin

qp = QueryParser("title", schema)
qp.add_plugin(FuzzyTermPlugin())

qp.parse("render~2")   # title:render~2  → matches "render", "renders", "rander"...
Enter fullscreen mode Exit fullscreen mode

~2 allows up to two edits. It's the cheapest way to forgive typos without building a separate spell-check pass.

Alias field names for humans

Your schema field is tags, but users type tag: or label:. FieldAliasPlugin maps friendly names onto real fields:

from whoosh.qparser import FieldAliasPlugin

qp = QueryParser("title", schema)
qp.add_plugin(FieldAliasPlugin({"tags": ["tag", "label"]}))

qp.parse("tag:python")   # tags:python
Enter fullscreen mode Exit fullscreen mode

This is great when your storage schema and your public query syntax shouldn't be the same thing.

Real date ranges from plain English

DateParserPlugin (in whoosh.qparser.dateparse) parses human date expressions against a DATETIME field and rewrites them to numeric ranges:

from whoosh.fields import Schema, TEXT, DATETIME
from whoosh.qparser import QueryParser
from whoosh.qparser.dateparse import DateParserPlugin

schema = Schema(title=TEXT, created=DATETIME)
qp = QueryParser("title", schema)
qp.add_plugin(DateParserPlugin())

qp.parse("created:[2023-01 to 2023-06]")
qp.parse("created:2020")   # expands to the whole year's range
Enter fullscreen mode Exit fullscreen mode

created:2020 automatically becomes the range covering all of 2020 — you don't have to spell out boundaries.

Redefine the operators themselves

Whoosh's default boolean operators are the terse AND/OR/ANDNOT/ANDMAYBE. If you want a stricter, SQL-ish language you can replace the operator plugin entirely:

from whoosh.qparser import QueryParser, OperatorsPlugin

# Only AND / OR, nothing else
ops = OperatorsPlugin(And=" AND ", Or=" OR ",
                      Not=None, AndNot=None, AndMaybe=None, clean=False)

qp = QueryParser("title", schema)
qp.replace_plugin(ops)

qp.parse("cat AND dog OR fish")
# ((title:cat AND title:dog) OR title:fish)
Enter fullscreen mode Exit fullscreen mode

Passing None for an operator removes it. This is how you constrain the query language to exactly the surface you want to support — no surprises from syntax you never documented.

Why this design is worth copying

The plugin list is the reason Whoosh's query language feels both powerful and safe to expose to end users. You start from a sensible default, then:

  • add capabilities users expect (+/-, ~, date ranges),
  • alias away your internal schema, and
  • remove anything you don't want people typing into a search box.

No regex spelunking, no forked grammar. If you've been reaching for Elasticsearch just to get a decent query syntax over a modest corpus, this is a big part of what you'd be reimplementing anyway.

Everything above runs on the maintained fork. Install with pip install whoosh3, and the source (plus a roadmap and good-first-issues if you'd like to contribute) is at github.com/priya-sundaram-dev/whoosh. If you build something with the plugin system, I'd love to see it.

— Priya

Top comments (0)