The median across all 70 tested cases is 7.7x.
BeautifulSlop is the compatibility API on top of a real DOM. It's part of
domonic, a Python implementation of the
browser's DOM. Same methods you already know: select, find_all, get_text,
decompose, wrap, all of it. It's not soup. It's slop.
pip install domonic turbohtml
from domonic.bs4 import BeautifulSlop
soup = BeautifulSlop(html)
soup.select_one("#content .mw-parser-output p") # this is the 8k one
soup.find_all("a", href=True)
soup.get_text(" ", strip=True)
The numbers
A 500 KB Wikipedia page, 70 cases, median speed-up per group. Beautiful Soup
on html.parser, BeautifulSlop on turbohtml.
| What | BeautifulSlop is |
|---|---|
| Parse | 3.4x faster |
| CSS selectors, simple | 245x faster |
| CSS selectors, combinators | 28x faster |
| CSS selectors, pseudo-classes | 16x faster |
find / find_all
|
8x faster |
find_all with filters |
11x faster |
Navigation (siblings, parents, find_next) |
12x faster |
Mutation (decompose, wrap, append) |
2x faster |
| Serialisation | 3x faster |
| Text extraction | 1.8x faster |
Every one of the 360 parity-checked cases across six fixture pages returns the
same result as Beautiful Soup. The benchmark script is in the repo. Run it on
your machine:
python scripts/benchmark_bs4.py --all-pages --check
Why it's faster
Beautiful Soup walks the tree for every query. BeautifulSlop keeps a real DOM
with indexes, so #id, .class and tag lookups don't walk anything, and
selectors run on a native engine. When you change the tree the indexes drop
themselves, so you never get a stale answer.
Text extraction was the one group Beautiful Soup used to win, because its
strings are its nodes. As of this release the parser hands the document a
list of its text nodes in order, so get_text() joins a list instead of
walking a tree.
What you get for free
Because it's not a fake DOM, the rest of the browser is there when you need it.
querySelector and XPath. Events and MutationObserver. A document you can
write() into as a page downloads, and stop early once you've seen what you
came for. Pages in windows-1252 or Shift_JIS decoded the way a browser does
it, from the bytes.
Repo: https://github.com/byteface/domonic
Docs: https://domonic.readthedocs.io
So if you have a project that already uses soup, try swapping it now and see huge performance gains!
Top comments (0)