In RAG pipelines, text chunking usually forces an uncomfortable trade-off:
-
Fast window splitters (like a raw
tiktokenloop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers. -
Structure-aware splitters try to respect document layout, but they are often 4×–11× slower, pull in dozens of dependencies, and frequently overshoot hard token limits (returning 513–514 tokens on a 512-token budget when headings or BPE boundary joins aren't re-counted).
With text-chipper v1.1.0, the goal was to eliminate that trade-off: build the fastest chunker that guarantees both strict token budgets and Markdown/table structural awareness, backed by reproducible benchmarks.
Here is what the benchmark numbers look like, the two major problems we solved in
v1.1.0(Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets. --- ## 1. The Benchmark: Speed + Strict Token Compliance + Structure We benchmarkedtext-chipperagainst seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict 512-token budget (cl100k_base,overlap=0).
### Throughput (Single-Core Geometric Mean across 12 Workloads)
-
text-chipper(default Markdown- and table-aware strict mode): 37.1 MiB/s -
Plain
tiktokenwindow (no structure, cuts words/code): 15.5 MiB/s -
Other chunking libraries: 3.2 to 9.5 MiB/s (
text-chipperis 3.9× to 11.6× faster) At the same time: -
0 chunks over 512 tokens across all 12 workloads (compared to 15 over-limit chunks in LangChain up to 514 tokens, and 18 each in
tiktokenwindow and Chonkie'sTokenChunker). -
100% of measured small code fences kept intact (392 in
markdown_docand 3,434 incode_heavy).
- 0 runtime dependencies (~4 MB installed wheel size, ~2 ms import time).
2. New in v1.1.0: Markdown Table-Aware Splitting (Without Breaking Source Offsets)
Consider a 200-row financial or inventory table in Markdown:
| Row | Item | Category | Price |
| --- | --- | --- | ---: |
| 001 | Product-1 | Hardware | $10 |
...
| 200 | Product-200 | Hardware | $2000 |
When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:
- A cut happens in the middle of a row (
| 045 | Prodin chunk 1,uct-45 | Hardware |in chunk 2). - Chunks 2..N lose the table header (
| Row | Item | Category | Price |), leaving the embedding model and LLM with floating rows of numbers that lack column names. ### Howtext-chipperv1.1.0 solves it—while keepingdoc[start:end] == chunk.textA core invariant oftext-chipper(verified across 26.9 million checks in our release test suite) is thatchunk.textis always an exact contiguous slice of the normalized input:
assert doc[chunk.start_index : chunk.end_index] == chunk.text
If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.
Instead, text-chipper treats the repeated table header as structural context, just like section headings (chunk.headers):
-
Tables that fit stay whole: If the entire GFM table fits within
max_length, it is never split. -
Row-boundary splitting: If an oversized table must be split, row boundaries are promoted to top-level structural cut points (
R_ITEM), so rows are kept whole whenever they fit. -
Chunk 1 of the table starts at the table's opening line in the source text, so
chunk.textalready begins with the verbatim header and| --- |delimiter row (chunk.table_headerisNone). -
Chunks 2..N of the table keep
chunk.textas the exact contiguous slice of data rows, attach the header + delimiter lines aschunk.table_header, and automatically prepend them inchunk.contextualized_text. -
Budget-enforced: The token cost of the repeated table header is reserved up front during row packing and re-verified on
chunk.contextualized_textby the strict final gate.
from text_chipper import TextChipper
header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |"
rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)]
doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n"
chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base")
chunks = chipper.chunk(doc)
for idx, c in enumerate(chunks):
assert c.token_count <= 120
assert chipper.count_tokens(c.contextualized_text) == c.token_count
assert header in c.contextualized_text # Every chunk has the table header!
assert doc[c.start_index : c.end_index] == c.text # And exact source offsets still hold!
3. New in v1.1.0: Zero-GIL Hugging Face Tokenizers & Multi-Tokenizer Guards
If you run local embedding models (BAAI/bge-large-en-v1.5, nomic-embed-text, jina-embeddings) or local LLMs (Qwen, Llama-3, Mistral), counting tokens with OpenAI's cl100k_base is dangerous: a 500-token chunk under cl100k_base can easily be 610+ tokens under your model's actual tokenizer.
Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.
Native Rust handoff for AutoTokenizer and from_pretrained
In text-chipper v1.1.0:
- You can call
TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510)directly (resolvingtokenizer.jsonfrom a local folder/file or viahuggingface_hub/tokenizers/transformers). - Or you can pass an already-loaded
transformers.PreTrainedTokenizerFast(AutoTokenizer.from_pretrained(...)) ortokenizers.Tokenizerviatokenizer=. - At initialization,
text-chipperextracts the serialized JSON from.backend_tokenizer.to_str()and loads it directly into Hugging Face's native Rusttokenizerscrate (Backend::from_tokenizer_bytes). During chunking, zero Python calls are made and the GIL is completely released. - Need a custom Python counter? You can also pass any
callable(text) -> intor object with.encode(text). ### Multi-Tokenizer Guards (guard_tokenizers) What if your pipeline sends the same chunk to both an embedder (e.g.bge-large-en-v1.5) and a cross-encoder reranker (bge-reranker-v2-m3) or LLM with a different vocabulary?
from transformers import AutoTokenizer
from text_chipper import TextChipper
embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5")
rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")
chipper = TextChipper(
max_length=510, # leave 2 tokens for [CLS] and [SEP] under a 512 limit
overlap=32,
tokenizer=embed_tok,
guard_tokenizers=[rerank_tok],
)
for chunk in chipper.chunk(markdown_text):
# Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3!
print(chunk.token_count, chunk.contextualized_text)
4. Bounded-Memory Streaming on 500+ MiB Files
When processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling chunk() spikes process memory by 4.6 GB.
Because text-chipper's BlockParser and SmartChunker are incremental line-driven state machines, from_file_stream(path) streams the 514.3 MiB file in 25 seconds (20.5 MiB/s on a single core) with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.
Try It Out
text-chipper v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:
pip install -U text-chipper
- GitHub Repository & Benchmarks: https://github.com/kameh0use/text-chipper
- PyPI: https://pypi.org/project/text-chipper/
-
Rust Crate (
text_chipper): https://crates.io/crates/text_chipper If you hit any strange Markdown tables or tokenizer edge cases in your RAG corpus, open an issue—I'd love to add them to the benchmark and fuzz suite!
Top comments (0)