text-chipper is a Rust RAG chunker with strict token budgets, structure-aware chunks, and bounded-memory streaming for large files.
A token budget is only a hard limit if it is checked on the text actually returned. I built text-chipper to do that while preserving Markdown structure and keeping large-file memory use in view.
In one benchmark using each library’s documented defaults and a 512-token limit, the tiktoken window and Chonkie’s TokenChunker produced chunks of up to 513 tokens; LangChain reached 514. text-chipper’s strict mode re-counted every completed chunk, and none exceeded the limit.
That is a result from a specific benchmark, not a claim that those libraries always exceed their limits. I have not isolated why those particular tiktoken and Chonkie chunks reached 513 tokens; the benchmark counted the delivered chunks but did not trace the libraries’ internal split and reconstruction steps. Adding heading context after an initial count is one separate way a pipeline can exceed its budget, and it is the failure mode text-chipper’s final recount guards against. I am not claiming it explains the benchmark overages.
Why I built it
Chunking determines what enters an index and how information is packaged. A boundary can split a code example, separate a heading from its explanation, or create input that an embedding model cannot accept. These are ingestion problems that can later surface as missing or confusing retrieval context.
I wanted a chunker that handled those details without requiring an entire large file to fit in memory. I also suspected that preserving document structure could help retrieval, but I have not measured retrieval quality or hallucination rates. That remains an open question, and one I hope to explore with people building real RAG systems.
A token budget should be a real limit
A chunker may estimate token counts as it builds chunks. But the text it delivers can differ from the text it counted during construction. This matters when the chunk includes extra context, such as a heading path: text that fit before contextualization can exceed the budget afterward.
For an embedding model with a hard input limit, an oversized input may be rejected or truncated, depending on the API and the surrounding pipeline.
In text-chipper, the default strict mode re-tokenizes each completed chunk in the exact form it returns, including heading context. For the configured tokenizer, every returned chunk satisfies:
token_count <= max_length
If a single Unicode grapheme cluster is larger than the budget, strict mode raises ValueError instead of returning an oversized chunk. This guarantee applies to the tokenizer you configure; it matches an embedding model’s limit only if that model uses the same tokenizer. A non-strict mode is available when an approximate budget is acceptable and speed matters more.
Text has structure, not just tokens
Markdown contains headings, paragraphs, lists, metadata, and fenced code. A generic token window does not recognize these as meaningful units. For example, it can split a code sample halfway through a parameter or detach it from the heading that explains it.
text-chipper divides a document into blocks at blank lines, tracks heading paths, and keeps fenced code blocks and list items whole whenever they fit within the budget. It does not split a Unicode grapheme cluster, such as an emoji sequence or combining mark, and its offsets map back to the original input.
I chose a lightweight Markdown block parser rather than a complete CommonMark implementation so files can be processed as a stream. That choice has limits: tables, block quotes, and HTML blocks receive no special handling. The README describes the supported constructs and compares its block boundaries with pulldown-cmark on the CommonMark examples.
What this looks like in an API docs assistant
Imagine a team indexing a few hundred Markdown pages with short code examples. Its embedding model accepts 512 tokens per input.
A fixed token window can split an example between chunks, leaving a retrieved result with half a call. It can also separate a heading from its explanation, so the returned text arrives without its topic.
text-chipper keeps fenced blocks intact when they fit. The measured comparison with other chunkers appears below; these results apply to the tested fixtures.
Headings are available as headers metadata, and contextualized_text prefixes the heading path so a chunk about retry settings still carries its section context. Strict mode counts that context against the token budget. Each chunk also includes orig_start_index and orig_end_index, which an application can use to link an answer back to its source.
for chunk in chipper.chunk(page_text):
index.add(
text=chunk.contextualized_text, # heading path + body
headers=chunk.headers,
source_span=(chunk.orig_start_index, chunk.orig_end_index),
)
Only the splitting step changes; embedding, storage, and retrieval can stay as they are.
Streaming a 514 MiB file
Peak memory can determine whether ingestion fits on a worker at all. I tested from_file_stream() on a 514.3 MiB WikiText-103 file. Across three single-core runs, including file reading, it reached 20.5 MiB/s. After warm-up, memory grew by 56 MB and the process peaked at about 109 MB. Separately, I ran a whole-file test: reading the complete file into one Python string and then calling text-chipper’s chunk() increased memory by about 4.6 GB. Other libraries also showed multi-gigabyte peaks in this whole-file test; several implementations were within 2 MB of one another. That suggests that materializing the file contributes much of the memory use, but I have not verified the cause.
For the separate large-file comparison, I fed the libraries that require complete strings roughly 8 MiB blocks; text-chipper was also run in block mode. Its memory grew by 140 MB in blocks, compared with 56 MB in direct streaming. SHA-256 hashes of the delivered text from from_file_stream() and whole-file chunk() matched. Block boundaries can change chunk output because chunks cannot cross them. These measurements came from a shared virtual machine, so treat throughput as indicative.
What I measured
The repository includes a benchmark harness, raw results, and charts you can regenerate.
The main comparison covered 12 workloads: synthetic prose, Markdown, code, multilingual text, real documents such as PubMed, finance, Wikipedia, and chat logs, plus stress cases. Each library was asked for chunks of at most 512 cl100k_base tokens with no overlap. Results are medians of five cold runs in fresh processes on one core; only the chunking call was timed. Each library used its documented token-bounded route and defaults. I did not tune the other libraries, so a Markdown-specific configuration may do better than the defaults measured here. The tested text-chipper version was 0.2.3 at commit d392ce3; library versions and hardware are in the raw results.
This is not an equal-feature comparison. text-chipper’s default mode parses Markdown, adds heading context, and recounts every delivered chunk. The other libraries do not do all of that work. The speed results compare documented default routes, not implementations doing identical work.
Results at a glance
| Measure | Result in these workloads |
|---|---|
| Token limit | No text-chipper chunk exceeded 512 tokens. LangChain had 15 over-limit chunks; the tiktoken window and Chonkie’s TokenChunker had 18 each (maximums: 514 and 513). |
| Throughput | 37.1 MiB/s geometric mean; fastest on 10 of 12 workloads. That was 2.4× the plain tiktoken window and 3.9–11.6× the other chunkers. |
| Word boundaries | One measured mid-word cut by text-chipper out of 1,259 cut points (<0.1%); token windows cut words at up to 35.5% of measured cut points. The harness excludes scripts without whitespace-separated words. |
| Small code fences | text-chipper, semantic-text-splitter, and semchunk kept all measured fences intact. Token windows and Chonkie broke roughly 21–23%. This result is specific to these fixtures, not unique to text-chipper. |
| Working memory | During chunking, text-chipper added 1.7 MB on average and about 11.7 MB on its most memory-intensive workload. Whole-process peak memory includes Python and the tokenizer. |
The comparisons ran on one shared virtual machine. Treat the ratios as indicative, especially on small files.
Unicode and replacement characters
On a 2 MiB CJK stress workload with no separators, the tiktoken window, Chonkie’s TokenChunker, and Chonkie’s RecursiveChunker each produced 751 chunks; 245 chunks from each method contained U+FFFD replacement characters. text-chipper produced 774 chunks and none contained U+FFFD.
A chunk containing U+FFFD does not mean all of its characters were lost. In a separate diagnostic on a synthetic CJK file of about 1.7 million non-whitespace characters, 546 of 2,833 chunks (19.3%) from a 512-token tiktoken window contained U+FFFD; 286 characters were lost (0.017% of the text). That diagnostic is separate from the benchmark above. text-chipper’s flat mode produced no replacement characters and lost nothing on the same file. The counting script is available in the repository.
Where text-chipper may not fit
- Mostly plain prose: there may be less structure to preserve, though strict budgets and word-aware cuts still apply.
- Non-Markdown input: text-chipper does not read HTML, PDF, or DOCX. Convert those formats to Markdown or text first.
- Need a full syntax tree: it is not a full CommonMark parser and does not split code by function or class. Tables, block quotes, and HTML blocks are not special-cased; long tables do not repeat their header row.
-
Need another Python tokenizer: Python supports built-in OpenAI encodings plus
charsandwords. Hugging Facetokenizer.jsonsupport requires a build with thehffeature; custom counters are currently available only from Rust. -
Need flat mode on a very large file:
include_context=Falsedoes not parse Markdown and reads the whole file; it is limited to 32 MiB. Bounded-memory streaming uses the default Markdown-aware mode.
The question I have not answered yet
These benchmarks measure speed and chunk behavior, not retrieval quality. I have not tested how these chunks affect search results with controlled embeddings, retrieval settings, and relevance judgments. Structure-aware chunks may preserve useful context without improving search; that needs a separate evaluation. I also have not tested whether they reduce hallucinations.
Retrieval determines what evidence a RAG system places in front of a model. I want to measure how chunking affects that evidence before making stronger claims.
That is where I would especially value help. If you work on retrieval evaluation, code-heavy documentation, multilingual corpora, or large knowledge bases, I would welcome feedback, reproducible test cases, and collaboration on a controlled comparison against established approaches.
Next on my list: retrieval-quality comparisons, better handling for Markdown tables and other constructs, and more tokenizer integrations.
Try it
text-chipper is licensed under Apache-2.0.
pip install text-chipper
from text_chipper import TextChipper
chipper = TextChipper(max_length=512, overlap=50, encoding="cl100k_base")
for chunk in chipper.chunk(markdown_text):
print(chunk.token_count)
print(chunk.headers)
print(chunk.contextualized_text)
This example uses overlap=50 to show the option; the benchmark comparison above used no overlap.
For Rust projects, add the crate with Cargo. The Rust crate uses an underscore in its name; the Python package uses a hyphen.
cargo add text_chipper
The library also supports file processing, batch operations, streaming, built-in tokenizer backends, and LangChain and LlamaIndex adapters.
If you try text-chipper on your documents, I’d value hearing where it works well and where it falls short. A small, reproducible example in a GitHub issue is especially useful.
If you work on RAG systems, what retrieval test would convince you that a chunker is helping on your data?



Top comments (0)