Recently, Gigatoken started showing up everywhere I look. The headline is difficult to ignore: a Rust-backed tokenizer claiming roughly 1,000x the throughput of Hugging Face Tokenizers in some workloads, with output measured in gigabytes per second. The project is interesting, so I did not want to dismiss the number just because it was large.
I maintain Splintr, a Rust-first tokenizer with Python bindings. A few months ago, I had already done the familiar comparison against Hugging Face Tokenizers, tiktoken, riptoken, and other Rust tokenizers. Splintr was ahead by enough that I got bored.
That was not the same as believing it was finished. I knew I could optimize it further. I was just waiting for bug reports and feature requests while I focused on my other projects.
I still use Splintr in another machine-learning project. It is one of my important crates. It just was not where I wanted to spend most of my time optimizing every last instruction.
Then Gigatoken made the question interesting again.
The 1,000x number pulled me back in
My first reaction was simple:
Really?
Was the benchmark reproducible? Was it comparing the same work? Was it an optimized native Rust pipeline against a Python pipeline that happened to call Rust underneath?
I had a high opinion of the author from the way the project was presented. The performance did not look like a random microbenchmark. But the headline still needed context, because a tokenizer can move a lot of work across the boundary between the file, the Rust core, Python, and the returned output objects.
The Gigatoken README explains an important part of the result. Its fastest API lets the Rust implementation read a large text file directly, find document boundaries, and parallelize the work without repeatedly handing Python strings across the boundary. Its compatibility APIs are more convenient, but the README explicitly says they do not deliver the same 1,000x result.
That is not a criticism. It is the center of gravity of the project.
The impressive number is answering a real question: how fast can this implementation tokenize a large corpus when it owns the file and the parallel pipeline? It is not automatically the answer to a different question: how fast does a Python application encode a batch of strings and receive token ids back as Python objects?
I wanted to compare both projects, but I had to start with the second question.
I chose to compete in Python - Since that's one public
My first idea was Rust versus Rust. Splintr is a Rust library, and that is still the comparison I want most.
But Gigatoken does not currently expose its Rust implementation as a published crate that I can just add to a Cargo benchmark.
I could clone the repository, inspect its internal build, configure my machine, and make the native harness work.
Honestly, I was too lazy for that. I should just wait for it.
So I compared what both projects currently expose to the people most likely to try them: their Python APIs.
Splintr is Rust-first. The tokenizer core, the BPE and SentencePiece paths, the tokenizer.json loader, the GGUF vocabulary path, and the batch implementation live on the Rust side.
Python bindings are an additional interface I conveniently add so Python users can benefit from it.
It is part of the product, but it is not the reason I started the project.
The basic use is deliberately small:
from splintr import Tokenizer
tokenizer = Tokenizer.from_pretrained("qwen3")
tokens = tokenizer.encode("The interesting work is in the boundary.")
The benchmark then compares equivalent operations through the exposed APIs. For a Python list result, that means a batch ending in list[list[int]]. For a flat result, it means a contiguous id buffer plus the information needed to recover each row.
Gigatoken gave me a useful target
When I first tried Splintr against Gigatoken, it performs roughly the same in Python, but often behind it by something like 10–20%.
That was an interesting result.
It was close enough that the remaining difference looked like something I could investigate.
And I found several places where I had left performance on the table.
The changelog tells the story better than a list of vague optimization claims.
- The pre-tokenizer moved from a regex-driven path to a direct scanner.
- JSON-loaded tokenizers stopped allocating a string for every pre-token.
- Normalizers that left the text unchanged stopped copying it.
The merge path also changed. Short pieces use a direct scan instead of paying for a heap that is only useful for larger inputs. Two-byte merge ranks use a direct-indexed table instead of a hash lookup. Chunks can write into one output buffer instead of building a temporary vector and copying it. The pre-tokenizer yields subslices instead of copying the input at every stage.
Those changes are not one clever trick. They are the usual uncomfortable performance work: remove a copy, stop allocating a temporary object, pick a data structure for the input size that actually reaches the loop, and check whether a lock is protecting something the hot path does not need to mutate.
I've recorded those changes, including the measured improvements.
I made the benchmark reproducable
I had been running comparisons on my Linux and macOS machines, but a local benchmark is easy to trust too much. I wanted every release to have a repeatable comparison against other tokenizers, not one number I ran once and remembered incorrectly.
The performance workflow now runs on both Ubuntu and macOS. It records the machine, the versions, the vocabulary form, the output form, and the parity checks before it reports timings.
The run I am using here is the v0.16.1 benchmark on GitHub Actions. The Python portion used:
Python 3.13.14
splintr-rs 0.16.1
tokenizers 0.23.1
tiktoken 0.13.0
gigatoken 0.10.0
rounds 3, interleaved
The Ubuntu job used an AMD EPYC 7763 with four available cores. The macOS job used a virtual Apple M1 runner with three cores.
The Rust report built the checked-out v0.16.1 source at commit 6d468b9, while the Python report installed the published splintr-rs 0.16.1 wheel. The workflow records that distinction instead of pretending the two reports are one binary.
The workflow has two vocabulary families. The *-ranks family gives every engine the same .tiktoken rank file and the corresponding pre-tokenizer. That covers cl100k, o200k, qwen3, and kimi.
The *-json family gives every compatible engine the same Hugging Face tokenizer.json. That covers cl100k, o200k, qwen3, and glm4. tiktoken is absent from this family because it does not load a tokenizer.json file.
The Rust suite also avoids making the cache look like the tokenizer. It downloads WikiText-103, extracts 6,741 documents, and builds a 120 MB prose corpus instead of repeating a small set of sentences. Splintr has a chunk cache, so repeated synthetic text could measure cache hits rather than tokenization.
The Python report uses a mixed corpus of 1,000 texts. Each result is the median of three interleaved rounds. The interleaving matters because a hosted runner can change speed while a long benchmark is running.
This is not the same workload as Gigatoken's headline file benchmark. It is a deliberate comparison of the APIs a Python users use.
The Python result is not a Gigatoken landslide
Here is the Ubuntu result for 1,000 mixed texts. Throughput is in MB/s. The list columns return list[list[int]]; the flat columns return a contiguous buffer where that API exists.
| Input form | Splintr list | Gigatoken list | Splintr flat | Gigatoken flat |
|---|---|---|---|---|
| cl100k-ranks | 78.8 | 79.5 | 189.8 | 152.8 |
| o200k-ranks | 77.7 | 85.6 | 162.2 | 145.3 |
| qwen3-ranks | 82.6 | 84.6 | 184.2 | 144.1 |
| kimi-ranks | 71.9 | 84.4 | 139.0 | 144.1 |
| cl100k-json | 72.6 | 78.6 | 156.2 | 145.7 |
| o200k-json | 72.1 | 84.7 | 130.9 | 140.5 |
| qwen3-json | 71.8 | 74.0 | 134.2 | 117.9 |
| glm4-json | 72.1 | 76.1 | 142.5 | 121.9 |
The ordinary list result is close. Gigatoken has the advantage on the rank-file family, while Splintr has the advantage on the JSON family in this Ubuntu run. Neither result justifies a universal winner.
The flat result changes the picture. Splintr wins six of the eight rows, with Gigatoken ahead on Kimi ranks and o200k JSON. The largest Splintr result is Qwen3 ranks at 184.2 MB/s against Gigatoken at 144.1 MB/s.
That does not mean I can write “Splintr is faster than Gigatoken” and stop there. It means returning Python lists is a different measurement from returning ids in a contiguous buffer. Once the output representation is normalized, the two projects are still close, and the winner depends on the vocabulary and machine.
The macOS runner tells the same story in a different shape. On the virtual M1, the normal list results are mostly close, and the flat results trade wins. Gigatoken wins some JSON and Kimi cases; Splintr wins other rank and JSON cases.
The charts below use the macOS Python summary from this run, not the raw artifact. The summary reports one list-output batch row at 1,000 texts, so these charts compare all vocabularies rather than inventing a batch-size sweep. The rank-file charts include Splintr, tiktoken, and Gigatoken. The tokenizer.json charts include Splintr, Hugging Face Tokenizers, and Gigatoken.
Batch encoding (.tiktoken) - tiktoken vs splintr vs gigatoken.
Batch encoding (from JSON) - huggingface vs splintr vs gigatoken.
Flat batch encoding (.tiktoken) - splintr vs gigatoken.
Flat batch encoding (JSON) - splintr vs gigatoken.
Latency and loading are where Splintr pulled ahead
The more consistent Splintr advantage in this run was not giant-corpus throughput. It was the cost around an individual call and the cost of getting a vocabulary ready.
On the four-core Ubuntu runner, the median single-text latency and vocabulary load times looked like this:
| Suite | Single text: Splintr | Single text: Gigatoken | Load: Splintr | Load: Gigatoken |
|---|---|---|---|---|
cl100k-ranks |
1.46 ms | 2.27 ms | 27.6 ms | 79.6 ms |
qwen3-ranks |
1.46 ms | 2.37 ms | 44.5 ms | 119.1 ms |
o200k-json |
1.80 ms | 2.39 ms | 239.9 ms | 459.0 ms |
glm4-json |
1.81 ms | 2.77 ms | 300.0 ms | 499.8 ms |
The virtual M1 had smaller absolute latency differences, but vocabulary loading still favored Splintr across the tested external rank-file and JSON paths. This result is using the external loader, not Splintr's packed bundled-vocabulary path to be fair.
The JSON path has separate loading improvements, including avoiding a string allocation for every pre-token and sizing buffers up front, but this benchmark does not isolate how much each change contributes.
This is useful for a different class of program: a serverless worker, a CLI, a short-lived process, a worker that reloads models, or an application that handles many small requests instead of one enormous corpus.
It is the kind of result I would love to see as a library.
The fair Rust comparison is still missing
The benchmark answers a real question, but it does not answer every question I started with.
Gigatoken's headline benchmark is built around its native file-oriented API. Splintr's comparison above goes through Python, and its flat Python API is still not the same thing as a native Rust program.
I still want to run that experiment:
same machine
same corpus
same vocabulary and pre-tokenizer
same token ids
no Python on either side
native Rust batch or file/stream input
cold and warm runs separated
cache effects measured instead of hidden
I would also want to reproduce the large-corpus shape behind the viral number, not quietly substitute a small batch of Python strings and call it equivalent. The current CI already has the beginnings of the methodology: real prose, explicit parity, recorded hardware, separate output forms, and repeated interleaved rounds.
What it does not have yet is a published Gigatoken crate that I can put beside Splintr in the same Cargo harness.
I could build gigatoken from its repository. I probably will when I have the time. For now, I am waiting for the author to publish the Rust crate so I can run the comparison properly.
Splintr already covers the workload I care about
I should be clearer about one thing: I am not looking at Gigatoken to substitute Splintr.
I already use Splintr to load and tokenize gigabytes of corpus for training in my machine-learning projects. It does that workload well enough that tokenization is not where I spend my time worrying. I do not need to replace it with another tokenizer just to get through a large dataset. Splintr already done the job well and fast.
That changes how I see Gigatoken. Its native file path is interesting, but it is not a missing capability in Splintr. It is another serious contender in the same performance market, and it gave me a useful reason to inspect the parts of Splintr I had left less optimized.
I like what the Gigatoken author has done. The project is not just a Python wrapper with an optimistic benchmark. Its native pipeline owns more of the input and parallelization path, and the author has clearly spent time on pre-tokenization, caching, and the cost of crossing into Python. That is a good effort, and I am glad it gave me a reason to look at my own hot paths again.
For my own work today, I would still use Splintr. It is already the Rust tokenizer in my training and inference workload. Unlike Gigatoken, it is available as a normal Cargo dependency, and it's Python bindings let me use it in Python workflow.
It supports byte-level BPE, SentencePiece BPE, Unigram, and WordPiece behind one handle.
It can load bundled vocabularies, Hugging Face tokenizer.json files, raw .tiktoken files, and GGUF vocabulary metadata on the Rust side.
The current evidence says that Splintr and Gigatoken are in the same performance class for comparable Python calls.
I still do not have the native Rust comparison, and I am not going to pretend that the Python table settles it.
Try Splintr before you choose a tokenizer
If you are already considering Gigatoken, take a look at Splintr too. You can install the Rust library with cargo add splintr, or install the Python bindings with pip install splintr-rs. The shortest paths are the repository, crates.io package, Python package, and API documentation.
I would be happy to receive an issue, a benchmark result, or a pull request. If the Gigatoken author publishes the Rust crate, I would also be very interested in running the Rust-versus-Rust test this article still owes.
What does your tokenizer workload look like?
Are you tokenizing a large file, encoding batches of application strings, serving one request at a time, or loading a vocabulary repeatedly in short-lived workers?
Would you be interested in using something like Gigatoken or Splintr, instead of Huggingface Tokenizer or Tiktoken?
Top comments (0)