I wrote hf_tokenizers because every time I needed a real tokenizer in Dart, the
options on pub.dev were pure-Dart reimplementations of one algorithm each, and
they drifted from the reference vocabulary in the corners. That drift is quiet
and it is expensive: your token ids stop matching the ids the model was trained
on, and the model's output degrades in ways that are hard to trace back to the
tokenizer.
If you run a model on device, count tokens before a call, or chunk text for
retrieval, you need the same tokenizer the model shipped with. Not a lookalike.
The idea: bind, do not reimplement
HuggingFace already maintains the tokenizer. It is a Rust crate, tokenizers,
and it is what the Python library calls under the hood. So hf_tokenizers does
not reimplement anything. It binds that crate through a thin C ABI and dart:ffi,
so the normalizer, pre-tokenizer, model (BPE, byte-level BPE, WordPiece, Unigram),
and post-processor in your tokenizer.json all run exactly as the reference runs
them.
dart pub add hf_tokenizers
Using it
import 'package:hf_tokenizers/hf_tokenizers.dart';
final tk = Tokenizer.fromFile('assets/tokenizer.json');
final ids = tk.encode('hello world'); // [101, 7592, 2088, 102] (bert)
final text = tk.decode(ids); // "hello world"
print(tk.vocabSize); // 30522
tk.close();
encode adds the model's special tokens by default (BERT's [CLS]/[SEP], for
example); pass addSpecialTokens: false to skip them. Tokenizer.fromBytes
takes the tokenizer.json bytes directly, for assets you load at runtime.
Byte-exact, and checked
The whole point is that the ids match. So the test suite loads a real
bert-base-uncased tokenizer.json and asserts against known-good values from
HuggingFace:
encode("hello world", special: true) -> [101, 7592, 2088, 102]
decode([101, 7592, 2088, 102]) -> "hello world"
[101, 7592, 2088, 102] is [CLS] hello world [SEP], identical to what Python
produces. The suite also checks WordPiece splitting and a decode round-trip. Ids
that match the reference are the reason to use this at all, so they are asserted,
not assumed.
What the drift actually costs you
That argument stays abstract until you measure it, so I built a small tool that
walks a repository and reports what sending it to a model would cost. Pointed at
the image package it reports 684,054 tokens across 479 Dart files, which is
three and a half 200,000-token windows. One embedded font file, arial_48.dart,
is 37,936 tokens by itself, a fifth of a whole window.
Useful numbers. But the tool ships bert-base-uncased as its default tokenizer,
and that is where this package earns its keep:
tokenizer.encode('aZ9' * 400); // [101, 100, 102]
Twelve hundred bytes. Three tokens. BERT maps any unrecognized contiguous run to
a single [UNK], so with the wrong tokenizer a minified bundle, a base64 blob or
a lockfile all measure as nearly free. Budget with that and you build a request
that overruns the window, and the failure arrives later, somewhere else, looking
like a model problem.
That is the drift I opened with, with a number attached. It is the reason the
package loads your model's tokenizer.json instead of approximating one.
No toolchain to install
The Rust crate would normally mean asking every user to install a Rust toolchain
at build time. That is a non-starter, so it does not work that way. CI
cross-compiles the native library for macOS, Linux, and Windows and attaches the
binaries to the release. A Dart build hook downloads the right binary for your
platform and bundles it. Building from source with cargo is only the fallback
if no prebuilt binary exists for your target.
I verified this end to end: a fresh project that only depends on hf_tokenizers,
with no Rust toolchain on the machine, runs dart pub add hf_tokenizers and the
native library loads and tokenizes. Nothing to install.
What it is not
- It is a binding, not a reimplementation. Correctness comes from the Rust crate; this package is the boundary that carries it into Dart faithfully.
- It is proven on macOS (arm64), with prebuilt binaries for Linux and Windows as well. The platform table stays honest about what is verified.
- It does the tokenization, not the model. You still bring the model; this gives it ids it was trained to understand, and turns ids back into text.
Try it
- pub.dev: https://pub.dev/packages/hf_tokenizers
- source: https://github.com/Yusufihsangorgel/tokenizers
Point it at any model's tokenizer.json (BERT, GPT-2, a sentence-transformer,
whatever you are running) and check the ids against Python. They should match.
That is the whole promise.
Top comments (0)