A zero byte in the input was silently truncating my tokenizer

hf_tokenizers binds the HuggingFace tokenizers Rust crate into Dart over FFI. The whole reason to use it instead of a pure-Dart reimplementation is that the token ids come back byte-exact with the reference, so they match what the model was trained on. It was byte-exact, except on one class of input, and the failure was silent.

The bug: text containing a U+0000 byte got cut off at the first NUL. Everything after it was dropped. encode, encodeWithOffsets, and the token-budget chunking built on top of them all returned counts for a shorter string than the one you handed in. Nothing threw. The numbers were just wrong, and wrong quietly is the worst kind.

Where it came from

The Dart side handed text to the native side as a C string:

final input = text.toNativeUtf8(allocator: calloc);
final ptr = tkEncode(_handle, input, addSpecialTokens, outLen);

toNativeUtf8 writes the UTF-8 bytes followed by a trailing NUL. On the Rust side the string came back out with CStr::from_ptr:

let s = match unsafe { CStr::from_ptr(text) }.to_str() {
    Ok(s) => s,
    Err(_) => return ptr::null_mut(),
};

A C string is defined as "the bytes up to the first zero." That is the entire contract of CStr. So a string that is hello, then a NUL byte, then world becomes, at the native boundary, just encode('hello'). The world is still sitting in memory right after the NUL, but CStr stops at the zero and never reads it.

For most text this never fires, because most text has no zero byte in it. But U+0000 is a legal Unicode code point and legal UTF-8 (a single 0x00 byte), so it does turn up: text pulled out of a binary format, a field that was zero-padded, a truncated or corrupted stream, a fuzzer's input. The moment one of those reaches a tokenizer you are using to measure a context-window budget, the count is short and the cut lands in the wrong place.

The fix is to stop using the NUL as the terminator

The bytes were never the problem. The length was. A NUL-terminated string throws the length away and asks the reader to recover it by scanning for a zero. The fix is to carry the length across the boundary instead.

On the Dart side, write the UTF-8 bytes into a plain buffer and pass the byte count next to the pointer:

(Pointer<Uint8>, int) _allocUtf8(String text) {
  final bytes = utf8.encode(text);
  final ptr = calloc<Uint8>(bytes.isEmpty ? 1 : bytes.length);
  if (bytes.isNotEmpty) {
    ptr.asTypedList(bytes.length).setAll(0, bytes);
  }
  return (ptr, bytes.length);
}

On the Rust side, build the &str from a length-bounded slice instead of a NUL scan:

unsafe fn str_from_raw<'a>(ptr: *const u8, len: usize) -> Option<&'a str> {
    if ptr.is_null() {
        return None;
    }
    let bytes = std::slice::from_raw_parts(ptr, len);
    std::str::from_utf8(bytes).ok()
}

Now a zero byte is just a byte. from_raw_parts(ptr, len) reads exactly len bytes, zeros included, and from_utf8 accepts them because 0x00 is valid UTF-8. The whole input reaches the tokenizer.

The signature change is the actual work. tk_encode went from (*const c_char, ...) to (*const u8, usize, ...), and the Dart @Native binding moved from Pointer<Utf8> to Pointer<Uint8> plus an IntPtr length. That is an ABI change, so the prebuilt native binaries have to be rebuilt and re-released in lockstep with the package. A consumer that paired the new Dart bindings with an old binary would be calling a three-argument function with four arguments, which is its own quiet corruption. The release that ships the new bindings and the release that ships the rebuilt binaries move together, or not at all.

The test that would have caught it

test('a U+0000 byte does not truncate the input', () {
  final withNul = 'hello' + String.fromCharCode(0) + 'world'; // a NUL in the middle
  final ids = tk.encode(withNul, addSpecialTokens: false);
  // The bytes after the NUL contribute tokens the truncating version could
  // never see.
  expect(ids.length,
      greaterThan(tk.encode('hello', addSpecialTokens: false).length));
  // And that content survives the round trip.
  expect(tk.decode(ids), contains('world'));
});

On the old code, encoding that hello-NUL-world string returned the ids for hello and nothing else, so its length equalled encode('hello') and the first expectation failed. With the length passed explicitly it returns tokens for the whole string, and the decoded output contains world again.

The rule I took from it is small and worth keeping. At an FFI boundary, if the text can come from outside your program, pass its length. NUL termination is a bet on the data that costs nothing until the day the data has a zero in it, and then it loses without telling you.