DEV Community

Cover image for Build a BPE Tokenizer in 80 Lines of TypeScript and Watch JSON Eat Your Tokens
Gabriel Anhaia
Gabriel Anhaia

Posted on Edited on AI-assisted

Build a BPE Tokenizer in 80 Lines of TypeScript and Watch JSON Eat Your Tokens

#ai #llm #typescript #tutorial

The maintainers of the TOON format ran three flat datasets through OpenAI's o200k_base encoding and counted 164,643 tokens for the pretty-printed JSON and 104,908 for the same data minified. Nothing was dropped. The minified version holds every key and every value. It stops paying for indentation and line breaks, and that alone takes about 36% off the count.

Tokens are the unit your LLM bill is written in. Anthropic's pricing table, for one, quotes input and output separately, per million tokens. Yet most developers who pay that bill have never watched a string turn into tokens. You can build the machine that does it in 80 lines of TypeScript and train it on your own JSON.

What happens to your prompt before the model sees it

OpenAI's tokenizers use byte-pair encoding. BPE began as a compression algorithm, and Sennrich, Haddow and Birch adapted it to split words into subword units for neural machine translation (ACL 2016). The tiktoken README says the encoding is reversible and lossless and works on any text, even text the tokenizer never saw in training. On average, a token covers about 4 bytes.

Encoding runs in three steps.

  1. A regex cuts the text into chunks, so letters, digits, punctuation and whitespace end up in separate runs. Karpathy's minbpe credits this to the GPT-2 paper and notes it "ensures that no merges will happen across category boundaries."
  2. Each chunk becomes its UTF-8 bytes. The starting vocabulary is the 256 possible byte values, so nothing is ever out of vocabulary.
  3. Learned merges glue adjacent byte pairs into bigger tokens, in the order they were learned.

Training is where the merges come from. Count every adjacent pair across the corpus, take the pair with the highest count, give it the next free id (256, then 257, and so on), replace it everywhere, repeat. The minbpe README walks through aaabdaaabac: three merges (aa, then ab, then the pair of those two new tokens) turn 11 bytes into five tokens, [258, 100, 258, 97, 99].

Rows of small square byte tiles fusing pairwise into larger tiles, layer by layer
Byte-pair encoding: start from single bytes, then fuse the most frequent neighbours, one merge at a time.

The tokenizer in 80 lines

This is the whole trainer and encoder. It uses GPT-2's pre-split pattern, minus the contraction rules ('s, 'll, and friends), copied from minbpe's regex.py.

// bpe.ts: a toy byte-level BPE trainer and encoder
type Pair = [number, number];
// GPT-2's pre-split pattern, minus its contraction rules
const SPLIT =
  / ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+/gu;

const utf8 = new TextEncoder();
const text8 = new TextDecoder();
const key = (a: number, b: number) => `${a},${b}`;
const show = (b: Uint8Array) => JSON.stringify(text8.decode(b));
const chunksOf = (text: string) =>
  (text.match(SPLIT) ?? []).map((c) => Array.from(utf8.encode(c)));

function countPairs(ids: number[], counts: Map<string, number>) {
  for (let i = 0; i < ids.length - 1; i++) {
    const k = key(ids[i], ids[i + 1]);
    counts.set(k, (counts.get(k) ?? 0) + 1);
  }
}

function merge(ids: number[], [a, b]: Pair, id: number) {
  const out: number[] = [];
  for (let i = 0; i < ids.length; i++) {
    if (ids[i] === a && ids[i + 1] === b) {
      out.push(id);
      i++; // the pair's second half is consumed too
    } else out.push(ids[i]);
  }
  return out;
}

export function train(text: string, numMerges: number) {
  let chunks = chunksOf(text);
  const merges = new Map<string, number>();
  const vocab = new Map<number, Uint8Array>();
  for (let b = 0; b < 256; b++) vocab.set(b, Uint8Array.of(b));

  for (let m = 0; m < numMerges; m++) {
    const counts = new Map<string, number>();
    for (const ids of chunks) countPairs(ids, counts);
    let best = "";
    let bestCount = 1; // a pair seen once earns no token
    for (const [k, c] of counts) {
      if (c > bestCount) { best = k; bestCount = c; }
    }
    if (!best) break;
    const [a, b] = best.split(",").map(Number) as Pair;
    const id = 256 + m;
    chunks = chunks.map((ids) => merge(ids, [a, b], id));
    merges.set(best, id);
    const l = vocab.get(a)!, r = vocab.get(b)!;
    const bytes = new Uint8Array([...l, ...r]);
    vocab.set(id, bytes);
    console.log(`#${m}`, show(l), "+", show(r),
      "->", show(bytes), `${bestCount}x`);
  }
  return { merges, vocab };
}

export function encode(text: string, merges: Map<string, number>) {
  const out: number[] = [];
  for (let ids of chunksOf(text)) {
    while (ids.length > 1) {
      // earliest-learned merge first, the order training used
      let pair: Pair | null = null;
      let rank = Infinity;
      for (let i = 0; i < ids.length - 1; i++) {
        const r = merges.get(key(ids[i], ids[i + 1]));
        if (r !== undefined && r < rank) {
          rank = r;
          pair = [ids[i], ids[i + 1]];
        }
      }
      if (!pair) break;
      ids = merge(ids, pair, rank);
    }
    out.push(...ids);
  }
  return out;
}
Enter fullscreen mode Exit fullscreen mode

Pairs are keyed as "a,b" strings because a Map compares array keys by reference, so [1, 2] would never find [1, 2]. Ties go to the pair seen first, since Map iterates in insertion order and the comparison is strict. Training works chunk by chunk, so a merge can never span a letter and a digit, or a word and the punctuation after it, which matches the real thing. And encode always applies the lowest-ranked merge available, the same rule minbpe's _encode_chunk uses: replay the merges in the order training learned them.

The show helper decodes a token's bytes for printing. When a token holds only part of a multi-byte character, TextDecoder prints �, because the token boundary falls inside the character. The Japanese sample further down does the same.

Train it on JSON and read the merges

Feed it the kind of text your app actually sends: an array of records, pretty-printed.

// demo.ts
import { train, encode } from "./bpe";

const records = Array.from({ length: 40 }, (_, i) => ({
  customer_id: 1000 + i,
  status: i % 4 ? "active" : "suspended",
  country: ["DE", "BR", "JP", "US"][i % 4],
}));
const pretty = JSON.stringify(records, null, 2);
const { merges } = train(pretty, 40);

const one = records[0];
const asPretty = encode(JSON.stringify(one, null, 2), merges);
const asMin = encode(JSON.stringify(one), merges);
console.log({ pretty: asPretty.length, minified: asMin.length });
Enter fullscreen mode Exit fullscreen mode

Run it with tsx: npx tsx demo.ts. Each log line is one merge, with the two parts, the new token, and how often the pair appeared. Read all 40 and sort them in your head: how many are fragments of the keys customer_id, status and country, how many are values and digits, and how many are quotes, colons and runs of spaces? On a corpus made of nothing but JSON, every merge has to come from one of those. The keys repeat in every single record, and that repetition is what earns them merges. A production tokenizer learns its merges from a huge general corpus instead, so common English words have a good chance of getting a token of their own (the next script shows you whether customer did). Your acctBalanceAdjFlagV2 pays for every piece.

Then try the experiments the script makes cheap. Train on JSON.stringify(records) instead of the pretty version and compare the merge log. Rename customer_id to cid and see how many merges it frees up. Compare the two counts in the last line, and remember that this toy only knows 40 merges, so the ratio between them tells you more than either number.

Ask a real encoder: o200k_base

Your toy has 40 merges. o200k_base has a vocabulary of about 200,000 tokens, roughly double GPT-4's ~100,000, according to Simon Willison's notes from the GPT-4o launch. tiktoken's model table maps gpt-4o, gpt-4.1, o3, o4-mini and gpt-5 to it. js-tiktoken loads it in Node; the lite import pulls in only the one rank file you need.

// compare.ts
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
const pieces = (s: string) =>
  enc.encode(s).map((id) => enc.decode([id]));

const samples = [
  '{\n  "customer_id": 1000\n}',
  '{"customer_id":1000}',
  "customerShippingAddressLine2",
  "1730505600000",
  "Your order has been cancelled.",
  "ご注文はキャンセルされました。",
];
for (const s of samples) {
  const p = pieces(s);
  console.log(p.length, JSON.stringify(p));
}
Enter fullscreen mode Exit fullscreen mode

You can predict part of what it prints by reading the o200k pre-split regex in tiktoken's openai_public.py. Every chunk the regex produces costs at least one token, so the regex sets a floor.

  • camelCase splits at every capital. The letter rule matches a run of uppercase letters (possibly none) followed by lowercase, so customerShippingAddressLine2 cuts into customer, Shipping, Address, Line and 2 before BPE starts. Five chunks, at least five tokens.
  • Digits go in groups of three. The number rule is \p{N}{1,3}. A 13-digit millisecond timestamp is five chunks, so at least five tokens, every time it appears in a log line you paste.
  • Whitespace is its own chunk. Indentation runs, minus their last space, are cut out separately from the keys they sit in front of. One small quirk: the space after "customer_id": in the pretty version cannot attach to a following number, because the digit rule (\p{N}{1,3}) takes no leading character and the punctuation rule's optional space needs a symbol after it. It becomes a chunk on its own. Before a string value it rides along with the opening quote instead.
  • Non-English text gets fewer merges. Byte-level BPE can encode anything, but the merges go to whatever was common in training data. Petrov et al. (NeurIPS 2023) found the same text translated into different languages could differ in token length by up to 15 times on the tokenizers they studied. OpenAI's larger vocabulary cut non-English counts: its launch figures, as quoted in Willison's notes, put Gujarati at 4.4x fewer tokens and Japanese at 1.4x fewer than the previous tokenizer. When a token boundary still lands inside a Japanese character, pieces prints � for those fragments.

A long accordion of curly braces and indentation blocks next to a compact stack of the same blocks
Same data, same keys and values; only the whitespace and repetition change, and that is where the tokens go.

One payload, three shapes

Now measure the thing you actually send. This script builds one payload and counts it as pretty JSON, minified JSON and TOON, a tabular encoding of the JSON data model that declares the keys of a uniform array once (@toon-format/toon on npm).

// shapes.ts
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";
import { encode as toToon } from "@toon-format/toon";

const enc = new Tiktoken(o200k_base);
const orders = Array.from({ length: 50 }, (_, i) => ({
  orderId: 5000 + i,
  customerId: 70 + (i % 9),
  status: i % 3 ? "shipped" : "pending",
  totalCents: 1999 + i * 37,
}));
const payload = { orders };

const shapes: Record<string, string> = {
  pretty: JSON.stringify(payload, null, 2),
  minified: JSON.stringify(payload),
  toon: toToon(payload),
};
for (const [name, text] of Object.entries(shapes)) {
  console.log(name.padEnd(9), enc.encode(text).length);
}
Enter fullscreen mode Exit fullscreen mode

Swap payload for a real request body from your logs and you have your own numbers. I can't give you numbers for your data, so here are published ones for theirs. TOON's benchmark, counted with o200k_base:

Track Pretty JSON Minified JSON TOON
Flat data, 3 datasets 164,643 104,908 68,030
Mixed structure, 5 datasets 393,637 260,451 264,734

On flat, uniform records TOON came in 58.7% under pretty JSON and 35.2% under minified. On mixed and nested data the picture flips: minified JSON beat TOON by 1.6% overall, and the README says plainly that "for deeply nested or non-uniform data, JSON may be more efficient." It also notes that absolute counts are tokenizer-specific, since other providers tokenize differently, and that you have to measure latency yourself. Fewer tokens is also not the same as better answers, so run your evals on the new shape before you ship it.

One win held on both tracks: minifying the JSON cut about a third of the tokens, so do that to anything a program produced before it goes into a prompt. Dropping keys the model does not need helps whatever the shape. Save the tabular format for long arrays of identical objects, where TOON won, and keep JSON for nested data, where it lost. Then put shapes.ts next to your prompt builder and check the count every time someone adds a field.

If this was useful

Token cost is the first bill an LLM app runs up, and it is one of the threads in AI That Answers, the first book of my AI in TypeScript series. It covers your first LLM app in TypeScript end to end: prompts, structured output, and keeping an eye on what each call costs before the invoice does it for you.

AI That Answers — AI in TypeScript

The other four books go from retrieval and tool calling to agents in production: the whole AI in TypeScript series is here.

Top comments (3)

Collapse
 
reidmarlow profile image
Reid Marlow •

The other hidden token tax with JSON in agent loops is string escaping inside tool arguments. When an agent passes code blocks, SQL queries, or nested markdown inside a JSON string field, every quote and newline gets escaped with a backslash. That backslash routinely splits what would have been a single multi-byte or common word token into two or three fragments.

We saw similar overhead with repeated keys in arrays of objects. Shipping an array of 50 records where each row repeats five verbose field names burns more tokens on schema boilerplate than on the actual payload. Switching those payloads to a columnar format with a single header row and an array of arrays dropped the payload token count by almost half before touching compression.

Collapse
 
alexcodebytes profile image
Oleksandr •

The 'minifying JSON cut about a third of the tokens' is the simplest takeaway here. It's a one-line change that saves real money, and most people don't bother because pretty-printed JSON is easier to read. But the model doesn't care about indentation.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Pretty-printing and long keys both add tokens, and the file is read again with each call while it is kept.