DEV Community: Talha Anwar

LangSmith Just Shipped an LLM Gateway. Here's the 3-Year Mess That Made Me Care

Talha Anwar — Sat, 08 Aug 2026 13:33:14 +0000

Just found that langsmith has launched gateway feature and I decided to share my experience. It is about how messy real LLM tooling decisions actually are, and why a small feature announcement from LangSmith made me sit up this month.

Where it all started

In August 2023, I onboarded my first RAG (retrieval-augmented generation — you fetch relevant documents first, then hand them to the model as context) chatbot client as an AI lead. Back then the models we had access to were much weaker and much less reliable than what you get today. Hallucinations were common, context windows were small, and you spent half your time just making the model behave.

The setup was simple, though. One client, one model, one hardcoded prompt in the code. No observability tool. No gateway. It just worked because there was nothing to break.

The step that looked solved

Then we got client number two. Then three. The obvious move was to copy the same folder, swap the API key, and ship it. For a while, that felt like scaling.

It was not. Our prompts lived scattered across different codebases. Our logs sat in our own Postgres database, one table per project, with no shared view across clients. When something went wrong for one client, we had no fast way to tell if it was that client's prompt, a model change, or something else entirely.

That is the real friction nobody warns you about. A broken prompt does not throw an error. It just quietly returns worse answers, and you might not notice for days — by which time a client has already noticed for you.

The decision that forced

That friction split into two separate problems we had to solve on purpose:

Where do we store and version prompts, and see what is actually happening across every client's traces (a trace is the full record of one request — every step, every token, every millisecond)?
How do we talk to multiple model providers through one shape, so a provider outage does not become our outage?

The first is an observability or "LLM ops" platform. The second is a gateway. Here is the order we actually went through both, mistakes included.

Round one: LangSmith, and giving up on it

LangSmith is LangChain's platform for tracing LLM calls, storing and versioning prompts, and running evaluations. We picked it up early because it was the obvious default in the LangChain ecosystem, and we were already leaning on some LangChain pieces.

We stuck with it for a few months before we dropped it. It kept breaking in ways that made us stop trusting the data it showed us — traces that did not line up with what we already had in our own Postgres logs. When your monitoring tool is less reliable than the thing it is monitoring, you stop using it.

Round two: the platform Anthropic later swallowed whole

So we moved to Humanloop, a platform built for the same prompt-management-and-evaluation job. It worked well enough for a good stretch.

Then, in August 2025, Anthropic acqui-hired Humanloop's founders and most of its team — no product, no IP, just the people. Humanloop as a platform sunset shortly after. If you have never seen an acqui-hire before: a bigger company hires the team, not the company, and the product itself just stops being maintained. We had to move a second time, through no fault of our own engineering choices.

Round three: back to LangSmith, warts and all

By then LangSmith had matured, and honestly, we did not have many other doors left to knock on. We are still on it today. It is not the platform I would design from scratch — there are features we need that are still missing — but it is stable enough now, and it gives us the cross-client visibility we were missing back in 2023.

Meanwhile, the gateway problem: why we picked LiteLLM

While all that platform-hopping was happening, we separately needed a gateway (a layer that sits between your app and every model provider, so your code talks to one consistent API instead of five different ones). We landed on LiteLLM, which we host ourselves on our own server.

LiteLLM is not perfect. We know about specific bugs in it, and we live with them, because the thing it does right is the thing we cannot do without: automatic fallback to another model or provider when the primary one errors out or gets rate-limited. That single feature is probably the most-used piece of LiteLLM in our whole stack. When a provider has a bad day, our clients never know.

This month: LangSmith adds its own Gateway

Then this month I noticed LangSmith has its own LLM Gateway — announced in private beta on May 13, 2026, and opened to public beta on July 30, 2026. My first reaction was genuinely good: one platform for traces, prompts, and provider routing instead of two separate systems to keep in sync.

I went in assuming the one feature we could not give up — fallback — would still be missing, because the original May announcement listed model fallback under "where this is going," as a future plan rather than a shipped feature. I was wrong about that. By the July public beta, LangSmith Gateway already ships configurable model fallbacks, chained across models, providers, and hosts, triggered by rate limits or provider errors. That is genuinely good news, since fallback is the one LiteLLM feature we would have refused to give up.

But here is the part I still cannot skip: latency. Our LiteLLM instance is self-hosted, on our own server. A request goes from our app to a local process and out to the provider — no extra hop over the public internet. LangSmith Gateway is a hosted, external service. Routing every completion through it adds a real network round trip we do not currently pay.

A fallback that reads well on a feature page has not been through a real outage on our own traffic yet. Before we retire LiteLLM, I want to see two numbers with our own eyes: the added latency per request, and how the fallback behaves the first time a provider actually goes down on us, not in a demo.

What I would tell you, if you were the one making this call

Do not trust a feature list over battle scars, even when the feature list is telling the truth. Test the exact thing you depend on most, under the exact load you actually run, before you let go of the tool that is already working.

Three years, four tools, one lesson that keeps repeating: the tool that wins is not the one with the longest feature list. It is the one that is still standing after the outage nobody planned for.

So tell me your version of this. Has anyone already pushed LangSmith Gateway through a real provider outage or a hard rate limit on real production traffic? Or did you go through your own round of tool-hopping — what made you leave a platform, and what finally made one stick? I would genuinely like to hear it before we run our own test.

Every Check Said Our GA4 Analytics Worked. It Had Never Recorded a Visit

Talha Anwar — Sat, 08 Aug 2026 10:06:48 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Adding analytics to a website is the most boring task in web development. Google hands you eight lines, you paste them above </head>, and traffic shows up in the dashboard about a minute later.

On a real app you tidy it up a little. The measurement ID moves into an environment variable, the snippet becomes a module, and you add a cookie banner with Consent Mode so nothing is tracked before a visitor agrees. Still a small job, and it still looks finished when the banner renders.

Here's the part nobody warns you about: every layer you just added can fail without saying a word. A build tool can drop the env var. A queue can accept commands it will never run. A CI secret can point at a property nobody is watching. None of them throws. None of them logs. The site loads fine, the banner works, and the dashboard stays at zero.

So the only check that means anything is the last one — did a hit actually arrive? This is the story of the three silent failures I had to clear before the answer was yes, and the second one nearly had me file a bug report against Google.

The setup

I work on AcruxCore, an LLM ops platform. Two sites: the marketing site and dashboard (apps/web, React + Vite, deployed in Docker), and the docs site (apps/docs, Docusaurus, deployed to Cloudflare Pages).

I shipped Google Analytics 4 to both in one PR. Consent Mode v2 with analytics_storage denied by default, a cookie banner, consent stored as a first-party cookie shared across subdomains.

I tested it properly before merging:

Playwright against a real dev server — accept, decline, reload, reopen from the footer
built the bundle with the env var set and grepped the output for the measurement ID
built it again with the var unset and confirmed the tag never loads

All green. I merged it, rebuilt the production image, and opened Google's tag inspector on the live site.

Tag not detected.

Bug 1 — the tag was never in the bundle

The cookie banner rendered on production, so the new frontend had definitely deployed. That's what made this confusing: the same PR shipped both, and only half of it arrived.

I SSH'd into the VPS and worked backward from the artifact rather than forward from the config.

# search the actually-served bundle for Google's script host
docker exec acruxcore-web grep -rc googletagmanager /usr/share/nginx/html/assets/
# → 0

Zero. Not a wrong ID, not a consent problem — the tag was never compiled in. A rebuild with docker compose build web still gave zero, and a full docker build --no-cache reproduced the same empty result.

At that point I'd checked the Docker ARG/ENV lines, the args: block in docker-compose.yml, and the define in vite.config.ts. All three were correct. So I stopped reading config and ran the build one layer down, inside the same built image:

# bypass the orchestrator, call the bundler directly
GA4_MEASUREMENT_ID=G-XXXXXXXXXX npx vite build
# → googletagmanager IS in the output

vite build produced the tag. turbo run build did not. Same image, same env, same everything else. The bug wasn't in Docker, Compose or Vite — it was in the layer between them.

What Turborepo was doing

Our Dockerfile builds through the monorepo task runner:

RUN npx turbo run build --filter=@acruxcore/web

Turborepo's default envMode is strict. Strict means a task's process starts with a filtered environment. It sees only the variables you declared in that task's env config, plus any auto-inferred from a recognized framework's naming convention. For Vite, that's anything prefixed VITE_.

Our variable is called GA4_MEASUREMENT_ID. It is deliberately not VITE_-prefixed, because vite.config.ts remaps it at build time:

define: {
  'import.meta.env.VITE_GA4_MEASUREMENT_ID':
    JSON.stringify(process.env.GA4_MEASUREMENT_ID ?? ''),
},

That ?? '' is the whole tragedy in two characters. Turbo stripped GA4_MEASUREMENT_ID before Vite's config file ever ran, so process.env.GA4_MEASUREMENT_ID was undefined, the fallback kicked in, and the bundle compiled a perfectly valid empty string. Analytics reads the ID, finds '', and correctly decides not to load. No error, because nothing went wrong — every layer did exactly what it was told.

Docker and Compose passing the variable in correctly made no difference. Turbo removed it after they handed it over.

npx turbo run build --dry=json --filter=@acruxcore/web \
  | jq '.tasks[0].envMode, .tasks[0].environmentVariables.specified.env'
# → "strict"
# → []      ← nothing declared, so nothing gets through

The fix is one line in turbo.json:

 "build": {
   "dependsOn": ["^build"],
   "outputs": ["dist/**", "build/**", ".docusaurus/**"],
+  "env": ["SENTRY_WEB_DSN", "GA4_MEASUREMENT_ID"]
 }

Mirror file: turbo.json

The part that actually scared me

GA4_MEASUREMENT_ID wasn't alone in that env array. SENTRY_WEB_DSN was there too, and I only added it because it has the same shape — a non-VITE_ name remapped in the same define block.

Then I reverted the fix and re-ran the build with SENTRY_WEB_DSN set, the same way I'd reproduced the GA4 failure. Identical result: stripped, empty string, no SDK.

Sentry had been wired into the web app six days earlier. In that window, the browser SDK on the deployed site had almost certainly never sent a single error report.

And nothing would ever have told me. A quiet analytics dashboard is at least ambiguous — maybe nobody visited. A quiet error monitor looks like good news. That is the failure mode I keep thinking about: the tool whose whole job is telling you when something is wrong can be the one thing that's broken, and its silence reads as success.

What wasn't affected. The docs site builds in CI with npm run build -w @acruxcore/docs — never through Turbo — so it picked up the measurement ID on the first deploy. Half the platform worked from day one, which is exactly why I initially suspected something specific to the marketing site.

Bug 2 — the tag loaded, the queue filled, nothing was sent

New image deployed. googletagmanager now in the bundle, script loading in the network tab, cookie banner working, consent granted.

GA4 Realtime: zero users.

I went through the browser console expecting an obvious break, and instead found every single signal healthy:

the https://www.googletagmanager.com/gtag/js?id=… script loads with a 200
window.google_tag_manager is defined
window.dataLayer.push !== Array.prototype.push — gtag.js has replaced the queue's push, so it is watching
the queue fills up with my consent, js and config commands

And no request to /g/collect. No _ga cookie. Nothing dispatched, ever.

I also ran GA4's own "Test your website" check from the admin panel. It went green — which I now know only means the tag script is present on the page. It never checks whether a hit arrives.

The wrong turn

Everything on my side was correct, so I started building the case that the problem was Google's.

I set up a second, completely unrelated GA4 property and pointed the site at it. Identical zero-dispatch behaviour. That felt conclusive: not my account, not my property, not my tag config.

I got as far as writing a standalone ga4-zero-hits-repro.html to hand over, drafting the issue report, and opening a PR to track the escalation. The report was finished and queued to post.

The moment it broke

Before posting I re-read my own repro page, and the thing I'd been ignoring finally registered. I had hand-written that page from Google's documentation rather than copying the code my product actually runs.

Which means it contained Google's canonical stub:

function gtag(){ window.dataLayer.push(arguments); }

And my product contained mine:

function gtag(...args: unknown[]): void {
  window.dataLayer = window.dataLayer ?? [];
  window.dataLayer.push(args);
}

Those look equivalent. TypeScript is happy, ESLint prefers the second one, and every reviewer including me read straight past it. My "repro of the bug" had quietly swapped out the one line that was the bug.

So I stopped reasoning and isolated the single variable. Two pages served from the same local origin, same measurement ID, same Consent Mode sequence, differing only in the push form:

stub	hits to `/g/collect`	`_ga` cookie
`push(args)` — what shipped	0	not set
`push(arguments)` — canonical	2 (`page_view`, `scroll`)	set

One line. Everything else identical.

Why `arguments` is load-bearing

gtag.js replaces dataLayer.push with its own processor. That queue carries two different kinds of thing, and the processor has to tell them apart:

commands — gtag('config', 'G-…'), gtag('consent', 'update', {…})
data-layer objects — ordinary values other tags push in for their own use

It tells them apart by checking whether the pushed value is a real arguments object. A rest parameter gives you a plain Array instead, which fails that check — so the value gets filed away as data and no command ever runs.

That includes config. Without config, the tag is initialised with no measurement ID, so it has nothing to send and no reason to complain. Everything downstream still looks alive because everything downstream is alive — the script really did load, the queue really is hooked, the commands really are in it. They're just sitting there.

I didn't get this from the docs. It's just the only difference between the two rows of that table. It's also a decent rule of thumb: when a vendor snippet uses an unfashionable function(){} where a modern one would do, assume the old form is doing something.

Lint rules encode what's usually true. prefer-rest-params is right nearly every time, and this is the case where "nearly" costs you all your data.

The fix

Keep Google's original stub verbatim, and leave a note loud enough to survive the next cleanup:

/**
 * **`arguments` here is load-bearing — do not "modernize" it into a rest
 * parameter.** gtag.js tells a command apart from a data-layer object by
 * checking the pushed value is a real `arguments` object. A plain array is
 * silently ignored: the tag loads, the queue fills, and not one hit is sent.
 */
export const gtag: GtagFn = function () {
  window.dataLayer = window.dataLayer ?? [];
  // eslint-disable-next-line prefer-rest-params -- a rest array is not a command
  window.dataLayer.push(arguments);
};

Mirror file: apps/web/src/lib/analytics.ts

A comment alone wasn't enough for me here, because the failure is invisible and the "cleanup" that reintroduces it is a one-character edit. So the test asserts the exact property that matters:

it('queues each command as an `arguments` object, which is what gtag.js recognizes', () => {
  vi.stubGlobal('window', {} as Window);
  gtag('config', 'G-TEST123');
  const queue = window.dataLayer ?? [];
  expect(Object.prototype.toString.call(queue[0])).toBe('[object Arguments]');
  expect(Array.from(queue[0] as IArguments)).toEqual(['config', 'G-TEST123']);
});

Rewrite the stub as a rest parameter and that test fails immediately with [object Array].

Verified end to end against a real vite build: accepting the consent banner produces a gcs=G101 hit and sets _ga.

Bug 3 — the site that worked was reporting to nobody

While isolating bug 2 I went back to the docs site as a control, because it had looked healthy from day one. Its snippet was hand-written directly into docusaurus.config.ts and never shared code with apps/web, so it had the correct arguments form all along.

It was sending hits. It was sending them to the wrong property.

Its measurement ID comes from a GitHub Actions secret that was never updated when the property changed, so every visit it recorded had been landing somewhere nobody was looking. No code change — update the secret, re-run the workflow — but it made the point for a third time in one week.

Working and reporting where you're watching are two different states, and only one of them shows up on a dashboard.

Before and after

	before	after
GA4 tag in the production bundle	absent	present
Sentry browser SDK in the production bundle	absent	present
gtag commands recognized by gtag.js	0 of 3	3 of 3
hits reaching `/g/collect`	0	`page_view`, `scroll`, consent hits
docs-site hits landing in the watched property	0	all of them
checks that had reported a problem	0	1 failing unit test if it regresses

That last row is the one I'd frame. Before the fix, the total number of automated signals warning me about any of this was zero.

What I'm carrying to the next codebase

A repro has to copy the product's code path, not your memory of it. I re-typed Google's snippet from the docs, which silently replaced the variable under test, and that one shortcut is the entire reason I wrote a bug report against Google. Paste the real function in and the two-page test finds it in ten minutes.

Testing a second account felt like ruling something out. It ruled out nothing. Both accounts ran the same broken client code, so the experiment could only ever return the same answer. A control group that shares the defect isn't a control group — and the confidence it gave me is what pushed me toward blaming Google instead of looking closer.

Assert the last link, not the layers. I had a check for the env var, a check for the bundle, a check for the banner, and Google's own check for the tag. Four green lights, zero data. Every one of them verified a precondition for a hit; none verified a hit.

When one config gap swallows a variable, check every variable of the same shape. SENTRY_WEB_DSN was found only because it looked like GA4_MEASUREMENT_ID, and it was the more dangerous of the two — nobody files a ticket saying "I'm getting suspiciously few errors."

If you've got analytics or error monitoring wired into a monorepo build, two things are worth ten minutes today: grep your deployed bundle for the tag host, and check the type of what your gtag stub actually pushes.

What did yours turn out to be — an Array, or Arguments?

Our Token Counter Took 26 Seconds on a Single Prompt

Talha Anwar — Fri, 07 Aug 2026 23:37:36 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with an Express API gateway. Every completion passes a budget pre-check: estimate the token cost, compare against the team's remaining spend, reject if it doesn't fit.

For OpenAI-family models, that estimate used js-tiktoken:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch {
    // Fall through to the heuristic if the encoder rejects the input.
  }
}
return Math.ceil(text.length / 4);

Three lines. They do the right thing. They just don't always do it quickly.

Bug Fix or Performance Improvement

Performance bug — a quadratic BPE tokenizer on the synchronous request path.

I didn't find this through a user complaint. Two datasets tests started timing out at Jest's 5-second limit. I assumed I'd broken something, stashed my changes, checked out clean staging, and ran them again. Still failed.

The tests were fine. The code under them was just brutally slow.

The BPE encoder is quadratic on whitespace-free input. BPE works by repeatedly merging the most common adjacent pair of characters. Each pass scans the entire string to find the best pair. One merge per pass, full scan each time. A string of N characters with no whitespace means roughly N passes over N characters — N² work.

Normal text never hits this because the pre-tokenizer splits on whitespace first. Every piece is one short word (~5 chars), so N² is nothing. A long unbroken string has no whitespace to split on, so the whole thing becomes one piece and N becomes the full length.

Measured on cl100k_base (GPT-4's tokenizer):

single piece	encode time
256 chars	4 ms
512 chars	18 ms
1,024 chars	70 ms
2,048 chars	281 ms
9,000 chars	5.4 s
20,000 chars	26 s

Doubling the length roughly quadruples the time. Meanwhile, 20,000 characters of ordinary prose encodes in 3 ms. It's not the length — it's the lack of whitespace.

This function runs on every single request. Node runs JavaScript on one thread. A 26-second synchronous encode doesn't just slow down one request — it freezes every other user on that process.

And the input that triggers it isn't exotic: a base64 blob in a prompt, a minified JSON payload, a corrupted copy-paste. Any authenticated caller could stall the whole process.

gateway completion, 2-char rendered variable:      82 ms
gateway completion, 9000-char rendered variable:  5506 ms

Same code path. Same mocked provider. One template variable different.

Why not just remove the tokenizer?

My first thought. But estimateTokens has two jobs:

Budget pre-check — reject requests that would exceed the spend cap. Overestimate and you reject valid requests. Underestimate and teams blow past their budget.
Billing — when a provider streams without usage data, this estimate becomes the number on the user's bill.

And chars / 4 is a rough rule of thumb for English prose. It breaks on other inputs. Our test suite uses a synthetic 400 KB input (1,600 repetitions of 250 x characters plus a space — the same one that caused the timeouts):

Characters: 1,600 × 251 = 401,600
Exact BPE:  52,800 tokens  →  401,600 / 52,800 = 7.6 chars/token
chars / 4:  100,400 tokens  →  401,600 / 4 = 100,400 (nearly 2× the truth)

If your budget allows 80,000 tokens, the real prompt fits (52,800) but chars / 4 says it doesn't (100,400) and rejects it. Using chars / 4 everywhere would overestimate so aggressively that it would reject roughly half of all valid prompts.

So the tokenizer stays. It just doesn't get to run unbounded anymore.

Code

The work happened in a private repo, so the PR link won't help. The fixed file is public in our mirror, with the measurement table in the doc comment:

apps/api/src/gateway/providers/token-estimate.ts

Before — the entire string in one encoder call:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

After — two decisions: how much text to encode, and how to chop it up first:

if (isOpenAiFamily) {
  try {
    // Bound 2: past 20,000 chars, encode a sample and scale the rest.
    if (text.length <= MAX_BPE_SAMPLE_CHARS) return boundedEncode(text);
    const sampleTokens = boundedEncode(text.slice(0, MAX_BPE_SAMPLE_CHARS));
    const ratio = sampleTokens / MAX_BPE_SAMPLE_CHARS;
    return sampleTokens + Math.ceil((text.length - MAX_BPE_SAMPLE_CHARS) * ratio);
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

My Improvements

Two bounds. No new dependencies. No change to what it returns for normal input.

Bound 1 — cap each piece at 256 characters

Nothing over 256 characters reaches the encoder. Longer pieces get ceil(chars / 4) as a quick estimate instead.

This removes the quadratic cost entirely, because 256 chars keeps each encoder call under ~4 ms.

Important: 256 is a threshold, not a chunk size. A 1,024-character piece is not split into four 256-char chunks. It skips the encoder entirely and gets ceil(1024 / 4) = 256 directly. Zero encoder calls.

The text is only split at whitespace boundaries — to keep token counts accurate:

function boundedEncode(text: string): number {
  let tokens = 0;
  let pendingSeparator = '';
  for (const part of text.split(/(\s+)/)) {
    if (part.length === 0) continue;
    if (/^\s+$/.test(part)) {
      pendingSeparator += part;
      continue;
    }
    const piece = pendingSeparator + part;
    pendingSeparator = '';
    tokens += piece.length > MAX_BPE_PIECE_CHARS
      ? heuristic(piece.length)
      : encoder().encode(piece).length;
  }
  if (pendingSeparator.length > 0) {
    tokens += pendingSeparator.length > MAX_BPE_PIECE_CHARS
      ? heuristic(pendingSeparator.length)
      : encoder().encode(pendingSeparator).length;
  }
  return tokens;
}

The easy line to get wrong is pendingSeparator. Whitespace must be attached to the following word because tiktoken groups " word" as one token. Encode the space on its own and " the" becomes two tokens instead of one — every prose prompt gets overcounted by roughly its word count, and budget checks silently start rejecting requests they shouldn't.

A correctness bug from a performance fix is a bad trade.

Effect on the 20,000-character blob

	before	after
encoder calls	one, on 20,000 chars	none (20,000 > 256)
time	26,952 ms	0 ms
tokens returned	2,500	5,000

The count doubles — that's deliberate. ceil(chars / 4) overestimates, but 20,000 identical characters isn't a real prompt, and for a budget check, overestimating is the safe direction.

Ordinary text is untouched. "Hello world" splits into "Hello" and " world", both under 256, both encoded exactly as before.

Bound 2 — cap the total at 20,000 characters

Bound 1 only looks at one piece at a time. It never catches the case of many short pieces adding up.

1,600 repetitions of 250 xs plus a space = 400 KB. Each piece is 250 chars (under the 256 cap), so Bound 1 lets every one through. At ~4 ms each, that's 7 seconds. Bound 1 never triggers once.

Bound 2 fixes this: encode at most 20,000 characters total. For anything longer, encode the first 20K, measure the tokens-per-char ratio, and scale that ratio across the rest.

Step by step:

Take the first 20,000 characters
Run boundedEncode on them (still uses Bound 1 internally)
Get the token count from the sample
Divide by 20,000 to get a tokens-per-char ratio
Multiply the ratio by the remaining characters
Add sample tokens + tail estimate = return value

Effect on the 400 KB input

	bound 1 only	both bounds
characters encoded	401,600	20,000
time	7,018 ms	334 ms
tokens returned	52,800	52,771

The first 20K characters gave 2,628 tokens. Applied to the remaining 381,600 characters: 50,143 tokens. Total: 52,771 vs exact 52,800.

Sampled ratio:  2,628 / 20,000 = 0.13 tokens per character
chars / 4 ratio:            1 / 4 = 0.25 tokens per character

0.13 is roughly half of 0.25. Using chars / 4 would claim 95,400 tokens on the tail instead of 50,143 — nearly double the truth. The sample already measured the real ratio, so using it is almost as accurate as encoding everything, without the 7-second cost.

0.05% off, for 5% of the work.

Why 20,000 characters? At the 256-char piece cap, worst case is ~80 pieces (20,000 / 256). At ~4 ms each, that's ~320 ms — fast enough to stay on the request path. Anything bigger gets the sample treatment. The number isn't magic; it's where "encode everything" becomes "too slow."

Worst case is now roughly 350 ms for any input size.

Error direction

Bound 1 can only overestimate — ceil(chars / 4) always returns more tokens than real BPE on the inputs that trigger it. For a budget check, that's the safe direction.

Bound 2 is different: it extrapolates from a sample, so it can land slightly above or below the true count. The 400 KB case came out 0.05% low. But that only applies past 20,000 characters, and a fraction of a percent drift on a huge prompt is not a budget problem. A 7-second event-loop stall is.

Proving it didn't change the answer

A bounded estimator that returns different numbers is a new bug, not a fix. Tests pin exactness first, speed second.

Exactness — must match whole-string BPE on all normal input (under 20K chars):

it('matches whole-string BPE counts for ordinary text', () => {
  const encoder = getEncoding('cl100k_base');
  const samples = [
    'Hi Alice, what is the weather in London?',
    'The quick brown fox jumps over the lazy dog. '.repeat(114),
    'const x = foo.bar(baz, 42); // comment here\n'.repeat(100),
    JSON.stringify({ a: 'hello world', b: [1, 2, 3], c: { d: 'nested value here' } }).repeat(50),
  ];
  for (const sample of samples) {
    expect(estimateTokens(sample, 'gpt-4o-mini')).toBe(encoder.encode(sample).length);
  }
});

Prose, code, JSON: identical counts. No existing estimate moved.

Direction — pathological input must never come back low:

it('never underestimates the pathological input', () => {
  const encoder = getEncoding('cl100k_base');
  const text = 'x'.repeat(2000);
  expect(estimateTokens(text, 'gpt-4o-mini')).toBeGreaterThanOrEqual(encoder.encode(text).length);
});

Speed — one test per bound, so removing either guard fails immediately:

it('stays fast on a long unbroken run of characters', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const started = Date.now();
  const n = estimateTokens('x'.repeat(50_000), 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

it('stays fast when many pieces sit just under the per-piece cap', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const text = `${'x'.repeat(250)} `.repeat(1600);
  const started = Date.now();
  const n = estimateTokens(text, 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

The 1-second limit is deliberately generous — won't flake on loaded CI, but still fails instantly if someone removes a guard (the unfixed path takes minutes).

Results

	before	after
50,000-char unbroken run	minutes	< 1 s
400 KB of 250-char pieces	7,018 ms	334 ms
completion, 9,000-char variable (provider mocked)	5,506 ms	~82 ms
`datasets` test A	6,340 ms	410 ms
`datasets` test B	6,243 ms	323 ms

The timing-out tests were never touched. They passed because the code underneath stopped being slow.

What I'd carry to another codebase

Test adversarial input, not your own. Every prompt I'd written was ordinary prose — the tokenizer's best case. Timing 'x'.repeat(20000) took ten minutes and found the whole bug.
A synchronous function on a request path is a shared resource. "It's just a pure function over a string" stops being reassuring when the string is caller-controlled and the cost is superlinear.
Pin the old answer before you optimize. An exact-match test against whole-string encoding is how I caught the pendingSeparator detail that would have silently overcounted every prompt.

If you're running a BPE tokenizer anywhere synchronous, it's worth timing encoder.encode('x'.repeat(20000)) on your own setup today. What did you get?