DEV Community: Talha Anwar

I published AI research papers and have 6 YoE building backends. Why do AI coding agents make me feel completely dumb?

Talha Anwar — Sat, 22 Aug 2026 08:02:26 +0000

There is a running joke that AI researchers can't write production software, and backend engineers don't understand research math.

I happen to do both. I’ve published deep learning papers in top conferences, won ML competitions, and spent the last 6 years as a founding engineer building production APIs, async queues, and distributed systems. I read dense math papers and trace database deadlocks for fun.

Yet, on any given day, an AI coding agent can "fix a bug" in 20 minutes and make me question if I even know how to read English.

The "10x Productivity" Myth vs. Management Expectations

Management reads one article on Twitter and suddenly expects everyone to ship features at 10x speed.

Here is what that "10x velocity" actually looks like when hooking up a RAG chatbot to external platform webhooks and syncing vector stores:

1 hour: Planning the sync logic, payload schema, and chunking strategy.
20 minutes: The AI agent writes 400 lines of code across 5 files to fix a streaming concurrency bug. Tests pass. The app runs. Pure magic.
4 hours: Me sitting frozen in front of my monitor trying to reverse-engineer why it bypassed our API gateway, restructured our entire websocket pipeline, and injected three layers of mysterious custom middleware.

The code is "done" in 20 minutes, but now I’m in a full sweat because tomorrow morning in standup, someone is going to ask:

"Hey, what was actually broken with the webhook listener?"

And what am I supposed to say?

"Honestly? The AI wizard cast a spell on line 42 and now it works. Please don't ask me anything else."

The 4-Stage Spiral of Shame

When I don't understand why the agent chose a specific architecture to fix the bug, I ask it to explain. That’s when the psychological damage begins, with my confidence dropping on every single attempt:

Attempt 1: The Pretentious Professor

I ask: "Why did you restructure the vector search callback and event listener this way?"

It replies with a 500-word essay containing words I haven't seen since a GRE test. I re-read it four times. My brain goes numb. Is my English broken? Am I secretly bad at backend engineering?

Attempt 2: The Polite Request

I say: Please explain in simpler, direct terms without the jargon.

It uses everyday words, but strings them together so backwards that the logic eats its own tail:

Of course! We just make sure that when the webhook fires, the thing checking the data knows that the data being checked isn't the same data that triggered the check in the first place.

No fancy jargon, but my brain is completely melted trying to parse the sentence structure. Is my reading comprehension broken? Confidence drops another 25%.

Attempt 3: The Desperation Move

I type: "Explain like I'm 15."

The AI instantly decides I have the cognitive ability of a toddler:

"Imagine your vector database is a magical post office! When a user sends a message, a friendly owl named Barnaby flies across the forest to find the matching letter..."

Attempt 4: Installing the "Fixes"

At this point, I started searching for community prompt hacks. I even installed the popular ayghri/i-have-adhd skill to force the agent into concise, no-fluff, step-by-step bullet points.

And you know what happened? Now the agent gives me blunt, hyper-compressed steps, and I still spend 30 minutes staring at bullet point #2 trying to understand the mathematical jump it made. Now I feel twice as dumb because I can't even digest the ADHD-friendly version.

The Real Problem: Reverse-Engineering Fatigue

Getting an AI-generated integration or bug fix to run is easy. Owning the code in a team of senior engineers is the hard part.

When a teammate looks at the PR and asks:

"Why did we switch to this custom event-emitter pattern instead of our standard message queue?"

I can't reply:

"Well, Barnaby the Owl needed a place to rest his wings."

So you spend half your day decoding AI logic just to translate it into normal human engineer terms. We didn't eliminate the coding bottleneck—we just traded writing code for full-time reverse-engineering.

Let's Settle This in the Comments:

How do you handle standup when AI fixed a bug but you have zero idea how? Do you fake confidence or admit the robot did it?
What is the most insulting analogy an AI has ever given you when you asked it to "simplify"?
What exact system prompt do you use to make your agent talk like a normal, grounded senior dev without getting a PhD thesis or a fairy tale?

Our SDK Cached One User's Answer and Served It to the Next One Who Asked

Talha Anwar — Tue, 11 Aug 2026 19:21:55 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with a published SDK in TypeScript and Python. Both SDKs render prompts client-side against a small in-memory cache, so a hot path doesn't make a network round trip on every call:

const rendered = await hub.renderPrompt('rag-chat', 'production', { question });

renderPrompt fetches the templated messages once, then serves that same result out of the cache for cacheTtl milliseconds (60 seconds by default) before checking the API again. That's the whole feature. Two bugs were hiding in how "serve that same result" actually decided what counted as "the same."

Bug Fix or Performance Improvement

Correctness bug — the cache key didn't include the one thing that changes on every real call: the variables.

The key was built from three fields:

const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;

API key, prompt name, alias. Nothing about what you're actually asking. Two different renders of the same prompt with different variables, inside the same TTL window, hashed to the identical key — so the second call got the first call's cached answer back, not its own.

This wasn't a hypothetical. It was live in our own docs. The RAG-without-a-gateway tutorial renders the same prompt per question:

rendered = await hub.prompts.render("rag-chat", "production",
                                     {"context": context, "question": question})

Ask "Where is my order?", then ask "How do I refund?" within 60 seconds, and the second call's rendered.messages still says "Where is my order?" — while the trace viewer, which logs what you passed in, correctly shows "How do I refund?" as the input. The trace and the actual prompt sent to the model disagree, and nothing errors. It's also exactly why a sibling tutorial, the tool-calling agent guide, never puts the question in a template variable at all — it appends it to the messages array in code instead, a design forced by this bug rather than a stylistic choice.

The second bug was in the same file, one function down:

if (age < this.cacheTtl) {
  return cached.value; // fresh — serve it directly
}
// stale — serve the old value anyway, refresh in the background

Setting cacheTtl: 0 to turn caching off did the opposite. age < 0 is never true, so every single call took the stale branch: return the cached value immediately, and kick off a background refetch nobody's waiting on. A TTL of 0 didn't mean "don't cache" — it meant "always serve whatever was cached first, forever." There was no setting that actually disabled the cache.

Two bugs, one root cause

Both come from the same gap: the cache didn't have a way to represent "this input is different" or "this call opted out." One key was too coarse; one flag had no matching branch. Fixing them meant adding both.

Code

The fix is public in our mirror — same logic in both SDKs, shown here in TypeScript:

packages/sdk/src/client.ts

Bug 1 — hash the variables into the key:

/**
 * Serialises a value with object keys in sorted order, at every depth, so that
 * two variable maps that differ only in insertion order produce one string.
 */
function stableStringify(value: unknown): string {
  if (value === null || typeof value !== 'object') {
    return JSON.stringify(value) ?? 'null';
  }
  if (Array.isArray(value)) {
    return `[${value.map(stableStringify).join(',')}]`;
  }
  const entries = Object.entries(value as Record<string, unknown>)
    .filter(([, v]) => v !== undefined)
    .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
  return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(',')}}`;
}

function hashVariables(variables: Record<string, unknown>): string {
  return createHash('sha256').update(stableStringify(variables)).digest('hex').slice(0, 16);
}

- const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;
+ const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}:${hashVariables(variables)}`;

Bug 2 — make a non-positive TTL actually skip the cache, on read and write:

  async renderPrompt(name, alias, variables = {}) {
+   // A non-positive TTL means "never serve a cached render" — skip the read and
+   // pass a null key so the write side skips too, instead of filling the cache
+   // with entries no read path will ever consult.
+   if (this.cacheTtl <= 0) {
+     return this._fetchAndCache(name, alias, variables, null);
+   }
    const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
    const cacheKey = `...`;
    ...
  }

  async _fetchAndCache(name, alias, variables, cacheKey /* now: string | null */) {
    ...
-   const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
-   cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   if (cacheKey !== null) {
+     const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
+     cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   }
    return value;
  }

Technical Approach

Why hash sorted keys at every depth instead of `JSON.stringify(variables)`

JSON.stringify preserves insertion order. { name: 'Alice', city: 'Lahore' } and { city: 'Lahore', name: 'Alice' } are the same request to any caller, but they'd stringify to two different strings — and split into two cache entries for what should be one. A caller that builds the variables object by spreading defaults first ({ ...defaults, question }) versus spreading them last would silently halve their own cache hit rate. Sorting keys at every depth, including inside nested objects, makes key order invisible to the cache. A dedicated unit test pins exactly this:

it('treats the same variables in a different key order as one cache entry', async () => {
  (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValue(makeOkResponse());
  await hub.renderPrompt('my-prompt', 'production', { name: 'Alice', city: 'Lahore' });
  await hub.renderPrompt('my-prompt', 'production', { city: 'Lahore', name: 'Alice' });
  expect(global.fetch).toHaveBeenCalledTimes(1);
});

Why not just delete the "serve stale, refresh in background" path

That's a real reliability feature, not accidental complexity: if the API is briefly unreachable, a caller with a warm cache keeps getting an answer instead of an error. Deleting it to fix the cacheTtl: 0 bug would trade one bug for a regression. The fix instead adds a third state — "don't cache at all" — that bypasses the stale-serving branch entirely rather than replacing it, so cacheTtl: 30_000 still falls back to a stale value during an outage, and only cacheTtl: 0 gives that up on purpose. Both behaviors are documented side by side in the README so the trade-off is explicit, not discovered during an incident.

Proving the fix actually fixes it, against a real API

Unit tests mock fetch. The two bugs above are about what a real render pipeline does with real prompt versions, so the integration suite hits an actual Express app and a real Postgres database instead:

it('renderPrompt renders the new variables inside the cache window instead of replaying the first render', async () => {
  // ...creates a real prompt, a real version with "Question: {{ question }}"...
  const hub = new acruxcore({ apiKey, baseUrl, cacheTtl: 600_000, maxRetries: 0 });

  const first = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });
  const second = await hub.renderPrompt(name, 'production', { question: 'How do I refund?' });
  const repeat = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });

  expect(first.messages[0].content).toBe('Question: Where is my order?');
  expect(second.messages[0].content).toBe('Question: How do I refund?');
  expect(repeat.messages[0].content).toBe('Question: Where is my order?'); // still cached, not evicted
});

I reverted src/client.ts and re-ran this test before writing the fix. It failed on exactly the assertion you'd expect — second.messages[0].content came back as 'Question: Where is my order?', the first question, not the second. That's the failure this whole post describes, caught by a test instead of a support ticket.

A second integration test does the same for Bug 2 — cacheTtl: 0, promote a new prompt version, confirm the very next call sees it instead of serving the version that was cached before the promotion.

Two existing tests were asserting the bug

The old unit test for caching was itself proof the bug had never been tested against a variable change:

- it('caches the result — second call does not hit fetch', async () => {
+ it('caches the result — repeating the same variables does not hit fetch', async () => {
    await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
-   await hub.renderPrompt('my-prompt', 'production', { name: 'Bob' });
+   await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
    expect(global.fetch).toHaveBeenCalledTimes(1);
  });

It rendered Alice, then Bob, and asserted one fetch call — which is precisely the bug, written down as a passing test. Fixing the implementation without touching this test would have left it green while lying about what "caching" meant. It had to be rewritten to assert the correct behavior — different inputs, different fetches — before the fix could be trusted.

Results

	before	after
second `renderPrompt` call, different variables, inside TTL	first call's cached content (wrong)	its own rendered content
repeated call, same variables as the first, inside TTL	cached content (correct, coincidentally)	cached content (correct, by design)
`cacheTtl: 0`, two calls	1st content served both times	fresh API call both times
entries written to the cache with `cacheTtl: 0`	one per call	zero
variables passed in a different key order	new cache entry (cache pressure ↑)	same entry, one hit

Test suite after the fix: 111 TS unit, 14 TS integration (both new ones failing-then-passing verified against the revert), 112 Python, clean tsc --noEmit, clean docs build.

What I'm carrying to the next cache I write

A cache key has to include everything the caller varies, not just everything the caller identifies with. API key and prompt name answer "who and what" — they don't answer "with what input," and that's the part that actually changes per call.
"Off" needs its own code path, not a boundary value on an existing one. 0 looked like it should mean "never fresh, so never serve cached" — the actual check (age < ttl) made it mean the opposite. A dedicated bypass, not a clever value, is what you want for a real off-switch.
A passing cache test that only ever sends one input is testing the plumbing, not the cache key. The bug survived in production because the one existing test for "does caching work" happened to use two different inputs and asserted the outcome the bug produces.

If your SDK caches a rendered or computed result keyed by anything less than the full input, it's worth a five-minute check today: call it twice with different arguments inside your TTL window, and read back what you actually get. What did yours return?

LLM Observability Tools Compared: Langfuse vs Helicone vs Opik vs Phoenix

Talha Anwar — Tue, 11 Aug 2026 19:00:00 +0000

The first trace looks the same everywhere

Wrap your LLM client with any open-source observability SDK — Langfuse, Helicone, Opik, Phoenix, doesn't matter which — and the first result is identical: a request goes out, a span shows up in a dashboard with the prompt, the response, and how long it took. That part is solved. Every landing page in this space makes the same claim, and for a first trace, they're all telling the truth.

The next question every team asks is also the same one: can this tool act on a trace, not just show me one? Score it. Alert someone. Block it before it ships. On the surface that looks solved too — all four platforms have an "Evaluation" tab and a "Settings" page, so it's tempting to assume they converge there as well.

They don't. I have all four self-hosted right now, and going through each dashboard side by side, four specific capabilities stopped me — each one a genuine standout for exactly one platform, and absent from at least two of the other three. Picking a tool because "it does observability" and finding out six months later that the one feature you actually needed lives in a dashboard you didn't choose is an expensive way to learn this.

TL;DR

Langfuse — an LLM judge that scores every live production trace on its own, no manual review step required.
Helicone — a dedicated rate-limit rule builder, a different object from a simple spend cap.
Opik — topic and PII guardrails that inspect a call's input or output directly, configurable per project.
Phoenix (built by Arize) — PXI, an AI agent docked inside the dashboard that already knows the trace you're looking at.

At a glance

Feature	Langfuse	Helicone	Opik	Phoenix
Auto-scores live traces, no manual trigger	✅ LLM-as-a-judge	❌	✅ Online evaluation rules (same idea, different name)	❌
Dedicated rate-limit rule builder	❌ (alerts only)	✅	❌	❌
Topic / PII guardrails on the call itself	❌	❌	✅	❌
AI agent built into the dashboard	❌	❌	❌	✅ PXI

Opik is the only platform that clears two of the four rows. Full breakdown below.

Langfuse: the judge that never sleeps

Most platforms' "evaluation" means a human clicks run against a dataset. Langfuse's Evaluators page does something different: you write an eval template — a scoring prompt, a model, and the trace variables it should read — and it fires that template against every new production trace as it lands. There's no "run experiment" button in the loop at all.

That's the deciding detail if you're comparing tools on this feature: is scoring something you trigger, or something that's already running before you open the dashboard?

Langfuse's own docs walk through setting one up. Opik's Online evaluation rules do the same job under a different name (see the table above), so Langfuse doesn't own this category outright — but it's the platform where this run found it first, with execution tracing on the evaluator itself as a bonus.

Takeaway: if unattended, always-on trace scoring is the requirement, Langfuse and Opik are the two platforms here built for it. Helicone and Phoenix both still need someone to explicitly trigger a run.

Helicone: a rate-limit rule, not a spend cap

A spend budget answers one question: how much can this app cost before it stops. A rate-limit rule answers a different one: how many requests can this specific user make in this window, independent of what anything costs.

Helicone's Monitor → Rate Limits page has a "Create Rule" builder for exactly that second question. A rule is a quota, a time window, a unit (requests, tokens, or dollars), and a segment — user, team, or global — expressed as a header like 10;w=1000;u=cents;s=user, which reads as "10 cents of requests per 1000 seconds, per user." Helicone's docs cover the full syntax.

Takeaway: if one noisy user needs throttling without shutting the whole app's budget down for everyone else, that's a rate-limit rule, and Helicone is the only one of the four that ships one as its own object.

Opik: guardrails that inspect the call itself

Everything above scores or alerts after a call happens. Opik's guardrails act on the call directly — before or after the model sees it, depending on which check you enable.

The "Set a guardrail" panel has two checks: a Topic guardrail (a sensitivity slider plus a comma-separated restricted-topics list) and a PII guardrail (per-category toggles — credit card number, phone number, email, and more — each with its own sensitivity threshold). Both ship with a ready opik.guardrails Python snippet. Comet's announcement post covers what triggers each one.

Takeaway: if the requirement is blocking a category of content before it ships, not scoring it afterward, guardrails are the only feature in this list built for that — and Opik is the only platform that has both this and an auto-scoring rule.

Phoenix: PXI, the assistant that already knows your traces

PXI is a chat panel that lives on every Phoenix page, seeded with suggestions like "Find critical issues" and "Explain a concept." The detail that makes it more than a docs chatbot bolted onto the product: it reads from whatever page you're already on, so it can answer questions about your own traces without you pasting a trace ID or copying a payload into it. Arize's docs describe it as an agent that debugs traces, builds evaluators, and runs experiments from the context you're already viewing.

Takeaway: if "explain what's happening in this trace, right now, without switching tools" is the requirement, Phoenix is the only platform here with an assistant scoped to the page in front of you.

Verdict

None of these four features decide which tool is "best" — they decide which tool fits a requirement you may not have hit yet. A team that never needs unattended scoring won't miss Langfuse's judge or Opik's online evaluation. A team running one internal chatbot for five people won't miss Helicone's rate-limit rules.

But if you're picking a platform today assuming "observability" is a solved, interchangeable checkbox, this is the list of four ways that assumption breaks: automated live-trace scoring, a real rate-limit object, content guardrails, and an in-dashboard agent. Check which of the four you'll actually need before you wrap your first client, not after.

Which of these four would you actually use — and is there a fifth dashboard feature I missed that deserves its own row?

Verdict

This piece only had room for one standout feature per platform. AcruxCore has compared 8 LLM observability tools in more depth, aspect by aspect.

Which of these four would you actually use — and is there a fifth dashboard feature I missed that deserves its own row?

Opik vs Langfuse: Where Two Open-Source LLM Observability Tools Actually Agree (and Where They Don't)

Talha Anwar — Sun, 09 Aug 2026 19:55:19 +0000

The box both of them check

If you're adding your first bit of visibility into an LLM app, the simplest version is a print() statement before the API call. That's enough while you're the only one testing it.

The natural next step is to swap that print statement for a real tracing SDK. You wrap your client, make a call, and a UI shows you a span with the prompt, the response, and how long it took. That feels solved — until your team grows past one prompt and one person testing it by hand.

That's the point where the real friction shows up: conditional prompt logic, a way to diff two prompt versions before you ship one, a way to catch a call your app got wrong before a user complains, and a currently-live guardrail against risky output. "It produces a trace" doesn't cover any of that. And the frustrating part is that this doesn't fail loudly — most open-source, self-hosted LLM observability tools look interchangeable from their landing pages, because they'd all check the same three boxes: open source, self-hosted, traces your calls.

Opik (built by Comet ML) and Langfuse are two of the most-used tools in that box. So we tested the actual boundary: we built the exact same prompt — a support-triage agent that changes tone for VIP customers and lists their open tickets — on both, independently, then set the two runs side by side.

TL;DR

Both platforms hit the exact same wall on prompt templating — neither supports real conditional logic, only flat variable substitution.
Both confirmed, independently, that their Playground alone does not produce a trace — only a wrapped SDK call does.
They diverge sharply on prompt versioning: Opik ships a real diff view and environment labels, Langfuse's version history has neither.
Their measured client-side overhead is different in kind, not just number — and the two numbers come from two separate test sessions, not one shared benchmark.
Both have a feature that does the same job under a different name — auto-scoring live production traffic.
They've drifted into genuinely different territory: guardrails and regression test suites (Opik) versus threshold alerting (Langfuse).
And as of five months ago, they're no longer comparable as two independent startups — Langfuse was acquired by ClickHouse in January 2026.

At a glance

Aspect	Opik	Langfuse
Prompt templating	Flat `{{variable}}` only	Flat `{{variable}}` only
Playground produces a trace	No — confirmed hands-on	No — confirmed hands-on
Version diff	Real red/green diff view	No diff control found
Environment labels	"Deploy to" tags a version	`production`/`latest` labels
Measured SDK overhead	+102ms (real, CI doesn't cross zero)	+15ms (not distinguishable from zero)
Auto-scores live traffic	Online Evaluation rules	LLM-as-a-judge evaluators
Standout feature	Guardrails (topic + PII) and test suites	Monitors — alerts to Slack/Webhooks/GitHub
GitHub stars (checked today)	21,252	32,778
Owned by	Comet ML (in-house product)	ClickHouse (acquired Jan 2026)

Full breakdown below.

Where they hit the exact same wall: templating

Both platforms' prompt editors only do flat {{variable}} substitution — no {% if %}, no {% for %}. We needed a VIP-customer branch and a rendered list of open tickets, and on both platforms we had to flatten that logic into plain text ourselves before the prompt ever saw it, keeping only the simple fields (like {{company}}) as real template variables.

Neither product hides this — it's a real, shared design choice, not a bug in either one. If your prompts need branching logic, both push that logic back onto your own code (a pre-render step) rather than into the template itself.

Where they hit the exact same wall, again: the Playground doesn't trace

This one surprised us enough to check twice on each platform. Running a prompt from Opik's Playground left its Logs tab at "No traces yet." Running a prompt from Langfuse's Playground did the same thing — no trace appeared until we called the SDK-wrapped client instead. On both, the interactive "try it in the UI" surface and the "get a trace out of it" surface are two separate things, which is easy to miss if you only ever click around the Playground before wiring in the SDK.

Where they genuinely diverge: prompt versioning

Opik has a real "Diff" button — edit a prompt to create v2, and it shows the old system message in red strikethrough next to the new one in green. It also has a "Deploy to" menu that tags a specific version production, staging, or development with a colored badge.

Langfuse has environment labels too (production / latest), but we could not find any diff or compare control anywhere in its version history UI — each version sits in a flat list, and comparing two of them means opening both and reading side by side yourself.

Where the numbers genuinely differ (and why you shouldn't stack them)

We measured client-side SDK overhead for both platforms — but in two separate benchmark sessions, run on different days as part of two different posts, each with its own direct-to-provider baseline. That matters: network conditions differ session to session, so these two numbers aren't from one shared interleaved test and shouldn't be read as a single ranked table.

With that caveat stated plainly:

	Opik SDK	Langfuse SDK
Measured overhead	+102ms	+15ms
95% bootstrap CI	[+14, +191]ms	[-90, +150]ms
Statistically real?	Yes — CI does not cross zero	No — CI crosses zero, indistinguishable from noise

Comparison with other platform can be found at AcruxCore

Opik's track_openai() wrapper adds overhead that clears the noise floor in its own test session. Langfuse's instrumentation, in its own separate test session, didn't clear that floor at all — its "cost" was statistically zero. Both are real findings about client-side instrumentation, not about which product is "faster" than the other, since neither routes a call through a gateway the way a proxy-style tool would.

Same idea, different name: scoring production traffic automatically

Both platforms have a feature that does the same underlying job — score live traffic as it lands, with no manual review step — under completely different names and UI.

Langfuse's LLM-as-a-judge Evaluators walk you through writing an eval template: a scoring prompt, a model to run it, and the trace variables it reads.

Opik's Online Evaluation page, under Production, lets you create a rule that scores every matching trace as it arrives.

If a platform-neutral feature list only checked for the name "online evaluation" or "LLM-as-a-judge," it would miss that both products actually offer this.

Where they've genuinely diverged: guardrails vs. alerting

Opik ships a guardrails panel — a topic-restriction check with a sensitivity slider and a comma-separated restricted-topics list, plus a PII (personally identifiable information) check that flags categories like credit card numbers, phone numbers, and email addresses, each with its own threshold. It also has a dedicated Test suites object for pre-deployment regression testing, distinct from its Experiments feature — import cases from a CSV/JSON file or the SDK, each with an expected output and a scoring method.

Langfuse doesn't ship anything like that. What it has instead is Monitors — set a threshold on cost, quality, or latency once, and get a notification routed to Slack, a webhook, or a GitHub Action when a metric moves outside it.

Neither one has the other's feature. If you need to block a call before it reaches a user for containing PII, Langfuse has nothing for that today. If you need a Slack ping when your P95 latency creeps up, Opik has nothing for that today.

Who's behind them, and how big is each

This is where the story changed since we first tested these products. Numbers below are live as of this week, pulled straight from each project's GitHub API.

	Opik	Langfuse
GitHub stars	21,252	32,778
Forks	1,690	3,524
Contributors	~136	~190
Open issues	178	764
Repo created	May 2023	May 2023

Both repos were created within a week of each other in May 2023 and are still receiving commits daily — this isn't a "one's abandoned" story. Langfuse has the larger community footprint on every raw count, including a notably higher open-issue count, which can mean more active usage surfacing more edge cases, or a smaller team keeping up with triage — we didn't dig into issue-close rates to tell which.

The bigger difference is who owns each project. Opik is a product inside Comet ML, a New York-based ML platform company founded in 2017, with $6.8M in disclosed seed funding (2018, 2020) and nothing disclosed since. Langfuse was an independent, Y Combinator-backed startup — until ClickHouse acquired it on January 16, 2026, the same day ClickHouse announced a $400M Series D at a $15B valuation. Both companies have publicly committed to keeping Langfuse open source and self-hostable going forward.

For a reader picking a platform to build on, that's a real signal worth weighing alongside the feature list: one tool is a side product of a smaller, steady company; the other just became an integration point inside a much larger, newly-flush infrastructure company. Neither is inherently the safer bet — but they're different bets.

We also compared each project's documentation footprint via its own sitemap, with a caveat: Langfuse's sitemap covers its entire domain (blog, changelog, academy, and docs together — 822 URLs, of which only 112 are strictly /docs), while Opik's docs live under Comet's site and its sitemap is scoped to just that docs subtree (709 URLs). So "709 vs 822" isn't an apples-to-apples count of documentation depth — it's closer to "Opik's docs alone" versus "Langfuse's docs plus its blog, changelog, and handbook combined."

What we're not calling a winner

We picked seven points, not the full feature list either platform ships, and skipped a symmetric "pro/con" for every row — some things really are one-sided (neither platform gained a fake counterweight just to look fair). Templating and Playground-tracing are genuine ties. Versioning favors Opik. The overhead numbers aren't comparable to each other, only to their own baselines. Guardrails/test-suites and Monitors/alerting are two teams building in different directions, not one being behind the other.

If you're choosing between them today, the feature list would point you toward whichever gap matters more to your team — guardrails and regression testing, or threshold alerting. The ownership question is the one that didn't exist a year ago, and it might now matter just as much as either feature list.

Sources: comet-ml/opik and langfuse/langfuse on GitHub (checked 2026-08-10); ClickHouse's acquisition announcement; Langfuse's own "joining ClickHouse" post. Product screenshots and hands-on findings are from our own independent testing of both platforms.

LangSmith Just Shipped an LLM Gateway. Here's the 3-Year Mess That Made Me Care

Talha Anwar — Sat, 08 Aug 2026 13:33:14 +0000

Just found that langsmith has launched gateway feature and I decided to share my experience. It is about how messy real LLM tooling decisions actually are, and why a small feature announcement from LangSmith made me sit up this month.

Where it all started

In August 2023, I onboarded my first RAG (retrieval-augmented generation — you fetch relevant documents first, then hand them to the model as context) chatbot client as an AI lead. Back then the models we had access to were much weaker and much less reliable than what you get today. Hallucinations were common, context windows were small, and you spent half your time just making the model behave.

The setup was simple, though. One client, one model, one hardcoded prompt in the code. No observability tool. No gateway. It just worked because there was nothing to break.

The step that looked solved

Then we got client number two. Then three. The obvious move was to copy the same folder, swap the API key, and ship it. For a while, that felt like scaling.

It was not. Our prompts lived scattered across different codebases. Our logs sat in our own Postgres database, one table per project, with no shared view across clients. When something went wrong for one client, we had no fast way to tell if it was that client's prompt, a model change, or something else entirely.

That is the real friction nobody warns you about. A broken prompt does not throw an error. It just quietly returns worse answers, and you might not notice for days — by which time a client has already noticed for you.

The decision that forced

That friction split into two separate problems we had to solve on purpose:

Where do we store and version prompts, and see what is actually happening across every client's traces (a trace is the full record of one request — every step, every token, every millisecond)?
How do we talk to multiple model providers through one shape, so a provider outage does not become our outage?

The first is an observability or "LLM ops" platform. The second is a gateway. Here is the order we actually went through both, mistakes included.

Round one: LangSmith, and giving up on it

LangSmith is LangChain's platform for tracing LLM calls, storing and versioning prompts, and running evaluations. We picked it up early because it was the obvious default in the LangChain ecosystem, and we were already leaning on some LangChain pieces.

We stuck with it for a few months before we dropped it. It kept breaking in ways that made us stop trusting the data it showed us — traces that did not line up with what we already had in our own Postgres logs. When your monitoring tool is less reliable than the thing it is monitoring, you stop using it.

Round two: the platform Anthropic later swallowed whole

So we moved to Humanloop, a platform built for the same prompt-management-and-evaluation job. It worked well enough for a good stretch.

Then, in August 2025, Anthropic acqui-hired Humanloop's founders and most of its team — no product, no IP, just the people. Humanloop as a platform sunset shortly after. If you have never seen an acqui-hire before: a bigger company hires the team, not the company, and the product itself just stops being maintained. We had to move a second time, through no fault of our own engineering choices.

Round three: back to LangSmith, warts and all

By then LangSmith had matured, and honestly, we did not have many other doors left to knock on. We are still on it today. It is not the platform I would design from scratch — there are features we need that are still missing — but it is stable enough now, and it gives us the cross-client visibility we were missing back in 2023.

Meanwhile, the gateway problem: why we picked LiteLLM

While all that platform-hopping was happening, we separately needed a gateway (a layer that sits between your app and every model provider, so your code talks to one consistent API instead of five different ones). We landed on LiteLLM, which we host ourselves on our own server.

LiteLLM is not perfect. We know about specific bugs in it, and we live with them, because the thing it does right is the thing we cannot do without: automatic fallback to another model or provider when the primary one errors out or gets rate-limited. That single feature is probably the most-used piece of LiteLLM in our whole stack. When a provider has a bad day, our clients never know.

This month: LangSmith adds its own Gateway

Then this month I noticed LangSmith has its own LLM Gateway — announced in private beta on May 13, 2026, and opened to public beta on July 30, 2026. My first reaction was genuinely good: one platform for traces, prompts, and provider routing instead of two separate systems to keep in sync.

I went in assuming the one feature we could not give up — fallback — would still be missing, because the original May announcement listed model fallback under "where this is going," as a future plan rather than a shipped feature. I was wrong about that. By the July public beta, LangSmith Gateway already ships configurable model fallbacks, chained across models, providers, and hosts, triggered by rate limits or provider errors. That is genuinely good news, since fallback is the one LiteLLM feature we would have refused to give up.

But here is the part I still cannot skip: latency. Our LiteLLM instance is self-hosted, on our own server. A request goes from our app to a local process and out to the provider — no extra hop over the public internet. LangSmith Gateway is a hosted, external service. Routing every completion through it adds a real network round trip we do not currently pay.

A fallback that reads well on a feature page has not been through a real outage on our own traffic yet. Before we retire LiteLLM, I want to see two numbers with our own eyes: the added latency per request, and how the fallback behaves the first time a provider actually goes down on us, not in a demo.

What I would tell you, if you were the one making this call

Do not trust a feature list over battle scars, even when the feature list is telling the truth. Test the exact thing you depend on most, under the exact load you actually run, before you let go of the tool that is already working.

Three years, four tools, one lesson that keeps repeating: the tool that wins is not the one with the longest feature list. It is the one that is still standing after the outage nobody planned for.

So tell me your version of this. Has anyone already pushed LangSmith Gateway through a real provider outage or a hard rate limit on real production traffic? Or did you go through your own round of tool-hopping — what made you leave a platform, and what finally made one stick? I would genuinely like to hear it before we run our own test.

Every Check Said Our GA4 Analytics Worked. It Had Never Recorded a Visit

Talha Anwar — Sat, 08 Aug 2026 10:06:48 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Adding analytics to a website is the most boring task in web development. Google hands you eight lines, you paste them above </head>, and traffic shows up in the dashboard about a minute later.

On a real app you tidy it up a little. The measurement ID moves into an environment variable, the snippet becomes a module, and you add a cookie banner with Consent Mode so nothing is tracked before a visitor agrees. Still a small job, and it still looks finished when the banner renders.

Here's the part nobody warns you about: every layer you just added can fail without saying a word. A build tool can drop the env var. A queue can accept commands it will never run. A CI secret can point at a property nobody is watching. None of them throws. None of them logs. The site loads fine, the banner works, and the dashboard stays at zero.

So the only check that means anything is the last one — did a hit actually arrive? This is the story of the three silent failures I had to clear before the answer was yes, and the second one nearly had me file a bug report against Google.

The setup

I work on AcruxCore, an LLM ops platform. Two sites: the marketing site and dashboard (apps/web, React + Vite, deployed in Docker), and the docs site (apps/docs, Docusaurus, deployed to Cloudflare Pages).

I shipped Google Analytics 4 to both in one PR. Consent Mode v2 with analytics_storage denied by default, a cookie banner, consent stored as a first-party cookie shared across subdomains.

I tested it properly before merging:

Playwright against a real dev server — accept, decline, reload, reopen from the footer
built the bundle with the env var set and grepped the output for the measurement ID
built it again with the var unset and confirmed the tag never loads

All green. I merged it, rebuilt the production image, and opened Google's tag inspector on the live site.

Tag not detected.

Bug 1 — the tag was never in the bundle

The cookie banner rendered on production, so the new frontend had definitely deployed. That's what made this confusing: the same PR shipped both, and only half of it arrived.

I SSH'd into the VPS and worked backward from the artifact rather than forward from the config.

# search the actually-served bundle for Google's script host
docker exec acruxcore-web grep -rc googletagmanager /usr/share/nginx/html/assets/
# → 0

Zero. Not a wrong ID, not a consent problem — the tag was never compiled in. A rebuild with docker compose build web still gave zero, and a full docker build --no-cache reproduced the same empty result.

At that point I'd checked the Docker ARG/ENV lines, the args: block in docker-compose.yml, and the define in vite.config.ts. All three were correct. So I stopped reading config and ran the build one layer down, inside the same built image:

# bypass the orchestrator, call the bundler directly
GA4_MEASUREMENT_ID=G-XXXXXXXXXX npx vite build
# → googletagmanager IS in the output

vite build produced the tag. turbo run build did not. Same image, same env, same everything else. The bug wasn't in Docker, Compose or Vite — it was in the layer between them.

What Turborepo was doing

Our Dockerfile builds through the monorepo task runner:

RUN npx turbo run build --filter=@acruxcore/web

Turborepo's default envMode is strict. Strict means a task's process starts with a filtered environment. It sees only the variables you declared in that task's env config, plus any auto-inferred from a recognized framework's naming convention. For Vite, that's anything prefixed VITE_.

Our variable is called GA4_MEASUREMENT_ID. It is deliberately not VITE_-prefixed, because vite.config.ts remaps it at build time:

define: {
  'import.meta.env.VITE_GA4_MEASUREMENT_ID':
    JSON.stringify(process.env.GA4_MEASUREMENT_ID ?? ''),
},

That ?? '' is the whole tragedy in two characters. Turbo stripped GA4_MEASUREMENT_ID before Vite's config file ever ran, so process.env.GA4_MEASUREMENT_ID was undefined, the fallback kicked in, and the bundle compiled a perfectly valid empty string. Analytics reads the ID, finds '', and correctly decides not to load. No error, because nothing went wrong — every layer did exactly what it was told.

Docker and Compose passing the variable in correctly made no difference. Turbo removed it after they handed it over.

npx turbo run build --dry=json --filter=@acruxcore/web \
  | jq '.tasks[0].envMode, .tasks[0].environmentVariables.specified.env'
# → "strict"
# → []      ← nothing declared, so nothing gets through

The fix is one line in turbo.json:

 "build": {
   "dependsOn": ["^build"],
   "outputs": ["dist/**", "build/**", ".docusaurus/**"],
+  "env": ["SENTRY_WEB_DSN", "GA4_MEASUREMENT_ID"]
 }

Mirror file: turbo.json

The part that actually scared me

GA4_MEASUREMENT_ID wasn't alone in that env array. SENTRY_WEB_DSN was there too, and I only added it because it has the same shape — a non-VITE_ name remapped in the same define block.

Then I reverted the fix and re-ran the build with SENTRY_WEB_DSN set, the same way I'd reproduced the GA4 failure. Identical result: stripped, empty string, no SDK.

Sentry had been wired into the web app six days earlier. In that window, the browser SDK on the deployed site had almost certainly never sent a single error report.

And nothing would ever have told me. A quiet analytics dashboard is at least ambiguous — maybe nobody visited. A quiet error monitor looks like good news. That is the failure mode I keep thinking about: the tool whose whole job is telling you when something is wrong can be the one thing that's broken, and its silence reads as success.

What wasn't affected. The docs site builds in CI with npm run build -w @acruxcore/docs — never through Turbo — so it picked up the measurement ID on the first deploy. Half the platform worked from day one, which is exactly why I initially suspected something specific to the marketing site.

Bug 2 — the tag loaded, the queue filled, nothing was sent

New image deployed. googletagmanager now in the bundle, script loading in the network tab, cookie banner working, consent granted.

GA4 Realtime: zero users.

I went through the browser console expecting an obvious break, and instead found every single signal healthy:

the https://www.googletagmanager.com/gtag/js?id=… script loads with a 200
window.google_tag_manager is defined
window.dataLayer.push !== Array.prototype.push — gtag.js has replaced the queue's push, so it is watching
the queue fills up with my consent, js and config commands

And no request to /g/collect. No _ga cookie. Nothing dispatched, ever.

I also ran GA4's own "Test your website" check from the admin panel. It went green — which I now know only means the tag script is present on the page. It never checks whether a hit arrives.

The wrong turn

Everything on my side was correct, so I started building the case that the problem was Google's.

I set up a second, completely unrelated GA4 property and pointed the site at it. Identical zero-dispatch behaviour. That felt conclusive: not my account, not my property, not my tag config.

I got as far as writing a standalone ga4-zero-hits-repro.html to hand over, drafting the issue report, and opening a PR to track the escalation. The report was finished and queued to post.

The moment it broke

Before posting I re-read my own repro page, and the thing I'd been ignoring finally registered. I had hand-written that page from Google's documentation rather than copying the code my product actually runs.

Which means it contained Google's canonical stub:

function gtag(){ window.dataLayer.push(arguments); }

And my product contained mine:

function gtag(...args: unknown[]): void {
  window.dataLayer = window.dataLayer ?? [];
  window.dataLayer.push(args);
}

Those look equivalent. TypeScript is happy, ESLint prefers the second one, and every reviewer including me read straight past it. My "repro of the bug" had quietly swapped out the one line that was the bug.

So I stopped reasoning and isolated the single variable. Two pages served from the same local origin, same measurement ID, same Consent Mode sequence, differing only in the push form:

stub	hits to `/g/collect`	`_ga` cookie
`push(args)` — what shipped	0	not set
`push(arguments)` — canonical	2 (`page_view`, `scroll`)	set

One line. Everything else identical.

Why `arguments` is load-bearing

gtag.js replaces dataLayer.push with its own processor. That queue carries two different kinds of thing, and the processor has to tell them apart:

commands — gtag('config', 'G-…'), gtag('consent', 'update', {…})
data-layer objects — ordinary values other tags push in for their own use

It tells them apart by checking whether the pushed value is a real arguments object. A rest parameter gives you a plain Array instead, which fails that check — so the value gets filed away as data and no command ever runs.

That includes config. Without config, the tag is initialised with no measurement ID, so it has nothing to send and no reason to complain. Everything downstream still looks alive because everything downstream is alive — the script really did load, the queue really is hooked, the commands really are in it. They're just sitting there.

I didn't get this from the docs. It's just the only difference between the two rows of that table. It's also a decent rule of thumb: when a vendor snippet uses an unfashionable function(){} where a modern one would do, assume the old form is doing something.

Lint rules encode what's usually true. prefer-rest-params is right nearly every time, and this is the case where "nearly" costs you all your data.

The fix

Keep Google's original stub verbatim, and leave a note loud enough to survive the next cleanup:

/**
 * **`arguments` here is load-bearing — do not "modernize" it into a rest
 * parameter.** gtag.js tells a command apart from a data-layer object by
 * checking the pushed value is a real `arguments` object. A plain array is
 * silently ignored: the tag loads, the queue fills, and not one hit is sent.
 */
export const gtag: GtagFn = function () {
  window.dataLayer = window.dataLayer ?? [];
  // eslint-disable-next-line prefer-rest-params -- a rest array is not a command
  window.dataLayer.push(arguments);
};

Mirror file: apps/web/src/lib/analytics.ts

A comment alone wasn't enough for me here, because the failure is invisible and the "cleanup" that reintroduces it is a one-character edit. So the test asserts the exact property that matters:

it('queues each command as an `arguments` object, which is what gtag.js recognizes', () => {
  vi.stubGlobal('window', {} as Window);
  gtag('config', 'G-TEST123');
  const queue = window.dataLayer ?? [];
  expect(Object.prototype.toString.call(queue[0])).toBe('[object Arguments]');
  expect(Array.from(queue[0] as IArguments)).toEqual(['config', 'G-TEST123']);
});

Rewrite the stub as a rest parameter and that test fails immediately with [object Array].

Verified end to end against a real vite build: accepting the consent banner produces a gcs=G101 hit and sets _ga.

Bug 3 — the site that worked was reporting to nobody

While isolating bug 2 I went back to the docs site as a control, because it had looked healthy from day one. Its snippet was hand-written directly into docusaurus.config.ts and never shared code with apps/web, so it had the correct arguments form all along.

It was sending hits. It was sending them to the wrong property.

Its measurement ID comes from a GitHub Actions secret that was never updated when the property changed, so every visit it recorded had been landing somewhere nobody was looking. No code change — update the secret, re-run the workflow — but it made the point for a third time in one week.

Working and reporting where you're watching are two different states, and only one of them shows up on a dashboard.

Before and after

	before	after
GA4 tag in the production bundle	absent	present
Sentry browser SDK in the production bundle	absent	present
gtag commands recognized by gtag.js	0 of 3	3 of 3
hits reaching `/g/collect`	0	`page_view`, `scroll`, consent hits
docs-site hits landing in the watched property	0	all of them
checks that had reported a problem	0	1 failing unit test if it regresses

That last row is the one I'd frame. Before the fix, the total number of automated signals warning me about any of this was zero.

What I'm carrying to the next codebase

A repro has to copy the product's code path, not your memory of it. I re-typed Google's snippet from the docs, which silently replaced the variable under test, and that one shortcut is the entire reason I wrote a bug report against Google. Paste the real function in and the two-page test finds it in ten minutes.

Testing a second account felt like ruling something out. It ruled out nothing. Both accounts ran the same broken client code, so the experiment could only ever return the same answer. A control group that shares the defect isn't a control group — and the confidence it gave me is what pushed me toward blaming Google instead of looking closer.

Assert the last link, not the layers. I had a check for the env var, a check for the bundle, a check for the banner, and Google's own check for the tag. Four green lights, zero data. Every one of them verified a precondition for a hit; none verified a hit.

When one config gap swallows a variable, check every variable of the same shape. SENTRY_WEB_DSN was found only because it looked like GA4_MEASUREMENT_ID, and it was the more dangerous of the two — nobody files a ticket saying "I'm getting suspiciously few errors."

If you've got analytics or error monitoring wired into a monorepo build, two things are worth ten minutes today: grep your deployed bundle for the tag host, and check the type of what your gtag stub actually pushes.

What did yours turn out to be — an Array, or Arguments?

Our Token Counter Took 26 Seconds on a Single Prompt

Talha Anwar — Fri, 07 Aug 2026 23:37:36 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with an Express API gateway. Every completion passes a budget pre-check: estimate the token cost, compare against the team's remaining spend, reject if it doesn't fit.

For OpenAI-family models, that estimate used js-tiktoken:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch {
    // Fall through to the heuristic if the encoder rejects the input.
  }
}
return Math.ceil(text.length / 4);

Three lines. They do the right thing. They just don't always do it quickly.

Bug Fix or Performance Improvement

Performance bug — a quadratic BPE tokenizer on the synchronous request path.

I didn't find this through a user complaint. Two datasets tests started timing out at Jest's 5-second limit. I assumed I'd broken something, stashed my changes, checked out clean staging, and ran them again. Still failed.

The tests were fine. The code under them was just brutally slow.

The BPE encoder is quadratic on whitespace-free input. BPE works by repeatedly merging the most common adjacent pair of characters. Each pass scans the entire string to find the best pair. One merge per pass, full scan each time. A string of N characters with no whitespace means roughly N passes over N characters — N² work.

Normal text never hits this because the pre-tokenizer splits on whitespace first. Every piece is one short word (~5 chars), so N² is nothing. A long unbroken string has no whitespace to split on, so the whole thing becomes one piece and N becomes the full length.

Measured on cl100k_base (GPT-4's tokenizer):

single piece	encode time
256 chars	4 ms
512 chars	18 ms
1,024 chars	70 ms
2,048 chars	281 ms
9,000 chars	5.4 s
20,000 chars	26 s

Doubling the length roughly quadruples the time. Meanwhile, 20,000 characters of ordinary prose encodes in 3 ms. It's not the length — it's the lack of whitespace.

This function runs on every single request. Node runs JavaScript on one thread. A 26-second synchronous encode doesn't just slow down one request — it freezes every other user on that process.

And the input that triggers it isn't exotic: a base64 blob in a prompt, a minified JSON payload, a corrupted copy-paste. Any authenticated caller could stall the whole process.

gateway completion, 2-char rendered variable:      82 ms
gateway completion, 9000-char rendered variable:  5506 ms

Same code path. Same mocked provider. One template variable different.

Why not just remove the tokenizer?

My first thought. But estimateTokens has two jobs:

Budget pre-check — reject requests that would exceed the spend cap. Overestimate and you reject valid requests. Underestimate and teams blow past their budget.
Billing — when a provider streams without usage data, this estimate becomes the number on the user's bill.

And chars / 4 is a rough rule of thumb for English prose. It breaks on other inputs. Our test suite uses a synthetic 400 KB input (1,600 repetitions of 250 x characters plus a space — the same one that caused the timeouts):

Characters: 1,600 × 251 = 401,600
Exact BPE:  52,800 tokens  →  401,600 / 52,800 = 7.6 chars/token
chars / 4:  100,400 tokens  →  401,600 / 4 = 100,400 (nearly 2× the truth)

If your budget allows 80,000 tokens, the real prompt fits (52,800) but chars / 4 says it doesn't (100,400) and rejects it. Using chars / 4 everywhere would overestimate so aggressively that it would reject roughly half of all valid prompts.

So the tokenizer stays. It just doesn't get to run unbounded anymore.

Code

The work happened in a private repo, so the PR link won't help. The fixed file is public in our mirror, with the measurement table in the doc comment:

apps/api/src/gateway/providers/token-estimate.ts

Before — the entire string in one encoder call:

if (isOpenAiFamily) {
  try {
    return encoder().encode(text).length;
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

After — two decisions: how much text to encode, and how to chop it up first:

if (isOpenAiFamily) {
  try {
    // Bound 2: past 20,000 chars, encode a sample and scale the rest.
    if (text.length <= MAX_BPE_SAMPLE_CHARS) return boundedEncode(text);
    const sampleTokens = boundedEncode(text.slice(0, MAX_BPE_SAMPLE_CHARS));
    const ratio = sampleTokens / MAX_BPE_SAMPLE_CHARS;
    return sampleTokens + Math.ceil((text.length - MAX_BPE_SAMPLE_CHARS) * ratio);
  } catch { /* fall through to heuristic */ }
}
return heuristic(text.length);

My Improvements

Two bounds. No new dependencies. No change to what it returns for normal input.

Bound 1 — cap each piece at 256 characters

Nothing over 256 characters reaches the encoder. Longer pieces get ceil(chars / 4) as a quick estimate instead.

This removes the quadratic cost entirely, because 256 chars keeps each encoder call under ~4 ms.

Important: 256 is a threshold, not a chunk size. A 1,024-character piece is not split into four 256-char chunks. It skips the encoder entirely and gets ceil(1024 / 4) = 256 directly. Zero encoder calls.

The text is only split at whitespace boundaries — to keep token counts accurate:

function boundedEncode(text: string): number {
  let tokens = 0;
  let pendingSeparator = '';
  for (const part of text.split(/(\s+)/)) {
    if (part.length === 0) continue;
    if (/^\s+$/.test(part)) {
      pendingSeparator += part;
      continue;
    }
    const piece = pendingSeparator + part;
    pendingSeparator = '';
    tokens += piece.length > MAX_BPE_PIECE_CHARS
      ? heuristic(piece.length)
      : encoder().encode(piece).length;
  }
  if (pendingSeparator.length > 0) {
    tokens += pendingSeparator.length > MAX_BPE_PIECE_CHARS
      ? heuristic(pendingSeparator.length)
      : encoder().encode(pendingSeparator).length;
  }
  return tokens;
}

The easy line to get wrong is pendingSeparator. Whitespace must be attached to the following word because tiktoken groups " word" as one token. Encode the space on its own and " the" becomes two tokens instead of one — every prose prompt gets overcounted by roughly its word count, and budget checks silently start rejecting requests they shouldn't.

A correctness bug from a performance fix is a bad trade.

Effect on the 20,000-character blob

	before	after
encoder calls	one, on 20,000 chars	none (20,000 > 256)
time	26,952 ms	0 ms
tokens returned	2,500	5,000

The count doubles — that's deliberate. ceil(chars / 4) overestimates, but 20,000 identical characters isn't a real prompt, and for a budget check, overestimating is the safe direction.

Ordinary text is untouched. "Hello world" splits into "Hello" and " world", both under 256, both encoded exactly as before.

Bound 2 — cap the total at 20,000 characters

Bound 1 only looks at one piece at a time. It never catches the case of many short pieces adding up.

1,600 repetitions of 250 xs plus a space = 400 KB. Each piece is 250 chars (under the 256 cap), so Bound 1 lets every one through. At ~4 ms each, that's 7 seconds. Bound 1 never triggers once.

Bound 2 fixes this: encode at most 20,000 characters total. For anything longer, encode the first 20K, measure the tokens-per-char ratio, and scale that ratio across the rest.

Step by step:

Take the first 20,000 characters
Run boundedEncode on them (still uses Bound 1 internally)
Get the token count from the sample
Divide by 20,000 to get a tokens-per-char ratio
Multiply the ratio by the remaining characters
Add sample tokens + tail estimate = return value

Effect on the 400 KB input

	bound 1 only	both bounds
characters encoded	401,600	20,000
time	7,018 ms	334 ms
tokens returned	52,800	52,771

The first 20K characters gave 2,628 tokens. Applied to the remaining 381,600 characters: 50,143 tokens. Total: 52,771 vs exact 52,800.

Sampled ratio:  2,628 / 20,000 = 0.13 tokens per character
chars / 4 ratio:            1 / 4 = 0.25 tokens per character

0.13 is roughly half of 0.25. Using chars / 4 would claim 95,400 tokens on the tail instead of 50,143 — nearly double the truth. The sample already measured the real ratio, so using it is almost as accurate as encoding everything, without the 7-second cost.

0.05% off, for 5% of the work.

Why 20,000 characters? At the 256-char piece cap, worst case is ~80 pieces (20,000 / 256). At ~4 ms each, that's ~320 ms — fast enough to stay on the request path. Anything bigger gets the sample treatment. The number isn't magic; it's where "encode everything" becomes "too slow."

Worst case is now roughly 350 ms for any input size.

Error direction

Bound 1 can only overestimate — ceil(chars / 4) always returns more tokens than real BPE on the inputs that trigger it. For a budget check, that's the safe direction.

Bound 2 is different: it extrapolates from a sample, so it can land slightly above or below the true count. The 400 KB case came out 0.05% low. But that only applies past 20,000 characters, and a fraction of a percent drift on a huge prompt is not a budget problem. A 7-second event-loop stall is.

Proving it didn't change the answer

A bounded estimator that returns different numbers is a new bug, not a fix. Tests pin exactness first, speed second.

Exactness — must match whole-string BPE on all normal input (under 20K chars):

it('matches whole-string BPE counts for ordinary text', () => {
  const encoder = getEncoding('cl100k_base');
  const samples = [
    'Hi Alice, what is the weather in London?',
    'The quick brown fox jumps over the lazy dog. '.repeat(114),
    'const x = foo.bar(baz, 42); // comment here\n'.repeat(100),
    JSON.stringify({ a: 'hello world', b: [1, 2, 3], c: { d: 'nested value here' } }).repeat(50),
  ];
  for (const sample of samples) {
    expect(estimateTokens(sample, 'gpt-4o-mini')).toBe(encoder.encode(sample).length);
  }
});

Prose, code, JSON: identical counts. No existing estimate moved.

Direction — pathological input must never come back low:

it('never underestimates the pathological input', () => {
  const encoder = getEncoding('cl100k_base');
  const text = 'x'.repeat(2000);
  expect(estimateTokens(text, 'gpt-4o-mini')).toBeGreaterThanOrEqual(encoder.encode(text).length);
});

Speed — one test per bound, so removing either guard fails immediately:

it('stays fast on a long unbroken run of characters', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const started = Date.now();
  const n = estimateTokens('x'.repeat(50_000), 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

it('stays fast when many pieces sit just under the per-piece cap', () => {
  estimateTokens('warm up the encoder', 'gpt-4o-mini');
  const text = `${'x'.repeat(250)} `.repeat(1600);
  const started = Date.now();
  const n = estimateTokens(text, 'gpt-4o-mini');
  expect(n).toBeGreaterThan(0);
  expect(Date.now() - started).toBeLessThan(1000);
});

The 1-second limit is deliberately generous — won't flake on loaded CI, but still fails instantly if someone removes a guard (the unfixed path takes minutes).

Results

	before	after
50,000-char unbroken run	minutes	< 1 s
400 KB of 250-char pieces	7,018 ms	334 ms
completion, 9,000-char variable (provider mocked)	5,506 ms	~82 ms
`datasets` test A	6,340 ms	410 ms
`datasets` test B	6,243 ms	323 ms

The timing-out tests were never touched. They passed because the code underneath stopped being slow.

What I'd carry to another codebase

Test adversarial input, not your own. Every prompt I'd written was ordinary prose — the tokenizer's best case. Timing 'x'.repeat(20000) took ten minutes and found the whole bug.
A synchronous function on a request path is a shared resource. "It's just a pure function over a string" stops being reassuring when the string is caller-controlled and the cost is superlinear.
Pin the old answer before you optimize. An exact-match test against whole-string encoding is how I caught the pendingSeparator detail that would have silently overcounted every prompt.

If you're running a BPE tokenizer anywhere synchronous, it's worth timing encoder.encode('x'.repeat(20000)) on your own setup today. What did you get?

DEV Community: Talha Anwar

I published AI research papers and have 6 YoE building backends. Why do AI coding agents make me feel completely dumb?

The "10x Productivity" Myth vs. Management Expectations

The 4-Stage Spiral of Shame

The Real Problem: Reverse-Engineering Fatigue

Let's Settle This in the Comments:

Our SDK Cached One User's Answer and Served It to the Next One Who Asked

Project Overview

Bug Fix or Performance Improvement

Two bugs, one root cause

Code

Technical Approach

Why hash sorted keys at every depth instead of JSON.stringify(variables)

Why not just delete the "serve stale, refresh in background" path

Proving the fix actually fixes it, against a real API

Two existing tests were asserting the bug

Results

What I'm carrying to the next cache I write

LLM Observability Tools Compared: Langfuse vs Helicone vs Opik vs Phoenix

The first trace looks the same everywhere

TL;DR

At a glance

Langfuse: the judge that never sleeps

Helicone: a rate-limit rule, not a spend cap

Opik: guardrails that inspect the call itself

Phoenix: PXI, the assistant that already knows your traces

Verdict

Verdict

Opik vs Langfuse: Where Two Open-Source LLM Observability Tools Actually Agree (and Where They Don't)

The box both of them check

TL;DR

At a glance

Where they hit the exact same wall: templating

Where they hit the exact same wall, again: the Playground doesn't trace

Where they genuinely diverge: prompt versioning

Where the numbers genuinely differ (and why you shouldn't stack them)

Same idea, different name: scoring production traffic automatically

Where they've genuinely diverged: guardrails vs. alerting

Who's behind them, and how big is each

What we're not calling a winner

LangSmith Just Shipped an LLM Gateway. Here's the 3-Year Mess That Made Me Care

Where it all started

The step that looked solved

The decision that forced

Round one: LangSmith, and giving up on it

Round two: the platform Anthropic later swallowed whole

Round three: back to LangSmith, warts and all

Meanwhile, the gateway problem: why we picked LiteLLM

This month: LangSmith adds its own Gateway

What I would tell you, if you were the one making this call

Every Check Said Our GA4 Analytics Worked. It Had Never Recorded a Visit

The setup

Bug 1 — the tag was never in the bundle

What Turborepo was doing

The part that actually scared me

Bug 2 — the tag loaded, the queue filled, nothing was sent

The wrong turn

The moment it broke

Why arguments is load-bearing

The fix

Bug 3 — the site that worked was reporting to nobody

Before and after

What I'm carrying to the next codebase

Our Token Counter Took 26 Seconds on a Single Prompt

Project Overview

Bug Fix or Performance Improvement

Why not just remove the tokenizer?

Code

My Improvements

Bound 1 — cap each piece at 256 characters

Effect on the 20,000-character blob

Bound 2 — cap the total at 20,000 characters

Effect on the 400 KB input

Error direction

Proving it didn't change the answer

Results

What I'd carry to another codebase

Why hash sorted keys at every depth instead of `JSON.stringify(variables)`

Why `arguments` is load-bearing