DEV Community: Himanshu Sharma

When to Use an AI Agent Instead of a Simple LLM Call

Himanshu Sharma — Tue, 13 Oct 2026 08:57:00 +0000

The word “agent” is being attached to almost every AI feature right now. A prompt calls a model, the model returns JSON, and suddenly the architecture diagram has an agent in it.

That label is not harmless. An agent brings a loop, tool access, state, new failure modes, more latency, and a larger surface to evaluate.

A useful default is much simpler:

Use the least autonomous architecture that can solve the problem reliably.

Sometimes that is one LLM call. Sometimes it is a workflow controlled by your code. An agent becomes useful only when the model must decide what to do next based on what it discovers.

This article gives you a practical way to choose.

Three architectures that are often confused

1. A direct LLM call

Your application provides an input. The model returns an answer or a structured object. Your code validates the result and continues.

Input → Model → Validated output

Examples include:

Classifying a support ticket
Extracting fields from an invoice
Rewriting text for a specific audience
Generating a summary from supplied documents
Producing a typed object from a prompt

There may be retrieval or context preparation before the call. That still does not make the system an agent if the model is not controlling the execution path.

2. A deterministic workflow

A workflow uses multiple steps, but your application decides the order.

Input → Classify → Retrieve → Draft → Validate → Output

A model can power one or more steps. The important point is that code owns the control flow.

For example, a document processing system might:

Extract text from a file
Classify the document
Choose a schema with normal application logic
Ask a model to extract structured data
Validate the result
Send invalid results to review

This is more capable than one model call, but it is still predictable. You can inspect every stage, retry a specific step, and write focused tests for each transition.

3. A tool using agent

An agent receives a goal and controls at least part of the execution path. It chooses an action, observes the result, and then decides what to do next.

Goal → Decide → Use tool → Observe → Decide again → Finish

OpenAI describes an agent as a system that independently accomplishes tasks and uses an LLM to manage workflow execution. Anthropic makes a similar distinction between workflows with predefined code paths and agents where the model directs its own process and tool use.

The idea is closely related to the ReAct pattern, which combines reasoning with actions and observations from an external environment.

The key difference is control.

A model inside a workflow follows the path your code selected. An agent helps select the path.

A practical comparison

Question	Direct call	Fixed workflow	Agent
Who controls the next step?	Application	Application	Model within guardrails
Number of model turns	Usually one	Known in advance	Varies by task
Tool choice	None or fixed	Chosen by code	Chosen at runtime
Predictability	High	High to medium	Medium to low
Latency	Lowest	Bounded	Variable
Cost	Lowest	Bounded	Variable
Debugging	One request and response	Inspect each stage	Inspect the full trace
Best fit	Transformation or judgment	Known business process	Open ended investigation and action

This table is not a maturity ladder. An agent is not the advanced version of every LLM feature. Each architecture solves a different control problem.

The question that usually decides it

Ask this:

Can my application know the next step before it sees the current result?

If the answer is yes, keep the control flow in code.

If the answer is no because the result determines which source to inspect, which tool to use, or whether the plan must change, an agent may be justified.

Consider a production incident assistant.

A fixed workflow can always fetch the same dashboard, read the latest deployment, summarize recent logs, and prepare a report. That works when the investigation procedure is stable.

An agent is useful when the evidence changes the investigation. A database timeout might lead it to query slow operations. A deployment error might lead it to inspect a change set. A regional spike might lead it to compare infrastructure metrics. The next useful action is discovered during the task.

That is agent shaped work.

When one LLM call is enough

Use a direct call when the task has one clear transformation and the application already has the required context.

Good examples include:

Turn these notes into a concise update
Map this ticket to one of eight categories
Extract these fields using this schema
Compare two pieces of text
Draft a reply from the facts provided

A direct call is easier to operate because it has a small failure surface. You can validate structured output, set a timeout, retry a safe request, and measure quality against a focused data set.

Do not add an agent just because the prompt contains several instructions. A capable model can perform substantial reasoning inside one call. Internal reasoning is not the same as autonomous execution.

When a workflow is the better design

Use a workflow when the task has several steps but the allowed path is known.

Imagine an internal request system:

Classify the request
Retrieve the relevant policy
Draft a response
Check the response against required rules
Route uncertain cases to a person

The model still provides valuable judgment. Your code keeps control of sequencing, permissions, retries, and approval.

Workflows are especially strong when:

The steps map to a business process
Each stage has a clear input and output
You need predictable latency or cost
Different failures need different recovery logic
Compliance requires an auditable path
A write action must always pass through approval

A workflow can also include model based routing. If the model chooses one of a small set of validated routes, the overall system can remain deterministic enough to test and operate.

When an agent earns its complexity

An agent becomes useful when several of these conditions are true:

The task is open ended and has more than one valid path
The model must choose among tools using intermediate evidence
The number of steps cannot be known in advance
The system must recover from failed actions or incomplete information
Success can be checked with tests, rules, or other clear evidence
The available tools and permissions can be tightly bounded
The value of completing the task exceeds the extra cost and latency

Strong examples include:

Investigating a software failure

The agent can search logs, inspect recent changes, run safe diagnostics, compare symptoms, and revise its theory as evidence arrives.

Updating a codebase toward a clear goal

The agent can inspect the repository, edit files, run tests, study failures, and repeat until the acceptance criteria pass.

Research across changing sources

The agent can search, open relevant sources, follow useful leads, reject weak evidence, and stop when the research question has enough support.

In each case, a fixed sequence would either be too rigid or would require you to encode a large decision tree that the model is better placed to navigate.

The cost of an agent is more than tokens

Every additional turn can add:

Model latency
Tool latency
More tokens and provider cost
Another opportunity to choose the wrong action
More state to store and inspect
A larger security boundary

There is also a simple reliability effect. If a ten step process had a 98 percent chance of success at every step, a simplified independent model would give the whole path only about an 82 percent chance of completing without an error.

Real failures are not independent, so this is not a production formula. It is a reminder that small uncertainties compound across a long run.

For agents, monitoring one final answer is not enough. You need the trace: model decisions, tool calls, results, guardrails, retries, and handoffs. OpenAI's guidance on agent evaluations recommends evaluating the complete workflow so you can see whether the right tools were selected and whether instructions were followed.

Autonomy also changes the security model

A direct call produces content. An agent may read private data or take actions in other systems.

That makes prompt injection, excessive permissions, accidental writes, and data leakage architectural concerns rather than prompt quality concerns.

OpenAI's agent safety guidance recommends structured data between steps, tool approvals, guardrails, and evaluation of traces. These controls matter because untrusted text can influence later tool calls.

A useful permission model is:

Give each tool the narrowest capability it needs
Separate read tools from write tools
Validate every tool argument in application code
Require approval before consequential actions
Set limits on turns, cost, and elapsed time
Define when the agent must stop and ask for help
Keep an audit trail that explains what happened

An agent should never receive broad access merely because it might be convenient.

When an agent is unnecessary

You probably do not need an agent when:

One response can solve the task
The sequence of steps is already known
Tool choice can be expressed with ordinary conditions
The result must arrive within a tight latency budget
Cost must be predictable per request
A wrong action would be difficult to reverse
You cannot define what successful completion looks like
You do not yet have evaluations for the simpler design

A common warning sign is building an agent loop before establishing a reliable direct call. Start with the smallest working baseline. Measure where it fails. Add retrieval, a validation step, or a fixed workflow if that solves the real problem. Introduce autonomy only when the remaining failure is about choosing the path.

A decision checklist

Before building an agent, answer these questions:

What exact outcome must the system produce?
Can one model call produce it from the available context?
If not, are the required steps known in advance?
Does new evidence genuinely need to change the next action?
Which tools are required, and can their permissions be narrowed?
How will the system verify progress and completion?
Which actions require human approval?
What are the limits for turns, time, tokens, and cost?
Can a failed action be retried or reversed safely?
Do you have examples that test both success and failure paths?

The result usually points to one of three choices:

Can one response solve the task?

Yes → Use a direct LLM call

No → Are the steps known in advance?

Yes → Use a deterministic workflow

No → Must intermediate results determine the next tool or action?

Yes → Consider an agent with bounded tools and clear stop rules

No → Simplify the problem before adding autonomy

Final thought

The best AI architecture is not the one with the most autonomy. It is the one that places judgment in the right layer.

Let the model generate, interpret, and reason where those abilities create value. Let normal code own the path when the path is already known. Use an agent when the work truly requires exploration, adaptation, and action across tools.

Start with one call. Add a workflow when the process needs structure. Add an agent when the process itself must be discovered.

Calendar Math vs. Duration Math: Lessons From an Open-Source Bug

Himanshu Sharma — Sun, 11 Oct 2026 07:19:32 +0000

“Repeat every year” sounds like a simple instruction—until the original date is February 29.

While contributing to Super Productivity, I worked on a recurrence bug involving a task that started on a leap day. In a non-leap year, the application could create an extra overdue task before the next occurrence was due.

The bug demonstrated an important distinction:

Duration arithmetic measures elapsed time. Calendar arithmetic models human schedules.

Those concepts overlap on ordinary dates, but they can produce different answers around leap years, month boundaries, daylight-saving transitions, and multi-year recurrence rules.

The bug

The reported task repeated yearly from February 29.

Because February 29 does not exist every year, the application needed a policy for non-leap years. In this case, the intended behavior was:

February 29, 2024
February 28, 2025
February 28, 2026
February 28, 2027
February 29, 2028

In the failing path, setting the day to 29 on a February 2025 date overflowed to March 1. The calculation then stepped back one year while preserving March 1, making March 1, 2024 appear later than the already-created February 29, 2024 occurrence.

That disagreement caused the application to materialize another overdue 2024 task.

The visible problem was a duplicate task, but the underlying problem was more interesting: different parts of the scheduling logic disagreed about which calendar occurrence existed.

JavaScript normalizes out-of-range date components

JavaScript’s Date setters normalize out-of-range components instead of rejecting them.

For example:

const date = new Date(2025, 1, 1); // February 1, 2025

date.setDate(29);

console.log(date);
// March 1, 2025

There is no February 29 in 2025, so JavaScript carries the extra day into March.

This behavior is valid and sometimes useful. But it is not necessarily the recurrence policy your product intends.

For a yearly task, several behaviors could be reasonable:

Move the occurrence to February 28.
Move it to March 1.
Skip non-leap years.
Ask the user to select a policy.

The important part is not choosing one universally correct answer. The important part is making the rule explicit and applying it consistently.

Super Productivity’s expected rule was to clamp the date to the last valid day of the month: February 28.

Calendar math and duration math answer different questions

Consider the phrase “one year later.”

It could mean an elapsed duration:

const ONE_YEAR_IN_MS = 365 * 24 * 60 * 60 * 1000;
const later = new Date(original.getTime() + ONE_YEAR_IN_MS);

But that assumes every year represents the same duration. Leap years immediately break that assumption.

It could instead mean a calendar operation:

Keep the original month and day.
Move to the target calendar year.
If the day does not exist, apply the recurrence policy.

These are different models.

Duration math is appropriate for questions such as:

Has a 30-minute timeout expired?
How long did this request take?
Has a 24-hour retention period passed?

Calendar math is more appropriate for:

Repeat this task every year.
Run payroll on the last day of each month.
Renew this subscription every three months.
Schedule an event every Monday.
Trigger an anniversary reminder.

A recurring task expresses human calendar intent. It is not merely a number of milliseconds after another timestamp.

Separate the recurrence interval from the occurrence date

The fix became clearer when I separated the problem into two questions.

1. Is this an eligible recurrence year?

Suppose the task started in 2024 and repeats every two years.

The eligible years should remain anchored to the original year:

2024, 2026, 2028, 2030...

Conceptually:

const yearsSinceStart = candidateYear - startYear;

const isEligibleYear =
  yearsSinceStart >= 0 &&
  yearsSinceStart % repeatEvery === 0;

This uses calendar-year positions rather than asking whether a fixed duration has elapsed.

2. What is the actual occurrence in that year?

Once the year is eligible, the application needs to construct a valid date.

A simplified model looks like this:

function getOccurrence(startDate, targetYear) {
  const month = startDate.getMonth();
  const requestedDay = startDate.getDate();
  const lastValidDay = getLastDayOfMonth(targetYear, month);

  return new Date(
    targetYear,
    month,
    Math.min(requestedDay, lastValidDay),
  );
}

This is illustrative pseudocode rather than the project’s implementation.

For a February 29 start date:

Target year: 2025
Requested day: 29
Last valid day in February: 28
Actual occurrence: February 28, 2025

The application can then compare normalized calendar dates in the same timezone:

const isDue = occurrenceDay <= todayDay;

That is safer than comparing against an out-of-range component and relying on automatic normalization. In production code, occurrenceDay and todayDay should use the same calendar and timezone policy.

Why multi-year repeats matter

A fix that only handles yearly repetition can still introduce another bug.

Consider a task that starts on February 29, 2024 and repeats every two years.

The expected sequence is:

February 29, 2024
February 28, 2026
February 29, 2028

If the algorithm advances from each normalized result, it may accidentally treat February 28 as the new permanent anchor. It can also drift into the wrong recurrence year if eligibility is based on fully elapsed durations rather than calendar-year intervals.

The original rule therefore needs to remain the source of truth:

Original month: February
Original day: 29
Original year: 2024
Repeat interval: 2 calendar years

Each occurrence should be projected from that rule. It should not be derived solely from the previous adjusted occurrence.

This was one reason the change needed to cover both the forward and backward recurrence calculations. Fixing only the “next occurrence” path would still leave another part of the system interpreting the same rule differently.

Testing the boundaries

Date bugs often hide in the boundaries, so testing only an ordinary date such as June 15 would provide little confidence.

The regression coverage included cases around:

February 28 in a non-leap year
February 29 in a leap year
March 1 after the adjusted occurrence
An occurrence that had already been created
Yearly repetition
Two-year repetition
Forward and backward recurrence calculations
Different timezone configurations

One particularly important scenario was:

Start date: February 29
Non-leap-year occurrence: February 28
Current date: March 1
February 28 occurrence already created
Expected result: no additional occurrence

That test protects the user-visible behavior rather than only checking an internal date transformation.

It asks the question that matters:

Given the task’s history and today’s date, should another task be created?

The answer must be no.

Timezones are a separate source of risk

Leap years were central to this bug, but timezone testing still mattered.

A calendar date is not always equivalent to a fixed period from another calendar date. Around daylight-saving transitions, a local day may contain 23 or 25 hours.

That means code like this can also be dangerous for calendar scheduling:

const nextDay = new Date(
  current.getTime() + 24 * 60 * 60 * 1000,
);

It adds exactly 24 elapsed hours. It does not necessarily mean “the same local time tomorrow.”

Testing the recurrence logic under more than one timezone helped confirm that the result represented the intended calendar date rather than an accidental property of one development environment.

What I learned

1. Recurrence rules are domain logic

“Every year” is not a low-level timestamp operation. It is a product rule that needs defined behavior for invalid dates.

2. Invalid dates should be handled intentionally

JavaScript’s normalization rules are predictable, but predictable does not mean correct for every domain.

3. Preserve the original anchor

Adjusted occurrences should not silently replace the original recurrence rule.

4. Forward and backward calculations must agree

If one function searches for the next occurrence while another searches for the newest possible due occurrence, both must implement the same calendar policy.

5. Test the dates around the boundary

For a February 29 rule, the most valuable tests are usually February 28, February 29, and March 1—not an arbitrary date in the middle of the year.

6. Test outcomes, not only transformations

Checking that February 29 becomes February 28 is useful. Checking that no duplicate task is created after that occurrence already exists is better.

A practical rule

First decide what the value represents. Use duration arithmetic for elapsed time—timeouts, performance measurements, and expiry windows. Use calendar arithmetic for human schedules—billing dates, anniversaries, weekday meetings, and recurring reminders.

Then define what should happen when the requested calendar date does not exist.

Final thoughts

This was a small open-source contribution, but it reinforced a broader engineering lesson: date bugs are often specification bugs hiding inside arithmetic.

The difficult question was not how to manipulate a JavaScript Date. It was deciding what “repeat every year” meant and ensuring every code path implemented that meaning consistently.

The proposed fix is available in Super Productivity PR #10724, created for issue #10679.

At the time of writing, the pull request is still open.

If there is one rule worth remembering, it is this:

Use duration arithmetic for elapsed time. Use calendar arithmetic for human schedules.

One call, from fetch to a validated object — streaming structured output without the plumbing

Himanshu Sharma — Fri, 09 Oct 2026 06:35:27 +0000

You asked the model for a JSON object and set stream: true, because you want the UI (or the next step) to start the moment the first field lands instead of blocking on the last token. Then you're here:

const res = await fetch(url, { method: "POST", body });
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buf = "", json = "";

for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });
  for (const line of buf.split("\n")) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6);
    if (payload === "[DONE]") continue;
    json += JSON.parse(payload).choices[0].delta.content ?? ""; // provider-specific
    try { setState(JSON.parse(json)); } catch {} // 💥 throws on every partial
  }
  buf = buf.slice(buf.lastIndexOf("\n") + 1);
}

const final = Schema.parse(JSON.parse(json)); // 💥 and throws here if the model added a trailing comma

Three separate problems are tangled together in that loop:

The transport. SSE framing by hand — splitting on newlines, handling a frame split across two network chunks, skipping [DONE].
The partial parse. JSON.parse(json) throws on every chunk until the object is complete, so you get nothing to render in the meantime.
The validation. At the end you still have to validate — and models emit trailing commas, wrap JSON in Markdown code fences, and send "true" where you wanted true, so a strict .parse() rejects output that's almost right.

Each of those is a small, well-defined job. I built one tiny zero-dependency package for each — and then one more that wires them together.

One call

import { structured } from "trickle-structured";
import { z } from "zod";

const User = z.object({ id: z.number(), name: z.string(), active: z.boolean() });

const stream = structured(
  {
    url: "https://api.openai.com/v1/chat/completions",
    method: "POST",
    headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
    body: JSON.stringify({
      model: "gpt-4o",
      stream: true,
      response_format: { type: "json_object" },
      messages,
    }),
  },
  User,
);

for await (const partial of stream) {
  render(partial);              // {}, { id: 42 }, { id: 42, name: "Ad" }, … typed as DeepPartial<User>
}

const user = await stream.value; // User — validated, and repaired if the model slipped

That's the whole thing. No reader loop, no try/catch around JSON.parse, no provider-specific envelope code, no separate validation step.

One stream, two surfaces

The return value is both an async iterable of partial snapshots and a holder for the final promise:

const stream = structured(request, User);

// iterate for live partials…
for await (const partial of stream) updateUI(partial);

// …and/or await the validated result
const user = await stream.value;        // User
const repairs = await stream.changes;   // what coerce-json had to fix

Each partial is a DeepPartial<User> — every field optional at every depth, because the object is still being built. Perfect for rendering skeletons that fill in.
.value resolves once the stream ends, with the validated and coerced result. If the output can't satisfy the schema even after repair, it rejects with a typed CoercionError carrying everything that was attempted.
The underlying fetch runs exactly once, no matter which surface you touch. Awaiting .value without iterating still drains the stream; iterating without awaiting .value never leaves a dangling rejection.

It composes three tiny packages

trickle-structured is the capstone of a small stack. The pipeline is literally the dependency list:

fetch → SSE (sse-wire) → parse partial JSON (trickle-json) → validate + repair (coerce-json)

sse-wire — fetch-based SSE: the POST, custom headers, and AbortSignal that the browser's EventSource can't do.
trickle-json — an incremental partial-JSON parser that returns the best valid value on every chunk, in O(n) total regardless of how the bytes are split.
coerce-json — repairs and coerces almost-valid model JSON to fit your schema (strips fences, fixes "7"→7), and logs every change.

Each is useful on its own; trickle-structured is the one import when you want all three wired correctly.

Providers are handled

OpenAI and Anthropic wrap model output in different SSE envelopes. trickle-structured decodes both, and auto-detects from the first event when you don't say which:

structured(request, Schema);                          // sniffs the first event
structured(request, Schema, { provider: "anthropic" }); // or be explicit
structured(request, Schema, { extract: (e) => JSON.parse(e.data).output?.text ?? null }); // or bring your own

Tool calls too

More and more, models return structured data as a tool/function call, where the object lands in the call's arguments rather than in message content. Same pipeline, one flag:

const WeatherArgs = z.object({ city: z.string(), units: z.enum(["c", "f"]) });

const stream = structured(request, WeatherArgs, {
  provider: "openai",
  mode: "tool",
  tool: "get_weather",
});

const args = await stream.value; // { city: "Paris", units: "c" }

Using React?

trickle-structured is deliberately framework-agnostic — a plain async iterable plus a promise. For React, trickle-react wraps the same pipeline in a useStreamingObject hook with per-field status and lifecycle-aware abort. Same stack, different top layer.

Try it

npm install trickle-structured
npm install zod   # optional peer, only if you pass a Zod schema

npm: https://www.npmjs.com/package/trickle-structured
GitHub: https://github.com/H1manshu01/trickle-structured

The orchestration layer is ~1.2 kB min+brotli (its three dependencies are separate, deduped packages). ESM + CJS + .d.ts, Node ≥ 18 and modern browsers, published with provenance. Tested with a 200-run byte-split fuzz that proves the final value is identical no matter how the stream is chunked.

If it mis-parses something, open an issue with the stream and the schema — a ⭐ is appreciated if it saves you a reader loop and a pile of try/catch.

Fit any chat history into the context window, bring your own tokenizer

Himanshu Sharma — Fri, 09 Oct 2026 05:58:21 +0000

You ship a chat feature. It works. Then a conversation gets long enough, a couple
of chunky tool results land in the middle, and the next request crosses the
model's context window. You get one of these:

400 This model's maximum context length is 128000 tokens, however you requested 131402...

So you write a trimmer. Keep the system prompt, keep the last few turns, drop the
oldest stuff until it fits. Easy — except the first version drops the system
prompt by accident, the second one orphans a tool result from the assistant
call that produced it (which the provider rejects), and the third one is off by
enough tokens that you still get the 400 sometimes. Everyone writes this. Almost
everyone writes it slightly wrong.

The trimming shape is always the same

Whatever the app, fitting a history into the window is the same move:

Keep the system prompt (it defines the behavior).
Keep the most recent turns (they are the live conversation).
Keep anything you pinned (a few-shot example, the current user turn).
Drop or fold the rest until the total is under the budget.
Never separate a tool result from the tool call it answers.

Get it wrong and you ship one of three failures: a hard 400 from the
provider, a conversation that is silently truncated at the wrong place, or a
request that quietly drops the system prompt and changes how the model
behaves. The shape is simple; the edges are where it breaks.

context-budgeter

context-budgeter is that shape
done once and correctly. You bring a token counter; it brings the strategies and
a hard guarantee the result is within budget with your pinned messages intact — or
it throws, rather than handing you something over budget.

import { fit } from "context-budgeter";
import { estimate, limitFor } from "context-budgeter/models";

const { messages, dropped, tokens, fits } = fit(history, {
  count: estimate,                          // bring your own; a real tokenizer is more accurate
  budget: limitFor("gpt-4o") ?? 128_000,
  reserve: 1_000,                           // hold back for the reply
  strategy: ["keep-system", "middle-out"],  // try in order until it fits
  keep: { last: 4 },                        // always keep the 4 most recent
});

fit is synchronous and deterministic. messages comes back in original order
with the system prompt still first and the last four turns kept; dropped is what
was removed; fits tells you whether it all fit. The input is never mutated, and
every passthrough field on your messages (name, tool_calls, tool_call_id,
anything) is carried through untouched.

Bring your own tokenizer — on purpose

There is no universal tokenizer. OpenAI, Anthropic, and Llama all count
differently, real BPE tokenizers are large, and sometimes a provider's
token-count API is the only exact source. So context-budgeter bundles no
tokenizer and takes a count function instead:

import { encode } from "gpt-tokenizer";
import { fit } from "context-budgeter";

const count = (text: string) => encode(text).length; // exact, for OpenAI-family models
fit(history, { count, budget: 128_000, reserve: 1_000 });

gpt-tokenizer and js-tiktoken are counters you pair with this library, not
competitors. If you just want to prototype, the optional context-budgeter/models
module ships a rough estimate (~4 chars/token) and a small limitFor(model)
table — both approximate, both deliberately kept out of the core so it stays
tiny (the core is ~971 B min+brotli).

Strategies, as a list tried in order

A strategy decides the order droppable messages are removed in. Pass one, or
an array that is applied in sequence until the history fits:

"drop-oldest" (default) — classic sliding window, drops from the front.
"middle-out" — drops from the center outward, so you keep both the oldest framing and the newest turns and sacrifice the middle.
"keep-system" — force-pins every system message for the run.
"summarize" — folds the dropped span into a summary (async; see below).

So ["keep-system", "middle-out", "summarize"] reads exactly as it should: always
keep system, prefer to drop the middle, and if it still doesn't fit, summarize
what was dropped.

Pinning, and the tool-call edge nobody remembers

Pins combine — a message survives if any keep rule matches it:

fit(history, {
  count,
  budget,
  keep: {
    system: true,                          // keep every system message (default)
    first: 2,                              // keep a leading few-shot pair
    last: 4,                               // keep the 4 most recent
    pin: (m) => m.name === "important",    // keep anything your predicate matches
  },
});

And the edge that trips hand-rolled trimmers: a tool result is grouped with the
assistant message that called it and they drop together. A tool result with no
preceding call is a request the provider rejects — context-budgeter never
produces one.

Summarize instead of discard (async)

When you would rather fold the dropped span into a summary than lose it, set the
"summarize" strategy and a hook, and call fitAsync:

import { fitAsync } from "context-budgeter";

const r = await fitAsync(history, {
  count,
  budget: 4_000,
  strategy: ["drop-oldest", "summarize"], // drop first; summarize what dropped
  summarize: async (dropped) => ({
    role: "system",
    content: await myLLM.summarize(dropped),
  }),
});

The summary is inserted where the dropped span used to be, so order reads
naturally. (fit throws if a summarize hook would run — a nudge to reach for the
async entry point deliberately.)

The guarantees are the point

Covered by 15 tests, including a fast-check property test over hundreds of
random inputs:

The result never exceeds the budget.
Pinned messages are never dropped.
If the pinned set alone is over budget, fit throws a RangeError — it never silently returns an over-budget result.
Order and passthrough fields are preserved; the input is never mutated.
Tool-call and tool-result messages drop together.
Deterministic and synchronous — the only async is your own summarize hook.

Part of a small line of LLM dev tools

context-budgeter is one of a set of zero-dependency tools, each useful on its
own:

context-budgeter — fit a chat history into the context window. (this one)
sse-wire — fetch-based SSE client: POST, headers, abort, opt-in reconnection.
retry-wire — provider-aware retry + throttle for the request (429 / 529 / Retry-After).
trickle-json — assemble streamed token deltas into the best valid partial value on every chunk.
coerce-json — repair and coerce that value to your Zod / JSON Schema, logging every fix.
trickle-react — React bindings for the streaming parse.
expect-llm — assertions for LLM output in tests.

Try it

npm install context-budgeter

npm: https://www.npmjs.com/package/context-budgeter
GitHub: https://github.com/H1manshu01/context-budgeter
Zero runtime dependencies, ESM + CJS, full types, core ~971 B min+brotli, published with provenance. Runs on Node 18+ and in the browser.

If it trims something it shouldn't — or keeps something it shouldn't, or your
budget math disagrees with your tokenizer — open an issue with the messages and
the count you used. The guarantees are the whole point, so I want to know. A
star helps if it saves you a hand-rolled trimmer.

Make your LLM calls survive 429s and overloads, with any client

Himanshu Sharma — Fri, 09 Oct 2026 05:24:59 +0000

You ship an LLM feature. It works. Then, in production, you start seeing this:

429 Too Many Requests
529 overloaded_error
503 Service Unavailable

So you reach for a retry library — p-retry, async-retry, a hand-rolled
backoff loop — and wrap the call. And it mostly works, until you notice it's
retrying a 400 Bad Request (pointless), ignoring the Retry-After: 2 the
server politely sent (so you wait the wrong amount), and hammering a 429 that
you could have paced under in the first place.

The problem is that an LLM endpoint doesn't fail like a generic HTTP service, and
generic retry libraries don't know the difference.

What's actually different about LLM failures

A 429 might be a quota limit or a momentary burst. Either way the server usually tells you how long to wait, in a Retry-After (seconds or an HTTP date) or a retry-after-ms header. A blind exponential backoff ignores it and guesses.
Anthropic adds a 529 overloaded_error — "the service is saturated, try again." That's retryable, and it's a different thing from a quota 429. Most retry predicates don't even have 529 in their list.
Providers report your remaining budget in x-ratelimit-remaining-requests / -tokens (OpenAI) and anthropic-ratelimit-* headers. You can pace against that instead of waiting to be told "no."

The official SDKs know all of this. The openai and @anthropic-ai/sdk clients
retry 429/5xx and honor retry-after out of the box. The catch: that logic
lives inside the client. The moment you call through a gateway, a proxy, your
own fetch, a second provider, or a streaming client, you leave it behind and
you're back to a generic loop.

retry-wire

retry-wire is that provider-aware
retry logic, unbundled — a zero-dependency policy you wrap around any fetch
or any async call. Two functions, same knowledge inside both.

A drop-in fetch wrapper:

import { retryFetch } from "retry-wire";

const rfetch = retryFetch(fetch, { provider: "openai", maxRetries: 5 });

const res = await rfetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  body,
  signal,
});

rfetch has the exact fetch signature, so it drops in anywhere a fetch is
accepted. It retries 429 / 529 / 5xx / 408 / 409 and network errors, honors
Retry-After, drains the response body before retrying so nothing leaks, and
returns non-retryable responses (200, 400, …) untouched — just like fetch.

Or wrap an SDK method — anything that returns a promise:

import { withResilience } from "retry-wire";

const out = await withResilience(
  (signal) => client.messages.create(params, { signal }),
  {
    provider: "anthropic", // treats 529 overloaded as retryable, distinct from 429
    maxRetries: 5,
    backoff: { initial: 500, max: 60_000, jitter: "full" },
    onRetry: ({ attempt, delayMs, reason }) =>
      console.warn(`retry ${attempt} in ${delayMs}ms: ${reason}`),
  },
);

It reads the thrown error's status and headers (matching how the SDK error
objects are shaped), forwards an AbortSignal to your call, and never retries an
abort or a non-retryable error.

Rule packs: the provider knowledge, as data

What makes it provider-aware is a rule pack — plain data, not hardcoded
branching:

interface RulePack {
  name?: string;
  isRetryable(ctx): boolean;                 // is this outcome worth retrying?
  retryAfterMs(headers): number | undefined; // the server's advised wait
  readLimits?(headers): RateLimitInfo;        // remaining request/token budget
}

genericPack, openaiPack, and anthropicPack ship built in. anthropicPack
is the one that knows 529; the OpenAI and Anthropic packs also parse the
provider's rate-limit headers. For a gateway, Bedrock, Vertex, or anything with
its own rules, defineRulePack gives you a typed custom pack:

import { defineRulePack, parseRetryAfter } from "retry-wire";

const bedrockPack = defineRulePack({
  name: "bedrock",
  isRetryable: (ctx) => ctx.status === 429 || ctx.status === 503,
  retryAfterMs: (headers) => parseRetryAfter(headers),
});

Don't just react to 429 — avoid it

Retrying reacts to a rate limit. Throttling avoids it: pace your own calls
under the tier before the provider has to push back. It's opt-in, built on token
buckets over requests- and tokens-per-minute:

const rfetch = retryFetch(fetch, {
  provider: "openai",
  throttle: {
    rpm: 50,
    tpm: 40_000,
    estimateTokens: (input, init) => countTokens(init?.body),
  },
});

A call that would outrun the budget waits on an abortable timer until a bucket
token is available — so you glide under the limit instead of bouncing off it.

It composes with sse-wire

If you stream completions, you've probably met
sse-wire — a fetch-based SSE client
that reconnects a stream only when the connection actually drops. The two solve
different halves of the same problem: sse-wire reconnects the stream;
retry-wire resilience-wraps the request that opens the stream, so a 429 or
529 on the way in is retried before a byte streams.

import { retryFetch } from "retry-wire";
import { sse } from "sse-wire";

const rfetch = retryFetch(fetch, { provider: "openai", maxRetries: 5 });

for await (const event of sse("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
  body: JSON.stringify({ model, stream: true, messages }),
  fetch: rfetch, // open the stream through the resilient fetch
})) {
  if (event.data === "[DONE]") break;
  render(JSON.parse(event.data));
}

retry-wire is part of a small line of zero-dependency LLM dev tools, each
useful on its own:

retry-wire — provider-aware retry + throttle for the request. (this one)
sse-wire — fetch-based SSE client: POST, headers, abort, opt-in reconnection.
trickle-json — assemble streamed token deltas into the best valid partial value on every chunk.
coerce-json — repair and coerce that value to your Zod / JSON Schema, logging every fix.
trickle-react — React bindings for the streaming parse.
expect-llm — assertions for LLM output in tests.

Why not just use the SDK's retries?

Because they only cover calls made through that SDK. One app that talks to
OpenAI and Anthropic has two retry implementations with different knobs; add a
gateway or a raw fetch and you have a third, generic one. retry-wire is one
policy — openai / anthropic / generic packs plus custom — across every
transport: SDKs, gateways, raw fetch, and streaming.

Try it

npm install retry-wire

npm: https://www.npmjs.com/package/retry-wire
GitHub: https://github.com/H1manshu01/retry-wire
Zero runtime dependencies, ESM + CJS, full types, ~1.68 kB min+brotli, published with provenance. Runs on Node ≥ 18, the browser, and Bun.

If it retries something it shouldn't — or doesn't retry something it should, or
misreads a Retry-After — open an issue with the status/headers you saw. The
provider-aware part is the whole point, so I want to know. A star helps if it
saves you a hand-rolled backoff loop.

Assert LLM output in the test runner you already use

Himanshu Sharma — Fri, 09 Oct 2026 05:02:42 +0000

You have an LLM call in your product, and a test that exercises it. Now you have
to assert on the output. If it is like most LLM output, it comes back looking
something like this:

{"total": 42, "currency": "USD"}

JSON, probably valid, maybe wrapped in a markdown fence, with fields you need and
phrases you would rather not see. So you write the assertion by hand:

let parsed: unknown;
try {
  parsed = JSON.parse(raw.replace(/```
{% endraw %}
json\n?|\n?
{% raw %}
```/g, ""));
} catch {
  throw new Error("not valid JSON"); // the real error is now gone
}
const result = Invoice.safeParse(parsed);
expect(result.success).toBe(true);   // and when it fails? "expected false to be true"
expect(raw.includes("total")).toBe(true);
expect(raw.includes("as an AI")).toBe(false);

Every line of that is a small papercut. The try/catch swallows the parser's
message. expect(result.success).toBe(true) throws away every per-field reason
Zod just computed. The includes checks can't tell you which string was
missing. And you will write it again, slightly differently, in the next test.

The usual escape hatch is to adopt an eval framework — promptfoo, DeepEval,
Braintrust, Evalite. Those are great tools for a real eval suite. But for
"assert this one response in my unit test," they are a CLI, a config file, maybe
an account, and a second mental model that lives outside the vitest/jest run
you already have.

There is a middle option.

expect-llm

expect-llm is a set of
expect(...) matchers for the things LLM output actually gets wrong. Register
them once, then assert inline, in the runner you already run:

import { expect } from "vitest";
import { z } from "zod";
import { llmMatchers } from "expect-llm";

expect.extend(llmMatchers); // once, in a setup file

const Invoice = z.object({ total: z.number(), currency: z.string() });

expect(out).toBeValidJSON({ allowFences: true });
expect(out).toMatchSchema(Invoice);             // out may be an object OR a JSON string
expect(out).toContainAll(["total", "currency"]);
expect(out).toContainNone(["TODO", "as an AI"]);
expect(decision).toBeOneOf(["refund", "escalate", "deny"]);

The whole papercut block from the top collapses into that. And when something
fails, the message tells you what — the missing items, the mismatched field, the
parser's actual error — instead of expected false to be true.

It is zero runtime dependencies, ships ESM + CJS + types, runs on Node >= 18,
and is roughly 1.41 kB min+brotli. .not works on every matcher.

The matchers

Eight matchers. Seven are deterministic; one is the opt-in judge.

toBeValidJSON(options?) — the value is a parseable JSON string. Pass { allowFences: true } to strip a surrounding markdown code fence first, which is exactly what models wrap JSON in.
toContainAll / toContainNone / toContainAny — substring checks over the (stringified) value. toContainNone(["as an AI", "TODO"]) is the banned- phrase guard you keep writing by hand; the failure names the phrase that leaked.
toBeOneOf(allowed) — the value deep-equals one of a closed set. For a routing or classification decision: toBeOneOf(["refund", "escalate", "deny"]).
toMatchStructure(reference) — same keys and value types as a reference, recursively, ignoring the values and array length. A shape snapshot that survives a prompt refactor changing the numbers.
toMatchSchema(schema) — validate against a Zod schema or any Standard Schema. A string value is JSON-parsed first, fences tolerated — so you assert on the raw output.
toSatisfy(rubric, judge) — the opt-in judge, below.

Assert on raw output, fences and all

Because toBeValidJSON and toMatchSchema tolerate a code fence and parse a
string for you, you assert on exactly what the model returned — no pre-cleaning:

const out = "```json\n{\"total\":42,\"currency\":\"USD\"}\n```";

expect(out).toBeValidJSON({ allowFences: true }); // passes
expect(out).toMatchSchema(Invoice);               // parses, unfences, then validates

Shape, not values

toMatchStructure is the one I reach for most. It checks the shape of a
response — keys and types — so a test does not break every time the model picks a
different number:

const reference = { id: 1, name: "x", tags: ["a"], meta: { active: true } };

expect({ id: 9, name: "y", tags: ["p", "q"], meta: { active: false } })
  .toMatchStructure(reference);     // passes: same keys, same types

expect({ id: "9", name: "y", tags: ["p"], meta: { active: true } })
  .not.toMatchStructure(reference); // fails: id is a string, not a number

Deterministic by default; the judge is opt-in

Those seven matchers are pure, synchronous, and make no network call — a
guard test fails the build if any of them touches fetch. Use them for anything
objectively checkable, which is most of what breaks: valid JSON, schema shape,
required and forbidden content, a closed decision set, a stable structure.

For the subjective checks — "is this a polite refusal?" — there is toSatisfy,
and it is deliberately bring your own model. expect-llm ships no SDK, no
key handling, and no default provider. You pass the model call in:

import type { Judge } from "expect-llm";

const judge: Judge = async (output, rubric) => {
  const verdict = await myModel(`Does this satisfy "${rubric}"?\n\n${output}\n\nAnswer yes or no.`);
  return { pass: /yes/i.test(verdict), reason: verdict };
};

await expect(answer).toSatisfy("is a concise, polite refusal", judge);

Two things fall out of that design. One: the deterministic matchers never cost a
token, so your CI stays fast and reproducible and the model call happens only
where you opt in. Two: the judge is a plain function, so a stub judge lets you
test the wiring in CI without a network call, and the judge's reason shows up
in the failure message when the assertion fails.

Vitest or Jest — same matchers

// Vitest — vitest.setup.ts
import { expect } from "vitest";
import { llmMatchers } from "expect-llm";
expect.extend(llmMatchers);

// Jest — jest.setup.ts
import { llmMatchers } from "expect-llm/jest";
expect.extend(llmMatchers);

The Vitest entry augments Vitest's expect types; the Jest entry augments
jest.Matchers. Same matcher set, typed on both.

It pairs with coerce-json and zod

expect-llm is the assertion end of a small line of zero-dependency LLM
dev-tools I maintain. In a structured-output flow it sits right after
coerce-json, which repairs and
coerces almost-valid model output to fit a schema. Coerce first, then assert the
result — toMatchSchema takes the same Zod schema:

import { coerce } from "coerce-json/zod";

const { value, ok } = coerce(rawModelOutput, Invoice);
expect(ok).toBe(true);
expect(value).toMatchSchema(Invoice);
expect(value).toContainAll(["total", "currency"]);

The full suite, each piece zero-dependency and useful on its own:

fetch -> SSE (sse-wire) -> parse partial JSON (trickle-json) -> coerce to schema (coerce-json) -> assert (expect-llm)

sse-wire — fetch-based SSE client for LLM streams.
trickle-json — incremental partial-JSON parser for those streams.
coerce-json — repair and coerce to your schema, logging every fix.
expect-llm — assert the result in Vitest or Jest. (this one)

Try it

npm install -D expect-llm

npm: https://www.npmjs.com/package/expect-llm
GitHub: https://github.com/H1manshu01/expect-llm
Zero runtime dependencies, ESM + CJS, full types, published with provenance. vitest, jest, and zod are optional peers — bring only the ones you use.

If a matcher's failure message is ever less helpful than the hand-rolled
assertion it replaced, open an issue — readable failures are the whole point. A
star is appreciated if it saves you a pile of try/catch boilerplate.

Stream an LLM object into React, field by field — in one hook

Himanshu Sharma — Thu, 08 Oct 2026 16:05:53 +0000

You asked the model for a JSON object and set stream: true so the UI can fill
in as it arrives. Now you're in the usual mess:

useEffect(() => {
  const ctrl = new AbortController();
  (async () => {
    let buf = "";
    for await (const chunk of stream(ctrl.signal)) {
      buf += chunk;
      try {
        setData(JSON.parse(buf)); // 💥 throws on every partial chunk
      } catch {}
    }
  })();
  return () => ctrl.abort();
}, [prompt]);

JSON.parse throws until the very last token, you can't tell which fields are
done, and if you're not careful you'll tear or leak the stream. Let's fix it.

One hook

import { z } from "zod";
import { sse } from "sse-wire";
import { useStreamingObject, openAIContent } from "trickle-react";

const Profile = z.object({ name: z.string(), city: z.string(), bio: z.string() });

function ProfileCard({ prompt }: { prompt: string }) {
  const { data, status, done } = useStreamingObject(
    Profile,
    (signal) => openAIContent(sse("/api/profile", { method: "POST", body: prompt, signal })),
    { deps: [prompt] },
  );

  return (
    <article>
      <h2>{data.name ?? <Skeleton />}</h2>
      <p data-loading={!status.city}>{data.city}</p>
      <p data-loading={!status.bio}>{data.bio}</p>
      {done && <Badge>complete</Badge>}
    </article>
  );
}

data is a DeepPartial<Profile> that updates on every chunk — never throws.
status.city turns true once city has fully arrived (not just appeared).
done flips at the end; the stream aborts automatically on unmount or when deps change.

Under the hood it uses useSyncExternalStore, so renders are tearing-free and
React-18 concurrent-safe, and the partial JSON parsing is handled by
trickle-json.

It's the top of a small stack

trickle-react is the adoption layer over three other tiny, zero-dependency
packages I built — each solves one step of the streaming structured-output
pipeline, and each is useful on its own:

fetch → SSE (sse-wire) → parse partial JSON (trickle-json) → repair/coerce (coerce-json) → render (trickle-react)

sse-wire — fetch-based SSE: the POST + headers EventSource can't do.
trickle-json — incremental partial-JSON parser; best valid value on every chunk.
coerce-json — repairs & coerces the model's almost-valid JSON to fit your schema, logging every fix.

You don't have to adopt the whole stack — trickle-react only needs
trickle-json. Bring sse-wire for the stream and coerce-json/zod for
validation when you want them.

There's more than one object

// rows as they stream
const { items } = useStreamingList<Suggestion>(startStream, { path: "suggestions" });

// function-call arguments as they fill in
const { toolCalls } = useToolCalls((signal) => sse("/api/agent", { method: "POST", body, signal }));

Try it

npm install trickle-react

npm: https://www.npmjs.com/package/trickle-react
GitHub: https://github.com/H1manshu01/trickle-react

react is a peer; zod and coerce-json are optional. ~1.5 kB, ESM + CJS,
fully typed, published with provenance. If it mis-renders something, open an
issue with the stream and the schema — ⭐ appreciated if it saves you a pile of
useEffect plumbing.

EventSource can't POST. Your LLM stream needs it to.

Himanshu Sharma — Thu, 08 Oct 2026 15:30:44 +0000

You want to stream a chat completion. The response is a text/event-stream, so
the browser's built-in EventSource seems like the obvious tool:

const es = new EventSource("https://api.openai.com/v1/chat/completions");

Except that line can't work, and not for a small reason. EventSource can only
issue a GET, and it can't set a single header. The chat-completions
endpoint needs the exact opposite: a POST, an Authorization: Bearer …
header, and a JSON body with your messages. The one tool the platform gives you
for SSE is structurally incapable of making an LLM request.

So you drop to fetch — and now you own the parser:

const res = await fetch(url, { method: "POST", headers, body });
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buf = "";
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });
  // now split on \n\n... or was it \r\n\r\n? what about a \r\n split
  // across two chunks? and a multibyte character split mid-sequence?
  // and when I break early, did I cancel the reader, or leak the connection?
}

Every one of those comments is a real bug people ship. This is the part of every
streaming integration nobody writes a blog post about. Let's fix it properly.

The usual options (and where they hurt)

1. Native EventSource. GET only, no headers — a non-starter for LLM APIs.
And when it does work, it reconnects on any stream end, including a clean
one. For an LLM response that's wrong: a clean end means the model finished.

2. @microsoft/fetch-event-source. The well-known "fetch SSE that can POST."
Genuinely good — but it's a callback API (onmessage, onclose), and you drive
the reconnection policy and the Last-Event-ID bookkeeping yourself.

3. Polyfills — eventsource, launchdarkly-eventsource. These bring the
EventSource interface to Node. That's the catch: you inherit the callback
model, .close() cancellation, and always-on reconnection — the semantics built
for a persistent stream that should reconnect, not a one-shot completion that
ends when the answer is done.

The common thread: either you can't POST, or you get a callback API with
reconnection tuned for the wrong shape of stream.

sse-wire

sse-wire is a zero-dependency,
fetch-based SSE client you consume with for await:

import { sse } from "sse-wire";

const controller = new AbortController();

for await (const event of sse("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
  body: JSON.stringify({ model, stream: true, messages }),
  signal: controller.signal,
})) {
  if (event.data === "[DONE]") break;      // provider sentinel — not JSON
  const delta = JSON.parse(event.data);     // { event?, data, id? }
  render(delta);
}

That's the whole surface for the common case. Three things make it pull its
weight:

It POSTs with headers. It's fetch underneath, so method, body, and headers are first-class. It auto-sets accept: text/event-stream if you didn't, and passes your signal straight through — so the AbortController you already have cancels the request and the stream.
It's an async generator. for await (const event of sse(...)). break when you hit [DONE], wrap it in try/finally, compose it like any other iterable. No listener wiring, no .close().
A clean end means done. When the stream closes normally, the loop ends — because the model finished. It does not silently reconnect and re-request a completed generation.

Cancellation that doesn't leak

Abort, and sse-wire throws the standard AbortError, cancels the in-flight
fetch, and cancels the stream reader. Same if you just break out of the loop
early — the reader is cancelled in a finally, so the connection is never left
dangling. It's covered by tests that assert the reader's cancel() actually ran
on abort and on early break.

const controller = new AbortController();
setTimeout(() => controller.abort(), 5_000); // or AbortSignal.timeout(5_000)

Reconnection, only when it's real

Reconnection is opt-in and fires on transport errors only — a dropped
connection mid-answer, never a clean end:

for await (const event of sse(url, { method: "POST", headers, body, reconnect: true })) {
  handle(event);
}

When the connection actually drops, it waits an equal-jitter exponential backoff
(a server retry: directive becomes the base delay), re-issues the request with
Last-Event-ID set to the last event it saw, and keeps yielding into the same
loop. Your consumer never notices the seam.

The parser is the hard-won part

The wire format is fiddly, and sse-wire handles it so you don't: CR / LF /
CRLF terminators (including a CRLF split across two chunks), multi-line
data: joined with \n, comments, a leading BOM, a NUL-in-id, and a final
line with no newline. It buffers across chunk and UTF-8 boundaries, so a
multibyte character split mid-sequence decodes correctly. There's a property
test that splits the same payload at every single byte offset and asserts the
events come out identical to a one-shot parse.

You can use that parser directly on any byte/text stream, not just a fetch body:

import { parseSSE } from "sse-wire";

for await (const event of parseSSE(someByteStream)) {
  // { event?, data, id? }
}

It's the front of a pipeline

sse-wire is the transport step of a streaming structured-output flow — and
it sits first. Its two siblings pick up from the events it yields:

fetch → SSE (sse-wire) → parse partial JSON (trickle-json) → repair/coerce to schema (coerce-json) → validate

sse-wire — POST the request, stream the events. (this one)
trickle-json — assemble the streamed token deltas into the best valid partial value available on every chunk, without throwing.
coerce-json — repair and coerce that value to your Zod / JSON Schema, logging every fix.

Here's all three together — stream a completion, assemble the JSON as it arrives,
and coerce the result to a schema:

import { sse } from "sse-wire";
import { StreamingJsonParser } from "trickle-json";
import { coerce } from "coerce-json/zod";
import { z } from "zod";

const Answer = z.object({
  sentiment: z.enum(["positive", "neutral", "negative"]),
  score: z.number(),
  tags: z.array(z.string()).default([]),
});

const parser = new StreamingJsonParser();
parser.on("snapshot", renderPreview); // progressive UI, every chunk

for await (const event of sse(endpoint, { method: "POST", headers, body })) {
  if (event.data === "[DONE]") break;
  const text = JSON.parse(event.data).choices?.[0]?.delta?.content;
  if (text) parser.write(text);
}

const { value, ok, changes } = coerce(parser.end(), Answer);
if (ok) save(value);
else console.warn("could not fully repair:", changes);

sse-wire owns the transport; trickle-json gives you the best value on every
chunk without throwing; coerce-json makes it fit your schema and hands you the
receipts. Each is zero-dependency and works on its own — adopt only the piece you
need.

"But doesn't provider structured output / the official SDK already do this?" The
SDKs help, but they couple the stream to their request lifecycle and their
provider's shapes. sse-wire is a pure, framework-agnostic fetch SSE client:
drop it into any pipeline, point it at any text/event-stream (LLM or not), mock
it in a test, and hand clean events to whatever's next.

Try it

npm install sse-wire

npm: https://www.npmjs.com/package/sse-wire
GitHub: https://github.com/H1manshu01/sse-wire
Zero runtime dependencies, ESM + CJS, full types, ~1.4 kB min+brotli, published with provenance. Runs on Node ≥ 18, the browser, and Bun.

If it mishandles some stream it shouldn't — a terminator edge case, a chunk
boundary, an abort that leaks — open an issue with a repro. The parser-parity and
no-leaked-reader guarantees are the whole point, so I want to know. ⭐ appreciated
if it saves you a hand-rolled fetch loop.

Your LLM returned almost-valid JSON. Stop hand-patching it.

Himanshu Sharma — Thu, 08 Oct 2026 15:09:22 +0000

You asked the model for { id: number, active: boolean }. Here's what came
back:

{"id": "42", "active": "true", "role": "admin"}

Every field is almost right. id is a string. active is a string. There's
a role you never asked for. And the whole thing is wrapped in a markdown code
fence. So this:

const user = User.parse(JSON.parse(raw)); // 💥

throws twice — once on the fence, and again on the types. So you start writing
the patches by hand:

const unfenced = raw.replace(/```
{% endraw %}
json\n?|\n?
{% raw %}
```/g, "");
const obj = JSON.parse(unfenced);
obj.id = Number(obj.id);
obj.active = obj.active === "true"; // and "yes"? and "on"? and "1"?
// ...and now do it again for the next schema

This is the part of every LLM integration nobody writes a blog post about. Let's
fix it properly.

The usual fixes (and where they hurt)

1. z.coerce. Zod can coerce primitives — z.coerce.number(),
z.coerce.boolean(). But you opt in field by field, it can't touch the fence or
the prose, and z.coerce.boolean() just calls Boolean(x) — so "false"
becomes true. Worse: when it does coerce, it does it silently. You can't
log what changed.

2. jsonrepair / json-repair. Great at fixing broken syntax — unclosed
braces, missing quotes, trailing commas. But they're schema-blind. Feed one
{"id":"42"} and it returns {"id":"42"}, because that's already valid JSON.
The string-vs-number mismatch is exactly the problem, and it can't see it.

3. instructor-js and re-prompting. Validate, and on failure send the error
back to the model to try again. It works — but you've turned a "42" → 42
cast, which is a few microseconds of local work, into another network round
trip, more tokens, and a non-deterministic retry.

The common thread: each tool owns one slice. Cast primitives, or fix syntax,
or re-prompt. None of them read the schema you already have, coerce the value
to fit it, and tell you what they changed.

coerce-json

coerce-json is a
zero-dependency library that takes almost-valid model output and makes it fit
your schema — reporting every fix:

import { coerce } from "coerce-json";
import { z } from "zod";

const User = z.object({ id: z.number(), active: z.boolean(), role: z.string().default("user") });

const { value, ok, changes } = coerce('```

json\n{"id":"42","active":"true"}\n

```', User);
// value   → { id: 42, active: true, role: "user" }
// ok      → true
// changes → strip-fence, string->number @ id, string->boolean @ active, fill-default @ role

Three things make that line pull its weight:

It's schema-aware. It reads your Zod schema (or a JSON Schema, or its own core spec) and coerces toward it — the fence comes off, "42" becomes a number because id wants a number, the missing role gets its documented default.
It reports every fix. changes is an ordered, auditable log. A coercion is never a silent black box — you can log it, warn on it, or gate on it.
It never fabricates. The only values it adds are documented schema defaults, and each one is logged. A missing optional field is left absent, not guessed.

How much does that actually buy you? I built a corpus of 37 representative LLM
mistakes — numbers as strings, "yes" booleans, fenced and prose-wrapped
objects, enum casing, missing defaults. Under plain Zod, 13.5% of them
validate. Run them through coerce with defaults, and 86.5% validate. Flip
on fuzzy: true and it's 100%. (It's a hand-built, illustrative corpus, not
a scientific sample — the harness and the full breakdown are in
BENCHMARKS.md,
so re-run it before quoting the numbers.)

It handles what models actually emit

// prose-wrapped output
coerce('Sure! Here it is: {"a":"1"}', z.object({ a: z.number() }));
// → { a: 1 }   (extract-json, then string->number)

// booleans the way models spell them
coerce('{"active":"yes"}', z.object({ active: z.boolean() })); // → { active: true }

// enum casing, always safe
coerce('{"status":"ACTIVE"}', z.object({ status: z.enum(["active","inactive"]) }));
// → { status: "active" }   (case-insensitive, on by default)

Lossier guesses — enum near-misses ("activ" → "active") and key re-casing
(first_name → firstName) — are opt-in behind { fuzzy: true }, because
they can change meaning. And an ambiguous fuzzy match is refused, not guessed.

JSON Schema, too

Not a Zod shop? Same API, with an optional Ajv hook for authoritative
validation:

import { coerce, coerceWithAjv } from "coerce-json/json-schema";

coerce('{"id":"5","active":"yes"}', {
  type: "object",
  properties: { id: { type: "integer" }, active: { type: "boolean" }, role: { type: "string", default: "user" } },
  required: ["id", "active"],
  additionalProperties: false,
});
// → { id: 5, active: true, role: "user" }, ok: true

The part I care about: trust

If a library silently rewrites your data, you can't trust it in a pipeline. So
coerce-json holds four invariants, checked by property tests:

Never fabricates beyond documented defaults.
Every mutation is logged — changes is empty if and only if the output equals the input.
Idempotent — coercing an already-valid value is a no-op, and re-coercing any output changes nothing (it reaches a fixpoint).
Prototype-pollution safe — __proto__ / constructor / prototype keys are dropped and logged.

It's the second half of a pipeline

This is a companion to trickle-json,
my incremental partial-JSON parser for LLM streams. Together they're the
backbone of a streaming structured-output flow:

fetch → SSE → parse partial JSON (trickle-json) → repair/coerce to schema (coerce-json) → validate

trickle-json gives you the best value available on every streamed chunk
without throwing; coerce-json makes that value fit your schema and hands you
the receipts:

import { StreamingJsonParser } from "trickle-json";
import { coerce } from "coerce-json/zod";

const parser = new StreamingJsonParser();
parser.on("snapshot", renderPreview);
for await (const chunk of res.body) parser.write(chunk);

const { value, ok, changes } = coerce(parser.end(), Answer);
if (ok) save(value);
else console.warn("could not fully repair:", changes);

"But provider structured outputs already fix this!" — they help, a lot. But they
don't cover local and open models, older endpoints, streamed partials, or
anything wrapped in prose or a fence. That's exactly coerce-json's niche — and
when the output is already clean, it's a near no-op, so it's safe to leave in.

Try it

npm install coerce-json

npm: https://www.npmjs.com/package/coerce-json
GitHub: https://github.com/H1manshu01/coerce-json
Zero runtime dependencies, ESM + CJS, full types, published with provenance. zod and ajv are optional peers — bring them only if you use them.

If it mangles some input it shouldn't, open an issue with the string and the
schema — the change log and the "never fabricate" guarantee are the whole point,
so I want to know. ⭐ appreciated if it saves you a pile of hand-written casts.

JSON.parse throws on every token your LLM streams. Here's a cleaner fix.

Himanshu Sharma — Thu, 08 Oct 2026 11:29:25 +0000

If you've ever streamed structured output from an LLM, you've met this bug:

for await (const part of stream) {
  buffer += part.choices[0]?.delta?.content ?? "";
  const data = JSON.parse(buffer); // 💥 SyntaxError on literally every chunk
  render(data);
}

A model emits JSON one token at a time. Until the very last token arrives, the buffer is syntactically broken — {"city":"Par isn't valid JSON — so JSON.parse throws on every intermediate chunk. You want to show the UI filling in as it streams; the parser wants a complete document.

Let's look at the usual workarounds, why they fall short, and a small library I built to do it properly.

The usual fixes (and where they hurt)

1. "Just parse at the end." Accumulate everything, parse once when the stream finishes. Correct — but you've thrown away the entire point of streaming. The user stares at a spinner while a 2 KB object generates token by token.

2. Split on }. Tempting for a streamed array, but it breaks the instant an element contains a nested object or a } inside a string. You end up re-implementing a JSON tokenizer badly.

3. Repair-then-parse (jsonrepair and friends). Close the open braces/quotes, then JSON.parse. This mostly works, but:

It re-parses the whole buffer on every chunk — that's O(n²) over a stream. On a long response it gets genuinely slow.
Repair guesses. It can briefly surface values that aren't actually in the final object.

4. The incumbent, partial-json. It's the de-facto choice (~35M downloads/month) — but it hasn't shipped a release since May 2024 and is still on 0.1.x. And the better ideas (progressive snapshots, path subscriptions, schema typing, provider adapters) are scattered across a handful of tiny single-purpose packages.

These aren't hypothetical rough edges. Two real bugs from the Vercel AI SDK tracker show exactly what goes wrong when a parseability check meets a half-streamed buffer:

#12052 — a parseability check passes on a truncated tool-call argument, emitting a tool call whose input is still a half-string (Unterminated string in JSON at position 3337).
#6687 — an empty arguments string "" is treated as unparseable, so a zero-arg tool call never fires.

The common thread: deciding "is this done?" by asking "does it parse yet?" is fragile. You want a parser that always returns the best valid value so far and never throws — so you can call it on every chunk without a try/catch.

trickle-json

trickle-json is a zero-dependency, incremental partial-JSON parser built for LLM streams. The core idea:

import { parsePartial } from "trickle-json";

parsePartial('{"user":{"name":"Ja')  // → { user: { name: "Ja" } }
parsePartial("[1,2,3")               // → [1, 2, 3]
parsePartial("")                     // → undefined  (never throws)

It returns the most-complete valid value available right now. A truncated string comes back as the partial string; a member whose value hasn't arrived yet is dropped rather than guessed; empty input is undefined, not an exception.

Streaming, with progressive snapshots

import { StreamingJsonParser } from "trickle-json";

const parser = new StreamingJsonParser();
parser.on("snapshot", (value) => render(value)); // best-effort value after each chunk

for await (const part of stream) {
  const delta = part.choices[0]?.delta?.content ?? "";
  if (delta) parser.write(delta);
}
const final = parser.end();

React to one field, or to a list filling in

// fires as a single field grows
parser.on("path", "choices[0].message.content", (text) => appendToken(text));

// wildcards + per-element "append" events for streaming lists
parser.on("path", "items[*].id", (id, segments) => console.log(segments, id));
parser.on("append", "items", (item, index) => addRow(index, item));

Provider adapters (SSE + tool calls)

Parse a raw text/event-stream body directly, including tool/function-call arguments as they stream:

import { openAISSEToChunks, streamOpenAIToolCalls } from "trickle-json/openai";

for await (const calls of streamOpenAIToolCalls(openAISSEToChunks(res.body))) {
  // calls[0].name, calls[0].arguments — arguments fill in as they arrive
}

There's a matching trickle-json/anthropic (text + input_json_delta tool-use), and a trickle-json/zod entry point for typed partial snapshots.

The part I'm proud of: it's O(n), not O(n²)

The naive approach re-parses the whole accumulated buffer on every chunk. trickle-json uses a resumable, character-at-a-time parser that keeps its state across chunks and mutates a live value tree in place — so feeding N characters total costs O(N), no matter how they're split.

Streaming a growing document in small chunks (reproduce with npm run bench:stream):

records	trickle-json (incremental)	re-parse each chunk	jsonriver
250	1.0 ms	182 ms	2.3 ms
1,000	2.9 ms	2,927 ms	5.1 ms
2,000	5.7 ms	12,347 ms	9.0 ms

At 2,000 records the incremental engine is ~2,150× faster than re-parsing the buffer on every chunk — the quadratic cost simply disappears.

Correct, and tested like it

Partial parsing is only useful if it's right. The streaming engine is verified equivalent to the one-shot parser on every prefix of a broad corpus plus 200 randomly generated documents, with a never-throw guarantee checked against an adversarial/garbage corpus. Against the incumbents on a prefix-by-prefix sweep (npm run bench):

parser	throws on truncation	no invented data	empty input
trickle-json	0%	99.3%	`undefined`
partial-json	0.3%	99.4%	throws
best-effort-json-parser	0%	69.2%	`""`

trickle-json is the only one that never throws on truncation and returns undefined (not an exception) on empty input — so it's safe to call on every chunk.

(One honest caveat shared by every snapshot parser: a number truncated mid-exponent, like 1.5e-1 as a prefix of 1.5e-10, transiently reads as 0.15. It resolves as soon as the next digit arrives.)

Try it

npm install trickle-json

npm: https://www.npmjs.com/package/trickle-json
GitHub: https://github.com/H1manshu01/trickle-json
Zero runtime dependencies, ~2.6 kB min+gzip, ESM + CJS, full types, published with provenance.

If you're building streaming AI UIs, give it a spin — and if it breaks on some input, open an issue with the string; the never-throw guarantee is the whole point, so I want to know. ⭐ appreciated if it saves you a try/catch.