DEV Community: plasma

I Was Measuring LLM Latency Wrong

plasma — Mon, 27 Jul 2026 02:58:52 +0000

For a long time, I recorded one latency number for every LLM request:

latency_ms = response_finished_at - request_started_at

It looked useful. I could calculate an average, add a p95 chart, and see whether a model was getting slower.

But whenever a user complained that an AI feature felt slow, that number rarely told me what to fix.

Sometimes the request waited in a queue before reaching the provider. Sometimes the first token was slow, even though the rest of the response streamed quickly. Sometimes the model finished in two seconds and a tool call took another eight.

All of those cases appeared in my dashboard as “LLM latency.”

That was the problem. I was measuring a request, while the user was waiting for a feature.

One latency number hides several different problems

An LLM-powered feature usually has more stages than the API request itself:

User action
  -> application queue
  -> LLM request
  -> first token
  -> complete generation
  -> tool execution
  -> validation and persistence
  -> visible result

If I only record the time between sending the request and receiving the final token, I miss everything before and after it.

I now separate latency into at least five measurements:

queue_ms: time spent waiting before the request starts
ttft_ms: time from request start to the first useful token
generation_ms: time from the first token to the final token
tool_ms: time spent running tools or external actions
end_to_end_ms: time from the original user action to the completed feature

These numbers answer different questions.

A high queue_ms points toward application capacity or concurrency limits. A high ttft_ms affects how responsive streaming feels. A high generation_ms may be related to output length or model behavior. A high tool_ms has little to do with model speed.

The user only sees end_to_end_ms.

A small Node.js latency tracker

Here is a minimal example using the OpenAI Node.js SDK. It also works with OpenAI-compatible APIs by setting LLM_BASE_URL.

Install the dependency:

npm install openai

Save this as latency-tracker.mjs:

import OpenAI from "openai";
import { performance } from "node:perf_hooks";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL || undefined,
});

const model = process.env.LLM_MODEL;

if (!process.env.LLM_API_KEY || !model) {
  throw new Error(
    "Set LLM_API_KEY and LLM_MODEL before running this script."
  );
}

const round = (value) =>
  value == null ? null : Math.round(value * 100) / 100;

async function runMeasuredFeature({
  feature,
  messages,
  enqueuedAt = performance.now(),
  toolStep,
}) {
  const dequeuedAt = performance.now();
  const requestStartedAt = performance.now();

  let streamReadyAt = null;
  let firstTokenAt = null;
  let generationFinishedAt = null;
  let toolStartedAt = null;
  let toolFinishedAt = null;
  let output = "";

  try {
    const stream = await client.chat.completions.create({
      model,
      messages,
      stream: true,
    });

    streamReadyAt = performance.now();

    for await (const chunk of stream) {
      const text = chunk.choices[0]?.delta?.content ?? "";

      if (text && firstTokenAt === null) {
        firstTokenAt = performance.now();
      }

      output += text;
    }

    generationFinishedAt = performance.now();

    if (toolStep) {
      toolStartedAt = performance.now();
      await toolStep(output);
      toolFinishedAt = performance.now();
    }

    const featureFinishedAt = performance.now();

    const metrics = {
      feature,
      status: "success",
      queue_ms: round(dequeuedAt - enqueuedAt),
      stream_setup_ms: round(streamReadyAt - requestStartedAt),
      ttft_ms: round(
        firstTokenAt === null
          ? null
          : firstTokenAt - requestStartedAt
      ),
      generation_ms: round(
        firstTokenAt === null
          ? null
          : generationFinishedAt - firstTokenAt
      ),
      model_total_ms: round(
        generationFinishedAt - requestStartedAt
      ),
      tool_ms: round(
        toolStartedAt === null
          ? 0
          : toolFinishedAt - toolStartedAt
      ),
      end_to_end_ms: round(featureFinishedAt - enqueuedAt),
      output_characters: output.length,
    };

    console.log(JSON.stringify(metrics));

    return { output, metrics };
  } catch (error) {
    const failedAt = performance.now();

    const metrics = {
      feature,
      status: "failed",
      queue_ms: round(dequeuedAt - enqueuedAt),
      ttft_ms: round(
        firstTokenAt === null
          ? null
          : firstTokenAt - requestStartedAt
      ),
      elapsed_before_failure_ms: round(
        failedAt - requestStartedAt
      ),
      end_to_end_ms: round(failedAt - enqueuedAt),
      error_name: error.name,
      error_message: error.message,
    };

    console.error(JSON.stringify(metrics));
    throw error;
  }
}

const enqueuedAt = performance.now();

// In a real application, other queued work may happen here.

const result = await runMeasuredFeature({
  feature: "support_reply",
  enqueuedAt,
  messages: [
    {
      role: "user",
      content:
        "Explain why measuring only average LLM latency can be misleading.",
    },
  ],
});

console.log(result.output);

Run it with:

LLM_API_KEY="your-api-key" \
LLM_MODEL="your-model-name" \
node latency-tracker.mjs

A log entry will look like this:

{
  "feature": "support_reply",
  "status": "success",
  "queue_ms": 0.14,
  "stream_setup_ms": 482.31,
  "ttft_ms": 615.27,
  "generation_ms": 1388.42,
  "model_total_ms": 2003.69,
  "tool_ms": 0,
  "end_to_end_ms": 2004.18,
  "output_characters": 721
}

The values above are only an example of the log format. Actual results depend on the model, provider, prompt, output length, network, and application workload.

Time to first token is not total latency

Streaming made me realize that “fast” can mean two different things.

Consider two hypothetical requests:

Request A
TTFT:          300 ms
Generation:  4700 ms
Total:       5000 ms

Request B
TTFT:         2200 ms
Generation:  2800 ms
Total:       5000 ms

Both have the same total model latency. They do not feel the same.

Request A gives the user visible feedback almost immediately. Request B leaves the interface looking stuck for more than two seconds.

If I only chart total latency, I cannot see that difference. For interactive applications, I want a separate TTFT percentile instead of treating streaming as an implementation detail.

I also record the time of the first useful token, not simply the first stream event. Some APIs send metadata or an empty delta before content begins.

Average latency was hiding the requests I cared about

Averages are convenient and often misleading.

A few very slow requests can be hidden inside a reasonable average. Different features can also have completely different latency profiles.

A short classification call and a long report-generation call should not share the same latency chart just because they use the same model.

I now group latency by fields such as:

feature
model
provider
streaming
tool_name
status
prompt_version
output_size_bucket

Then I look at percentiles:

p50 shows the typical experience
p95 shows the slow experience that occurs regularly
p99 helps expose severe outliers

I avoid placing raw user prompts in latency logs. Operational metadata is usually enough to identify whether the delay belongs to queuing, model generation, tool execution, or another application stage.

Model latency and feature latency need separate dashboards

This distinction changed how I interpreted performance incidents.

Suppose the model finishes quickly, but the feature still feels slow because it calls a search service, updates a database, and waits for another workflow.

Calling this an “LLM latency problem” sends me toward the wrong fix. Switching models will not make a slow database write faster.

I keep two related views now.

The model view tracks:

time to first token
generation time
request failures
output size
model and provider

The feature view tracks:

queue time
full model time
tool execution time
validation time
persistence time
end-to-end completion time

The first view helps diagnose the inference layer. The second represents what the user actually experiences.

Failed requests need timing data too

Initially, I only emitted latency metrics after a successful response. That meant the slowest failures were absent from the dashboard.

A timeout after 30 seconds did not increase the latency chart because it never reached the success logging code.

That made the system look faster when it was less reliable.

Now failure logs include:

elapsed time before failure
whether any content had arrived
the stage where the failure occurred
retry attempt
error category
feature name

Success and failure latency should be analyzed together, while still being distinguishable. Otherwise, a dashboard can report healthy performance by quietly excluding the worst user experiences.

The metric I care about most

Provider latency still matters. I need it when evaluating models, investigating regressions, or configuring timeouts.

But it is no longer the main number I use to describe an AI feature.

The metric I care about most is:

Time from user action to a valid, visible, completed result

That definition forces me to include queues, retries, tools, validation, and persistence. It also prevents a fast model response from hiding a slow or broken workflow.

The model can be fast while the product is slow.

Once I measured the full path, latency stopped being a vague provider complaint and became a set of specific engineering problems I could actually investigate.

I work on TokenBay, so I spend a lot of time thinking about model and provider behavior. But the most useful latency lesson for me was broader: measure the feature boundary first, then break it down until every slow stage has an owner.

The Bug Only Happened After I Switched LLM Providers

plasma — Fri, 24 Jul 2026 02:51:53 +0000

The request succeeded.

The response was valid JSON.

The SDK did not throw an exception.

And the application still broke.

The bug appeared after I switched an LLM application to a different provider with an OpenAI-compatible endpoint. The integration looked almost too easy: change the base URL, replace the API key, keep the request body, and ship.

That worked for the simplest prompt.

It did not mean the providers behaved the same way.

The failure was not in the HTTP contract. It was in the assumptions my application had quietly built around one provider's behavior.

Compatible Requests Do Not Guarantee Compatible Behavior

When an API describes itself as OpenAI-compatible, it usually means that familiar requests are accepted:

{
  "model": "some-model",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this support ticket."
    }
  ]
}

That is useful compatibility. It lets us reuse clients, authentication patterns, and much of our integration code.

But production applications depend on more than whether the server accepts the request.

They may also depend on:

whether content is a string, an array, an empty value, or null
how tool-call arguments are serialized
which finish_reason values can appear
whether usage fields are always included
how streaming completion is signaled
whether structured-output constraints are enforced
how errors, timeouts, and unsupported parameters are reported

Two providers can accept the same request and return responses that are syntactically valid but operationally different.

That is where the interesting bugs live.

The Assumption Hidden in My Parser

My original response handler effectively assumed this:

const text = response.choices[0].message.content.trim();

It worked until the selected model returned a tool call.

In that response, message.content was null, while the actual output was inside message.tool_calls.

The API had not failed. My parser had.

The first fix was straightforward:

function parseAssistantMessage(response) {
  const choice = response?.choices?.[0];

  if (!choice?.message) {
    throw new Error("Response is missing choices[0].message");
  }

  const { content, tool_calls: toolCalls } = choice.message;

  if (Array.isArray(toolCalls) && toolCalls.length > 0) {
    return {
      type: "tool_calls",
      toolCalls,
      finishReason: choice.finish_reason ?? null
    };
  }

  if (typeof content === "string" && content.trim() !== "") {
    return {
      type: "text",
      text: content,
      finishReason: choice.finish_reason ?? null
    };
  }

  throw new Error(
    `Assistant returned neither text nor tool calls. finish_reason=${choice.finish_reason ?? "missing"}`
  );
}

That fixed one symptom, but it exposed the larger problem: I had treated the response shape I usually saw as if it were the entire protocol.

Tool Calls Are an Especially Fragile Boundary

Tool calling creates at least three compatibility questions.

1. Are the arguments valid JSON?

A tool call may contain an argument string like this:

{
  "name": "create_ticket",
  "arguments": "{\"priority\":\"high\",\"summary\":\"Login failure\"}"
}

The outer response can be valid JSON while the nested arguments string is not.

Never pass it directly to a tool:

function parseToolArguments(toolCall) {
  const raw = toolCall?.function?.arguments;

  if (typeof raw !== "string") {
    throw new Error("Tool arguments are missing or are not a string");
  }

  try {
    return JSON.parse(raw);
  } catch (error) {
    throw new Error(
      `Invalid JSON in tool arguments for ${toolCall.function?.name ?? "unknown tool"}: ${error.message}`
    );
  }
}

2. Do the arguments satisfy the business contract?

Valid JSON is not the same as a valid action.

function validateCreateTicket(args) {
  const allowedPriorities = new Set(["low", "medium", "high"]);

  if (typeof args.summary !== "string" || args.summary.trim() === "") {
    throw new Error("create_ticket.summary must be a non-empty string");
  }

  if (!allowedPriorities.has(args.priority)) {
    throw new Error(
      `create_ticket.priority must be low, medium, or high`
    );
  }

  return {
    summary: args.summary.trim(),
    priority: args.priority
  };
}

The model proposes an action. Application code decides whether that action is allowed.

3. What tells the application that generation is complete?

Some integrations assume that a particular finish_reason always accompanies a tool call. That assumption should be tested, not inherited from a single provider.

I now treat the presence and validity of tool_calls as one signal, and finish_reason as separate diagnostic information. If they disagree, I log the response and stop before executing anything with side effects.

I Started Testing Provider Behavior, Not Just Availability

A normal health check asks:

Can this endpoint return a response?

That is necessary, but too weak for switching providers.

A compatibility check should ask:

Does this provider preserve the behaviors my application relies on?

Here is a small Node.js script I use as a starting point.

It requires Node.js 18 or later and no external packages.

// provider-compatibility-test.mjs

const config = {
  name: process.env.PROVIDER_NAME ?? "candidate-provider",
  baseUrl: process.env.LLM_BASE_URL,
  apiKey: process.env.LLM_API_KEY,
  model: process.env.LLM_MODEL
};

for (const [key, value] of Object.entries(config)) {
  if (!value) {
    throw new Error(`Missing configuration value: ${key}`);
  }
}

async function createChatCompletion(body) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 20_000);

  try {
    const response = await fetch(
      `${config.baseUrl.replace(/\/$/, "")}/chat/completions`,
      {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: `Bearer ${config.apiKey}`
        },
        body: JSON.stringify(body),
        signal: controller.signal
      }
    );

    const rawBody = await response.text();

    let data;
    try {
      data = JSON.parse(rawBody);
    } catch {
      throw new Error(
        `Provider returned non-JSON content with HTTP ${response.status}`
      );
    }

    if (!response.ok) {
      throw new Error(
        `Provider returned HTTP ${response.status}: ${JSON.stringify(data)}`
      );
    }

    return data;
  } finally {
    clearTimeout(timeout);
  }
}

function firstChoice(response) {
  const choice = response?.choices?.[0];

  if (!choice || !choice.message) {
    throw new Error("Missing choices[0].message");
  }

  return choice;
}

function inspectCommonFields(response) {
  const choice = firstChoice(response);

  return {
    hasId: typeof response.id === "string",
    hasModel: typeof response.model === "string",
    hasUsage: response.usage !== undefined,
    contentType:
      choice.message.content === null
        ? "null"
        : Array.isArray(choice.message.content)
          ? "array"
          : typeof choice.message.content,
    toolCallCount: Array.isArray(choice.message.tool_calls)
      ? choice.message.tool_calls.length
      : 0,
    finishReason: choice.finish_reason ?? "missing"
  };
}

async function testTextResponse() {
  const response = await createChatCompletion({
    model: config.model,
    temperature: 0,
    messages: [
      {
        role: "user",
        content: 'Reply with exactly: PROVIDER_TEST_OK'
      }
    ]
  });

  const choice = firstChoice(response);
  const content = choice.message.content;

  if (typeof content !== "string") {
    throw new Error(
      `Expected string content, received ${
        content === null ? "null" : typeof content
      }`
    );
  }

  if (!content.includes("PROVIDER_TEST_OK")) {
    throw new Error(`Unexpected text response: ${content}`);
  }

  return inspectCommonFields(response);
}

async function testToolCall() {
  const response = await createChatCompletion({
    model: config.model,
    temperature: 0,
    messages: [
      {
        role: "user",
        content:
          "Create a high-priority ticket with the summary Login failure. Use the provided tool."
      }
    ],
    tools: [
      {
        type: "function",
        function: {
          name: "create_ticket",
          description: "Create a support ticket",
          parameters: {
            type: "object",
            additionalProperties: false,
            properties: {
              summary: {
                type: "string"
              },
              priority: {
                type: "string",
                enum: ["low", "medium", "high"]
              }
            },
            required: ["summary", "priority"]
          }
        }
      }
    ],
    tool_choice: {
      type: "function",
      function: {
        name: "create_ticket"
      }
    }
  });

  const choice = firstChoice(response);
  const toolCall = choice.message.tool_calls?.[0];

  if (!toolCall) {
    throw new Error("Expected a tool call but none was returned");
  }

  if (toolCall.function?.name !== "create_ticket") {
    throw new Error(
      `Expected create_ticket, received ${toolCall.function?.name}`
    );
  }

  let args;
  try {
    args = JSON.parse(toolCall.function.arguments);
  } catch (error) {
    throw new Error(`Tool arguments are not valid JSON: ${error.message}`);
  }

  if (
    typeof args.summary !== "string" ||
    !["low", "medium", "high"].includes(args.priority)
  ) {
    throw new Error(
      `Tool arguments failed validation: ${JSON.stringify(args)}`
    );
  }

  return {
    ...inspectCommonFields(response),
    toolName: toolCall.function.name,
    argumentKeys: Object.keys(args).sort()
  };
}

async function runTest(name, test) {
  const startedAt = Date.now();

  try {
    const observations = await test();

    return {
      test: name,
      status: "passed",
      durationMs: Date.now() - startedAt,
      observations
    };
  } catch (error) {
    return {
      test: name,
      status: "failed",
      durationMs: Date.now() - startedAt,
      error: error.message
    };
  }
}

const results = [];

results.push(await runTest("text response", testTextResponse));
results.push(await runTest("tool call", testToolCall));

console.log(
  JSON.stringify(
    {
      provider: config.name,
      model: config.model,
      testedAt: new Date().toISOString(),
      results
    },
    null,
    2
  )
);

if (results.some((result) => result.status === "failed")) {
  process.exitCode = 1;
}

Run it against the provider currently in production:

PROVIDER_NAME=current \
LLM_BASE_URL=https://current-provider.example/v1 \
LLM_API_KEY=your-key \
LLM_MODEL=your-model \
node provider-compatibility-test.mjs

Then run the same script against the candidate:

PROVIDER_NAME=candidate \
LLM_BASE_URL=https://candidate-provider.example/v1 \
LLM_API_KEY=your-key \
LLM_MODEL=your-model \
node provider-compatibility-test.mjs

The goal is not merely to get two green results. Save and compare the observations.

A different contentType, missing usage information, unexpected finish reason, or changed tool-call shape may be harmless. It may also reveal an assumption that will break elsewhere in the application.

What I Test Before Switching Now

I keep the suite small enough to run during deployment, but specific enough to represent the application.

For a text-only application, that may include:

a normal text response
an intentionally invalid request
a timeout or cancellation
a response near the output-token limit

For an agent or workflow, I also test:

a forced tool call
multiple available tools
malformed or incomplete tool arguments
a tool result returned to the model
a streaming tool call
an action that must not execute twice

For structured data, I test the actual schema used by the application—not a toy object with two string fields.

The important question is not whether every provider produces identical JSON. They probably will not.

The important question is whether those differences remain inside the boundaries the application knows how to handle.

Normalize at the Boundary

Once I know which differences are intentional, I normalize them before the rest of the application sees them.

function normalizeCompletion(response) {
  const choice = response?.choices?.[0];

  if (!choice?.message) {
    return {
      status: "invalid",
      reason: "missing_assistant_message",
      raw: response
    };
  }

  const toolCalls = Array.isArray(choice.message.tool_calls)
    ? choice.message.tool_calls
    : [];

  if (toolCalls.length > 0) {
    return {
      status: "tool_calls",
      toolCalls,
      finishReason: choice.finish_reason ?? null,
      usage: response.usage ?? null
    };
  }

  if (typeof choice.message.content === "string") {
    return {
      status: "text",
      text: choice.message.content,
      finishReason: choice.finish_reason ?? null,
      usage: response.usage ?? null
    };
  }

  return {
    status: "invalid",
    reason: "unsupported_message_shape",
    finishReason: choice.finish_reason ?? null,
    raw: response
  };
}

The rest of the application consumes this internal contract instead of depending directly on every provider's response details.

That makes provider-specific behavior visible in one place. It also makes failures easier to log, test, and stop safely.

A Provider Switch Is a Dependency Upgrade

Changing an LLM provider may look like a configuration update:

- LLM_BASE_URL=https://provider-a.example/v1
+ LLM_BASE_URL=https://provider-b.example/v1

Operationally, I treat it more like upgrading a major dependency.

The endpoint may be compatible while the behavior at the edges is not. Those edges include streaming, tool calls, structured outputs, token limits, cancellation, error semantics, and usage reporting.

None of this makes OpenAI-compatible APIs a bad idea. Shared interfaces remove a lot of unnecessary integration work. I work on TokenBay, where routing requests across different models makes this compatibility layer especially useful.

But a shared interface should make switching testable—not invisible.

Now, before I change a base URL in production, I run the candidate provider through the same behavioral tests as the current one.

The request being accepted is only the first test.

It is not the definition of compatibility.

I Couldn’t Fix My LLM Costs Until I Measured Tokens Per Feature

plasma — Wed, 22 Jul 2026 03:15:40 +0000

My LLM bill kept growing, so I did what seemed obvious: I looked for a cheaper model.

That helped a little, but it didn't explain why the bill was growing.

The dashboard could tell me how many tokens the application used. It couldn't tell me what those tokens were doing.

Were they coming from chat?

Document summaries?

Background classification?

An agent retrying the same tool call?

I was trying to optimize a total without knowing which product feature created it.

The useful unit wasn't tokens per model.

It was tokens per feature.

Model-level totals hid the real problem

A provider dashboard usually groups usage by model, API key, project, or time period.

That is useful for billing, but not always for product decisions.

Imagine an application with four LLM-powered features:

interactive chat
document summarization
support-ticket classification
an agent that prepares weekly reports

If the bill increases by 30%, the model name doesn't explain which feature changed.

Maybe chat traffic grew.

Maybe summarization started sending entire documents instead of selected sections.

Maybe the classifier received a much larger system prompt.

Maybe the report agent retried after tool failures and generated the same plan several times.

Those problems require completely different fixes.

Switching every request to a cheaper model would reduce the bill, but it could also hide the engineering mistake.

Tag every request with a feature

I started giving every LLM call a small amount of application context:

const context = {
  feature: "document_summary",
  operation: "initial_summary",
  customer_tier: "pro"
};

The model provider doesn't need these fields.

They belong in the application's usage record.

I avoid using individual user IDs as the primary grouping dimension. For cost analysis, a product feature, workflow, or operation is normally more useful and creates fewer privacy problems.

A practical record looks like this:

{
  "timestamp": "2026-07-22T03:12:48.201Z",
  "feature": "document_summary",
  "operation": "initial_summary",
  "model": "example-model",
  "input_tokens": 4821,
  "output_tokens": 614,
  "total_tokens": 5435,
  "latency_ms": 2834,
  "status": "success"
}

Once I had that record for every request, I could answer better questions:

Which feature uses the most tokens?
Which feature has the fastest usage growth?
How many tokens does one successful operation require?
Are retries increasing tokens without increasing completed work?
Is the input growing faster than the output?
Which feature is using an expensive model without needing it?

A small Node.js usage recorder

Here is a minimal implementation using an OpenAI-compatible chat-completions endpoint.

It uses only built-in Node.js modules and expects Node 18 or newer.

Create llm-client.mjs:

import { appendFile } from "node:fs/promises";
import { randomUUID } from "node:crypto";

const API_URL =
  process.env.LLM_API_URL ??
  "https://api.example.com/v1/chat/completions";

const API_KEY = process.env.LLM_API_KEY;
const USAGE_FILE =
  process.env.LLM_USAGE_FILE ?? "./llm-usage.jsonl";

if (!API_KEY) {
  throw new Error("LLM_API_KEY is required");
}

async function writeUsage(record) {
  await appendFile(
    USAGE_FILE,
    `${JSON.stringify(record)}\n`,
    "utf8"
  );
}

export async function createChatCompletion({
  feature,
  operation,
  model,
  messages,
  temperature = 0
}) {
  if (!feature || !operation) {
    throw new Error(
      "Every LLM request needs a feature and operation"
    );
  }

  const requestId = randomUUID();
  const startedAt = Date.now();

  try {
    const response = await fetch(API_URL, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${API_KEY}`,
        "x-client-request-id": requestId
      },
      body: JSON.stringify({
        model,
        messages,
        temperature
      })
    });

    const body = await response.json();

    if (!response.ok) {
      throw new Error(
        body?.error?.message ??
        `LLM request failed with status ${response.status}`
      );
    }

    const usage = body.usage ?? {};

    await writeUsage({
      timestamp: new Date().toISOString(),
      request_id: requestId,
      feature,
      operation,
      model,
      input_tokens:
        usage.prompt_tokens ??
        usage.input_tokens ??
        null,
      output_tokens:
        usage.completion_tokens ??
        usage.output_tokens ??
        null,
      total_tokens: usage.total_tokens ?? null,
      latency_ms: Date.now() - startedAt,
      status: "success"
    });

    return body;
  } catch (error) {
    await writeUsage({
      timestamp: new Date().toISOString(),
      request_id: requestId,
      feature,
      operation,
      model,
      input_tokens: null,
      output_tokens: null,
      total_tokens: null,
      latency_ms: Date.now() - startedAt,
      status: "error",
      error: error?.message ?? String(error)
    });

    throw error;
  }
}

A feature calls the wrapper like this:

import {
  createChatCompletion
} from "./llm-client.mjs";

const result = await createChatCompletion({
  feature: "document_summary",
  operation: "initial_summary",
  model: "example-model",
  messages: [
    {
      role: "system",
      content:
        "Summarize the document into five concise bullet points."
    },
    {
      role: "user",
      content: "Document content goes here."
    }
  ]
});

console.log(result.choices[0].message.content);

The wrapper writes one line to llm-usage.jsonl for every request.

It does not store the prompt or model response. For feature-level cost analysis, I usually need usage metadata, not user content.

Summarize tokens by feature

The raw JSONL file is useful for debugging, but the first report I want is much simpler:

Feature                  Requests   Input      Output     Total
document_summary         42         182,140    21,382     203,522
interactive_chat         391        96,241     44,829     141,070
weekly_report_agent      18         81,440     19,205     100,645
ticket_classification    804        51,462     8,214      59,676

Create summarize-usage.mjs:

import { readFile } from "node:fs/promises";

const file =
  process.env.LLM_USAGE_FILE ?? "./llm-usage.jsonl";

const content = await readFile(file, "utf8");

const records = content
  .split("\n")
  .filter(Boolean)
  .map(line => JSON.parse(line))
  .filter(record => record.status === "success");

const features = new Map();

for (const record of records) {
  const current = features.get(record.feature) ?? {
    feature: record.feature,
    requests: 0,
    input_tokens: 0,
    output_tokens: 0,
    total_tokens: 0,
    missing_usage: 0
  };

  current.requests += 1;

  if (record.total_tokens == null) {
    current.missing_usage += 1;
  } else {
    current.input_tokens += record.input_tokens ?? 0;
    current.output_tokens += record.output_tokens ?? 0;
    current.total_tokens += record.total_tokens;
  }

  features.set(record.feature, current);
}

const result = [...features.values()]
  .sort((a, b) => b.total_tokens - a.total_tokens);

console.table(result);

Run it with:

node summarize-usage.mjs

The absolute totals are only the first layer.

I also calculate tokens per successful operation:

const tokensPerRequest =
  feature.total_tokens / feature.requests;

For agent workflows, I prefer tokens per completed workflow rather than tokens per API request.

One user action might trigger five model calls. If I optimize each request separately without tracking the completed action, I can make the request-level metrics look better while the workflow still wastes tokens.

Add operation-level detail

A feature tag tells me where the usage came from.

An operation tag tells me what happened inside that feature.

For example:

weekly_report_agent
├── create_plan
├── call_data_tool
├── repair_tool_arguments
├── draft_report
└── revise_report

Suppose weekly_report_agent consumes 100,000 tokens.

That total alone doesn't reveal much.

If 45,000 tokens come from repair_tool_arguments, I probably don't need a cheaper writing model. I need to understand why the tool call keeps failing.

If draft_report input tokens keep growing, I might be sending too much raw source material.

If create_plan runs three times for a single report, the retry or state-management logic needs attention.

The feature tells me where to look.

The operation tells me what to fix.

Measure retries separately

Retries are easy to miss because the successful response looks normal.

I add an attempt number to each record:

{
  feature: "weekly_report_agent",
  operation: "draft_report",
  attempt: 2
}

Then I compare:

total requests
unique operation IDs
successful operations
retry tokens
tokens per successful operation

This prevents a misleading result where traffic appears stable but token usage doubles because requests are being repeated internally.

An operation ID can be created once at the beginning of the workflow:

const operationId = randomUUID();

Every retry keeps the same operation ID but increments the attempt:

{
  operation_id: operationId,
  attempt: 2
}

Now retry waste can be measured directly instead of inferred from a monthly bill.

Convert tokens to cost outside the request path

I don't hardcode model prices inside the API wrapper.

Prices change, and different providers may expose different input, cached-input, and output rates.

Instead, I keep a separate rate table:

const rates = {
  "example-model": {
    input_per_million: 1.00,
    output_per_million: 4.00
  }
};

Then estimate cost during reporting:

function estimateCost(record, rate) {
  const inputCost =
    ((record.input_tokens ?? 0) / 1_000_000) *
    rate.input_per_million;

  const outputCost =
    ((record.output_tokens ?? 0) / 1_000_000) *
    rate.output_per_million;

  return inputCost + outputCost;
}

The numbers above are placeholders, not current pricing.

Before using the report for billing decisions, I replace them with the current rates from the provider and record the effective date of that rate table.

Keeping pricing outside the request wrapper also lets me recalculate historical usage after a pricing change without modifying the original token records.

Missing usage is a metric too

Not every API response includes token usage in the same format.

Streaming responses may require an additional option to return usage. Some providers expose different field names. Failed requests may not return usage at all.

I don't silently convert missing usage to zero.

Zero means the request used no tokens.

null means I don't know.

Those are very different statements.

The report includes a missing_usage count for each feature. If that number grows, the cost report is becoming less trustworthy even if the visible totals look stable.

What I optimize first

Once usage is grouped by feature and operation, I work down this list:

Unnecessary calls

Is the feature calling the model when a cached result, deterministic function, or database query would work?

Repeated context

Is every request sending the same large document, tool schema, conversation history, or instructions?

Retry waste

Are timeouts, invalid tool arguments, or parsing failures causing the same operation to run again?

Oversized outputs

Does a classification task need 800 generated tokens, or would a small structured response be enough?

Model selection

After fixing the request shape and workflow behavior, is the current model still necessary for this operation?

Model selection matters. It just isn't always the first problem.

The metric I was missing

A monthly LLM bill tells me the result.

Tokens per feature tell me where the result came from.

Tokens per successful operation go one step further: they connect infrastructure usage to something the product actually accomplished.

That changed the questions I ask.

Instead of:

Which model should I replace?

I can ask:

Why did document summarization input grow by 40%?

Why does one completed report require nine model calls?

Why are retry tokens increasing while completed workflows stay flat?

Those questions lead to engineering fixes, not just cheaper invoices.

I work on TokenBay, so I regularly deal with multiple models behind an OpenAI-compatible interface. Model-level usage is still useful, but feature and operation tags are what make that usage actionable inside an application.

The next thing I'm adding is a small budget guardrail: not a global monthly limit, but a token budget for each completed feature operation.

The LLM Response Looked Fine. My Parser Disagreed.

plasma — Mon, 20 Jul 2026 06:22:53 +0000

The response looked perfectly reasonable in the logs.

It had the fields I asked for. The values made sense. Nothing looked obviously broken.

Then my application rejected it.

That failure taught me to separate two questions I had been treating as one:

Does the response look correct to a human?
Is the response safe for software to consume?

Those are very different standards.

A person can understand JSON wrapped in a Markdown fence, infer a missing optional field, or ignore a sentence added before the object. A parser cannot—and in production, it should not have to guess.

The response was valid English, not valid data

Imagine asking a model for this:

{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice"
}

Instead, it returns:

Here is the JSON you requested:

json
{
"category": "billing",
"priority": "high",
"summary": "Customer was charged twice"
}

json

To a human, that is fine.

To JSON.parse(), it is a syntax error.

That is the obvious failure. The more annoying ones are valid JSON with the wrong shape:

{
  "category": "billing",
  "priority": "urgent",
  "summary": ""
}

The JSON parses successfully, but the application still has three problems:

urgent is not a supported priority
summary is present but unusable
the response passed syntax validation without passing business validation

This is why I no longer treat JSON.parse() as validation. It only proves that a string is valid JSON.

The small validation layer I use

For Node.js projects, I usually put a schema between the model response and the rest of the application.

Here is a minimal example using Zod.

Install it:

npm install zod

Create validate-llm-output.js:

import { z } from "zod";

const TicketSchema = z.object({
  category: z.enum([
    "billing",
    "technical",
    "account",
    "other"
  ]),
  priority: z.enum([
    "low",
    "medium",
    "high"
  ]),
  summary: z.string().trim().min(1).max(240)
}).strict();

function removeMarkdownFence(value) {
  const trimmed = value.trim();

  const match = trimmed.match(
    /^```
{% endraw %}
(?:json)?\s*([\s\S]*?)\s*
{% raw %}
```$/i
  );

  return match ? match[1].trim() : trimmed;
}

export function validateTicketOutput(rawOutput) {
  const cleaned = removeMarkdownFence(rawOutput);

  let parsed;

  try {
    parsed = JSON.parse(cleaned);
  } catch (error) {
    return {
      ok: false,
      stage: "json_parse",
      error: error.message,
      rawOutput
    };
  }

  const result = TicketSchema.safeParse(parsed);

  if (!result.success) {
    return {
      ok: false,
      stage: "schema_validation",
      error: result.error.flatten(),
      rawOutput
    };
  }

  return {
    ok: true,
    data: result.data
  };
}

Then test it:

import { validateTicketOutput } from "./validate-llm-output.js";

const response = `\`\`\`json
{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice"
}
\`\`\``;

console.log(validateTicketOutput(response));

The result is:

{
  ok: true,
  data: {
    category: "billing",
    priority: "high",
    summary: "Customer was charged twice"
  }
}

Now try a response with an unsupported priority:

const response = JSON.stringify({
  category: "billing",
  priority: "urgent",
  summary: "Customer was charged twice"
});

console.log(validateTicketOutput(response));

This time, the validation fails before the value reaches application logic.

I keep parsing and validation separate

It is tempting to put every cleanup rule into one large function. I found it easier to debug when each failure has a distinct stage:

raw model response
        ↓
transport check
        ↓
JSON parsing
        ↓
schema validation
        ↓
business validation
        ↓
side effect

This distinction matters when something breaks.

A truncated response is not the same failure as an unsupported enum. A structurally correct object with a prohibited action is not the same failure as malformed JSON.

If all three become Invalid model response, the log is technically correct and operationally useless.

I record at least:

{
  requestId,
  model,
  stage,
  schemaVersion,
  error,
  responseLength,
  finishReason
}

I do not automatically store the full prompt or response. Whether that is acceptable depends on the data involved. For sensitive inputs, metadata and a redacted failure sample are often safer than dumping raw content into logs.

What I do not silently repair

Some cleanup is deterministic.

Removing one complete Markdown fence is predictable. Trimming whitespace is predictable.

Inventing a missing field is not.

I avoid quietly doing things like:

if (!output.priority) {
  output.priority = "medium";
}

That converts a model failure into application behavior without leaving a clear trace.

The same applies to truncated JSON. If the response ends halfway through an object, I do not ask a regex to reconstruct what the model “probably meant.”

My rule is simple:

Normalize presentation noise when the transformation is unambiguous
Reject structural ambiguity
Never execute a side effect from an invalid response

A failed classification can usually be retried or reviewed.

An incorrect refund, email, database update, or account action is much harder to undo.

Schema validation is not the final gate

A schema can prove that this object has the expected shape:

{
  "action": "refund",
  "amount": 500,
  "reason": "duplicate charge"
}

It cannot prove that the refund is authorized.

That requires business rules:

function validateRefund(ticket, account) {
  if (ticket.action !== "refund") {
    return { ok: false, reason: "unsupported_action" };
  }

  if (ticket.amount > account.maxAutomaticRefund) {
    return { ok: false, reason: "human_approval_required" };
  }

  if (!account.hasDuplicateCharge) {
    return { ok: false, reason: "duplicate_charge_not_verified" };
  }

  return { ok: true };
}

I now treat model output as untrusted input, even when the model was instructed to return a strict format.

The model proposes data.

The application decides whether that data is valid and whether it is allowed to cause an action.

The failure categories worth tracking

Instead of one generic parse-error counter, I track separate categories:

empty response
truncated response
Markdown-wrapped JSON
invalid JSON
missing required field
unsupported enum
unexpected additional field
business-rule rejection
human approval required

This makes provider or model changes easier to evaluate.

If malformed JSON increases after a model switch, that is a format-reliability issue. If business-rule rejections increase, the JSON may be perfect while the decisions are getting worse.

Those problems need different fixes.

What changed in my production code

The biggest change was not a better prompt.

It was moving trust out of the prompt.

I still ask for a precise response format, but I assume the contract can be broken. Every response must pass parsing, schema validation, and business validation before the application uses it.

That makes the failure less mysterious:

The model returned 200.
The text looked fine.
The parser rejected it.
No side effect happened.
The failure was recorded at the correct stage.

That is a much better outcome than allowing a plausible-looking response to quietly become production state.

I work on TokenBay, where I spend a lot of time thinking about model and provider behavior. But this validation layer is provider-agnostic: if an LLM response is going to influence real application logic, I want code—not confidence—to decide whether it is safe.

A Tiny LLM Request Recorder I Use to Reproduce Production Failures

plasma — Fri, 17 Jul 2026 03:21:00 +0000

Most LLM failures are easy to describe and surprisingly hard to reproduce.

A user reports that the model returned an empty answer. A tool call disappeared halfway through a stream. One provider rejected a request that worked everywhere else.

Then I open the logs and find something like this:

LLM request failed: 400 Bad Request

Technically true. Operationally useless.

The missing piece is usually the exact request shape: model, parameters, message roles, tool definitions, timeout behavior, and the raw provider response.

I wanted something smaller than a full observability platform, so I built a request recorder around fetch. It stores enough information to inspect or replay a failed call without logging the API key.

What the recorder captures

For each request, I want:

a unique request ID
timestamp and duration
URL and model
sanitized request body
HTTP status
raw response body
network or timeout errors

I deliberately do not record the Authorization header.

Prompt content is also redacted by default. Full payload capture must be enabled explicitly because storing production prompts can create a much worse problem than the bug being investigated.

The recorder

This example runs on Node.js 18 or newer and has no external dependencies.

Create recorded-fetch.mjs:

import { randomUUID } from "node:crypto";
import { mkdir, writeFile } from "node:fs/promises";
import path from "node:path";

function sanitize(value, captureContent) {
  if (Array.isArray(value)) {
    return value.map((item) => sanitize(item, captureContent));
  }

  if (!value || typeof value !== "object") {
    return value;
  }

  const result = {};

  for (const [key, child] of Object.entries(value)) {
    const normalizedKey = key.toLowerCase();

    if (
      normalizedKey.includes("api_key") ||
      normalizedKey.includes("apikey") ||
      normalizedKey.includes("authorization")
    ) {
      result[key] = "[REDACTED]";
      continue;
    }

    if (
      !captureContent &&
      (normalizedKey === "content" ||
        normalizedKey === "prompt" ||
        normalizedKey === "input")
    ) {
      result[key] = "[REDACTED]";
      continue;
    }

    result[key] = sanitize(child, captureContent);
  }

  return result;
}

async function saveRecording(directory, recording) {
  await mkdir(directory, { recursive: true });

  const filename = `${recording.startedAt.replaceAll(":", "-")}-${recording.id}.json`;
  const destination = path.join(directory, filename);

  await writeFile(
    destination,
    JSON.stringify(recording, null, 2),
    "utf8"
  );

  return destination;
}

export function createRecordedFetch({
  directory = ".llm-recordings",
  captureContent = false,
} = {}) {
  return async function recordedFetch(url, options = {}) {
    const id = randomUUID();
    const startedAt = new Date().toISOString();
    const start = performance.now();

    let requestBody;

    try {
      requestBody =
        typeof options.body === "string"
          ? JSON.parse(options.body)
          : options.body;
    } catch {
      requestBody = "[UNPARSEABLE BODY]";
    }

    const recording = {
      id,
      startedAt,
      request: {
        url: String(url),
        method: options.method ?? "GET",
        body: sanitize(requestBody, captureContent),
      },
    };

    try {
      const response = await fetch(url, options);
      const rawResponse = await response.text();

      recording.durationMs = Math.round(performance.now() - start);
      recording.response = {
        status: response.status,
        statusText: response.statusText,
        headers: Object.fromEntries(response.headers.entries()),
        body: rawResponse,
      };

      const savedTo = await saveRecording(directory, recording);

      if (!response.ok) {
        throw new Error(
          `LLM request ${id} failed with ${response.status}. Recording: ${savedTo}`
        );
      }

      return {
        requestId: id,
        response,
        rawResponse,
        savedTo,
      };
    } catch (error) {
      if (!recording.response) {
        recording.durationMs = Math.round(performance.now() - start);
        recording.error = {
          name: error.name,
          message: error.message,
        };

        await saveRecording(directory, recording);
      }

      throw error;
    }
  };
}

This wrapper reads the response body once and returns it as rawResponse. That is important because a fetch response body cannot normally be consumed twice.

Using it for an LLM request

Create example.mjs:

import { createRecordedFetch } from "./recorded-fetch.mjs";

const recordedFetch = createRecordedFetch({
  directory: ".llm-recordings",
  captureContent: false,
});

const baseUrl =
  process.env.LLM_BASE_URL ?? "https://api.openai.com/v1";

const apiKey = process.env.LLM_API_KEY;

if (!apiKey) {
  throw new Error("LLM_API_KEY is required");
}

const payload = {
  model: process.env.LLM_MODEL ?? "gpt-4.1-mini",
  messages: [
    {
      role: "user",
      content: "Explain idempotency in one paragraph.",
    },
  ],
  temperature: 0.2,
};

const result = await recordedFetch(
  `${baseUrl}/chat/completions`,
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(payload),
    signal: AbortSignal.timeout(30_000),
  }
);

const data = JSON.parse(result.rawResponse);

console.log({
  requestId: result.requestId,
  recording: result.savedTo,
  answer: data.choices?.[0]?.message?.content,
});

Run it with:

LLM_API_KEY="your-api-key" node example.mjs

Every call now produces a JSON file under .llm-recordings.

A failed request might look like this:

{
  "id": "9e50d5a9-4a0d-42a2-94c9-711d36b2057d",
  "startedAt": "2026-07-17T08:14:32.442Z",
  "request": {
    "url": "https://api.example.com/v1/chat/completions",
    "method": "POST",
    "body": {
      "model": "example-model",
      "messages": [
        {
          "role": "user",
          "content": "[REDACTED]"
        }
      ],
      "temperature": 0.2
    }
  },
  "durationMs": 418,
  "response": {
    "status": 400,
    "statusText": "Bad Request",
    "headers": {
      "content-type": "application/json"
    },
    "body": "{\"error\":{\"message\":\"Unsupported parameter: temperature\"}}"
  }
}

That is already more useful than a generic 400.

It tells me the failure was probably caused by a provider compatibility difference, not the prompt, network, or model output.

Replaying a request

For local or staging environments, I can temporarily set:

const recordedFetch = createRecordedFetch({
  captureContent: true,
});

The resulting request body can then be replayed with a small script.

Create replay.mjs:

import { readFile } from "node:fs/promises";

const recordingPath = process.argv[2];

if (!recordingPath) {
  throw new Error("Usage: node replay.mjs <recording.json>");
}

const recording = JSON.parse(
  await readFile(recordingPath, "utf8")
);

const serializedBody = JSON.stringify(recording.request.body);

if (serializedBody.includes("[REDACTED]")) {
  throw new Error(
    "This recording contains redacted values and cannot be replayed."
  );
}

const apiKey = process.env.LLM_API_KEY;

if (!apiKey) {
  throw new Error("LLM_API_KEY is required");
}

const response = await fetch(recording.request.url, {
  method: recording.request.method,
  headers: {
    "Authorization": `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: serializedBody,
  signal: AbortSignal.timeout(30_000),
});

console.log("Status:", response.status);
console.log(await response.text());

Then run:

LLM_API_KEY="your-api-key" \
node replay.mjs .llm-recordings/example.json

I only enable full-content recording with synthetic or approved test data. I would not turn it on globally in production.

What this has helped me distinguish

The recorder is most useful when several failures look identical from the outside.

It helps separate:

Request-shape failures

Unsupported parameters, invalid tool schemas, wrong message roles, or model-specific limits.

Transport failures

Timeouts, connection resets, interrupted streams, and responses that never reach the application.

Provider failures

A structured error response, unexpected content type, or different interpretation of an OpenAI-compatible field.

Application failures

The provider returned a valid response, but my parser, state machine, or tool executor mishandled it.

Those categories lead to very different fixes. Retrying all four of them is not a debugging strategy.

Production considerations

This recorder is intentionally small, so it leaves several decisions to the application using it.

Before running something like this in production, I would add:

a retention period for recordings
sampling instead of recording every successful request
encryption at rest
size limits for request and response bodies
stricter redaction for user and tool data
access controls around the recording directory
correlation with the application trace or operation ID

I would also record failed calls more aggressively than successful ones. Keeping every response forever is an expensive way to create a privacy problem.

Why I keep this wrapper small

There are good observability products that can collect richer traces, token usage, latency distributions, and model-level metrics.

I still like having a tiny recorder close to the HTTP boundary.

It gives me a provider-neutral artifact I can inspect before data passes through SDK abstractions, response parsers, retry policies, or agent frameworks.

When I test multiple OpenAI-compatible endpoints—including my work on TokenBay—that raw boundary is often where compatibility problems become easiest to see.

The goal is not to log everything.

It is to make the next production failure reproducible enough that I do not have to debug it from a single error message.

My LLM Provider Returned 200. The Workflow Still Failed.

plasma — Wed, 15 Jul 2026 06:57:55 +0000

The request looked completely normal.

The server returned 200 OK. The response body was valid JSON. The latency was acceptable. Nothing appeared in the error logs.

So I marked the provider integration as working.

A few days later, the workflow failed in production.

The API request had succeeded, but the task had not.

That incident changed how I test LLM providers. I no longer treat a successful HTTP request as proof that an LLM operation succeeded.

The response that passed my first test

My original health check was embarrassingly simple:

const response = await fetch(`${baseUrl}/chat/completions`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model,
    messages: [
      {
        role: "user",
        content: "Reply with the word READY.",
      },
    ],
  }),
});

if (!response.ok) {
  throw new Error(`Provider returned ${response.status}`);
}

console.log("Provider is working");

It verified a few useful things:

the endpoint was reachable
the API key was accepted
the model name was recognized
the server returned a successful HTTP status

But it did not verify the thing I actually cared about:

Can this provider complete the operation my application depends on?

The answer, in my case, was no.

HTTP success and task success are different states

An LLM request can return 200 OK while still being unusable to the application.

I have learned to think about success in layers:

Transport success — Did the request reach the server?
Protocol success — Did the server return the expected response format?
Model success — Did the model produce a complete and valid result?
Workflow success — Can the application safely continue?
User success — Did the action the user requested actually happen?

A status code mostly tells me about the first two layers.

My product usually depends on the last three.

For example, all of these responses might arrive with a 200:

an empty choices array
a missing message.content
truncated JSON
a tool call with invalid arguments
a stream that ends without a terminal event
a refusal represented differently from what the application expects
a response that is syntactically valid but violates the required schema

If my code checks only response.ok, every one of those cases can be mislabeled as success.

Validate the response you actually need

The first improvement was to validate the full response shape.

function validateChatCompletion(data) {
  if (!data || typeof data !== "object") {
    throw new Error("Response is not an object");
  }

  if (!Array.isArray(data.choices) || data.choices.length === 0) {
    throw new Error("Response contains no choices");
  }

  const choice = data.choices[0];

  if (!choice.message || typeof choice.message !== "object") {
    throw new Error("Response contains no assistant message");
  }

  if (
    typeof choice.message.content !== "string" ||
    choice.message.content.trim() === ""
  ) {
    throw new Error("Assistant message is empty");
  }

  if (!choice.finish_reason) {
    throw new Error("Response does not include a finish reason");
  }

  return choice.message.content;
}

Then the request check became:

async function runProviderCheck({ baseUrl, apiKey, model }) {
  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model,
      messages: [
        {
          role: "user",
          content: "Reply with exactly: READY",
        },
      ],
      temperature: 0,
    }),
  });

  if (!response.ok) {
    throw new Error(`HTTP failure: ${response.status}`);
  }

  const data = await response.json();
  const content = validateChatCompletion(data);

  if (content.trim() !== "READY") {
    throw new Error(`Unexpected model output: ${content}`);
  }

  return {
    status: "passed",
    model,
    finishReason: data.choices[0].finish_reason,
    usage: data.usage ?? null,
  };
}

This is still a small test, but it checks more than endpoint availability. It checks whether the provider can satisfy a precise application-level contract.

Tool calls need a different definition of success

Text validation is not enough when an agent can call tools.

Suppose the model is expected to create this tool call:

{
  "name": "create_support_ticket",
  "arguments": {
    "customer_id": "cus_123",
    "priority": "high",
    "summary": "Customer cannot access the dashboard"
  }
}

A provider could return a tool call with:

malformed JSON arguments
the wrong tool name
required fields missing
strings where the schema expects numbers
duplicated tool calls
a different tool-call structure than the SDK expects

The HTTP request still succeeded. Executing the tool call blindly would be the actual failure.

I now validate tool calls before allowing them anywhere near a real side effect:

function validateSupportTicketToolCall(toolCall) {
  if (toolCall?.function?.name !== "create_support_ticket") {
    throw new Error("Unexpected tool name");
  }

  let args;

  try {
    args = JSON.parse(toolCall.function.arguments);
  } catch {
    throw new Error("Tool arguments are not valid JSON");
  }

  if (typeof args.customer_id !== "string") {
    throw new Error("Missing or invalid customer_id");
  }

  if (!["low", "medium", "high"].includes(args.priority)) {
    throw new Error("Invalid priority");
  }

  if (
    typeof args.summary !== "string" ||
    args.summary.trim().length === 0
  ) {
    throw new Error("Missing ticket summary");
  }

  return args;
}

The model proposes an action. The application decides whether that action is valid and safe to execute.

Those should never be the same step.

Streaming has its own false-success cases

Streaming responses made this problem less obvious.

A stream can begin successfully, deliver several tokens, and then stop before the completion is usable. If the UI has already displayed partial output, both the user and the application may believe the task finished.

For streaming requests, I check for more than the first chunk:

Was at least one content or tool-call delta received?
Did the stream reach the expected terminal state?
Was the final structured output complete?
Were tool-call arguments fully assembled?
Did the connection close normally?
If the stream failed, was the partial state discarded or clearly marked?

The exact terminal signal depends on the API and SDK. The important part is not to interpret “some tokens arrived” as “the operation completed.”

A partial answer can be useful in a chat interface. A partial tool call can be dangerous in a production workflow.

My provider smoke test is now a small behavior suite

I used to test a provider with one prompt.

Now I run several small checks that represent the behaviors my application relies on:

a deterministic text response
structured JSON matching a schema
a valid tool call
a streaming completion
timeout behavior
rate-limit behavior
malformed-request errors
usage metadata, if billing or monitoring depends on it

I do not test every possible model behavior. I test the contracts that could break my application.

A simplified test runner looks like this:

const checks = [
  ["text completion", checkTextCompletion],
  ["structured output", checkStructuredOutput],
  ["tool calling", checkToolCalling],
  ["stream termination", checkStreaming],
];

async function runSmokeTests(provider) {
  const results = [];

  for (const [name, check] of checks) {
    const startedAt = Date.now();

    try {
      await check(provider);

      results.push({
        name,
        passed: true,
        durationMs: Date.now() - startedAt,
      });
    } catch (error) {
      results.push({
        name,
        passed: false,
        durationMs: Date.now() - startedAt,
        error: error instanceof Error ? error.message : String(error),
      });
    }
  }

  return results;
}

I run this before enabling a new provider or model in a production route. I also rerun the relevant checks after SDK changes or response-format changes.

Do not let “success” hide the evidence

Another mistake I made was logging only failed HTTP requests.

That meant the misleading 200 left almost no useful evidence.

For every important LLM operation, I now try to record:

provider and model
operation ID
HTTP status
latency
finish reason
response validation result
tool name, without sensitive arguments
token usage when available
retry count
final workflow outcome

I avoid logging raw prompts or sensitive model output by default. The goal is to preserve enough metadata to distinguish:

request failed
response was invalid
model output was rejected
tool execution failed
workflow completed

Without that separation, several very different incidents become the same vague log message: “LLM call failed.”

The status code is evidence, not the verdict

I still check HTTP status codes. They are useful.

I just no longer let them make the final decision.

For an LLM application, success is a contract defined by the workflow:

Was the response complete?
Did it match the expected shape?
Was the requested tool call valid?
Is the proposed action safe?
Did the workflow reach its intended state?

A clean 200 answers none of those questions on its own.

This matters even more when an application can switch between multiple models or providers. OpenAI-compatible endpoints can reduce integration work, but compatibility at the request level does not guarantee identical runtime behavior.

I work on TokenBay, so provider routing is part of what I think about day to day. The lesson I keep coming back to is simple:

Test provider behavior, not just provider availability.

The request can succeed while the workflow fails. Your success check needs to know the difference.

My Agent Said “Done” After One Tool Call Failed Silently

plasma — Mon, 13 Jul 2026 06:10:11 +0000

The agent had one job: update a customer record, issue a refund, and send a confirmation email.

It called all three tools.

Then it replied:

Done — the customer record was updated, the refund was issued, and the confirmation email was sent.

There was just one problem: the refund tool had failed.

The HTTP request timed out, the tool returned an error object, and the agent continued as if nothing had happened. From the user's perspective, the workflow succeeded. From the payment system's perspective, no refund existed.

That was the moment I stopped treating a tool call as proof that an action happened.

A tool call is only an attempt

A lot of agent workflows follow roughly this pattern:

The model decides which tool to call.
The application executes the tool.
The tool returns something.
The result goes back into the conversation.
The model writes a final answer.

The dangerous assumption is between steps three and four:

If the tool returned, the operation must have succeeded.

That assumption breaks surprisingly easily.

A tool can return:

an HTTP 500 response;
a timeout;
malformed JSON;
{ success: false };
a partial result;
a valid-looking response for an operation that was never committed;
or an error that gets converted into an ordinary string.

If the application passes that result back without preserving its failure state, the model may interpret “the tool ran” as “the task completed.”

The model is not necessarily ignoring the error. Sometimes the surrounding code has already erased the distinction.

The version that failed silently

Here is a simplified version of the pattern that caused the problem:

async function updateCustomer() {
  return { customerId: "cus_123", updated: true };
}

async function issueRefund() {
  throw new Error("Payment provider timed out");
}

async function sendEmail() {
  return { messageId: "msg_456", sent: true };
}

async function runTool(name, fn) {
  try {
    const result = await fn();

    return {
      tool: name,
      result,
    };
  } catch (error) {
    return {
      tool: name,
      result: error.message,
    };
  }
}

const results = [];

results.push(await runTool("update_customer", updateCustomer));
results.push(await runTool("issue_refund", issueRefund));
results.push(await runTool("send_email", sendEmail));

console.log(results);

Run it with:

node agent.js

The output looks reasonable at first glance:

[
  {
    "tool": "update_customer",
    "result": {
      "customerId": "cus_123",
      "updated": true
    }
  },
  {
    "tool": "issue_refund",
    "result": "Payment provider timed out"
  },
  {
    "tool": "send_email",
    "result": {
      "messageId": "msg_456",
      "sent": true
    }
  }
]

But the failed tool has the same outer shape as the successful tools. They all produced a result.

If this array is turned into a prompt such as:

The tools have finished. Summarize the completed actions for the user.

the workflow is already encouraging the wrong conclusion.

The error became content instead of state.

I now make failure impossible to miss

The first fix is not a smarter prompt. It is a stricter result contract.

Every tool execution should return an explicit status that the orchestration layer can evaluate without asking the model to interpret arbitrary text.

async function runTool(name, fn) {
  const startedAt = Date.now();

  try {
    const data = await fn();

    return {
      tool: name,
      status: "succeeded",
      data,
      error: null,
      durationMs: Date.now() - startedAt,
    };
  } catch (error) {
    return {
      tool: name,
      status: "failed",
      data: null,
      error: {
        message: error.message,
        retryable: error.message.includes("timed out"),
      },
      durationMs: Date.now() - startedAt,
    };
  }
}

Now success and failure cannot accidentally share the same shape.

More importantly, the application can make a deterministic decision before the model writes anything:

function summarizeExecution(results) {
  const succeeded = results.filter(
    (result) => result.status === "succeeded"
  );

  const failed = results.filter(
    (result) => result.status === "failed"
  );

  return {
    status:
      failed.length === 0
        ? "completed"
        : succeeded.length === 0
          ? "failed"
          : "partially_completed",
    succeeded,
    failed,
  };
}

Using the same three tools:

async function main() {
  const results = [];

  results.push(
    await runTool("update_customer", updateCustomer)
  );

  results.push(
    await runTool("issue_refund", issueRefund)
  );

  results.push(
    await runTool("send_email", sendEmail)
  );

  console.dir(summarizeExecution(results), {
    depth: null,
  });
}

main().catch(console.error);

The workflow now has three possible outcomes:

completed
partially_completed
failed

“Partially completed” matters. Collapsing every workflow into success or failure hides the most uncomfortable state: some real-world actions happened and others did not.

The application should decide whether "done" is allowed

I used to let the model infer the final workflow status from tool messages.

I do not do that anymore.

The orchestration layer determines whether the completion claim is permitted:

function buildFinalResponse(execution) {
  if (execution.status === "completed") {
    return {
      completionAllowed: true,
      message:
        "Done. All requested actions completed successfully.",
    };
  }

  if (execution.status === "partially_completed") {
    const completedTools = execution.succeeded
      .map((item) => item.tool)
      .join(", ");

    const failedTools = execution.failed
      .map((item) => item.tool)
      .join(", ");

    return {
      completionAllowed: false,
      message:
        `I couldn't complete the full request. ` +
        `Completed: ${completedTools}. ` +
        `Failed: ${failedTools}.`,
    };
  }

  return {
    completionAllowed: false,
    message:
      "I couldn't complete the requested actions.",
  };
}

The model can still make the response clearer and more conversational, but it cannot upgrade partially_completed to completed.

That boundary belongs in code.

Tool planning and tool execution are different states

Another mistake I made was treating a planned action as if it were already underway.

An agent might produce a plan like this:

[
  {
    "tool": "update_customer",
    "status": "planned"
  },
  {
    "tool": "issue_refund",
    "status": "planned"
  },
  {
    "tool": "send_email",
    "status": "planned"
  }
]

Those entries should move through explicit states:

planned
  -> executing
  -> succeeded

planned
  -> executing
  -> failed

For operations with side effects, I also track an operation ID:

const operation = {
  operationId: crypto.randomUUID(),
  tool: "issue_refund",
  status: "planned",
  attempts: 0,
  externalReference: null,
};

This helps answer questions that become critical after a timeout:

Did the request reach the provider?
Did the provider complete it before the connection failed?
Is retrying safe?
Do we have an external transaction ID?
Could a retry issue the refund twice?

A timeout does not always mean “nothing happened.” Sometimes it means “the client does not know what happened.”

That is a different failure state, and it should not be retried blindly.

Do not continue after every failure

In my original workflow, the confirmation email still ran after the refund failed.

That created an even worse outcome: the customer received an email saying the refund had been processed when it had not.

Tool dependencies need to be represented in the workflow, not left for the model to guess.

async function runRefundWorkflow() {
  const update = await runTool(
    "update_customer",
    updateCustomer
  );

  if (update.status === "failed") {
    return summarizeExecution([update]);
  }

  const refund = await runTool(
    "issue_refund",
    issueRefund
  );

  if (refund.status === "failed") {
    return summarizeExecution([update, refund]);
  }

  const email = await runTool(
    "send_email",
    sendEmail
  );

  return summarizeExecution([
    update,
    refund,
    email,
  ]);
}

The confirmation email is now conditional on a successful refund.

This sounds obvious when written down. It becomes less obvious in an agent loop where the model dynamically chooses the next action and every tool result is appended to the same conversation.

What I log for every real-world action

For read-only tools, lightweight logging may be enough.

For tools that send emails, move money, change permissions, delete files, or update customer data, I want a much stronger audit trail:

{
  operationId: "op_8f23...",
  workflowId: "wf_19ab...",
  tool: "issue_refund",
  status: "failed",
  attempt: 1,
  startedAt: "2026-07-13T08:20:10.000Z",
  finishedAt: "2026-07-13T08:20:12.413Z",
  durationMs: 2413,
  retryable: true,
  idempotencyKey: "refund_order_789",
  externalReference: null,
  errorCode: "PROVIDER_TIMEOUT"
}

The exact schema is less important than being able to reconstruct what happened without relying on the agent's final sentence.

The final sentence is presentation.

The execution log is evidence.

Prompts still help, but they are not enforcement

I still tell the model to be explicit about partial failure:

Never claim that a task is complete unless every required
tool operation has status "succeeded".

If one or more required operations failed:

1. State that the request was not fully completed.
2. List which actions succeeded.
3. List which actions failed.
4. Do not imply that a failed external action occurred.
5. Do not retry actions with side effects unless the
   orchestration layer explicitly permits it.

This improves the wording and reduces ambiguous responses.

But a prompt is not where I enforce the rule.

The application should:

preserve structured tool status;
validate required results;
stop dependent actions after failure;
prevent unsafe retries;
and block completion claims when the workflow is incomplete.

The model should explain the state, not decide whether the state exists.

The test I wish I had written earlier

I now include a test that deliberately fails one required tool:

import assert from "node:assert/strict";

const execution = summarizeExecution([
  {
    tool: "update_customer",
    status: "succeeded",
  },
  {
    tool: "issue_refund",
    status: "failed",
  },
  {
    tool: "send_email",
    status: "succeeded",
  },
]);

const response = buildFinalResponse(execution);

assert.equal(
  execution.status,
  "partially_completed"
);

assert.equal(
  response.completionAllowed,
  false
);

assert.doesNotMatch(
  response.message,
  /^Done\b/
);

It is a small test, but it protects an important product promise:

The agent cannot say “done” unless the system has evidence that the required work is done.

"Done" is a production state

The hardest agent failures are not always crashes.

Sometimes the workflow keeps moving, produces a polished answer, and gives the user no reason to suspect that anything went wrong.

That is more dangerous than an obvious exception.

An exception asks for attention. A false success closes the conversation.

The rule I use now is simple:

A requested action is not complete because the model asked for a tool, or because the tool returned something. It is complete only when the application has validated the result.

I work on TokenBay, so I spend a lot of time thinking about what happens between model output, provider behavior, and real application state. The more actions we give agents permission to take, the less comfortable I am letting a fluent final answer serve as proof of execution.

My LLM Bill Kept Growing, but User Traffic Didn’t

plasma — Fri, 10 Jul 2026 08:47:23 +0000

My request count looked normal.

Traffic was mostly flat. There was no sudden wave of new users, no runaway background job, and no obvious model upgrade hiding in a deployment.

But the LLM bill kept climbing.

My first instinct was to look for the usual suspects: a pricing change, an expensive model accidentally pushed to production, or someone hammering an endpoint.

None of those explanations held up.

The problem was simpler: I was measuring requests, while the provider was charging me for tokens.

Those two graphs had slowly stopped telling the same story.

A request is not a useful cost unit

I had dashboards for request volume, latency, error rate, and model name. That felt like enough observability until cost started behaving differently from traffic.

A single LLM request can contain:

the system prompt
the current user message
conversation history
retrieved documents
tool definitions
JSON schemas
examples included in the prompt
output tokens
provider-specific reasoning tokens
another attempt after a retry

Two requests to the same endpoint can therefore have completely different costs.

One might classify a short support message. Another might resend a long conversation, twelve tool definitions, and several retrieved documents before producing a two-sentence answer.

My dashboard counted both as one request.

The invoice did not.

The first number I needed was tokens per operation

Logging total tokens was better than logging nothing, but it still left me with a pile of numbers and no explanation.

I needed to connect usage to the work my application was trying to perform.

For each model call, I started recording:

operation_id: the user-visible action
workflow: chat, summarization, extraction, agent step, and so on
model
attempt
input_tokens
output_tokens
cached_input_tokens, when reported
reasoning_tokens, when reported
latency_ms
status
whether the call was a retry or fallback

The distinction between a request and an operation matters.

If a user clicks “Generate” once and the application makes three model calls, I want to see one operation with three attempts or steps—not three unrelated requests.

That was the first change that made the cost graph useful.

A small usage logger

Here is a simplified Node.js example using an OpenAI-compatible Chat Completions endpoint.

The detailed usage fields are optional because providers and models do not all return the same response shape.

import crypto from "node:crypto";

const baseUrl = process.env.LLM_BASE_URL ?? "https://api.openai.com/v1";
const apiKey = process.env.LLM_API_KEY;
const model = process.env.LLM_MODEL;

if (!apiKey || !model) {
  throw new Error("Set LLM_API_KEY and LLM_MODEL");
}

function readUsage(usage = {}) {
  return {
    inputTokens: usage.prompt_tokens ?? 0,
    outputTokens: usage.completion_tokens ?? 0,
    totalTokens: usage.total_tokens ?? 0,
    cachedInputTokens:
      usage.prompt_tokens_details?.cached_tokens ?? 0,
    reasoningTokens:
      usage.completion_tokens_details?.reasoning_tokens ?? 0,
  };
}

async function callModel({
  messages,
  workflow,
  operationId = crypto.randomUUID(),
  attempt = 1,
}) {
  const startedAt = Date.now();

  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify({
      model,
      messages,
    }),
  });

  const body = await response.json();
  const usage = readUsage(body.usage);

  const event = {
    timestamp: new Date().toISOString(),
    operationId,
    workflow,
    model,
    attempt,
    status: response.ok ? "success" : "error",
    latencyMs: Date.now() - startedAt,
    ...usage,
  };

  console.log(JSON.stringify(event));

  if (!response.ok) {
    throw new Error(
      body.error?.message ?? `LLM request failed: ${response.status}`
    );
  }

  return {
    operationId,
    text: body.choices?.[0]?.message?.content ?? "",
    usage,
  };
}

const result = await callModel({
  workflow: "support_reply",
  messages: [
    {
      role: "system",
      content: "Write concise and accurate support replies.",
    },
    {
      role: "user",
      content: "How do I update my billing email?",
    },
  ],
});

console.log(result.text);

In production, I send this event to the same logging or tracing system that holds the rest of the workflow.

The important part is not the destination. It is preserving the relationship between usage and the operation that caused it.

Where the extra tokens were hiding

Once I grouped usage by workflow and operation, several patterns became visible.

Conversation history kept growing

The chat endpoint received the full conversation history on every turn.

A conversation that began cheaply became progressively more expensive even though the user was still sending short messages.

Request volume stayed flat. Input tokens per request did not.

The fix was not simply “truncate everything.” Older context sometimes mattered.

Instead, I separated context into three groups:

recent messages that should be preserved
durable facts that could be stored separately
older conversation that could be summarized or dropped

The important measurement was input tokens by conversation age. Without that breakdown, long-running chats looked like ordinary traffic.

Tool definitions were being sent repeatedly

My agent had access to several tools, each with a description and JSON schema.

Those definitions were part of the model input. I had been thinking about them as application configuration, not recurring context.

Some workflows exposed tools they could never use. Others included verbose parameter descriptions copied from internal documentation.

Reducing the tool set per workflow made the agent easier to reason about and reduced repeated input.

It also gave me a better question than “How do I shorten this prompt?”

The better question was:

Why am I sending this piece of context on this specific call?

Retries looked like new work

A timeout triggered a retry. A parsing failure triggered another retry. A fallback model sometimes ran after both.

At the infrastructure level, those were separate requests.

At the product level, the user had asked for one thing.

Without an operation ID and attempt number, I could not distinguish growing usage from growing demand.

I now track both:

requests per operation
tokens per operation

If requests per operation rises, I look at retries, fallbacks, agent loops, and validation failures before blaming user traffic.

I also avoid assuming that every failed request has zero cost. Whether usage is recorded depends on the provider and how far the request progressed, so application logs should be reconciled with provider-side usage data.

Outputs were longer than the product needed

Some workflows asked for structured data containing five fields but allowed the model to generate a long explanation around it.

Another workflow produced detailed internal reasoning-style text that the UI never displayed.

The model was doing more work than the product used.

Output limits helped, but clearer response contracts helped more. If the UI needs a category, confidence score, and short explanation, the prompt and schema should request exactly that.

“Be concise” is weaker than defining the output the application can actually consume.

The dashboard I use now

I no longer begin cost investigations with total spend.

I start with four ratios:

input tokens / operation
output tokens / operation
model calls / operation
cached input tokens / input tokens

Then I break them down by:

workflow
model
release version
customer or plan, using an internal non-sensitive identifier
success, retry, and fallback status

This makes different problems look different.

If input tokens per operation rise, context may be growing.

If calls per operation rise, retries or loops may be involved.

If output tokens rise, the response contract may have drifted.

If the cache ratio changes, a supposedly stable prompt prefix may no longer be stable. Prompt caching behavior and usage reporting vary by provider and model, so I treat the returned usage fields and provider documentation as the source of truth.

I stopped estimating cost from averages

A global “average cost per request” hid too much.

A cheap classification call and a long agent workflow should not share the same baseline. Neither should successful operations and operations that needed three attempts.

I now compare each workflow against its own recent behavior.

The alert is not:

LLM spend increased by 20%

It is closer to:

input tokens per successful support_reply operation
increased materially after the latest release

That points toward something I can investigate.

A total-spend alert tells me there is a fire. A workflow-level usage alert tells me which room might be burning.

Cost optimization came later

Before this work, my optimization ideas were mostly guesses:

use a cheaper model
shorten the system prompt
reduce output length
add caching
retrieve fewer documents

All of those can be reasonable. None of them tells you where your application is wasting tokens.

Once usage was tied to operations, the priorities changed.

The biggest opportunity was not necessarily switching models. Sometimes it was stopping an unnecessary second call. Sometimes it was removing irrelevant tools. Sometimes it was preventing the same context from growing forever.

The useful sequence became:

Measure usage per operation.
Find the workflow whose token intensity changed.
Separate input, output, retries, fallbacks, and caching.
Change one behavior.
Confirm that quality and reliability did not regress.

Only then did model pricing become a meaningful optimization variable.

The lesson

User traffic and LLM usage are related, but they are not the same metric.

Traffic tells me how many people are using the product.

Token telemetry tells me how much work the application is asking the models to do on their behalf.

When those numbers diverge, I no longer start by blaming the model or the provider. I look at context growth, tool payloads, retries, fallbacks, and workflow design.

The bill was not the first signal.

It was just the first signal I had bothered to watch.

I work on TokenBay, so multi-model usage and routing are problems I spend a lot of time thinking about. The most useful improvement, though, was provider-independent: attach every token to a workflow and an operation before trying to optimize it.

I Stopped Swapping LLM Providers Without a Smoke Test

plasma — Wed, 08 Jul 2026 10:25:06 +0000

For a while, my LLM provider migration test was embarrassingly simple:

Change the base URL
Send one normal prompt
Get a response
Call it done

Which is fine if your app only fails on days when it feels polite.

The problem is that a happy-path request does not tell you much. It tells you the provider can answer one boring prompt under normal conditions. It does not tell you what happens when the request times out, the stream cuts off, the rate limit response has a weird shape, or your app is about to retry something that may have already triggered a tool call.

So now, before I trust a new LLM provider in anything production-ish, I run a small smoke test first.

Not a full benchmark. Not a provider comparison. Just a quick "will this break my app in boring, predictable ways?" test.

The old test was basically useless

The old version looked something like this:

const response = await client.chat.completions.create({
  model: "some-model",
  messages: [
    { role: "user", content: "Say hello in one sentence." }
  ]
});

console.log(response.choices[0].message.content);

If that worked, I felt good.

That was the trap.

A normal request does not test the parts that usually hurt in production:

timeout behavior
retry safety
streaming completion
rate limit shape
token limit behavior
usage metadata
provider request IDs
tool call handling
error classification
whether the response envelope is actually complete

The provider can pass the hello-world test and still be a bad fit for your app.

What I actually want to know

Before swapping providers, I want answers to a few boring questions.

Boring is good here. Boring saves weekends.

I want to know:

Can I classify failures without parsing random strings?
Can I tell the difference between a retryable failure and a permanent one?
Does streaming end with enough metadata for me to trust the turn is complete?
Do timeouts cancel cleanly?
Does the provider return useful request IDs?
Does usage data exist where my billing and logging expect it?
If the call could trigger a side effect, will my retry policy avoid replaying it blindly?

That is the point of the smoke test.

It is not trying to prove that one provider is "better." It is trying to prove that my app can understand what happened.

My 10-minute smoke test

I usually run five small checks.

They are simple on purpose.

1. A boring normal request

First, I still run the basic request.

But now I log more than the text.

const started = Date.now();

const response = await client.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    { role: "user", content: "Reply with exactly one short sentence." }
  ]
});

console.log({
  latency_ms: Date.now() - started,
  content: response.choices?.[0]?.message?.content,
  finish_reason: response.choices?.[0]?.finish_reason,
  usage: response.usage,
  id: response.id
});

I am not looking for a beautiful answer.

I am looking for the envelope:

Did I get a response ID?
Did I get a finish reason?
Did I get usage?
Is the response shape what my app expects?
Is latency roughly sane?

If the normal path is already weird, I stop there.

2. A request that should timeout

Next, I force an aggressive timeout.

Not because I expect to run with a 200ms timeout in production, but because I want to see what failure looks like.

const controller = new AbortController();

const timeout = setTimeout(() => {
  controller.abort();
}, 200);

try {
  await client.chat.completions.create(
    {
      model: process.env.MODEL,
      messages: [
        {
          role: "user",
          content: "Write a detailed 1000-word explanation of database indexes."
        }
      ]
    },
    { signal: controller.signal }
  );
} catch (error) {
  console.log({
    name: error.name,
    message: error.message,
    code: error.code,
    status: error.status
  });
} finally {
  clearTimeout(timeout);
}

What I want to know:

Does the request actually cancel?
Does the SDK throw something I can classify?
Is the error clearly a timeout, or just a generic network failure?
Would my retry policy treat this correctly?

The retry decision depends on where the timeout happened.

A timeout before any response is usually retryable.

A timeout after partial output is much messier.

That difference matters a lot if the model is being used inside a workflow or agent.

3. A streaming request

Streaming is where a lot of "looks fine locally" bugs hide.

I want to know whether the stream gives me a clear ending.

const stream = await client.chat.completions.create({
  model: process.env.MODEL,
  stream: true,
  messages: [
    { role: "user", content: "Count from 1 to 5, one number per line." }
  ]
});

let text = "";
let finalChunkSeen = false;
let finishReason = null;

for await (const chunk of stream) {
  const delta = chunk.choices?.[0]?.delta?.content;
  if (delta) text += delta;

  const reason = chunk.choices?.[0]?.finish_reason;
  if (reason) {
    finishReason = reason;
    finalChunkSeen = true;
  }
}

console.log({
  text,
  finalChunkSeen,
  finishReason
});

For a normal app, partial text may be fine.

For a workflow, agent, or tool-calling system, partial text can be dangerous. The user may have already seen output. The app may have already started acting on it. The system may think the turn completed when it did not.

So I do not only ask, "Did I get tokens?"

I ask:

Did the stream end cleanly?
Did I get a finish reason?
Did I get the final metadata I expected?
Can my app distinguish "completed" from "stopped receiving chunks"?

A response can be useful and still not be complete enough to trust.

4. A rate limit or token limit check

I do not always try to actually hit a provider's rate limit. That can be annoying, noisy, and occasionally rude.

But I do want to know what rate limit and token limit failures look like.

If I can trigger one safely, great. If not, I test this through a mocked response in my wrapper.

The important part is classification.

function classifyLlmFailure(error) {
  const status = error.status;

  if (status === 429) {
    return {
      type: "rate_limit",
      retryable: true,
      action: "queue_or_backoff"
    };
  }

  if (status === 400 && /context|token/i.test(error.message || "")) {
    return {
      type: "token_limit",
      retryable: false,
      action: "shrink_or_fail"
    };
  }

  if (status >= 500) {
    return {
      type: "provider_error",
      retryable: true,
      action: "retry_or_fallback"
    };
  }

  return {
    type: "unknown",
    retryable: false,
    action: "fail_visibly"
  };
}

The detail I care about here is whether the failure is actionable.

"Too many requests" and "input too large" should not go through the same path.

Even token limits need a split:

If the context is shrinkable, reduce optional context and retry.
If the input is one large tool result, do not silently clip it.
If clipping changes the meaning, fail visibly or queue the job.

Silently truncating important input is how you get a confident answer based on damaged context.

That is not a retry. That is a new incident wearing a fake mustache.

5. A tool-call-shaped request

This one matters if your LLM app can do anything outside the model call.

Send an email. Create a ticket. Write to a database. Call a webhook. Charge a card. Trigger a workflow.

Even if the smoke test does not execute a real tool, I still want to test the policy path.

function shouldRetry({ failureType, sideEffectRisk, streamStarted }) {
  if (sideEffectRisk === "none") {
    return failureType === "timeout" || failureType === "provider_error";
  }

  if (sideEffectRisk === "possible" && streamStarted) {
    return false;
  }

  if (sideEffectRisk === "confirmed") {
    return false;
  }

  return false;
}

console.log(
  shouldRetry({
    failureType: "timeout",
    sideEffectRisk: "possible",
    streamStarted: true
  })
);

This is where a lot of retry logic gets too cute.

If a request is just generating a summary, retrying is usually fine.

If a request may have already emitted a tool call, retrying blindly can duplicate the real-world action.

The smoke test should confirm that your app knows the difference.

What I log from the smoke test

I keep the output boring and structured:

{
  provider: "example-provider",
  model: "example-model",
  base_url: "https://api.example.com/v1",
  test_name: "streaming_completion",
  latency_ms: 842,
  status: 200,
  response_id: "abc123",
  finish_reason: "stop",
  usage_present: true,
  stream_completed: true,
  error_type: null,
  retryable: false,
  side_effect_risk: "none"
}

The exact fields depend on the app, but I usually want:

provider
model
base URL
test name
latency
status
provider request ID
finish reason
usage presence
stream completion
classified error type
retryable or not
side-effect risk

If I cannot log the provider's behavior clearly, I probably cannot operate it clearly either.

The rule I use now

I do not need every provider to behave exactly the same.

That is unrealistic.

But I do need the differences to be visible.

If a provider has different rate limit semantics, fine. I can handle that.

If it has different streaming metadata, fine. I can adapt.

If it gives me failures I cannot classify, streams I cannot verify, or retry behavior that might duplicate side effects, I do not want to discover that from a user report.

The rule is simple:

Do not trust a provider because one normal prompt worked.

Trust it after you understand how it fails.

The smoke test is not about distrust

This might sound like paranoia, but I do not think of it that way.

LLM providers are complicated systems. So are the apps built on top of them. Weird edge cases are not a moral failure. They are just part of running software that depends on remote inference, streaming, rate limits, queues, SDKs, and user-facing workflows.

The smoke test is not there to prove that a provider is bad.

It is there to make sure my app and the provider agree on what "done," "failed," and "safe to retry" mean.

That is a much better thing to learn in 10 minutes than during an incident.

I work on TokenBay, so I spend a lot of time thinking about provider routing and OpenAI-compatible APIs. But this rule applies no matter what stack you use: swapping a base URL is easy. Trusting the new behavior should take a little more work.

A Small Node.js Wrapper for LLM API Retries, Timeouts, and Logging

plasma — Tue, 07 Jul 2026 08:25:13 +0000

Most LLM API integrations start with a direct SDK call.

That is fine for a demo.

But once the call is inside a real product, I usually want three things around it:

a timeout
retry rules
useful logs when something fails

Not a giant framework. Not a full observability platform. Just a small wrapper that makes the boring production stuff harder to forget.

Here is the Node.js wrapper pattern I usually reach for.

The problem

A normal LLM call often starts like this:

const response = await client.chat.completions.create({
  model: "gpt-4.1-mini",
  messages: [
    { role: "user", content: "Summarize this support ticket." }
  ]
});

That is readable, but it hides a few production questions:

How long can this request run before we give up?
Which errors should be retried?
How many times?
Are retries safe for this workflow?
What do we log when the request fails?
Can someone debug this later without reproducing the whole request?

The wrapper below is intentionally small. It does not try to solve every LLM reliability problem.

It just gives each request a basic failure policy.

A minimal wrapper

This version uses fetch, so it works on modern Node.js without extra dependencies.

const DEFAULT_TIMEOUT_MS = 30_000;

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function createTimeoutSignal(timeoutMs) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), timeoutMs);

  return {
    signal: controller.signal,
    clear: () => clearTimeout(timeout)
  };
}

function shouldRetry({ status, error, attempt, maxRetries }) {
  if (attempt >= maxRetries) return false;

  if (error?.name === "AbortError") {
    return true;
  }

  if (!status) {
    return true;
  }

  return status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
}

function retryDelayMs(attempt, retryAfterHeader) {
  if (retryAfterHeader) {
    const seconds = Number(retryAfterHeader);
    if (Number.isFinite(seconds)) {
      return seconds * 1000;
    }
  }

  const base = 500 * 2 ** attempt;
  const jitter = Math.floor(Math.random() * 250);

  return base + jitter;
}

export async function callLlmWithPolicy({
  url,
  apiKey,
  body,
  timeoutMs = DEFAULT_TIMEOUT_MS,
  maxRetries = 2,
  requestId = crypto.randomUUID(),
  logger = console
}) {
  let lastError;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const startedAt = Date.now();
    const timeout = createTimeoutSignal(timeoutMs);

    try {
      logger.info({
        event: "llm_request_started",
        requestId,
        attempt,
        model: body.model,
        timeoutMs
      });

      const response = await fetch(url, {
        method: "POST",
        signal: timeout.signal,
        headers: {
          "Authorization": `Bearer ${apiKey}`,
          "Content-Type": "application/json"
        },
        body: JSON.stringify(body)
      });

      const durationMs = Date.now() - startedAt;
      const text = await response.text();

      if (!response.ok) {
        const retryAfter = response.headers.get("retry-after");

        logger.warn({
          event: "llm_request_failed",
          requestId,
          attempt,
          status: response.status,
          durationMs,
          retryAfter,
          bodyPreview: text.slice(0, 500)
        });

        if (shouldRetry({
          status: response.status,
          attempt,
          maxRetries
        })) {
          await sleep(retryDelayMs(attempt, retryAfter));
          continue;
        }

        throw new Error(`LLM request failed with status ${response.status}: ${text}`);
      }

      logger.info({
        event: "llm_request_succeeded",
        requestId,
        attempt,
        status: response.status,
        durationMs
      });

      return JSON.parse(text);
    } catch (error) {
      const durationMs = Date.now() - startedAt;
      lastError = error;

      logger.warn({
        event: "llm_request_error",
        requestId,
        attempt,
        errorName: error.name,
        errorMessage: error.message,
        durationMs
      });

      if (shouldRetry({
        error,
        attempt,
        maxRetries
      })) {
        await sleep(retryDelayMs(attempt));
        continue;
      }

      throw error;
    } finally {
      timeout.clear();
    }
  }

  throw lastError;
}

Using it

Here is a basic chat completion request:

import { callLlmWithPolicy } from "./llm-wrapper.js";

const result = await callLlmWithPolicy({
  url: "https://api.openai.com/v1/chat/completions",
  apiKey: process.env.OPENAI_API_KEY,
  body: {
    model: "gpt-4.1-mini",
    messages: [
      {
        role: "user",
        content: "Write a one-paragraph summary of this incident report."
      }
    ]
  },
  timeoutMs: 20_000,
  maxRetries: 2
});

console.log(result.choices[0].message.content);

If you are using an OpenAI-compatible provider, the shape stays almost the same. The main change is usually the base URL and model name.

That is one reason I like keeping this wrapper close to the HTTP boundary instead of burying all of the logic inside application code.

What this wrapper logs

The logs are deliberately simple.

On start:

{
  event: "llm_request_started",
  requestId: "01...",
  attempt: 0,
  model: "gpt-4.1-mini",
  timeoutMs: 20000
}

On failure:

{
  event: "llm_request_failed",
  requestId: "01...",
  attempt: 0,
  status: 429,
  durationMs: 813,
  retryAfter: "2"
}

On success:

{
  event: "llm_request_succeeded",
  requestId: "01...",
  attempt: 1,
  status: 200,
  durationMs: 1420
}

That is enough to answer a few important questions later:

Did this fail once and recover?
Did it burn all retries?
Was it a timeout or provider response?
Which model was being called?
How long did each attempt take?
Did the provider send Retry-After?

For a lot of production debugging, that is already much better than “the AI call failed.”

The part I would not hide in the wrapper

I would be careful about making retries automatic for every LLM call.

Some calls are safe to repeat.

For example:

summarizing text
classifying a ticket
extracting fields from a document
generating a draft that has not been shown or saved yet

Other calls may not be safe to repeat.

For example:

sending an email
creating a ticket
charging a customer
calling a webhook
letting an agent execute a tool

The wrapper cannot know that from the HTTP status code.

So in real apps, I usually add one more input:

retryMode: "safe" | "unsafe"

Then the retry rule becomes stricter:

function shouldRetry({ status, error, attempt, maxRetries, retryMode }) {
  if (attempt >= maxRetries) return false;

  if (retryMode === "unsafe") {
    return false;
  }

  if (error?.name === "AbortError") {
    return true;
  }

  if (!status) {
    return true;
  }

  return status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
}

That small distinction matters.

A retry on a read-only summarization call is boring.

A retry after an agent may have already triggered an external action is a completely different failure mode.

Timeouts need product judgment

I usually avoid one global timeout for every LLM call.

Different workflows have different tolerance.

For example:

const timeoutByWorkflow = {
  autocomplete: 5_000,
  support_summary: 20_000,
  background_analysis: 90_000
};

A user-facing autocomplete should probably fail fast.

A background analysis job can wait longer.

A support summary might sit somewhere in the middle.

The point is not to find the perfect timeout. The point is to make timeout behavior intentional instead of accidental.

Keep the wrapper boring

It is tempting to keep adding features:

model fallback
schema validation
streaming recovery
circuit breakers
provider routing
prompt redaction
token accounting

Those can all be useful.

But I like starting with the boring layer first:

timeout each request
retry only known transient failures
honor Retry-After
log every attempt
avoid retrying unsafe operations
attach a request ID

That is the part you want in place before the first incident, not after.

Final thought

A small LLM wrapper will not make your app production-ready by itself.

But it does create a place for production policy to live.

Without that layer, retry behavior gets scattered across route handlers, background jobs, SDK calls, and agent tools. Eventually, nobody knows which LLM calls are safe to retry and which ones are quietly dangerous.

My rule now is simple:

If an LLM call matters to the product, it should not be a naked SDK call.

Put a small policy around it.

Your future debugging session will be shorter.

I work on TokenBay, so I spend a lot of time thinking about OpenAI-compatible APIs, multi-model routing, retries, and failure handling. The wrapper above is intentionally provider-agnostic: the same basic shape works whether you call one provider directly or route through a compatible API gateway.

The LLM API Failure Policy I Wish I Had Before My First Production Incident

plasma — Mon, 06 Jul 2026 07:33:55 +0000

Most LLM API error handling starts out looking like normal HTTP error handling.

If it is a 429, retry later.

If it is a 500, retry with backoff.

If it is a timeout, maybe retry once.

That works fine right up until your first real production incident, where the question is no longer:

"Did the API return an error?"

The question becomes:

"What should the system do now?"

That is where normal HTTP error handling starts to feel too shallow.

The problem is not the error code

A 429 from an LLM provider can mean very different things depending on the provider, account, model, and workload.

You might be hitting:

requests per minute
tokens per minute
daily quota
concurrency limits
temporary provider-side throttling
a model-specific capacity limit

Those are not the same operational problem.

If you hit requests per minute, slowing request frequency may help.

If you hit tokens per minute, retrying the same large prompt immediately is probably useless.

If the provider is overloaded, retrying aggressively can make the incident worse.

If your own queue is backing up, switching models may hide the symptom while the real issue keeps growing.

The HTTP status code tells you the broad class of failure. It does not tell you the correct response.

SDK error handling is not an incident policy

Most SDKs are designed to help you make a request and catch an exception.

That is useful, but it is not the same as a production policy.

An SDK can tell you:

the request failed
the provider returned 429
the provider returned 529
the request timed out
the response body had a certain error shape

But your application still has to decide:

should this be retried?
should the request be queued?
should we switch models?
should we degrade the feature?
should we show the user a specific message?
should this page someone?
should we stop retrying because this action has side effects?

That decision tree usually does not live in the SDK.

In a lot of teams, it lives in a runbook someone wrote after the first painful incident.

That is better than nothing, but it creates a gap: the code sees the error first, while the runbook explains what humans should do later.

The failure policy is the layer in between.

What I mean by a failure policy

A failure policy is a small layer near your LLM client that maps provider-specific errors into internal failure categories and actions.

Instead of spreading logic like this across your app:

if (error.status === 429) {
  retry();
}

I prefer something closer to this:

const decision = classifyLlmFailure(error, requestContext);

if (decision.action === "retry") {
  await retryWithBackoff();
}

if (decision.action === "queue") {
  await enqueueForLater();
}

if (decision.action === "fail_closed") {
  throw new UserVisibleError(decision.userMessage);
}

The important part is that the policy returns a decision, not just a label.

A useful failure policy usually answers four questions:

What happened?
How confident are we?
What should the system do first?
What should humans look at if this continues?

A small example

Here is a simplified JavaScript example.

This is not meant to cover every provider or SDK. The point is the shape of the layer.

function classifyLlmFailure(error, context = {}) {
  const status = error.status || error.response?.status;
  const headers = normalizeHeaders(error.headers || error.response?.headers || {});
  const provider = context.provider || "unknown";

  if (status === 429) {
    const resetRequests = headers["x-ratelimit-reset-requests"];
    const resetTokens = headers["x-ratelimit-reset-tokens"];
    const retryAfter = headers["retry-after"];

    if (resetTokens && !resetRequests) {
      return {
        category: "rate_limit_tokens",
        action: "queue",
        retryable: true,
        confidence: "medium",
        logLevel: "warn",
        runbook: "llm-rate-limit-tokens",
        reason: "Provider returned token reset metadata."
      };
    }

    if (resetRequests && !resetTokens) {
      return {
        category: "rate_limit_requests",
        action: "retry",
        retryable: true,
        confidence: "medium",
        retryAfterMs: parseRetryAfter(retryAfter),
        logLevel: "warn",
        runbook: "llm-rate-limit-requests",
        reason: "Provider returned request reset metadata."
      };
    }

    return {
      category: "rate_limit_unknown",
      action: "slow_down",
      retryable: true,
      confidence: "low",
      retryAfterMs: parseRetryAfter(retryAfter),
      logLevel: "warn",
      runbook: "llm-rate-limit-unknown",
      reason: "429 without enough metadata to distinguish request vs token limits."
    };
  }

  if (status === 529 || status === 503) {
    return {
      category: "provider_overloaded",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-provider-overload",
      reason: "Provider appears temporarily overloaded."
    };
  }

  if (error.name === "AbortError" || error.code === "ETIMEDOUT") {
    return {
      category: "timeout",
      action: context.hasSideEffects ? "fail_closed" : "retry",
      retryable: !context.hasSideEffects,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-timeouts",
      reason: "Request timed out before a reliable result was received."
    };
  }

  if (status >= 500) {
    return {
      category: "provider_5xx",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-provider-5xx",
      reason: "Provider returned a server-side error."
    };
  }

  if (status >= 400) {
    return {
      category: "bad_request_or_auth",
      action: "fail_fast",
      retryable: false,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-client-errors",
      reason: "Client-side request, auth, or configuration error."
    };
  }

  return {
    category: "unknown",
    action: "alert_human",
    retryable: false,
    confidence: "low",
    logLevel: "error",
    runbook: "llm-unknown-failures",
    reason: "Could not classify LLM failure."
  };
}

function normalizeHeaders(headers) {
  const result = {};

  for (const [key, value] of Object.entries(headers)) {
    result[key.toLowerCase()] = Array.isArray(value) ? value.join(",") : String(value);
  }

  return result;
}

function parseRetryAfter(value) {
  if (!value) return undefined;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(value);
  if (Number.isFinite(date)) {
    return Math.max(0, date - Date.now());
  }

  return undefined;
}

The exact categories will vary by app.

The useful part is not the specific code. It is the habit of turning provider errors into internal operational decisions.

Log the policy decision, not just the error

During an incident, a raw stack trace is rarely enough.

I want one log event that answers:

which provider failed?
which model was used?
what status code came back?
what raw response headers mattered?
how many input and output tokens were estimated?
what failure category did we assign?
what action did the system take?
which runbook should a human open?

A log line like this is much more useful than "LLM request failed":

logger.warn("llm_request_failed", {
  provider: "example-provider",
  model: "example-model",
  status: 429,
  category: decision.category,
  action: decision.action,
  confidence: decision.confidence,
  retryable: decision.retryable,
  retryAfterMs: decision.retryAfterMs,
  runbook: decision.runbook,
  estimatedInputTokens: context.estimatedInputTokens,
  estimatedOutputTokens: context.estimatedOutputTokens,
  requestId: context.requestId
});

The goal is simple: if someone is on call, they should not need 20 minutes to figure out whether this was a request-rate issue, token-rate issue, provider overload, timeout, or bad config.

If the first useful question during an incident is "what kind of failure is this?", your logs should answer that directly.

Do not retry every retryable error the same way

This is where a lot of LLM API code gets too optimistic.

"Retryable" does not mean "retry immediately."

A token-per-minute limit, request-per-minute limit, timeout, and provider overload may all be retryable, but they should not share the same retry behavior.

For example:

rate_limit_requests: retry after the reset window, possibly with queueing
rate_limit_tokens: reduce token usage, queue, or switch to a smaller request
provider_overloaded: retry with jitter, but avoid synchronized retry storms
timeout: retry only if the operation is safe to repeat
stream_interrupted: decide whether partial output is usable or must be discarded
tool_call_uncertain: do not blindly execute side effects twice

That last one matters a lot for agentic workflows.

If the LLM call was only generating text, a retry is usually fine.

If the LLM call was planning or triggering an external action, retrying can accidentally duplicate real work. Sending two emails, creating two tickets, charging twice, or calling the same webhook twice is a very different failure mode from "the answer took too long."

Keep the runbook close to the code

I do not think the whole incident runbook belongs inside the codebase.

Human context still matters.

But the first decision should be encoded near the client.

A practical compromise is:

code classifies the failure
code chooses the first action
logs include the runbook ID
the runbook explains investigation steps and business context

For example:

{
  category: "rate_limit_tokens",
  action: "queue",
  runbook: "llm-rate-limit-tokens"
}

Then the runbook can explain:

where to check token usage
which dashboards matter
whether to reduce max output tokens
whether fallback models are approved
who owns quota increases
what user-facing degradation is acceptable

That keeps institutional knowledge closer to the place where the incident starts, without turning your application code into a giant wiki page.

The checklist I use now

Before shipping a new LLM provider or model path, I try to answer these:

Do we log raw provider error metadata?
Can we distinguish request limits from token limits?
Do we classify overload separately from rate limits?
Are timeout retries safe for this workflow?
Do streaming failures preserve partial state intentionally?
Do tool calls have idempotency keys or operation IDs?
Does every failure category have a first action?
Does every common category point to a short runbook?
Can someone on call understand the failure from one log event?

If the answer is no, the integration may still work in development, but it is probably not ready for production traffic.

Final thought

LLM APIs look like normal APIs until they fail under real workload.

Then the hard part is not catching the exception. The hard part is knowing whether to retry, slow down, queue, fallback, fail closed, or wake someone up.

That decision should not live only in a forgotten runbook.

Put a small failure policy next to your LLM client.

Your future on-call self will be a lot less annoyed.

I work on TokenBay, so I spend a lot of time thinking about multi-model API behavior, routing, and failure handling. This post is less about any one provider and more about the production layer I wish more LLM apps had from day one.

The Retry Setup I Use for LLM APIs Without Accidentally Duplicating User Actions

plasma — Fri, 03 Jul 2026 06:00:00 +0000

Retries look simple until an LLM call is allowed to do something.

For a normal read-only API request, retrying is usually boring:

if (status === 429 || status >= 500) {
  retryWithBackoff();
}

But LLM APIs are often sitting inside workflows that are not purely read-only.

A failed LLM call might be part of:

sending an email
creating a support ticket
updating a CRM record
calling a tool
writing to a database
charging credits
generating a document
triggering another automation step

In those cases, retrying blindly can turn one user action into two, three, or four real-world actions.

That is the bug I try hardest to avoid.

The mistake

The first retry setup I used for LLM APIs was basically copied from normal HTTP APIs:

async function callWithRetry(fn: () => Promise<Response>) {
  let lastError: unknown;

  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      await sleep(2 ** attempt * 1000);
    }
  }

  throw lastError;
}

This works fine for some LLM calls.

For example:

summarize this paragraph
classify this ticket
rewrite this title
extract entities from this text

If the first attempt fails, retrying is probably fine.

But it gets risky once the LLM call is part of a workflow with side effects.

Imagine this agent step:

Read customer message
Decide whether refund is needed
Call issueRefund
Write reply draft
Send email

If the request times out after step 3, your retry code may run the whole thing again.

Now you might issue two refunds.

That is not a model quality problem.

That is a retry design problem.

My rule: retry transport, not user intent

The main principle I use now is:

Retry technical failures, but do not replay user intent unless the operation is idempotent.

A user clicked "Send invoice" once.

That user action should have one logical operation ID.

Even if the system retries internally, the outside world should still see one invoice send attempt, not three.

So before thinking about retry count or backoff, I separate LLM work into operation types.

type LlmOperationKind =
  | "read_only_generation"
  | "structured_extraction"
  | "streaming_chat"
  | "tool_planning"
  | "tool_execution"
  | "external_side_effect";

Then I treat them differently.

function canRetryOperation(kind: LlmOperationKind) {
  switch (kind) {
    case "read_only_generation":
    case "structured_extraction":
      return true;

    case "streaming_chat":
      return "only_if_no_tokens_received";

    case "tool_planning":
      return true;

    case "tool_execution":
    case "external_side_effect":
      return false;
  }
}

This is the core distinction.

Retrying the model's planning step is usually fine.

Retrying the step that actually sends the email, charges the card, or updates the external system is not fine unless that step is idempotent.

Use one operation ID per user action

Every user-triggered workflow gets an operation ID.

type LlmOperation = {
  operationId: string;
  userId: string;
  kind: LlmOperationKind;
  createdAt: number;
};

That operationId follows the request through:

LLM call logs
retry attempts
tool calls
database writes
external API calls
user-visible status updates

A retry is not a new operation.

It is another attempt inside the same operation.

type LlmAttemptLog = {
  operationId: string;
  attempt: number;
  provider: string;
  model: string;
  startedAt: number;
  finishedAt?: number;
  status: "success" | "failed";
  errorCategory?: string;
};

This makes debugging much easier.

Instead of seeing four unrelated LLM failures, I can see:

One user action created one operation, which made four attempts.

That difference matters a lot during incidents.

Make tool calls idempotent

If an LLM can call tools, the tools need idempotency keys.

For example, do not let this happen:

await issueRefund({
  customerId,
  amount
});

Prefer this:

await issueRefund({
  customerId,
  amount,
  idempotencyKey: `refund:${operationId}:${customerId}:${amount}`
});

Same for external actions:

await sendEmail({
  to,
  subject,
  body,
  idempotencyKey: `email:${operationId}:${to}`
});

The exact key depends on the action, but the idea is the same:

If the system retries, the external action should still happen at most once.

This is especially important because timeouts are ambiguous.

A timeout does not always mean "nothing happened."

It often means:

The client stopped waiting, but the provider or tool may still be working.

That ambiguity is where duplicate actions come from.

Split planning from execution

For agents, I try to split the workflow into two phases:

LLM decides what should happen.
Application code executes the action.

The LLM can retry the planning step.

The application controls execution.

Example:

type PlannedAction =
  | {
      type: "send_email";
      to: string;
      subject: string;
      body: string;
    }
  | {
      type: "create_ticket";
      title: string;
      priority: "low" | "medium" | "high";
    }
  | {
      type: "no_action";
      reason: string;
    };

The LLM returns a plan.

Then the app validates and executes it.

async function executePlannedAction(
  action: PlannedAction,
  operationId: string
) {
  if (action.type === "send_email") {
    return sendEmail({
      to: action.to,
      subject: action.subject,
      body: action.body,
      idempotencyKey: `send_email:${operationId}:${action.to}`
    });
  }

  if (action.type === "create_ticket") {
    return createTicket({
      title: action.title,
      priority: action.priority,
      idempotencyKey: `create_ticket:${operationId}:${action.title}`
    });
  }

  return { skipped: true, reason: action.reason };
}

This keeps retry logic away from irreversible side effects.

If the LLM plan fails due to a transient error, I can retry the planning step.

If execution starts, I stop treating the whole workflow as freely retryable.

Retry only known transient failures

I do not retry every LLM failure.

I usually retry:

temporary rate limits
provider 5xx errors
network timeouts
connection resets
stream interrupted before any useful output
malformed structured output, sometimes

I usually do not retry:

auth errors
quota exhaustion
context length errors
content refusals
invalid request shape
completed tool execution
user-cancelled operations

A simple classifier helps.

type LlmErrorCategory =
  | "rate_limited"
  | "provider_unavailable"
  | "timeout"
  | "network_error"
  | "stream_interrupted"
  | "malformed_structured_output"
  | "quota_exceeded"
  | "auth_error"
  | "context_window_exceeded"
  | "content_refused"
  | "invalid_request"
  | "unknown";

function isTransient(error: LlmErrorCategory) {
  return [
    "rate_limited",
    "provider_unavailable",
    "timeout",
    "network_error",
    "stream_interrupted"
  ].includes(error);
}

Then retry logic becomes more explicit.

function shouldRetryLlmCall(params: {
  operationKind: LlmOperationKind;
  error: LlmErrorCategory;
  attempt: number;
  receivedTokens?: number;
}) {
  if (params.attempt >= 3) return false;
  if (!isTransient(params.error)) return false;

  if (params.operationKind === "external_side_effect") {
    return false;
  }

  if (
    params.operationKind === "streaming_chat" &&
    params.receivedTokens &&
    params.receivedTokens > 0
  ) {
    return false;
  }

  return true;
}

This is not fancy.

But it prevents a lot of expensive nonsense.

Streaming retries need a different rule

Streaming responses are a special case.

If a stream fails before any tokens arrive, retrying is usually fine.

If a stream fails after the user has already seen half the answer, retrying from scratch may be weird.

The user might see:

partial answer
sudden reset
different answer
duplicate content

For chat UI, I track whether any content was received.

type StreamAttempt = {
  operationId: string;
  attempt: number;
  receivedTokens: number;
  partialText: string;
  completed: boolean;
  error?: LlmErrorCategory;
};

Then the behavior is:

no tokens received: retry automatically
some tokens received: show incomplete state
structured output expected: discard or repair
tool call involved: stop and ask for confirmation

For user-visible chat, I would rather show:

The response was interrupted. Continue?

than silently retry and produce a second answer.

Backoff still matters

Once I know a retry is safe, I still use normal backoff.

function retryDelayMs(attempt: number) {
  const base = Math.min(1000 * 2 ** attempt, 10000);
  const jitter = Math.floor(Math.random() * 500);
  return base + jitter;
}

I also respect provider retry hints when available.

function getRetryDelay(error: {
  retryAfterMs?: number;
}, attempt: number) {
  if (error.retryAfterMs) {
    return error.retryAfterMs;
  }

  return retryDelayMs(attempt);
}

The important part is that backoff is not the whole strategy.

Backoff answers:

How long should I wait before retrying?

It does not answer:

Is retrying safe?

That second question is where most LLM workflow bugs live.

Keep attempts visible to the user when needed

For background jobs, retries can be invisible.

For user-facing workflows, I prefer exposing state.

Example states:

type OperationStatus =
  | "queued"
  | "running"
  | "retrying"
  | "waiting_for_confirmation"
  | "completed"
  | "failed";

If an operation is safe to retry automatically, the UI can say:

Still working. Retrying after a temporary model error.

If the operation may cause a duplicate side effect, the UI should ask:

The email may have already been sent. Do you want to check status before trying again?

That sounds less smooth, but it is much better than accidentally sending two emails.

Log retries as part of one workflow

One of the worst debugging experiences is seeing retries as unrelated logs.

I want logs grouped under one operation.

{
  "operation_id": "op_8f23",
  "user_id": "user_123",
  "operation_kind": "tool_planning",
  "attempt": 2,
  "max_attempts": 3,
  "provider": "openai",
  "model": "gpt-4.1-mini",
  "error_category": "rate_limited",
  "retryable": true,
  "next_retry_ms": 2400,
  "external_side_effect_started": false
}

For side effects, I log the idempotency key too.

{
  "operation_id": "op_8f23",
  "action": "send_email",
  "idempotency_key": "send_email:op_8f23:customer@example.com",
  "status": "completed"
}

This gives me a clear answer to the scary question:

Did the user action happen once, multiple times, or not at all?

My retry setup

The setup I use now is basically this:

Create one operation ID per user action.
Classify the LLM operation type.
Classify the error.
Retry only known transient failures.
Do not retry external side effects unless they are idempotent.
Use idempotency keys for tools and external APIs.
Split LLM planning from application execution.
Treat streaming partial output as a special state.
Log all attempts under the same operation.
Ask for confirmation when the system cannot know whether an action already happened.

In code, the retry wrapper looks less like a generic HTTP helper and more like an operation-aware executor.

async function runLlmAttempt<T>(params: {
  operationId: string;
  operationKind: LlmOperationKind;
  maxAttempts: number;
  call: () => Promise<T>;
}) {
  let attempt = 0;
  let lastError: unknown;

  while (attempt < params.maxAttempts) {
    try {
      return await params.call();
    } catch (error) {
      const category = classifyLlmError(error);

      const retryable = shouldRetryLlmCall({
        operationKind: params.operationKind,
        error: category,
        attempt
      });

      logLlmAttempt({
        operationId: params.operationId,
        operationKind: params.operationKind,
        attempt,
        errorCategory: category,
        retryable
      });

      if (!retryable) {
        throw error;
      }

      await sleep(retryDelayMs(attempt));
      attempt += 1;
      lastError = error;
    }
  }

  throw lastError;
}

This is more code than a simple retry helper.

But it has saved me from treating dangerous operations like harmless API reads.

Final thought

Retries are not automatically reliability.

For LLM apps, retries can improve reliability when the operation is read-only, transient, and safe to repeat.

They can create serious bugs when the model is part of a workflow that changes the outside world.

The question I ask now is not just:

Did the LLM call fail?

It is:

What has already happened because of this user action?

If nothing happened, retry.

If something may have happened, check, dedupe, or ask for confirmation.

That one distinction makes LLM retry logic a lot less scary.

For multi-model workflows, centralizing retries, fallback, and routing also helps keep this logic consistent. I use TokenBay for that kind of setup because it keeps the API surface familiar while making model/provider choices easier to manage in one place.