DEV Community: plasma

I Stopped Swapping LLM Providers Without a Smoke Test

plasma — Wed, 08 Jul 2026 10:25:06 +0000

For a while, my LLM provider migration test was embarrassingly simple:

Change the base URL
Send one normal prompt
Get a response
Call it done

Which is fine if your app only fails on days when it feels polite.

The problem is that a happy-path request does not tell you much. It tells you the provider can answer one boring prompt under normal conditions. It does not tell you what happens when the request times out, the stream cuts off, the rate limit response has a weird shape, or your app is about to retry something that may have already triggered a tool call.

So now, before I trust a new LLM provider in anything production-ish, I run a small smoke test first.

Not a full benchmark. Not a provider comparison. Just a quick "will this break my app in boring, predictable ways?" test.

The old test was basically useless

The old version looked something like this:

const response = await client.chat.completions.create({
  model: "some-model",
  messages: [
    { role: "user", content: "Say hello in one sentence." }
  ]
});

console.log(response.choices[0].message.content);

If that worked, I felt good.

That was the trap.

A normal request does not test the parts that usually hurt in production:

timeout behavior
retry safety
streaming completion
rate limit shape
token limit behavior
usage metadata
provider request IDs
tool call handling
error classification
whether the response envelope is actually complete

The provider can pass the hello-world test and still be a bad fit for your app.

What I actually want to know

Before swapping providers, I want answers to a few boring questions.

Boring is good here. Boring saves weekends.

I want to know:

Can I classify failures without parsing random strings?
Can I tell the difference between a retryable failure and a permanent one?
Does streaming end with enough metadata for me to trust the turn is complete?
Do timeouts cancel cleanly?
Does the provider return useful request IDs?
Does usage data exist where my billing and logging expect it?
If the call could trigger a side effect, will my retry policy avoid replaying it blindly?

That is the point of the smoke test.

It is not trying to prove that one provider is "better." It is trying to prove that my app can understand what happened.

My 10-minute smoke test

I usually run five small checks.

They are simple on purpose.

1. A boring normal request

First, I still run the basic request.

But now I log more than the text.

const started = Date.now();

const response = await client.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    { role: "user", content: "Reply with exactly one short sentence." }
  ]
});

console.log({
  latency_ms: Date.now() - started,
  content: response.choices?.[0]?.message?.content,
  finish_reason: response.choices?.[0]?.finish_reason,
  usage: response.usage,
  id: response.id
});

I am not looking for a beautiful answer.

I am looking for the envelope:

Did I get a response ID?
Did I get a finish reason?
Did I get usage?
Is the response shape what my app expects?
Is latency roughly sane?

If the normal path is already weird, I stop there.

2. A request that should timeout

Next, I force an aggressive timeout.

Not because I expect to run with a 200ms timeout in production, but because I want to see what failure looks like.

const controller = new AbortController();

const timeout = setTimeout(() => {
  controller.abort();
}, 200);

try {
  await client.chat.completions.create(
    {
      model: process.env.MODEL,
      messages: [
        {
          role: "user",
          content: "Write a detailed 1000-word explanation of database indexes."
        }
      ]
    },
    { signal: controller.signal }
  );
} catch (error) {
  console.log({
    name: error.name,
    message: error.message,
    code: error.code,
    status: error.status
  });
} finally {
  clearTimeout(timeout);
}

What I want to know:

Does the request actually cancel?
Does the SDK throw something I can classify?
Is the error clearly a timeout, or just a generic network failure?
Would my retry policy treat this correctly?

The retry decision depends on where the timeout happened.

A timeout before any response is usually retryable.

A timeout after partial output is much messier.

That difference matters a lot if the model is being used inside a workflow or agent.

3. A streaming request

Streaming is where a lot of "looks fine locally" bugs hide.

I want to know whether the stream gives me a clear ending.

const stream = await client.chat.completions.create({
  model: process.env.MODEL,
  stream: true,
  messages: [
    { role: "user", content: "Count from 1 to 5, one number per line." }
  ]
});

let text = "";
let finalChunkSeen = false;
let finishReason = null;

for await (const chunk of stream) {
  const delta = chunk.choices?.[0]?.delta?.content;
  if (delta) text += delta;

  const reason = chunk.choices?.[0]?.finish_reason;
  if (reason) {
    finishReason = reason;
    finalChunkSeen = true;
  }
}

console.log({
  text,
  finalChunkSeen,
  finishReason
});

For a normal app, partial text may be fine.

For a workflow, agent, or tool-calling system, partial text can be dangerous. The user may have already seen output. The app may have already started acting on it. The system may think the turn completed when it did not.

So I do not only ask, "Did I get tokens?"

I ask:

Did the stream end cleanly?
Did I get a finish reason?
Did I get the final metadata I expected?
Can my app distinguish "completed" from "stopped receiving chunks"?

A response can be useful and still not be complete enough to trust.

4. A rate limit or token limit check

I do not always try to actually hit a provider's rate limit. That can be annoying, noisy, and occasionally rude.

But I do want to know what rate limit and token limit failures look like.

If I can trigger one safely, great. If not, I test this through a mocked response in my wrapper.

The important part is classification.

function classifyLlmFailure(error) {
  const status = error.status;

  if (status === 429) {
    return {
      type: "rate_limit",
      retryable: true,
      action: "queue_or_backoff"
    };
  }

  if (status === 400 && /context|token/i.test(error.message || "")) {
    return {
      type: "token_limit",
      retryable: false,
      action: "shrink_or_fail"
    };
  }

  if (status >= 500) {
    return {
      type: "provider_error",
      retryable: true,
      action: "retry_or_fallback"
    };
  }

  return {
    type: "unknown",
    retryable: false,
    action: "fail_visibly"
  };
}

The detail I care about here is whether the failure is actionable.

"Too many requests" and "input too large" should not go through the same path.

Even token limits need a split:

If the context is shrinkable, reduce optional context and retry.
If the input is one large tool result, do not silently clip it.
If clipping changes the meaning, fail visibly or queue the job.

Silently truncating important input is how you get a confident answer based on damaged context.

That is not a retry. That is a new incident wearing a fake mustache.

5. A tool-call-shaped request

This one matters if your LLM app can do anything outside the model call.

Send an email. Create a ticket. Write to a database. Call a webhook. Charge a card. Trigger a workflow.

Even if the smoke test does not execute a real tool, I still want to test the policy path.

function shouldRetry({ failureType, sideEffectRisk, streamStarted }) {
  if (sideEffectRisk === "none") {
    return failureType === "timeout" || failureType === "provider_error";
  }

  if (sideEffectRisk === "possible" && streamStarted) {
    return false;
  }

  if (sideEffectRisk === "confirmed") {
    return false;
  }

  return false;
}

console.log(
  shouldRetry({
    failureType: "timeout",
    sideEffectRisk: "possible",
    streamStarted: true
  })
);

This is where a lot of retry logic gets too cute.

If a request is just generating a summary, retrying is usually fine.

If a request may have already emitted a tool call, retrying blindly can duplicate the real-world action.

The smoke test should confirm that your app knows the difference.

What I log from the smoke test

I keep the output boring and structured:

{
  provider: "example-provider",
  model: "example-model",
  base_url: "https://api.example.com/v1",
  test_name: "streaming_completion",
  latency_ms: 842,
  status: 200,
  response_id: "abc123",
  finish_reason: "stop",
  usage_present: true,
  stream_completed: true,
  error_type: null,
  retryable: false,
  side_effect_risk: "none"
}

The exact fields depend on the app, but I usually want:

provider
model
base URL
test name
latency
status
provider request ID
finish reason
usage presence
stream completion
classified error type
retryable or not
side-effect risk

If I cannot log the provider's behavior clearly, I probably cannot operate it clearly either.

The rule I use now

I do not need every provider to behave exactly the same.

That is unrealistic.

But I do need the differences to be visible.

If a provider has different rate limit semantics, fine. I can handle that.

If it has different streaming metadata, fine. I can adapt.

If it gives me failures I cannot classify, streams I cannot verify, or retry behavior that might duplicate side effects, I do not want to discover that from a user report.

The rule is simple:

Do not trust a provider because one normal prompt worked.

Trust it after you understand how it fails.

The smoke test is not about distrust

This might sound like paranoia, but I do not think of it that way.

LLM providers are complicated systems. So are the apps built on top of them. Weird edge cases are not a moral failure. They are just part of running software that depends on remote inference, streaming, rate limits, queues, SDKs, and user-facing workflows.

The smoke test is not there to prove that a provider is bad.

It is there to make sure my app and the provider agree on what "done," "failed," and "safe to retry" mean.

That is a much better thing to learn in 10 minutes than during an incident.

I work on TokenBay, so I spend a lot of time thinking about provider routing and OpenAI-compatible APIs. But this rule applies no matter what stack you use: swapping a base URL is easy. Trusting the new behavior should take a little more work.

A Small Node.js Wrapper for LLM API Retries, Timeouts, and Logging

plasma — Tue, 07 Jul 2026 08:25:13 +0000

Most LLM API integrations start with a direct SDK call.

That is fine for a demo.

But once the call is inside a real product, I usually want three things around it:

a timeout
retry rules
useful logs when something fails

Not a giant framework. Not a full observability platform. Just a small wrapper that makes the boring production stuff harder to forget.

Here is the Node.js wrapper pattern I usually reach for.

The problem

A normal LLM call often starts like this:

const response = await client.chat.completions.create({
  model: "gpt-4.1-mini",
  messages: [
    { role: "user", content: "Summarize this support ticket." }
  ]
});

That is readable, but it hides a few production questions:

How long can this request run before we give up?
Which errors should be retried?
How many times?
Are retries safe for this workflow?
What do we log when the request fails?
Can someone debug this later without reproducing the whole request?

The wrapper below is intentionally small. It does not try to solve every LLM reliability problem.

It just gives each request a basic failure policy.

A minimal wrapper

This version uses fetch, so it works on modern Node.js without extra dependencies.

const DEFAULT_TIMEOUT_MS = 30_000;

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function createTimeoutSignal(timeoutMs) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), timeoutMs);

  return {
    signal: controller.signal,
    clear: () => clearTimeout(timeout)
  };
}

function shouldRetry({ status, error, attempt, maxRetries }) {
  if (attempt >= maxRetries) return false;

  if (error?.name === "AbortError") {
    return true;
  }

  if (!status) {
    return true;
  }

  return status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
}

function retryDelayMs(attempt, retryAfterHeader) {
  if (retryAfterHeader) {
    const seconds = Number(retryAfterHeader);
    if (Number.isFinite(seconds)) {
      return seconds * 1000;
    }
  }

  const base = 500 * 2 ** attempt;
  const jitter = Math.floor(Math.random() * 250);

  return base + jitter;
}

export async function callLlmWithPolicy({
  url,
  apiKey,
  body,
  timeoutMs = DEFAULT_TIMEOUT_MS,
  maxRetries = 2,
  requestId = crypto.randomUUID(),
  logger = console
}) {
  let lastError;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const startedAt = Date.now();
    const timeout = createTimeoutSignal(timeoutMs);

    try {
      logger.info({
        event: "llm_request_started",
        requestId,
        attempt,
        model: body.model,
        timeoutMs
      });

      const response = await fetch(url, {
        method: "POST",
        signal: timeout.signal,
        headers: {
          "Authorization": `Bearer ${apiKey}`,
          "Content-Type": "application/json"
        },
        body: JSON.stringify(body)
      });

      const durationMs = Date.now() - startedAt;
      const text = await response.text();

      if (!response.ok) {
        const retryAfter = response.headers.get("retry-after");

        logger.warn({
          event: "llm_request_failed",
          requestId,
          attempt,
          status: response.status,
          durationMs,
          retryAfter,
          bodyPreview: text.slice(0, 500)
        });

        if (shouldRetry({
          status: response.status,
          attempt,
          maxRetries
        })) {
          await sleep(retryDelayMs(attempt, retryAfter));
          continue;
        }

        throw new Error(`LLM request failed with status ${response.status}: ${text}`);
      }

      logger.info({
        event: "llm_request_succeeded",
        requestId,
        attempt,
        status: response.status,
        durationMs
      });

      return JSON.parse(text);
    } catch (error) {
      const durationMs = Date.now() - startedAt;
      lastError = error;

      logger.warn({
        event: "llm_request_error",
        requestId,
        attempt,
        errorName: error.name,
        errorMessage: error.message,
        durationMs
      });

      if (shouldRetry({
        error,
        attempt,
        maxRetries
      })) {
        await sleep(retryDelayMs(attempt));
        continue;
      }

      throw error;
    } finally {
      timeout.clear();
    }
  }

  throw lastError;
}

Using it

Here is a basic chat completion request:

import { callLlmWithPolicy } from "./llm-wrapper.js";

const result = await callLlmWithPolicy({
  url: "https://api.openai.com/v1/chat/completions",
  apiKey: process.env.OPENAI_API_KEY,
  body: {
    model: "gpt-4.1-mini",
    messages: [
      {
        role: "user",
        content: "Write a one-paragraph summary of this incident report."
      }
    ]
  },
  timeoutMs: 20_000,
  maxRetries: 2
});

console.log(result.choices[0].message.content);

If you are using an OpenAI-compatible provider, the shape stays almost the same. The main change is usually the base URL and model name.

That is one reason I like keeping this wrapper close to the HTTP boundary instead of burying all of the logic inside application code.

What this wrapper logs

The logs are deliberately simple.

On start:

{
  event: "llm_request_started",
  requestId: "01...",
  attempt: 0,
  model: "gpt-4.1-mini",
  timeoutMs: 20000
}

On failure:

{
  event: "llm_request_failed",
  requestId: "01...",
  attempt: 0,
  status: 429,
  durationMs: 813,
  retryAfter: "2"
}

On success:

{
  event: "llm_request_succeeded",
  requestId: "01...",
  attempt: 1,
  status: 200,
  durationMs: 1420
}

That is enough to answer a few important questions later:

Did this fail once and recover?
Did it burn all retries?
Was it a timeout or provider response?
Which model was being called?
How long did each attempt take?
Did the provider send Retry-After?

For a lot of production debugging, that is already much better than “the AI call failed.”

The part I would not hide in the wrapper

I would be careful about making retries automatic for every LLM call.

Some calls are safe to repeat.

For example:

summarizing text
classifying a ticket
extracting fields from a document
generating a draft that has not been shown or saved yet

Other calls may not be safe to repeat.

For example:

sending an email
creating a ticket
charging a customer
calling a webhook
letting an agent execute a tool

The wrapper cannot know that from the HTTP status code.

So in real apps, I usually add one more input:

retryMode: "safe" | "unsafe"

Then the retry rule becomes stricter:

function shouldRetry({ status, error, attempt, maxRetries, retryMode }) {
  if (attempt >= maxRetries) return false;

  if (retryMode === "unsafe") {
    return false;
  }

  if (error?.name === "AbortError") {
    return true;
  }

  if (!status) {
    return true;
  }

  return status === 429 || status === 500 || status === 502 || status === 503 || status === 504;
}

That small distinction matters.

A retry on a read-only summarization call is boring.

A retry after an agent may have already triggered an external action is a completely different failure mode.

Timeouts need product judgment

I usually avoid one global timeout for every LLM call.

Different workflows have different tolerance.

For example:

const timeoutByWorkflow = {
  autocomplete: 5_000,
  support_summary: 20_000,
  background_analysis: 90_000
};

A user-facing autocomplete should probably fail fast.

A background analysis job can wait longer.

A support summary might sit somewhere in the middle.

The point is not to find the perfect timeout. The point is to make timeout behavior intentional instead of accidental.

Keep the wrapper boring

It is tempting to keep adding features:

model fallback
schema validation
streaming recovery
circuit breakers
provider routing
prompt redaction
token accounting

Those can all be useful.

But I like starting with the boring layer first:

timeout each request
retry only known transient failures
honor Retry-After
log every attempt
avoid retrying unsafe operations
attach a request ID

That is the part you want in place before the first incident, not after.

Final thought

A small LLM wrapper will not make your app production-ready by itself.

But it does create a place for production policy to live.

Without that layer, retry behavior gets scattered across route handlers, background jobs, SDK calls, and agent tools. Eventually, nobody knows which LLM calls are safe to retry and which ones are quietly dangerous.

My rule now is simple:

If an LLM call matters to the product, it should not be a naked SDK call.

Put a small policy around it.

Your future debugging session will be shorter.

I work on TokenBay, so I spend a lot of time thinking about OpenAI-compatible APIs, multi-model routing, retries, and failure handling. The wrapper above is intentionally provider-agnostic: the same basic shape works whether you call one provider directly or route through a compatible API gateway.

The LLM API Failure Policy I Wish I Had Before My First Production Incident

plasma — Mon, 06 Jul 2026 07:33:55 +0000

Most LLM API error handling starts out looking like normal HTTP error handling.

If it is a 429, retry later.

If it is a 500, retry with backoff.

If it is a timeout, maybe retry once.

That works fine right up until your first real production incident, where the question is no longer:

"Did the API return an error?"

The question becomes:

"What should the system do now?"

That is where normal HTTP error handling starts to feel too shallow.

The problem is not the error code

A 429 from an LLM provider can mean very different things depending on the provider, account, model, and workload.

You might be hitting:

requests per minute
tokens per minute
daily quota
concurrency limits
temporary provider-side throttling
a model-specific capacity limit

Those are not the same operational problem.

If you hit requests per minute, slowing request frequency may help.

If you hit tokens per minute, retrying the same large prompt immediately is probably useless.

If the provider is overloaded, retrying aggressively can make the incident worse.

If your own queue is backing up, switching models may hide the symptom while the real issue keeps growing.

The HTTP status code tells you the broad class of failure. It does not tell you the correct response.

SDK error handling is not an incident policy

Most SDKs are designed to help you make a request and catch an exception.

That is useful, but it is not the same as a production policy.

An SDK can tell you:

the request failed
the provider returned 429
the provider returned 529
the request timed out
the response body had a certain error shape

But your application still has to decide:

should this be retried?
should the request be queued?
should we switch models?
should we degrade the feature?
should we show the user a specific message?
should this page someone?
should we stop retrying because this action has side effects?

That decision tree usually does not live in the SDK.

In a lot of teams, it lives in a runbook someone wrote after the first painful incident.

That is better than nothing, but it creates a gap: the code sees the error first, while the runbook explains what humans should do later.

The failure policy is the layer in between.

What I mean by a failure policy

A failure policy is a small layer near your LLM client that maps provider-specific errors into internal failure categories and actions.

Instead of spreading logic like this across your app:

if (error.status === 429) {
  retry();
}

I prefer something closer to this:

const decision = classifyLlmFailure(error, requestContext);

if (decision.action === "retry") {
  await retryWithBackoff();
}

if (decision.action === "queue") {
  await enqueueForLater();
}

if (decision.action === "fail_closed") {
  throw new UserVisibleError(decision.userMessage);
}

The important part is that the policy returns a decision, not just a label.

A useful failure policy usually answers four questions:

What happened?
How confident are we?
What should the system do first?
What should humans look at if this continues?

A small example

Here is a simplified JavaScript example.

This is not meant to cover every provider or SDK. The point is the shape of the layer.

function classifyLlmFailure(error, context = {}) {
  const status = error.status || error.response?.status;
  const headers = normalizeHeaders(error.headers || error.response?.headers || {});
  const provider = context.provider || "unknown";

  if (status === 429) {
    const resetRequests = headers["x-ratelimit-reset-requests"];
    const resetTokens = headers["x-ratelimit-reset-tokens"];
    const retryAfter = headers["retry-after"];

    if (resetTokens && !resetRequests) {
      return {
        category: "rate_limit_tokens",
        action: "queue",
        retryable: true,
        confidence: "medium",
        logLevel: "warn",
        runbook: "llm-rate-limit-tokens",
        reason: "Provider returned token reset metadata."
      };
    }

    if (resetRequests && !resetTokens) {
      return {
        category: "rate_limit_requests",
        action: "retry",
        retryable: true,
        confidence: "medium",
        retryAfterMs: parseRetryAfter(retryAfter),
        logLevel: "warn",
        runbook: "llm-rate-limit-requests",
        reason: "Provider returned request reset metadata."
      };
    }

    return {
      category: "rate_limit_unknown",
      action: "slow_down",
      retryable: true,
      confidence: "low",
      retryAfterMs: parseRetryAfter(retryAfter),
      logLevel: "warn",
      runbook: "llm-rate-limit-unknown",
      reason: "429 without enough metadata to distinguish request vs token limits."
    };
  }

  if (status === 529 || status === 503) {
    return {
      category: "provider_overloaded",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-provider-overload",
      reason: "Provider appears temporarily overloaded."
    };
  }

  if (error.name === "AbortError" || error.code === "ETIMEDOUT") {
    return {
      category: "timeout",
      action: context.hasSideEffects ? "fail_closed" : "retry",
      retryable: !context.hasSideEffects,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-timeouts",
      reason: "Request timed out before a reliable result was received."
    };
  }

  if (status >= 500) {
    return {
      category: "provider_5xx",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-provider-5xx",
      reason: "Provider returned a server-side error."
    };
  }

  if (status >= 400) {
    return {
      category: "bad_request_or_auth",
      action: "fail_fast",
      retryable: false,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-client-errors",
      reason: "Client-side request, auth, or configuration error."
    };
  }

  return {
    category: "unknown",
    action: "alert_human",
    retryable: false,
    confidence: "low",
    logLevel: "error",
    runbook: "llm-unknown-failures",
    reason: "Could not classify LLM failure."
  };
}

function normalizeHeaders(headers) {
  const result = {};

  for (const [key, value] of Object.entries(headers)) {
    result[key.toLowerCase()] = Array.isArray(value) ? value.join(",") : String(value);
  }

  return result;
}

function parseRetryAfter(value) {
  if (!value) return undefined;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(value);
  if (Number.isFinite(date)) {
    return Math.max(0, date - Date.now());
  }

  return undefined;
}

The exact categories will vary by app.

The useful part is not the specific code. It is the habit of turning provider errors into internal operational decisions.

Log the policy decision, not just the error

During an incident, a raw stack trace is rarely enough.

I want one log event that answers:

which provider failed?
which model was used?
what status code came back?
what raw response headers mattered?
how many input and output tokens were estimated?
what failure category did we assign?
what action did the system take?
which runbook should a human open?

A log line like this is much more useful than "LLM request failed":

logger.warn("llm_request_failed", {
  provider: "example-provider",
  model: "example-model",
  status: 429,
  category: decision.category,
  action: decision.action,
  confidence: decision.confidence,
  retryable: decision.retryable,
  retryAfterMs: decision.retryAfterMs,
  runbook: decision.runbook,
  estimatedInputTokens: context.estimatedInputTokens,
  estimatedOutputTokens: context.estimatedOutputTokens,
  requestId: context.requestId
});

The goal is simple: if someone is on call, they should not need 20 minutes to figure out whether this was a request-rate issue, token-rate issue, provider overload, timeout, or bad config.

If the first useful question during an incident is "what kind of failure is this?", your logs should answer that directly.

Do not retry every retryable error the same way

This is where a lot of LLM API code gets too optimistic.

"Retryable" does not mean "retry immediately."

A token-per-minute limit, request-per-minute limit, timeout, and provider overload may all be retryable, but they should not share the same retry behavior.

For example:

rate_limit_requests: retry after the reset window, possibly with queueing
rate_limit_tokens: reduce token usage, queue, or switch to a smaller request
provider_overloaded: retry with jitter, but avoid synchronized retry storms
timeout: retry only if the operation is safe to repeat
stream_interrupted: decide whether partial output is usable or must be discarded
tool_call_uncertain: do not blindly execute side effects twice

That last one matters a lot for agentic workflows.

If the LLM call was only generating text, a retry is usually fine.

If the LLM call was planning or triggering an external action, retrying can accidentally duplicate real work. Sending two emails, creating two tickets, charging twice, or calling the same webhook twice is a very different failure mode from "the answer took too long."

Keep the runbook close to the code

I do not think the whole incident runbook belongs inside the codebase.

Human context still matters.

But the first decision should be encoded near the client.

A practical compromise is:

code classifies the failure
code chooses the first action
logs include the runbook ID
the runbook explains investigation steps and business context

For example:

{
  category: "rate_limit_tokens",
  action: "queue",
  runbook: "llm-rate-limit-tokens"
}

Then the runbook can explain:

where to check token usage
which dashboards matter
whether to reduce max output tokens
whether fallback models are approved
who owns quota increases
what user-facing degradation is acceptable

That keeps institutional knowledge closer to the place where the incident starts, without turning your application code into a giant wiki page.

The checklist I use now

Before shipping a new LLM provider or model path, I try to answer these:

Do we log raw provider error metadata?
Can we distinguish request limits from token limits?
Do we classify overload separately from rate limits?
Are timeout retries safe for this workflow?
Do streaming failures preserve partial state intentionally?
Do tool calls have idempotency keys or operation IDs?
Does every failure category have a first action?
Does every common category point to a short runbook?
Can someone on call understand the failure from one log event?

If the answer is no, the integration may still work in development, but it is probably not ready for production traffic.

Final thought

LLM APIs look like normal APIs until they fail under real workload.

Then the hard part is not catching the exception. The hard part is knowing whether to retry, slow down, queue, fallback, fail closed, or wake someone up.

That decision should not live only in a forgotten runbook.

Put a small failure policy next to your LLM client.

Your future on-call self will be a lot less annoyed.

I work on TokenBay, so I spend a lot of time thinking about multi-model API behavior, routing, and failure handling. This post is less about any one provider and more about the production layer I wish more LLM apps had from day one.

The Retry Setup I Use for LLM APIs Without Accidentally Duplicating User Actions

plasma — Fri, 03 Jul 2026 06:00:00 +0000

Retries look simple until an LLM call is allowed to do something.

For a normal read-only API request, retrying is usually boring:

if (status === 429 || status >= 500) {
  retryWithBackoff();
}

But LLM APIs are often sitting inside workflows that are not purely read-only.

A failed LLM call might be part of:

sending an email
creating a support ticket
updating a CRM record
calling a tool
writing to a database
charging credits
generating a document
triggering another automation step

In those cases, retrying blindly can turn one user action into two, three, or four real-world actions.

That is the bug I try hardest to avoid.

The mistake

The first retry setup I used for LLM APIs was basically copied from normal HTTP APIs:

async function callWithRetry(fn: () => Promise<Response>) {
  let lastError: unknown;

  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      await sleep(2 ** attempt * 1000);
    }
  }

  throw lastError;
}

This works fine for some LLM calls.

For example:

summarize this paragraph
classify this ticket
rewrite this title
extract entities from this text

If the first attempt fails, retrying is probably fine.

But it gets risky once the LLM call is part of a workflow with side effects.

Imagine this agent step:

Read customer message
Decide whether refund is needed
Call issueRefund
Write reply draft
Send email

If the request times out after step 3, your retry code may run the whole thing again.

Now you might issue two refunds.

That is not a model quality problem.

That is a retry design problem.

My rule: retry transport, not user intent

The main principle I use now is:

Retry technical failures, but do not replay user intent unless the operation is idempotent.

A user clicked "Send invoice" once.

That user action should have one logical operation ID.

Even if the system retries internally, the outside world should still see one invoice send attempt, not three.

So before thinking about retry count or backoff, I separate LLM work into operation types.

type LlmOperationKind =
  | "read_only_generation"
  | "structured_extraction"
  | "streaming_chat"
  | "tool_planning"
  | "tool_execution"
  | "external_side_effect";

Then I treat them differently.

function canRetryOperation(kind: LlmOperationKind) {
  switch (kind) {
    case "read_only_generation":
    case "structured_extraction":
      return true;

    case "streaming_chat":
      return "only_if_no_tokens_received";

    case "tool_planning":
      return true;

    case "tool_execution":
    case "external_side_effect":
      return false;
  }
}

This is the core distinction.

Retrying the model's planning step is usually fine.

Retrying the step that actually sends the email, charges the card, or updates the external system is not fine unless that step is idempotent.

Use one operation ID per user action

Every user-triggered workflow gets an operation ID.

type LlmOperation = {
  operationId: string;
  userId: string;
  kind: LlmOperationKind;
  createdAt: number;
};

That operationId follows the request through:

LLM call logs
retry attempts
tool calls
database writes
external API calls
user-visible status updates

A retry is not a new operation.

It is another attempt inside the same operation.

type LlmAttemptLog = {
  operationId: string;
  attempt: number;
  provider: string;
  model: string;
  startedAt: number;
  finishedAt?: number;
  status: "success" | "failed";
  errorCategory?: string;
};

This makes debugging much easier.

Instead of seeing four unrelated LLM failures, I can see:

One user action created one operation, which made four attempts.

That difference matters a lot during incidents.

Make tool calls idempotent

If an LLM can call tools, the tools need idempotency keys.

For example, do not let this happen:

await issueRefund({
  customerId,
  amount
});

Prefer this:

await issueRefund({
  customerId,
  amount,
  idempotencyKey: `refund:${operationId}:${customerId}:${amount}`
});

Same for external actions:

await sendEmail({
  to,
  subject,
  body,
  idempotencyKey: `email:${operationId}:${to}`
});

The exact key depends on the action, but the idea is the same:

If the system retries, the external action should still happen at most once.

This is especially important because timeouts are ambiguous.

A timeout does not always mean "nothing happened."

It often means:

The client stopped waiting, but the provider or tool may still be working.

That ambiguity is where duplicate actions come from.

Split planning from execution

For agents, I try to split the workflow into two phases:

LLM decides what should happen.
Application code executes the action.

The LLM can retry the planning step.

The application controls execution.

Example:

type PlannedAction =
  | {
      type: "send_email";
      to: string;
      subject: string;
      body: string;
    }
  | {
      type: "create_ticket";
      title: string;
      priority: "low" | "medium" | "high";
    }
  | {
      type: "no_action";
      reason: string;
    };

The LLM returns a plan.

Then the app validates and executes it.

async function executePlannedAction(
  action: PlannedAction,
  operationId: string
) {
  if (action.type === "send_email") {
    return sendEmail({
      to: action.to,
      subject: action.subject,
      body: action.body,
      idempotencyKey: `send_email:${operationId}:${action.to}`
    });
  }

  if (action.type === "create_ticket") {
    return createTicket({
      title: action.title,
      priority: action.priority,
      idempotencyKey: `create_ticket:${operationId}:${action.title}`
    });
  }

  return { skipped: true, reason: action.reason };
}

This keeps retry logic away from irreversible side effects.

If the LLM plan fails due to a transient error, I can retry the planning step.

If execution starts, I stop treating the whole workflow as freely retryable.

Retry only known transient failures

I do not retry every LLM failure.

I usually retry:

temporary rate limits
provider 5xx errors
network timeouts
connection resets
stream interrupted before any useful output
malformed structured output, sometimes

I usually do not retry:

auth errors
quota exhaustion
context length errors
content refusals
invalid request shape
completed tool execution
user-cancelled operations

A simple classifier helps.

type LlmErrorCategory =
  | "rate_limited"
  | "provider_unavailable"
  | "timeout"
  | "network_error"
  | "stream_interrupted"
  | "malformed_structured_output"
  | "quota_exceeded"
  | "auth_error"
  | "context_window_exceeded"
  | "content_refused"
  | "invalid_request"
  | "unknown";

function isTransient(error: LlmErrorCategory) {
  return [
    "rate_limited",
    "provider_unavailable",
    "timeout",
    "network_error",
    "stream_interrupted"
  ].includes(error);
}

Then retry logic becomes more explicit.

function shouldRetryLlmCall(params: {
  operationKind: LlmOperationKind;
  error: LlmErrorCategory;
  attempt: number;
  receivedTokens?: number;
}) {
  if (params.attempt >= 3) return false;
  if (!isTransient(params.error)) return false;

  if (params.operationKind === "external_side_effect") {
    return false;
  }

  if (
    params.operationKind === "streaming_chat" &&
    params.receivedTokens &&
    params.receivedTokens > 0
  ) {
    return false;
  }

  return true;
}

This is not fancy.

But it prevents a lot of expensive nonsense.

Streaming retries need a different rule

Streaming responses are a special case.

If a stream fails before any tokens arrive, retrying is usually fine.

If a stream fails after the user has already seen half the answer, retrying from scratch may be weird.

The user might see:

partial answer
sudden reset
different answer
duplicate content

For chat UI, I track whether any content was received.

type StreamAttempt = {
  operationId: string;
  attempt: number;
  receivedTokens: number;
  partialText: string;
  completed: boolean;
  error?: LlmErrorCategory;
};

Then the behavior is:

no tokens received: retry automatically
some tokens received: show incomplete state
structured output expected: discard or repair
tool call involved: stop and ask for confirmation

For user-visible chat, I would rather show:

The response was interrupted. Continue?

than silently retry and produce a second answer.

Backoff still matters

Once I know a retry is safe, I still use normal backoff.

function retryDelayMs(attempt: number) {
  const base = Math.min(1000 * 2 ** attempt, 10000);
  const jitter = Math.floor(Math.random() * 500);
  return base + jitter;
}

I also respect provider retry hints when available.

function getRetryDelay(error: {
  retryAfterMs?: number;
}, attempt: number) {
  if (error.retryAfterMs) {
    return error.retryAfterMs;
  }

  return retryDelayMs(attempt);
}

The important part is that backoff is not the whole strategy.

Backoff answers:

How long should I wait before retrying?

It does not answer:

Is retrying safe?

That second question is where most LLM workflow bugs live.

Keep attempts visible to the user when needed

For background jobs, retries can be invisible.

For user-facing workflows, I prefer exposing state.

Example states:

type OperationStatus =
  | "queued"
  | "running"
  | "retrying"
  | "waiting_for_confirmation"
  | "completed"
  | "failed";

If an operation is safe to retry automatically, the UI can say:

Still working. Retrying after a temporary model error.

If the operation may cause a duplicate side effect, the UI should ask:

The email may have already been sent. Do you want to check status before trying again?

That sounds less smooth, but it is much better than accidentally sending two emails.

Log retries as part of one workflow

One of the worst debugging experiences is seeing retries as unrelated logs.

I want logs grouped under one operation.

{
  "operation_id": "op_8f23",
  "user_id": "user_123",
  "operation_kind": "tool_planning",
  "attempt": 2,
  "max_attempts": 3,
  "provider": "openai",
  "model": "gpt-4.1-mini",
  "error_category": "rate_limited",
  "retryable": true,
  "next_retry_ms": 2400,
  "external_side_effect_started": false
}

For side effects, I log the idempotency key too.

{
  "operation_id": "op_8f23",
  "action": "send_email",
  "idempotency_key": "send_email:op_8f23:customer@example.com",
  "status": "completed"
}

This gives me a clear answer to the scary question:

Did the user action happen once, multiple times, or not at all?

My retry setup

The setup I use now is basically this:

Create one operation ID per user action.
Classify the LLM operation type.
Classify the error.
Retry only known transient failures.
Do not retry external side effects unless they are idempotent.
Use idempotency keys for tools and external APIs.
Split LLM planning from application execution.
Treat streaming partial output as a special state.
Log all attempts under the same operation.
Ask for confirmation when the system cannot know whether an action already happened.

In code, the retry wrapper looks less like a generic HTTP helper and more like an operation-aware executor.

async function runLlmAttempt<T>(params: {
  operationId: string;
  operationKind: LlmOperationKind;
  maxAttempts: number;
  call: () => Promise<T>;
}) {
  let attempt = 0;
  let lastError: unknown;

  while (attempt < params.maxAttempts) {
    try {
      return await params.call();
    } catch (error) {
      const category = classifyLlmError(error);

      const retryable = shouldRetryLlmCall({
        operationKind: params.operationKind,
        error: category,
        attempt
      });

      logLlmAttempt({
        operationId: params.operationId,
        operationKind: params.operationKind,
        attempt,
        errorCategory: category,
        retryable
      });

      if (!retryable) {
        throw error;
      }

      await sleep(retryDelayMs(attempt));
      attempt += 1;
      lastError = error;
    }
  }

  throw lastError;
}

This is more code than a simple retry helper.

But it has saved me from treating dangerous operations like harmless API reads.

Final thought

Retries are not automatically reliability.

For LLM apps, retries can improve reliability when the operation is read-only, transient, and safe to repeat.

They can create serious bugs when the model is part of a workflow that changes the outside world.

The question I ask now is not just:

Did the LLM call fail?

It is:

What has already happened because of this user action?

If nothing happened, retry.

If something may have happened, check, dedupe, or ask for confirmation.

That one distinction makes LLM retry logic a lot less scary.

For multi-model workflows, centralizing retries, fallback, and routing also helps keep this logic consistent. I use TokenBay for that kind of setup because it keeps the API surface familiar while making model/provider choices easier to manage in one place.

Stop Treating LLM API Errors Like Normal HTTP Errors

plasma — Thu, 02 Jul 2026 06:38:41 +0000

Most backend engineers already know how to handle HTTP errors.

400 means the request is bad.

401 means auth failed.

429 means rate limited.

500 means something broke upstream.

Retry a few times, add exponential backoff, log the response body, move on.

That works fine for many APIs.

It works badly for LLM APIs.

LLM providers may use normal HTTP status codes, but the operational meaning behind those errors is different enough that treating them like ordinary REST failures can make your app slower, more expensive, and harder to debug.

The mistake I kept making

Early on, I handled LLM failures the same way I handled every other external API:

if (response.status === 429 || response.status >= 500) {
  retryWithBackoff();
}

Simple. Familiar. Dangerous.

That logic misses the actual question your app needs to answer:

What kind of LLM failure happened, and what should the product do next?

Because an LLM API failure is rarely just "one HTTP request failed."

It can break:

a user-facing chat response
a background agent run
a document generation job
a tool-calling workflow
a batch evaluation pipeline
a structured JSON generation step

And each one needs different handling.

Not all 429s mean the same thing

For a normal API, 429 Too Many Requests usually means:

Slow down and retry later.

With LLM APIs, 429 can mean several different things.

It might be a temporary rate limit:

{
  "error": {
    "message": "Rate limit reached",
    "type": "rate_limit_error"
  }
}

Retrying with backoff may help here.

But it might also mean quota exhaustion:

{
  "error": {
    "message": "You exceeded your current quota",
    "type": "insufficient_quota"
  }
}

Retrying this does not help. It just adds latency, noisy logs, and a worse user experience.

It could also be model-specific pressure. One model may be overloaded while another model from the same provider, or a different provider, would work fine.

So your handler should distinguish between:

temporary rate limit
hard quota exhaustion
model-level capacity issue
billing or account issue

Those should not all go through the same retry path.

Retrying can multiply the damage

Retries are one of the easiest ways to make an LLM system worse while thinking you made it more reliable.

Imagine this:

A user asks your app to generate a report.
Your backend calls an LLM.
The request times out after 30 seconds.
Your code retries three times.

That sounds reasonable.

But the first request may still be running provider-side. The second request may generate a duplicate answer. The third request may hit another rate limit. The user waits longer. Your token bill goes up. Your logs now show four attempts for one user action.

For simple completions, this is annoying.

For agent workflows, it can be dangerous.

If the LLM call can trigger tools, send emails, update tickets, write to a database, or call external systems, blind retries can create duplicate side effects.

Your retry policy needs to know what kind of operation it is retrying.

type LlmOperationType =
  | "simple_completion"
  | "streaming_chat"
  | "tool_calling_agent"
  | "background_batch_job"
  | "structured_output";

function shouldRetry(
  operation: LlmOperationType,
  status: number,
  errorType?: string
) {
  if (errorType === "insufficient_quota") return false;
  if (status === 401 || status === 403) return false;
  if (status === 400) return false;

  if (operation === "tool_calling_agent") {
    return status >= 500;
  }

  if (operation === "streaming_chat") {
    return status >= 500;
  }

  return status === 429 || status >= 500;
}

This is still simplified, but the idea matters:

retry policy should understand the LLM task, not just the HTTP status code.

Context length errors are product errors

A context length error often comes back as a 400.

For a normal API, a 400 often means the developer sent a malformed request.

For an LLM app, context length errors are often product-level failures.

It means your app did not manage the prompt budget properly.

Possible fixes include:

summarize older messages
trim retrieved documents
reduce tool output size
switch to a larger-context model
ask the user to narrow the task
split the job into multiple calls

The worst response is showing the raw provider error to the user.

A better product response is something like:

This request is too large to process in one pass. I can summarize the source material first, then continue.

At the system level, context errors should be classified separately from normal bad requests.

function classifyLlmError(status: number, message: string) {
  const text = message.toLowerCase();

  if (status === 400 && text.includes("context length")) {
    return "context_window_exceeded";
  }

  if (status === 429 && text.includes("quota")) {
    return "quota_exceeded";
  }

  if (status === 429) {
    return "rate_limited";
  }

  if (status >= 500) {
    return "provider_unavailable";
  }

  return "unknown_llm_error";
}

Once you classify this correctly, the rest of the app can respond correctly.

Streaming failures are partial product states

Streaming makes error handling more complicated.

With a normal request, the API either succeeds or fails.

With streaming, the model may produce 80% of the answer and then disconnect.

Now what?

You need to decide:

Should the partial answer be shown?
Should the UI mark it as incomplete?
Should the user be offered a continuation?
Should the backend retry from scratch?
Should the retry include the partial output?
Should partial structured output be discarded?

For chat products, partial output may still be useful.

For code generation, partial output may be misleading.

For JSON generation, partial output may be invalid and should not be consumed directly.

I like tracking stream state explicitly:

type StreamState = {
  requestId: string;
  startedAt: number;
  receivedTokens: number;
  partialText: string;
  completed: boolean;
  error?: string;
};

Then the app can make different decisions:

if no tokens were received, retry automatically
if some tokens were received, show an incomplete marker
if JSON was expected, discard or repair the output
if tools were involved, stop and ask for confirmation before continuing

A streaming disconnect is not just a network failure.

It is a partial product state.

Model fallback is not just retrying another URL

A common reliability pattern is:

If model A fails, try model B.

That can work, but only if you define safe fallback rules.

Different models may have different:

context windows
pricing
tool calling support
JSON reliability
latency profiles
refusal behavior
output style
reasoning quality

If you silently fall back from a stronger model to a smaller one, your request may "succeed" while the product quality quietly drops.

A better fallback strategy starts with task classes.

const fallbackModels: Record<string, string[]> = {
  high_reasoning: [
    "gpt-4.1",
    "claude-3-5-sonnet",
    "gemini-1.5-pro"
  ],
  fast_chat: [
    "gpt-4.1-mini",
    "claude-3-haiku",
    "gemini-1.5-flash"
  ],
  cheap_batch: [
    "gpt-4.1-mini",
    "gemini-1.5-flash"
  ]
};

Fallback should depend on what the task needs.

For example:

summarization can usually tolerate fallback
legal or financial analysis may not
tool-calling agents need models with compatible tool support
structured output workflows need models that follow schemas reliably
user-selected models should not be silently replaced

If you use an OpenAI-compatible routing layer, this is where it can help. You can centralize fallback and routing behavior instead of scattering provider-specific rules across the codebase.

This is one reason I like using a gateway such as TokenBay for multi-model apps: the API surface stays familiar, but you can still make model choice, fallback, and routing explicit.

Log the fields you will actually need later

For normal APIs, status code, endpoint, latency, and response body may be enough.

For LLM APIs, that is usually not enough.

Useful fields include:

{
  "provider": "openai",
  "model": "gpt-4.1-mini",
  "operation": "support_ticket_summary",
  "status": 429,
  "error_category": "rate_limited",
  "input_tokens_estimated": 18420,
  "output_tokens_received": 0,
  "streaming": false,
  "retry_count": 2,
  "fallback_used": false,
  "latency_ms": 12833
}

These fields help answer the questions you will care about during an incident:

Are failures concentrated on one provider?
Is one model causing most context errors?
Are retries helping or just increasing latency?
Are streaming failures happening after output starts?
Are background jobs consuming too much quota?
Are fallbacks hiding a provider outage?
Are structured output failures tied to one model?

Without this data, you end up debugging LLM reliability with screenshots, anecdotes, and vibes.

I have done that. Not fun.

Build an internal LLM error taxonomy

Instead of handling raw HTTP status codes everywhere, create your own internal error categories.

type LlmErrorCategory =
  | "auth_error"
  | "rate_limited"
  | "quota_exceeded"
  | "context_window_exceeded"
  | "content_refused"
  | "provider_unavailable"
  | "stream_interrupted"
  | "invalid_tool_call"
  | "malformed_structured_output"
  | "unknown";

type LlmError = {
  category: LlmErrorCategory;
  provider: string;
  model: string;
  status?: number;
  retryable: boolean;
  userVisibleMessage: string;
  rawMessage?: string;
};

Then map provider-specific errors into your own categories.

This gives the rest of your product one consistent way to decide:

whether to retry
whether to fallback
what to show the user
what to log
whether to alert the team
whether to pause a workflow

The goal is not to hide HTTP.

The goal is to stop pretending HTTP status codes contain all the operational context your LLM app needs.

A simple handling matrix

Here is a practical starting point.

Error category	Retry?	Fallback?	Product behavior
`auth_error`	No	No	Ask admin to check API key
`quota_exceeded`	No	Maybe	Explain capacity or billing issue
`rate_limited`	Yes, with backoff	Maybe	Queue or show short delay
`context_window_exceeded`	No	Maybe	Summarize, trim, or split task
`provider_unavailable`	Yes	Yes	Retry or switch model
`stream_interrupted`	Depends	Depends	Show incomplete state
`content_refused`	No	No	Ask user to revise request
`malformed_structured_output`	Yes, constrained	Maybe	Repair or regenerate
`invalid_tool_call`	Maybe	No	Validate schema and retry carefully

Your matrix will vary by product, but even a rough version is much better than this:

if (status >= 500 || status === 429) {
  retry();
}

Final thought

LLM APIs use HTTP, but LLM failures are not just HTTP failures.

A 429 might mean slow down, upgrade quota, switch models, queue the job, or stop retrying entirely.

A 400 might mean your app failed to manage context.

A streaming disconnect might mean the user already received half an answer.

A fallback might save the request, or quietly degrade quality.

If you are building anything serious on top of LLMs, create an error layer that understands LLM-specific failure modes.

Your users will see fewer broken states.

Your logs will become easier to reason about.

And your retry logic will stop turning small provider issues into expensive production incidents.

LLM API debugging checklist

plasma — Wed, 01 Jul 2026 06:27:21 +0000

When an LLM feature breaks in production, my first instinct used to be: "the model got worse."

That was usually the wrong place to start.

Most of the painful bugs I've debugged around LLM APIs had nothing to do with model quality. They were boring infrastructure problems hiding behind a model response: missing request metadata, silent timeouts, partial streaming output, retry logic that made things worse, or logs that captured the prompt but not the thing that actually failed.

Here's the checklist I use now before blaming the model.

1. Can I Reproduce the Exact Request?

The first question is simple:

Can I replay the exact same request that failed?

For LLM calls, "exact same" means more than just the user prompt.

I want to capture:

model name
provider
base URL
request timestamp
request ID if available
full message array
temperature / top_p / max_tokens
tool definitions
response format settings
streaming vs non-streaming mode
timeout settings
retry attempt number
user/session/job ID from my own system

The mistake I used to make was logging only the final prompt string. That works for simple demos, but production calls usually depend on system messages, tool schemas, routing logic, and runtime options.

A minimal structured log might look like this:

const requestLog = {
  event: "llm_request_started",
  request_id: crypto.randomUUID(),
  provider: "openai-compatible-provider",
  model: "gpt-4.1-mini",
  stream: true,
  temperature: 0.2,
  max_tokens: 800,
  message_count: messages.length,
  has_tools: tools.length > 0,
  timeout_ms: 30000,
  retry_attempt: 0,
  created_at: new Date().toISOString(),
};

I usually avoid logging raw user content unless I have a clear privacy policy and retention plan. But I do log enough metadata to know what shape of request failed.

2. Did the Request Actually Reach the Provider?

A surprising number of "LLM bugs" are not LLM bugs.

They are:

DNS issues
auth failures
proxy errors
gateway timeouts
invalid base URLs
SDK configuration mistakes
environment variables missing in one deploy target

Before looking at the model response, check whether the request reached the provider at all.

Log the transport layer separately from the model layer:

try {
  const startedAt = Date.now();

  const response = await client.chat.completions.create({
    model,
    messages,
    stream: false,
  });

  console.log({
    event: "llm_request_completed",
    latency_ms: Date.now() - startedAt,
    model,
    response_id: response.id,
    finish_reason: response.choices?.[0]?.finish_reason,
  });
} catch (error) {
  console.error({
    event: "llm_request_failed",
    model,
    error_name: error.name,
    error_message: error.message,
    status: error.status,
    code: error.code,
    type: error.type,
  });

  throw error;
}

The important part is separating:

network failure
HTTP error
provider error
model refusal
empty model output
malformed output
application parsing failure

Those are different problems. They should not all show up as LLM failed.

3. Was the Response Empty, Truncated, or Just Invalid?

There are three very different failure modes that often get mixed together:

The model returned nothing
The model returned a partial answer
The model returned something your app could not parse

They need different fixes.

For normal non-streaming calls, I log:

response ID
finish reason
output length
token usage if available
whether content was empty
whether JSON parsing failed
whether schema validation failed

Example:

const content = response.choices?.[0]?.message?.content ?? "";
const finishReason = response.choices?.[0]?.finish_reason;

console.log({
  event: "llm_response_received",
  response_id: response.id,
  model: response.model,
  finish_reason: finishReason,
  output_chars: content.length,
  prompt_tokens: response.usage?.prompt_tokens,
  completion_tokens: response.usage?.completion_tokens,
  total_tokens: response.usage?.total_tokens,
});

If finish_reason is length, I do not treat that the same as a model quality issue. It usually means my max_tokens value was too low, the prompt asked for too much, or the response format was too verbose.

If JSON parsing fails, I log that as an application-level failure:

try {
  const parsed = JSON.parse(content);
  return parsed;
} catch (error) {
  console.error({
    event: "llm_json_parse_failed",
    model: response.model,
    response_id: response.id,
    finish_reason: finishReason,
    output_preview: content.slice(0, 500),
  });

  throw error;
}

That distinction matters. A provider outage and a JSON parse failure should not trigger the same incident response.

4. If Streaming Failed, How Far Did It Get?

Streaming makes debugging harder because failure can happen after the request has already started successfully.

For streaming calls, I want to know:

did the stream open?
when did the first token arrive?
how many chunks arrived?
how many characters were received?
did the stream end cleanly?
was there a final usage chunk?
did the client disconnect first?
did my server timeout first?
did the provider close the stream?

Here's a simplified example:

let chunkCount = 0;
let outputChars = 0;
let firstChunkAt = null;
const startedAt = Date.now();

try {
  const stream = await client.chat.completions.create({
    model,
    messages,
    stream: true,
  });

  for await (const chunk of stream) {
    chunkCount += 1;

    if (!firstChunkAt) {
      firstChunkAt = Date.now();
    }

    const delta = chunk.choices?.[0]?.delta?.content ?? "";
    outputChars += delta.length;

    // Send delta to the client here
  }

  console.log({
    event: "llm_stream_completed",
    model,
    chunk_count: chunkCount,
    output_chars: outputChars,
    time_to_first_chunk_ms: firstChunkAt ? firstChunkAt - startedAt : null,
    total_latency_ms: Date.now() - startedAt,
  });
} catch (error) {
  console.error({
    event: "llm_stream_failed",
    model,
    chunk_count: chunkCount,
    output_chars: outputChars,
    time_to_first_chunk_ms: firstChunkAt ? firstChunkAt - startedAt : null,
    latency_before_error_ms: Date.now() - startedAt,
    error_name: error.name,
    error_message: error.message,
  });

  throw error;
}

This tells me whether I'm dealing with:

no response at all
slow first token
mid-stream failure
client disconnect
provider timeout
application-side streaming bug

Without this, all I know is "the stream broke," which is not enough.

5. Did My Retry Logic Make It Worse?

Retries are useful until they quietly multiply your problems.

For LLM APIs, I log every retry attempt with:

original request ID
retry attempt number
delay before retry
error that caused the retry
whether the retry used the same model
whether the retry used a fallback model
whether the operation was safe to retry

Example:

console.warn({
  event: "llm_retry_scheduled",
  original_request_id: requestId,
  retry_attempt: attempt + 1,
  delay_ms: delay,
  model,
  reason: error.message,
  status: error.status,
});

The big trap is retrying requests that already caused side effects.

For example, if the LLM call is part of a workflow that sends an email, creates a ticket, charges credits, or writes to a database, retries need idempotency keys and careful boundaries.

Otherwise, a timeout can turn into duplicate actions.

My rule is:

retry transport failures carefully
retry rate limits with backoff
do not blindly retry tool execution
do not retry user-visible side effects without idempotency
always log whether a response came back before retrying

6. Is the Failure Model-Specific or Provider-Specific?

When using OpenAI-compatible APIs, it is easy to swap models or providers and assume everything else is equivalent.

It usually is not.

Different models and providers can vary in:

streaming behavior
tool calling support
JSON mode support
context window limits
rate limit headers
error response shape
usage reporting
timeout behavior
finish reason semantics

So I try to isolate the failure:

same prompt, same provider, different model
same prompt, same model family, different provider if available
same prompt, non-streaming instead of streaming
same prompt, no tools
same prompt, smaller context
same prompt, lower max tokens

This usually tells me whether I'm looking at a model behavior issue, a provider compatibility issue, or my own integration bug.

7. Can I See the Full Timeline?

For production debugging, individual logs are not enough. I want the timeline.

A useful LLM request timeline looks like this:

00ms    request accepted by app
12ms    prompt assembled
18ms    provider selected
25ms    LLM request started
430ms   first token received
2840ms  stream chunk 50 received
5100ms  stream completed
5120ms  output parsed
5190ms  database write completed
5300ms  response sent to client

Or, for a bad request:

00ms     request accepted by app
10ms     prompt assembled
18ms     provider selected
30ms     LLM request started
30030ms  provider timeout
30035ms  retry scheduled
32040ms  retry started
62045ms  retry timeout
62050ms  request failed

This makes the real issue obvious.

Maybe the model was fine, but the timeout was too aggressive. Maybe the first token was fast, but the downstream parser failed. Maybe the app spent 4 seconds building context before the LLM call even started.

Without a timeline, it is too easy to blame the most mysterious part of the stack.

8. Do I Have a Small Test Case?

After I identify the likely failure, I try to reduce it to a tiny test.

For example:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL,
});

const response = await client.chat.completions.create({
  model: process.env.LLM_MODEL,
  messages: [
    { role: "system", content: "Return only valid JSON." },
    { role: "user", content: "Return an object with one key: status." },
  ],
  temperature: 0,
  response_format: { type: "json_object" },
});

console.log(response.choices[0].message.content);

A small test case helps answer:

is my production prompt too complex?
is tool calling the problem?
is JSON mode supported?
is streaming the problem?
is the SDK configured correctly?
is the provider returning the shape I expect?

If the small test fails, the issue is probably integration-level.

If the small test passes, the issue is probably in my application logic, prompt assembly, context size, or output parsing.

9. What I Put in My Default LLM Debug Log

My default log event for LLM calls usually includes this shape:

{
  event: "llm_call",
  request_id: "...",
  user_id: "...",
  provider: "...",
  model: "...",
  stream: true,
  status: "completed",
  latency_ms: 3200,
  time_to_first_token_ms: 420,
  input_tokens: 1200,
  output_tokens: 300,
  finish_reason: "stop",
  retry_attempts: 0,
  error_type: null,
  error_message: null
}

For failures:

{
  event: "llm_call",
  request_id: "...",
  provider: "...",
  model: "...",
  stream: true,
  status: "failed",
  latency_ms: 30000,
  time_to_first_token_ms: null,
  chunks_received: 0,
  output_chars: 0,
  retry_attempts: 2,
  error_type: "timeout",
  error_message: "Request timed out after 30000ms"
}

That is usually enough to start debugging without drowning in logs.

10. The Short Checklist

Before blaming the model, I check:

Can I replay the exact request?
Did the request reach the provider?
Was it a network, HTTP, provider, model, parsing, or app failure?
Was the response empty, truncated, malformed, or refused?
If streaming failed, how many chunks did I receive?
Did the first token arrive?
Did my timeout happen before the provider finished?
Did retry logic duplicate the problem?
Is the issue model-specific or provider-specific?
Does the same request work without streaming?
Does the same request work without tools?
Can I reproduce it with a tiny test case?
Do I have a timeline from app request to final response?

Final Thought

The model is the easy thing to blame because it feels like a black box.

But a lot of LLM production bugs are regular distributed systems problems wearing a very expensive hat: timeouts, retries, partial responses, schema mismatches, bad observability, and unclear ownership between the app, SDK, provider, and model.

The fix is not always a better model.

Sometimes it is just better logs.

I work on TokenBay, so I spend a lot of time thinking about this layer between apps and model providers. The more models and providers you route through, the more valuable boring debugging discipline becomes.

What I Log When an LLM API Call Fails Mid-Stream

plasma — Tue, 30 Jun 2026 07:13:48 +0000

Streaming looks great in demos.

The UI feels faster, users see tokens immediately, and nobody has to stare at a spinner while the model thinks.

Then production happens.

A request starts fine. The first few tokens arrive. Maybe half the answer is already on screen. Then the stream dies.

No clean final response. No full usage object. Sometimes no helpful error body. Just a broken connection, a timeout, or a user asking why the answer stopped in the middle.

That was the point where I realized my normal API logging was not enough.

For regular HTTP APIs, I used to care about simple things:

request started
request finished
status code
latency
error message

For streaming LLM APIs, that misses the most important part: how far the request got before it failed.

Why mid-stream failures are different

A non-streaming LLM call usually has a simple shape:

send request
wait
receive complete response or error

A streaming call has more states:

send request
wait for headers
wait for first token
receive chunks
maybe receive tool calls
maybe receive usage
maybe receive a final done event
maybe fail somewhere in the middle

That means "request failed" is too vague.

These are very different failures:

the provider never accepted the request
the provider accepted it but never sent the first token
the stream started and then disconnected
the model generated partial text but never sent a final event
tool call JSON started but was cut off halfway
the client closed the connection
my own server timed out while the provider was still streaming

If all of those show up as 500 stream failed, debugging becomes guesswork.

The minimum fields I log

I try not to log user prompts or raw model outputs unless there is a very explicit reason and consent path.

But I still want enough metadata to debug failures.

This is the baseline event I want for every LLM request:

{
  "request_id": "req_01J...",
  "provider": "example-provider",
  "model": "example-model",
  "operation": "chat.completions.stream",
  "stream": true,
  "started_at": "2026-06-30T10:12:41.120Z",
  "status": "failed",
  "failure_stage": "after_first_token",
  "error_type": "stream_interrupted",
  "http_status": 200,
  "time_to_first_token_ms": 842,
  "stream_duration_ms": 4810,
  "chunks_received": 37,
  "output_chars_received": 1842,
  "finish_reason": null,
  "usage_available": false
}

The most important field here is failure_stage.

Without that, the rest is just noise.

The failure stages I use

I like to classify streaming failures by where they happened.

type StreamFailureStage =
  | "before_request"
  | "request_rejected"
  | "before_first_token"
  | "after_first_token"
  | "during_tool_call"
  | "after_finish_reason"
  | "client_aborted"
  | "unknown";

Here is how I think about each one.

before_request

The request failed before it reached the provider.

Common causes:

invalid local config
missing API key
bad base URL
serialization error
invalid request body built by my own app

This is usually my bug, not the provider's.

I log:

provider
model
route or feature name
config version
validation error
whether the API key was present, never the key itself

request_rejected

The provider returned a normal error before the stream started.

Common causes:

401 invalid key
403 forbidden model
404 model not found
429 rate limit
400 invalid tool schema
context length exceeded

I log:

HTTP status
provider error code
provider error type
normalized error category
retryable or not
request size metadata

Example:

{
  "failure_stage": "request_rejected",
  "http_status": 429,
  "error_type": "rate_limit",
  "retryable": true,
  "input_tokens_estimated": 1200
}

This is the cleanest kind of failure. Annoying, but at least the provider told you what happened.

before_first_token

This is where a lot of streaming pain starts.

The provider accepted the request, but no token arrived before the timeout.

I log:

time to headers
first-token timeout
model
provider
region if available
whether this was a fallback attempt
whether the request was retried

Example:

{
  "failure_stage": "before_first_token",
  "error_type": "first_token_timeout",
  "time_to_headers_ms": 210,
  "first_token_timeout_ms": 15000,
  "chunks_received": 0
}

This is different from a normal request timeout.

For streaming, I usually want separate timeout budgets:

connection timeout
headers timeout
first token timeout
idle stream timeout
total stream timeout

One giant timeout: 60000 value is usually too blunt.

after_first_token

This is the classic mid-stream failure.

The model started responding, the user saw something, and then the stream broke.

I log:

time to first token
chunks received
output characters received
stream duration
last chunk timestamp
whether a final done event arrived
whether finish reason was present
whether usage was present

Example:

{
  "failure_stage": "after_first_token",
  "error_type": "stream_interrupted",
  "time_to_first_token_ms": 730,
  "stream_duration_ms": 6200,
  "chunks_received": 58,
  "output_chars_received": 2410,
  "done_event_received": false,
  "finish_reason": null,
  "usage_available": false
}

This tells me the request was not simply "slow" or "rejected".

It was accepted, generated partial output, and then the stream was interrupted.

That matters for UX too. You probably do not want to show the same message as a normal failure.

during_tool_call

Tool calls make stream logging more interesting.

Sometimes the model starts emitting tool call arguments and the stream fails halfway through the JSON.

If I only log "stream interrupted", I miss the actual risk: I might have an incomplete tool call that should never be executed.

For tool calls, I log:

tool call started
tool name if available
argument JSON completed or not
tool call id if available
whether tool execution started

Example:

{
  "failure_stage": "during_tool_call",
  "error_type": "incomplete_tool_arguments",
  "tool_name": "create_invoice",
  "tool_arguments_completed": false,
  "tool_execution_started": false
}

The key rule in my apps is simple:

Never execute a tool call unless the tool arguments are complete and validated.

A half-streamed JSON object is not an instruction. It is debris.

after_finish_reason

This one is subtle.

Sometimes the model sends a finish reason, but the client still fails before all cleanup logic runs.

For example:

the stream ended but usage metadata was missing
the final parser failed
my server failed while saving the assistant message
the client disconnected right after completion

I log:

finish reason
final event received
usage available
persistence status
response delivered to client or not

Example:

{
  "failure_stage": "after_finish_reason",
  "finish_reason": "stop",
  "usage_available": false,
  "message_persisted": false,
  "client_delivery_status": "unknown"
}

This helps separate model/provider failures from my own application failures.

client_aborted

Sometimes the user closes the tab. Sometimes the browser cancels the request. Sometimes a reverse proxy drops the connection.

That is not the same as the provider failing.

I log:

client abort timestamp
provider stream still active or not
generated tokens/chunks before abort
whether billing metadata was received
whether the server cancelled the upstream request

Example:

{
  "failure_stage": "client_aborted",
  "stream_duration_ms": 3100,
  "chunks_received": 22,
  "upstream_cancelled": true
}

This matters because client aborts can distort reliability metrics.

If 30% of your "failed" streams are just users navigating away, your provider is not necessarily the problem.

A small Node.js example

Here is a simplified wrapper around a streaming call.

This example does not depend on a specific vendor beyond using an OpenAI-compatible chat completions endpoint.

type StreamLog = {
  requestId: string;
  provider: string;
  model: string;
  startedAt: string;
  status: "ok" | "failed";
  failureStage?: string;
  errorType?: string;
  httpStatus?: number;
  timeToFirstTokenMs?: number;
  streamDurationMs?: number;
  chunksReceived: number;
  outputCharsReceived: number;
  doneEventReceived: boolean;
  finishReason?: string | null;
};

function nowMs() {
  return Date.now();
}

function classifyError(error: unknown) {
  if (error instanceof Error && error.name === "AbortError") {
    return "client_or_timeout_abort";
  }

  if (error instanceof Error && error.message.includes("timeout")) {
    return "timeout";
  }

  return "unknown_stream_error";
}

export async function streamChatCompletion({
  requestId,
  apiKey,
  baseUrl,
  model,
  messages,
  onToken
}: {
  requestId: string;
  apiKey: string;
  baseUrl: string;
  model: string;
  messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
  onToken: (token: string) => void;
}) {
  const started = nowMs();

  const log: StreamLog = {
    requestId,
    provider: new URL(baseUrl).hostname,
    model,
    startedAt: new Date().toISOString(),
    status: "ok",
    chunksReceived: 0,
    outputCharsReceived: 0,
    doneEventReceived: false,
    finishReason: null
  };

  let firstTokenAt: number | null = null;

  try {
    const response = await fetch(`${baseUrl}/chat/completions`, {
      method: "POST",
      headers: {
        "authorization": `Bearer ${apiKey}`,
        "content-type": "application/json"
      },
      body: JSON.stringify({
        model,
        messages,
        stream: true
      })
    });

    log.httpStatus = response.status;

    if (!response.ok || !response.body) {
      log.status = "failed";
      log.failureStage = "request_rejected";
      log.errorType = `http_${response.status}`;
      return log;
    }

    const reader = response.body.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { value, done } = await reader.read();

      if (done) {
        log.doneEventReceived = true;
        break;
      }

      const chunk = decoder.decode(value, { stream: true });
      log.chunksReceived += 1;

      for (const line of chunk.split("\n")) {
        const trimmed = line.trim();

        if (!trimmed.startsWith("data:")) continue;

        const data = trimmed.slice("data:".length).trim();

        if (data === "[DONE]") {
          log.doneEventReceived = true;
          continue;
        }

        try {
          const parsed = JSON.parse(data);
          const token = parsed.choices?.[0]?.delta?.content ?? "";
          const finishReason = parsed.choices?.[0]?.finish_reason ?? null;

          if (finishReason) {
            log.finishReason = finishReason;
          }

          if (token) {
            if (firstTokenAt === null) {
              firstTokenAt = nowMs();
              log.timeToFirstTokenMs = firstTokenAt - started;
            }

            log.outputCharsReceived += token.length;
            onToken(token);
          }
        } catch {
          log.status = "failed";
          log.failureStage = firstTokenAt === null
            ? "before_first_token"
            : "after_first_token";
          log.errorType = "invalid_stream_chunk";
          return log;
        }
      }
    }

    log.streamDurationMs = nowMs() - started;

    if (!log.doneEventReceived) {
      log.status = "failed";
      log.failureStage = firstTokenAt === null
        ? "before_first_token"
        : "after_first_token";
      log.errorType = "missing_done_event";
    }

    return log;
  } catch (error) {
    log.status = "failed";
    log.streamDurationMs = nowMs() - started;
    log.failureStage = firstTokenAt === null
      ? "before_first_token"
      : "after_first_token";
    log.errorType = classifyError(error);
    return log;
  }
}

This is not a complete production wrapper. It does not handle every SSE edge case, retry policy, proxy timeout, or tool call format.

But the shape is useful:

track whether the first token arrived
count chunks
track whether the final event arrived
separate request rejection from stream interruption
return structured logs instead of one generic error

What I avoid logging

The tempting thing is to log everything.

Raw prompt. Raw completion. Full request body. Full streamed chunks.

That makes debugging easy, but it creates a different problem.

For most production apps, I try to avoid logging:

raw user prompts
raw assistant responses
API keys or auth headers
full tool arguments if they may contain user data
uploaded file contents
personal identifiers unless explicitly needed

Instead, I prefer metadata:

prompt token estimate
output character count
message count
model name
feature name
latency buckets
error category
retry count
request id
user/account id only if your privacy policy allows it

You can debug a surprising amount without storing the actual prompt.

The dashboard I want

Once these fields are logged, the useful charts become obvious.

I want to see:

failure rate by provider
failure rate by model
failures before first token vs after first token
p50/p95 time to first token
average chunks before interruption
stream interruptions by route or feature
retry success rate
client abort rate
missing usage rate

That last one matters more than people expect.

If usage metadata is missing after failed streams, your cost dashboard can lie. Not always dramatically, but enough to make optimization work fuzzy.

Where OpenAI-compatible APIs get tricky

OpenAI-compatible APIs are great for switching providers, but compatibility does not mean every streaming behavior is identical.

In practice, I still check:

does the provider send [DONE]?
is finish_reason always present?
is usage included for streamed responses?
how are provider errors formatted?
what happens when the upstream model times out?
are tool calls streamed in the same structure?
does the SDK surface stream errors clearly?

This is one reason I like keeping my logging format provider-neutral.

Whether I am using a direct provider API or routing through something like TokenBay, I want my app logs to answer the same question:

Where did the stream fail, and how much work happened before it failed?

My current checklist

For every streamed LLM call, I want to know:

Did the provider reject the request?
Did we receive the first token?
How long did first token take?
How many chunks arrived?
Did the stream finish cleanly?
Did we get a finish reason?
Did we get usage metadata?
Was the failure caused by the provider, my app, or the client?
Is this safe to retry?
Did a tool call start?
Did a tool call complete?
Did a tool execute?

If my logs can answer those questions, debugging becomes much less painful.

Not effortless. This is still LLM infrastructure, after all.

But at least I am not staring at stream failed and pretending that is observability.

The LLM API Timeout Playbook I Wish I Had Before Production

plasma — Mon, 29 Jun 2026 07:58:20 +0000

The first time an LLM API timeout hurt us in production, I blamed the provider.

That was emotionally satisfying for about five minutes.

Then I looked at our own code and realized we had done the usual thing: one request, one timeout value, one retry policy, one optimistic assumption that every model call would behave like a normal REST API.

LLM APIs do not behave like normal REST APIs.

They can be slow because the model is busy, because your prompt is huge, because streaming is misconfigured, because the provider is degraded, because your own network path is weird, or because you asked a reasoning model to do something that takes real compute.

This is the timeout playbook I wish I had before shipping LLM calls into production.

The Problem

Most teams start with something like this:

const response = await fetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Summarize this document..." }],
  }),
});

It works in local testing.

Then production happens.

A few requests take 30 seconds. Some hang until the platform kills the request. Users refresh the page. Your queue backs up. Retries accidentally double your cost. Logs show "timeout" but not why.

The mistake is treating "timeout" as one problem.

In production, there are at least five different timeout problems:

connection timeout
first-token timeout
total request timeout
idle stream timeout
background job timeout

Each one needs a different response.

Timeout Types You Actually Need

1. Connection Timeout

This is how long you are willing to wait to establish the HTTP connection.

If this fails, the provider probably never received the request body. Retrying is usually safe.

Typical range: 2-5 seconds.

2. First-Token Timeout

For streaming requests, this is how long you wait before the first token arrives.

This is the timeout users feel most directly. If the UI is blank for 20 seconds, people assume the app is broken.

Typical range: 5-15 seconds, depending on model and task.

3. Total Request Timeout

This is the maximum time the entire request is allowed to run.

For short chat completions, maybe 20-60 seconds. For long analysis, code generation, or reasoning-heavy tasks, this may need to be much higher.

Typical range: 30-180 seconds.

4. Idle Stream Timeout

For streaming responses, this is how long you wait between chunks.

The request may have started successfully, but if no tokens arrive for a while, you need to decide whether to keep waiting or abort.

Typical range: 10-30 seconds.

5. Job Timeout

If the LLM call is part of a background workflow, the job itself needs a timeout too.

This protects your queue. Even if the HTTP client behaves badly, your worker should not be stuck forever.

Typical range: depends on the workflow, but always set one.

A Minimal Timeout Wrapper

Here is a small Node.js wrapper using AbortController.

It works with any OpenAI-compatible chat completions endpoint. Set LLM_BASE_URL to your provider's base URL and LLM_API_KEY to your key.

// timeout-chat.js
const DEFAULT_BASE_URL = "https://api.openai.com/v1";

class LLMTimeoutError extends Error {
  constructor(message, details = {}) {
    super(message);
    this.name = "LLMTimeoutError";
    this.details = details;
  }
}

async function chatCompletion({
  baseUrl = process.env.LLM_BASE_URL || DEFAULT_BASE_URL,
  apiKey = process.env.LLM_API_KEY,
  model = "gpt-4o-mini",
  messages,
  timeoutMs = 45_000,
}) {
  if (!apiKey) {
    throw new Error("Missing LLM_API_KEY");
  }

  const controller = new AbortController();
  const timeout = setTimeout(() => {
    controller.abort();
  }, timeoutMs);

  const startedAt = Date.now();

  try {
    const response = await fetch(`${baseUrl}/chat/completions`, {
      method: "POST",
      signal: controller.signal,
      headers: {
        "Authorization": `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model,
        messages,
        temperature: 0.2,
      }),
    });

    const elapsedMs = Date.now() - startedAt;

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(
        `LLM request failed: ${response.status} ${response.statusText} after ${elapsedMs}ms\n${errorText}`
      );
    }

    const data = await response.json();

    return {
      data,
      elapsedMs,
    };
  } catch (error) {
    const elapsedMs = Date.now() - startedAt;

    if (error.name === "AbortError") {
      throw new LLMTimeoutError(`LLM request timed out after ${elapsedMs}ms`, {
        timeoutMs,
        elapsedMs,
        model,
      });
    }

    throw error;
  } finally {
    clearTimeout(timeout);
  }
}

async function main() {
  const result = await chatCompletion({
    messages: [
      {
        role: "user",
        content: "Give me a 5-bullet checklist for debugging LLM API timeouts.",
      },
    ],
    timeoutMs: 30_000,
  });

  console.log(result.data.choices[0].message.content);
  console.log(`Completed in ${result.elapsedMs}ms`);
}

main().catch((error) => {
  console.error(error);
  process.exit(1);
});

Run it:

export LLM_API_KEY="your_api_key"
export LLM_BASE_URL="https://api.openai.com/v1"
node timeout-chat.js

If you use an OpenAI-compatible gateway, change only the base URL and model name.

That small abstraction already gives you three useful things:

every request has a deadline
timeout errors are distinguishable from provider errors
logs include model and elapsed time

But this is still only the first layer.

Do Not Retry Every Timeout

The most expensive timeout bug is the blind retry.

Something times out at 45 seconds, your app retries, the retry also runs for 45 seconds, and now you have doubled latency and maybe doubled spend.

Before retrying, ask one question:

Did the provider probably receive and start processing the request?

If the answer is "probably no," retrying is reasonable.

If the answer is "probably yes," retrying may create duplicate work.

A practical retry policy:

function shouldRetryLLMError(error) {
  if (error.name === "LLMTimeoutError") {
    const elapsed = error.details?.elapsedMs ?? 0;

    // Very short failures are often connection/network issues.
    // Longer timeouts may mean the model was already processing.
    return elapsed < 5_000;
  }

  const message = String(error.message || "");

  if (message.includes("429")) return true;
  if (message.includes("500")) return true;
  if (message.includes("502")) return true;
  if (message.includes("503")) return true;
  if (message.includes("504")) return true;

  return false;
}

This is intentionally conservative.

For user-facing chat, I would rather show a clear failure than silently retry a long-running request and make the user wait 90 seconds.

For background jobs, I may retry more aggressively, but only with idempotency and backoff.

Add Backoff With Jitter

If a provider is overloaded, retrying immediately is the worst possible move.

Use exponential backoff with jitter:

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function backoffMs(attempt) {
  const base = 500 * Math.pow(2, attempt);
  const jitter = Math.floor(Math.random() * 250);
  return Math.min(base + jitter, 8_000);
}

async function withRetries(fn, { maxRetries = 2 } = {}) {
  let lastError;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;

      if (attempt === maxRetries || !shouldRetryLLMError(error)) {
        throw error;
      }

      await sleep(backoffMs(attempt));
    }
  }

  throw lastError;
}

Usage:

const result = await withRetries(() =>
  chatCompletion({
    messages: [{ role: "user", content: "Explain API retries in one paragraph." }],
    timeoutMs: 30_000,
  })
);

For production, also log every retry attempt. A retry that saves a user request is good. A retry policy that hides provider degradation is not.

Stream When Humans Are Waiting

For user-facing requests, streaming is usually better than waiting for the full response.

Streaming does not make the model faster, but it makes the experience feel alive. More importantly, it lets you measure first-token latency separately from total latency.

Here is a minimal streaming example:

// stream-chat.js
const baseUrl = process.env.LLM_BASE_URL || "https://api.openai.com/v1";
const apiKey = process.env.LLM_API_KEY;

if (!apiKey) {
  throw new Error("Missing LLM_API_KEY");
}

const controller = new AbortController();
const totalTimeout = setTimeout(() => controller.abort(), 60_000);

const startedAt = Date.now();
let firstChunkAt = null;

try {
  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: "POST",
    signal: controller.signal,
    headers: {
      "Authorization": `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: process.env.LLM_MODEL || "gpt-4o-mini",
      stream: true,
      messages: [
        {
          role: "user",
          content: "Write a short explanation of first-token latency.",
        },
      ],
    }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`Streaming request failed: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();

    if (done) break;

    if (!firstChunkAt) {
      firstChunkAt = Date.now();
      console.error(`First chunk in ${firstChunkAt - startedAt}ms`);
    }

    const chunk = decoder.decode(value, { stream: true });
    process.stdout.write(chunk);
  }

  console.error(`\nTotal time: ${Date.now() - startedAt}ms`);
} finally {
  clearTimeout(totalTimeout);
}

Run it:

export LLM_API_KEY="your_api_key"
node stream-chat.js

A production streaming implementation should parse Server-Sent Events properly instead of printing raw chunks, but the point is the measurement:

time to first byte
time to first token
total completion time
stream interruptions
idle gaps between chunks

Those numbers tell you much more than "request timed out."

Set Different Timeouts By Task

One timeout value for every LLM call is usually wrong.

A better pattern is to define timeout classes:

const TIMEOUTS = {
  autocomplete: 8_000,
  chat: 45_000,
  documentSummary: 120_000,
  backgroundExtraction: 180_000,
};

Then make the caller choose intentionally:

await chatCompletion({
  model: "gpt-4o-mini",
  messages,
  timeoutMs: TIMEOUTS.chat,
});

This forces the product decision into code.

If a feature needs a 180-second timeout, it probably should not block an HTTP request from the browser. Put it in a job queue and notify the user when it is done.

Use Fallbacks Carefully

Fallbacks are useful, but they can also make debugging harder.

A fallback from one model to another may change:

output quality
latency
cost
context window
tool-calling behavior
JSON reliability

So fallback should be explicit.

Example:

const MODEL_CHAIN = [
  { model: "primary-model", timeoutMs: 45_000 },
  { model: "fallback-model", timeoutMs: 30_000 },
];

async function runWithFallback(messages) {
  const errors = [];

  for (const option of MODEL_CHAIN) {
    try {
      return await chatCompletion({
        model: option.model,
        messages,
        timeoutMs: option.timeoutMs,
      });
    } catch (error) {
      errors.push({
        model: option.model,
        name: error.name,
        message: error.message,
      });
    }
  }

  throw new Error(`All model attempts failed: ${JSON.stringify(errors)}`);
}

Do not silently fallback on every request without tracking it.

If your fallback rate jumps from 2% to 25%, that is an incident signal.

Log The Right Fields

This is the minimum I want in logs for every LLM request:

{
  "event": "llm_request_completed",
  "provider": "openai-compatible",
  "model": "gpt-4o-mini",
  "timeout_ms": 45000,
  "elapsed_ms": 12842,
  "streaming": false,
  "retry_count": 0,
  "fallback_used": false,
  "status": "success"
}

For failures:

{
  "event": "llm_request_failed",
  "provider": "openai-compatible",
  "model": "gpt-4o-mini",
  "timeout_ms": 45000,
  "elapsed_ms": 45021,
  "streaming": false,
  "retry_count": 1,
  "fallback_used": false,
  "status": "timeout",
  "error_type": "LLMTimeoutError"
}

Do not log raw prompts by default.

Prompts can contain user data, private business context, credentials, internal code, or documents. If you need prompt logging for debugging, put it behind explicit controls and retention rules.

My Production Checklist

Before I ship an LLM API call now, I check:

Does this request have a total timeout?
Is the timeout appropriate for this task?
Is there a separate first-token timeout for streaming UX?
Do retries use backoff and jitter?
Are long timeouts moved to background jobs?
Are timeout errors distinguishable from provider errors?
Do logs include model, elapsed time, retry count, and fallback status?
Is fallback behavior explicit and observable?
Are prompts and responses excluded from default logs?
Does the UI tell the user what happened when a timeout occurs?

The last one matters more than people think.

A good timeout message is not:

Something went wrong.

A better one is:

The model took too long to respond. You can try again, or shorten the input and run it again.

That gives the user a next move.

Where Gateways Help

One reason teams use OpenAI-compatible gateways is operational flexibility.

If your app talks to one OpenAI-style interface, you can route across different models or providers without rewriting every integration. That does not remove the need for timeouts, retries, logging, and fallbacks. It just makes those controls easier to centralize.

At TokenBay, this is one of the problems we care about: giving developers one API key and one OpenAI-compatible interface across multiple model providers.

But gateway or no gateway, the production rule is the same:

Do not let LLM calls be mysterious black boxes.

Give them deadlines. Measure them. Retry carefully. Stream when humans are waiting. Move long work into jobs. Make fallbacks visible.

Your future incident review will be much shorter.

My LLM API Calls Were Failing Silently. Here's the Logging Setup I Wish I Had Earlier

plasma — Fri, 26 Jun 2026 05:53:05 +0000

The first few LLM API bugs I hit in production were easy to notice.

The request failed. The user saw an error. I opened the logs, found the stack trace, fixed the obvious thing, and moved on.

The harder bugs were quieter.

The API still returned a response, but it was slower than usual. A fallback model kicked in without anyone noticing. Token usage crept up over a few days. A retry made the request succeed, but doubled the latency. Streaming worked most of the time, except when it didn't.

Nothing looked "down." The app just started feeling worse.

That was when I realized my LLM logging was too thin.

I was logging errors, but not enough context to understand behavior.

The problem with normal API logs

For a typical REST API call, I might log:

request path
status code
latency
error message
user ID or request ID

That is useful, but LLM calls have a few extra dimensions.

A successful LLM request can still be a problem if:

it used the wrong model
it silently retried
it fell back to another model
it returned fewer tokens than expected
it took 18 seconds instead of 2
it streamed partially, then stopped
it cost more than the normal path
it failed only for long prompts
it failed only for tool calling or JSON mode

If all I log is status: 200, I miss almost everything that matters.

What I log for every LLM call

This is the basic shape I try to capture now:

{
  "event": "llm_request",
  "request_id": "req_123",
  "provider": "tokenbay",
  "model": "gpt-4.1-mini",
  "operation": "chat_completion",
  "status": "success",
  "latency_ms": 1842,
  "input_tokens": 812,
  "output_tokens": 244,
  "estimated_cost_usd": 0.0019,
  "retry_count": 0,
  "fallback_from": null,
  "fallback_to": null,
  "streaming": false,
  "error_type": null,
  "error_message": null
}

For failed requests:

{
  "event": "llm_request",
  "request_id": "req_124",
  "provider": "tokenbay",
  "model": "some-model",
  "operation": "chat_completion",
  "status": "error",
  "latency_ms": 5000,
  "input_tokens": null,
  "output_tokens": null,
  "estimated_cost_usd": null,
  "retry_count": 2,
  "fallback_from": "some-model",
  "fallback_to": "backup-model",
  "streaming": false,
  "error_type": "rate_limit",
  "error_message": "Rate limit exceeded"
}

The exact fields depend on your app, but the categories matter more than the names.

I want to know:

what model I asked for
what provider handled it
how long it took
whether retries happened
whether fallback happened
how many tokens were used
whether the request streamed
what kind of failure happened
roughly how much the request cost

That is the difference between "the AI feature feels slow today" and "requests to model X are retrying twice after 429s, then falling back to model Y."

A small Node.js wrapper

Here is a simple version using the OpenAI SDK.

It works with OpenAI directly, or with any OpenAI-compatible endpoint by changing baseURL.

Install:

npm install openai

Create llm-client.js:

import OpenAI from "openai";
import crypto from "node:crypto";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL || "https://api.openai.com/v1"
});

function nowMs() {
  return Number(process.hrtime.bigint() / 1000000n);
}

function promptHash(messages) {
  const text = JSON.stringify(messages);
  return crypto.createHash("sha256").update(text).digest("hex").slice(0, 16);
}

function classifyError(error) {
  const status = error?.status;

  if (status === 400) return "invalid_request";
  if (status === 401 || status === 403) return "auth_or_permission";
  if (status === 413) return "request_too_large";
  if (status === 429) return "rate_limit";
  if (status === 503) return "service_unavailable";
  if (status === 504) return "upstream_timeout";
  if (status >= 500) return "provider_5xx";

  const message = String(error?.message || "").toLowerCase();

  if (message.includes("context length")) return "context_length";
  if (message.includes("timeout")) return "timeout";
  if (message.includes("content filter")) return "content_filter";

  return "unknown";
}

function logLLMEvent(event) {
  console.log(JSON.stringify(event));
}

export async function createLoggedChatCompletion({
  requestId,
  provider = "default",
  model,
  messages,
  temperature = 0.2,
  maxTokens = 500,
  streaming = false
}) {
  const startedAt = nowMs();

  const baseEvent = {
    event: "llm_request",
    request_id: requestId,
    provider,
    model,
    operation: "chat_completion",
    prompt_hash: promptHash(messages),
    streaming,
    retry_count: 0,
    fallback_from: null,
    fallback_to: null
  };

  try {
    const response = await client.chat.completions.create({
      model,
      messages,
      temperature,
      max_tokens: maxTokens,
      stream: streaming
    });

    const latencyMs = nowMs() - startedAt;

    if (streaming) {
      logLLMEvent({
        ...baseEvent,
        status: "success",
        latency_ms: latencyMs,
        input_tokens: null,
        output_tokens: null,
        estimated_cost_usd: null,
        error_type: null,
        error_message: null
      });

      return response;
    }

    logLLMEvent({
      ...baseEvent,
      status: "success",
      latency_ms: latencyMs,
      input_tokens: response.usage?.prompt_tokens ?? null,
      output_tokens: response.usage?.completion_tokens ?? null,
      estimated_cost_usd: null,
      error_type: null,
      error_message: null
    });

    return response;
  } catch (error) {
    const latencyMs = nowMs() - startedAt;

    logLLMEvent({
      ...baseEvent,
      status: "error",
      latency_ms: latencyMs,
      input_tokens: null,
      output_tokens: null,
      estimated_cost_usd: null,
      error_type: classifyError(error),
      error_message: error?.message || "Unknown error"
    });

    throw error;
  }
}

Use it like this:

import crypto from "node:crypto";
import { createLoggedChatCompletion } from "./llm-client.js";

const response = await createLoggedChatCompletion({
  requestId: crypto.randomUUID(),
  provider: "openai-compatible",
  model: "gpt-4.1-mini",
  messages: [
    {
      role: "user",
      content: "Explain retries and exponential backoff in one paragraph."
    }
  ]
});

console.log(response.choices[0].message.content);

Run it:

LLM_API_KEY="your-api-key" node app.js

If you use TokenBay, the OpenAI-compatible base URL is:

LLM_API_KEY="your-tokenbay-api-key" \
LLM_BASE_URL="https://api.tokenbay.com/v1" \
node app.js

Same SDK shape. Different base URL.

Do not log raw prompts by default

This part matters.

It is tempting to log the full prompt because it makes debugging easier. I try not to do that by default.

Prompts can contain:

user messages
emails
names
customer data
internal business logic
secrets that users accidentally pasted
private documents

Instead, I usually log a hash of the prompt and a few safe metadata fields:

{
  "prompt_hash": "a3f9c01de81b7a22",
  "message_count": 4,
  "has_system_prompt": true,
  "input_chars": 3821
}

That lets me group repeated failures without storing the actual content.

For local development, raw prompt logging can be useful. For production, I want it behind a very explicit flag, with retention rules and access control.

Provider logs are not enough

Provider-side usage logs are useful.

For example, TokenBay's Usage Logs page can show request-level details such as time, model, token count, and cost.

That is helpful, especially when you are using multiple models through one OpenAI-compatible API.

But provider logs usually do not know your application context.

They do not know that this request came from your support reply generator, or that the user had already waited through two failed attempts, or that the answer was discarded before being shown.

That is why I still keep app-side logs.

The provider can tell me what happened at the API layer.

My app logs tell me why it mattered.

The fields that helped me most

Some fields looked boring at first, but ended up being the most useful.

`model`

This sounds obvious until you have multiple models in production.

If your app can use GPT, Claude, Gemini, Qwen, DeepSeek, GLM, or smaller fallback models, you need to know which one actually handled the request.

Not which one the product team thinks is configured.

The actual model.

`provider`

This matters when using multiple vendors or an OpenAI-compatible API gateway.

The same model name can behave differently depending on the provider, gateway, account limits, or routing setup.

If latency spikes, I want to know whether it is model-specific or provider-specific.

`latency_ms`

Average latency is not enough.

I usually want p50, p95, and p99 by model and operation.

A chatbot can feel fine at p50 and awful at p95.

`retry_count`

Retries are sneaky.

They make reliability look better while quietly increasing latency and cost.

If a request succeeds after two retries, the user may not see an error, but the system still degraded.

`fallback_from` and `fallback_to`

Fallback is great until it hides the original problem.

If model A fails and model B saves the request, that is useful. But if it happens 30 percent of the time, I need to know.

Otherwise I might think model A is working fine.

`input_tokens` and `output_tokens`

Token usage explains a lot of cost surprises.

When a bill jumps, the cause is often not "the provider got expensive." It is more likely:

prompts got longer
retrieved context got larger
output limits were too high
retries increased
a more expensive model handled more traffic
tool calls caused extra rounds

You cannot see that from request count alone.

`error_type`

Raw error messages are messy.

One provider says rate_limit_exceeded. Another says Too many requests. Another gives you a 429 with a different body.

I normalize errors into categories:

const errorTypes = [
  "auth_or_permission",
  "invalid_request",
  "rate_limit",
  "request_too_large",
  "context_length",
  "content_filter",
  "provider_5xx",
  "service_unavailable",
  "upstream_timeout",
  "stream_interrupted",
  "unknown"
];

This makes dashboards and alerts much easier.

Silent failures I now watch for

The worst failures are not always exceptions.

These are the ones I try to catch with logs and metrics:

1. Retry storms

A provider starts returning intermittent 429s, 503s, or 504s.

Your retry logic hides it.

The app still works, but latency doubles and costs rise.

Watch:

retry count by provider
retry count by model
p95 latency after retries
final status after retry

2. Fallback becoming the main path

Fallback should be the backup plan.

If fallback becomes normal, you may have a provider issue, a bad timeout setting, or a model that is no longer suitable.

Watch:

fallback rate
fallback source model
fallback target model
quality complaints after fallback

3. Token creep

This is when prompts slowly get larger over time.

Maybe you added more retrieved documents. Maybe the system prompt grew. Maybe conversation history is not being trimmed.

Nothing breaks immediately. The bill just gets heavier.

Watch:

average input tokens by feature
p95 input tokens by route
output tokens by model
token usage per customer or workspace

4. Streaming interruptions

Streaming can fail differently from normal responses.

Sometimes the first tokens arrive, then the stream stops. If you only log the initial request success, you miss the failure.

Watch:

stream started
stream completed
stream duration
chunks received
interruption reason

5. Model mismatch

This happens when config changes, environment variables drift, or a gateway route points somewhere unexpected.

The app asks for one model, but production traffic goes somewhere else.

Watch:

requested model
resolved model, if available
provider
deployment environment

A slightly better log event

After a few rounds, my log event usually grows into something like this:

{
  "event": "llm_request",
  "timestamp": "2026-06-26T08:30:00.000Z",
  "request_id": "req_abc",
  "user_id_hash": "user_91ab",
  "environment": "production",
  "feature": "support_reply_generator",
  "provider": "tokenbay",
  "model": "gpt-4.1-mini",
  "operation": "chat_completion",
  "streaming": false,
  "status": "success",
  "latency_ms": 1842,
  "retry_count": 1,
  "fallback_from": null,
  "fallback_to": null,
  "input_tokens": 812,
  "output_tokens": 244,
  "estimated_cost_usd": 0.0019,
  "prompt_hash": "a3f9c01de81b7a22",
  "error_type": null
}

This is not fancy observability.

It is just enough structure to answer practical questions.

Which feature got slower?

Which model is causing errors?

Did fallback save us or hide a bigger issue?

Did the cost increase because of traffic, tokens, retries, or model choice?

Where TokenBay fits into this

Disclosure: I work on TokenBay, so I am biased here.

One reason I care about this logging shape is that TokenBay is built around using multiple AI models through one OpenAI-compatible API.

That makes it convenient to switch between models, but it also makes observability more important.

TokenBay can show usage details at the API layer. I still want my own application logs because my app knows things the API layer cannot always know:

which product feature triggered the request
whether the answer was shown to the user
whether the request was part of a retry chain
whether fallback was intentional
whether the user abandoned the flow before the model answered

The more flexible your model setup becomes, the more important boring logs become.

My current rule

For every production LLM call, I want enough information to debug four questions:

Why did this request fail?
Why was this request slow?
Why did this request cost more than expected?
Which model actually answered?

If my logs cannot answer those, I am probably flying blind.

The annoying part is that you usually do not notice this on day one.

You notice it later, when something is already weird and your only log line says:

LLM request completed

Ask me how I know.

OpenAI-Compatible APIs Are Great Until Streaming Breaks: What I Check Before Switching Providers

plasma — Thu, 25 Jun 2026 03:48:43 +0000

Swapping an AI provider looks easy on paper.

Change the baseURL, keep the OpenAI SDK, point your app at a different model, and you're done.

And honestly, for basic non-streaming chat completions, that often works.

But the first place I usually see things break is streaming.

Not because OpenAI-compatible APIs are bad. They're incredibly useful. But "compatible" can mean different things once you move beyond a simple request/response call:

chunks arrive in a slightly different shape
the first token takes longer than expected
proxies buffer the stream
usage data is missing from the final response
tool calls stream differently
errors happen halfway through a stream instead of before it starts

I work on TokenBay, so I spend a lot of time testing OpenAI-compatible model routing across providers. This is the checklist I use before moving a production app from one provider to another.

The part everyone tests

Most people test provider compatibility with something like this:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.API_KEY,
  baseURL: process.env.BASE_URL,
});

const response = await client.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    { role: "user", content: "Say hello in one sentence." }
  ],
});

console.log(response.choices[0].message.content);

If that works, great.

But it doesn't tell you whether streaming works in your actual app.

For a lot of AI products, streaming is not a nice-to-have. It's the difference between "this feels responsive" and "did the app freeze?"

So I test streaming separately.

A minimal streaming test

Here's the smallest script I usually start with.

Create a file called test-streaming.mjs:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.API_KEY,
  baseURL: process.env.BASE_URL,
  timeout: 30_000,
});

const model = process.env.MODEL;

if (!model) {
  throw new Error("Missing MODEL env var");
}

const startedAt = Date.now();
let firstTokenAt = null;
let chunkCount = 0;
let contentChunks = 0;
let emptyChunks = 0;
let finalText = "";

const stream = await client.chat.completions.create({
  model,
  stream: true,
  temperature: 0,
  messages: [
    {
      role: "user",
      content:
        "Write a short explanation of why streaming matters in AI apps. Keep it under 80 words.",
    },
  ],
});

for await (const chunk of stream) {
  chunkCount += 1;

  const delta = chunk.choices?.[0]?.delta;
  const content = delta?.content ?? "";

  if (content) {
    if (firstTokenAt === null) {
      firstTokenAt = Date.now();
    }

    contentChunks += 1;
    finalText += content;
    process.stdout.write(content);
  } else {
    emptyChunks += 1;
  }
}

const finishedAt = Date.now();

console.log("\n\n--- streaming diagnostics ---");
console.log({
  model,
  chunkCount,
  contentChunks,
  emptyChunks,
  firstTokenMs: firstTokenAt ? firstTokenAt - startedAt : null,
  totalMs: finishedAt - startedAt,
  chars: finalText.length,
});

Install the SDK:

npm install openai

Then run it against any OpenAI-compatible endpoint:

API_KEY="your_api_key" \
BASE_URL="https://your-provider.example/v1" \
MODEL="your-model-name" \
node test-streaming.mjs

If you're using OpenAI directly, the base URL is usually not needed:

API_KEY="your_openai_key" \
MODEL="gpt-4.1-mini" \
node test-streaming.mjs

If you're testing a gateway such as TokenBay, the idea is the same: keep the OpenAI SDK, change the baseURL, and test the model you actually plan to use.

What I check

I don't just check whether text prints.

That is the first pass, but not enough.

1. Does the first token arrive quickly?

The total response time matters, but streaming UX depends heavily on first-token latency.

If the full response takes 5 seconds but the first token arrives in 600ms, the app feels alive.

If the first token arrives after 5 seconds, streaming is technically working but the UX is basically the same as non-streaming.

In the script above, I look at:

firstTokenMs

For production apps, I usually compare this across:

the current provider
the new provider
the same model through a gateway
a fallback model

I don't need perfect lab numbers. I just want to know if the new route is obviously slower before I ship it.

2. Are chunks actually streaming, or being buffered?

This one is sneaky.

Sometimes the SDK receives a stream, but an intermediate layer buffers the response and releases it all at once.

That can happen because of:

reverse proxy settings
serverless platform behavior
CDN buffering
incorrect response headers
framework abstractions that collect the body before returning it

A rough smell test:

chunkCount
contentChunks
firstTokenMs
totalMs

If firstTokenMs and totalMs are almost identical, I get suspicious.

It doesn't always mean buffering, but it's worth checking.

3. Are empty chunks expected?

Some streaming APIs send chunks that don't contain text content.

That can happen for role metadata, finish signals, tool call deltas, or provider-specific fields.

So I don't treat this as a failure:

emptyChunks > 0

But I do check whether the final assembled text is correct.

The thing I care about is not "every chunk has content." The thing I care about is:

finalText.length > 0

and whether the text is complete.

4. Does your frontend parser assume too much?

A lot of streaming bugs are not provider bugs. They're parser bugs.

For example, a frontend might assume every chunk has this shape:

chunk.choices[0].delta.content

That works for simple text.

But once you add tool calls, JSON mode, or multimodal responses, the stream can include other delta fields.

A safer frontend parser should tolerate chunks where content is missing.

Bad:

const token = chunk.choices[0].delta.content;
render(token.toUpperCase());

Better:

const token = chunk.choices?.[0]?.delta?.content;

if (token) {
  render(token);
}

This sounds tiny, but it saves you from a lot of random "Cannot read properties of undefined" errors during provider migration.

5. What happens when the stream fails halfway through?

Non-streaming calls usually fail before you render anything.

Streaming can fail after you've already shown partial output.

That means your app needs to decide what to do with incomplete text.

I usually test three cases:

invalid model name
invalid API key
forced timeout during a long response

For the timeout case, ask for a longer answer and lower your client timeout.

Example:

const client = new OpenAI({
  apiKey: process.env.API_KEY,
  baseURL: process.env.BASE_URL,
  timeout: 2_000,
});

Then ask for something long:

{
  role: "user",
  content: "Write a detailed 1500-word explanation of streaming APIs."
}

The exact error shape may differ by provider or network path. Your app should not depend on one extremely specific error message.

In production, I care about:

showing a clear retry state
preserving partial output if useful
logging the provider, model, and request ID if available
not charging ahead as if the response completed normally

6. Does usage accounting still work?

This is easy to forget.

For non-streaming calls, usage usually comes back in the response object.

For streaming calls, usage may be missing, delayed, provider-specific, or only available from a dashboard/API after the request finishes.

If your product depends on per-request cost tracking, don't assume streaming usage works the same way.

Before switching providers, I check:

do I get prompt tokens?
do I get completion tokens?
do I get total tokens?
are streamed requests visible in the provider dashboard?
can I reconcile usage later if the stream response doesn't include it?

For internal tools, this may not matter much.

For SaaS apps where you meter customer usage, it matters a lot.

7. Do tool calls stream the same way?

Plain text streaming is the easy case.

Tool calls are where compatibility claims need more testing.

If your app uses tools/function calling, test that separately.

Things I check:

are tool call deltas streamed incrementally?
are tool names stable?
are arguments valid JSON after reconstruction?
does the provider send one tool call or multiple?
does the final message match what your agent framework expects?

A basic text streaming test passing does not mean your agent loop is safe.

I learned this the annoying way, which is usually how production lessons arrive.

My migration checklist

Before switching an app to a new OpenAI-compatible provider, I run through this:

basic non-streaming completion works
basic streaming completion works
first-token latency is acceptable
chunks are not being buffered by my hosting stack
empty chunks don't crash the parser
timeout and mid-stream errors are handled
usage tracking still works for streamed requests
tool calls work if the app uses tools
fallback behavior is tested with the real model names
logs include provider, model, latency, and error details

The main point: don't test only the happy path.

OpenAI-compatible APIs can make provider switching much easier, but streaming is where the abstraction gets tested for real.

Testing this with TokenBay

If you want to test this kind of provider switch without rewriting your OpenAI SDK code, TokenBay is one option.

The setup is intentionally boring:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL: "https://api.tokenbay.com/v1",
});

Then you can run the same streaming test against different models by changing the MODEL env var.

TOKENBAY_API_KEY="your_tokenbay_key" \
MODEL="your-model-name" \
node test-streaming.mjs

That's the main reason I like OpenAI-compatible routing: the migration surface area stays small. You can test GPT, Claude, Gemini, Qwen, GLM, or other supported models without changing the rest of your application code.

TokenBay also gives you one place to manage API usage across providers, which is helpful when you're comparing models or setting up fallbacks.

A good first test is:

run your current provider directly
run the same prompt through TokenBay
compare first-token latency
check whether chunks stream correctly
verify your app handles timeout and partial output states

If those checks pass, then you can start thinking about routing rules, fallback models, and cost optimization.

Try the boring test before switching

The test script in this post is not fancy.

That's the point.

Before I move a real app to a different provider, I want a repeatable check that answers:

does streaming actually work?
does it feel fast enough?
does my parser survive weird chunks?
does failure handling behave sanely?

If you want to try the same checklist with TokenBay, you can start here:

https://www.tokenbay.com

Run the script with your own prompts, models, and frontend stack. The useful result is not just "the API call worked." The useful result is knowing whether your app still feels good when streaming, retries, fallbacks, and real users get involved.

GLM 5.2 Is Now Available on TokenBay: Testing It with the OpenAI SDK

plasma — Wed, 24 Jun 2026 11:45:18 +0000

GLM 5.2 is now available on TokenBay.

If you're already using the OpenAI SDK, this means you can try GLM 5.2 without rewriting your application around a new provider-specific client.

For developers building agents, coding assistants, support automation, or internal AI tools, this is the part that matters most: model experiments should not require a full integration project every time.

Disclosure: I work on TokenBay. This post is a practical quickstart for developers who want to test GLM 5.2 through an OpenAI-compatible API.

Why GLM 5.2 Is Interesting

The GLM model family has been getting more attention from developers working on coding, long-context tasks, and agentic workflows.

I would be careful about treating any single benchmark or social media thread as the final answer on model quality. The useful question is more practical:

Can this model handle your workload well enough, at the right latency and cost, with minimal integration work?

That is exactly the kind of question an OpenAI-compatible gateway makes easier to answer.

Instead of changing SDKs, auth flows, request formats, and billing setups, you can test another model by changing the model name.

The Setup

TokenBay provides an OpenAI-compatible API endpoint, so the basic integration looks familiar if you have used the OpenAI JavaScript SDK before.

Install the SDK:

npm install openai

Create a file called glm52-test.js:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL: "https://api.tokenbay.com/v1"
});

async function main() {
  const response = await client.chat.completions.create({
    model: "glm-5.2",
    messages: [
      {
        role: "system",
        content: "You are a concise technical assistant."
      },
      {
        role: "user",
        content: "Explain when an AI app should use a multi-model gateway."
      }
    ]
  });

  console.log(response.choices[0].message.content);
}

main().catch(console.error);

Run it:

TOKENBAY_API_KEY=your_api_key_here node glm52-test.js

If your existing app already uses the OpenAI SDK, the important part is just the client configuration:

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL: "https://api.tokenbay.com/v1"
});

Then set the model to GLM 5.2:

model: "glm-5.2"

If the model ID appears differently in your dashboard, use the exact model name shown in TokenBay's model list.

Testing It on a Coding Task

A quick way to evaluate a model is to give it a small but realistic coding task.

Here is a simple example:

const response = await client.chat.completions.create({
  model: "glm-5.2",
  messages: [
    {
      role: "system",
      content: "You are a senior JavaScript engineer. Keep answers practical."
    },
    {
      role: "user",
      content: `
Review this function and suggest improvements:

function groupByUser(events) {
  const result = {};
  for (const event of events) {
    if (!result[event.userId]) {
      result[event.userId] = [];
    }
    result[event.userId].push(event);
  }
  return result;
}
`
    }
  ]
});

console.log(response.choices[0].message.content);

For coding tasks, I usually look for a few things:

Does the model understand the intent?
Does it suggest changes that actually matter?
Does it avoid overengineering the solution?
Does it explain tradeoffs clearly?
Does it produce code I would be willing to test?

That last part is important.

A model can sound confident and still give you code that breaks in boring ways.

Testing It on an Agent-Style Task

GLM 5.2 is also worth testing on multi-step reasoning and agent-style workflows.

For example:

const response = await client.chat.completions.create({
  model: "glm-5.2",
  messages: [
    {
      role: "system",
      content: `
You are an AI workflow planner.
Break tasks into clear steps.
Call out assumptions and risks.
Do not invent unavailable tools.
`
    },
    {
      role: "user",
      content: `
Design an automation workflow for handling inbound demo requests.

The workflow should:
- classify the lead
- extract company name and email
- store the contact
- send a personalized reply
- notify the sales team if the lead looks high intent
`
    }
  ]
});

console.log(response.choices[0].message.content);

For this kind of task, I care less about whether the model writes a pretty answer and more about whether it can produce a plan that maps cleanly to real systems.

Good signs:

It separates extraction, classification, storage, and notification.
It identifies missing fields.
It mentions error handling.
It avoids assuming every lead should be treated the same.
It gives steps that could become an n8n, Make, Zapier, or backend workflow.

Routing GLM 5.2 by Use Case

Once GLM 5.2 is available through the same API layer as your other models, you can start routing by task type.

Here is a simple example:

function selectModel(taskType) {
  switch (taskType) {
    case "coding_review":
      return "glm-5.2";

    case "workflow_planning":
      return "glm-5.2";

    case "cheap_summary":
      return "gpt-4o-mini";

    default:
      return "glm-5.2";
  }
}

Then use it in your completion call:

async function runTask(taskType, prompt) {
  const model = selectModel(taskType);

  const response = await client.chat.completions.create({
    model,
    messages: [
      {
        role: "user",
        content: prompt
      }
    ]
  });

  return {
    model,
    content: response.choices[0].message.content
  };
}

This is not fancy.

That is the point.

Start with simple routing rules. Measure the results. Then make the routing smarter only if you actually need to.

What I Would Measure

Before moving any new model into production, I would test it against your own workload.

For GLM 5.2, I would measure:

output quality on real prompts
coding accuracy
instruction following
latency
streaming behavior
failure rate
cost per successful task
behavior on long prompts
behavior with structured output
tool-calling or agent workflow compatibility if your app needs it

Do not rely only on generic benchmarks.

Benchmarks are useful for orientation, but your product has its own edge cases. Every real app does.

Adding a Fallback

If you are testing GLM 5.2 in a production-like workflow, a fallback is worth adding early.

async function completeWithFallback(messages) {
  const models = [
    "glm-5.2",
    "gpt-4o-mini"
  ];

  let lastError;

  for (const model of models) {
    try {
      const response = await client.chat.completions.create({
        model,
        messages
      });

      return {
        model,
        content: response.choices[0].message.content
      };
    } catch (error) {
      lastError = error;
      console.warn(`Model ${model} failed, trying next option...`);
    }
  }

  throw lastError;
}

This gives you a simple safety net while you test.

In a real production setup, you may also want:

request IDs
structured logs
timeout handling
retry limits
model-specific fallback rules
alerting when fallback usage spikes

Why Use TokenBay for This?

The main benefit is not that every model behaves the same.

They do not.

The benefit is that your integration surface stays simple while you compare models.

With TokenBay, you can access supported models through one OpenAI-compatible API, one API key, and one billing/usage surface.

That makes model evaluation less annoying.

Instead of asking, "Do we have time to integrate another provider?", you can ask the better question:

"Does this model actually work well for our use case?"

Final Thoughts

GLM 5.2 is another good reminder that model choice should stay flexible.

The best model for coding may not be the best model for summarization. The best model for agent planning may not be the best model for customer support. And the best model this month may not be the best model next month.

If your application is built around one hardcoded provider, every new model creates integration work.

If your application talks to an OpenAI-compatible API layer, testing GLM 5.2 becomes much simpler.

You can try GLM 5.2 through TokenBay.

I would be curious to hear how other developers are evaluating GLM 5.2. Are you testing it for coding, agents, long-context tasks, or something else entirely?

One API Key for GPT, Claude, Gemini, and Qwen: A Practical Guide to OpenAI-Compatible Model Routing

plasma — Wed, 24 Jun 2026 07:40:15 +0000

If you've built anything serious with LLM APIs, you've probably hit this pattern:

GPT is great for one task, but too expensive for another.
Claude is better at long-context reasoning, but you don't want to rewrite your whole client.
Gemini or Qwen may be good enough for cheaper background jobs.
Your app slowly turns into a pile of provider-specific SDKs, env vars, retry logic, and billing dashboards.

Not fun.

The cleanest version of this setup is simple: keep your app speaking the OpenAI API format, but route requests to different models behind the scenes.

That's the idea behind an OpenAI-compatible AI gateway.

Disclosure: I work on TokenBay, an AI model API gateway. This post is based on the setup I usually recommend when developers want access to multiple model families without rewriting their app every time they test a new provider.

The Problem

Most AI apps start with one model provider.

That works fine until you need to optimize for cost, latency, quality, availability, or task type.

For example:

Use a strong reasoning model for complex user-facing answers.
Use a cheaper model for summarization, tagging, extraction, or internal jobs.
Fall back to another provider when one API is slow or unavailable.
Test new models without refactoring half your codebase.

The painful part is not calling one API.

The painful part is maintaining five slightly different API integrations.

The Practical Pattern

Instead of wiring every provider directly into your app, you can use an OpenAI-compatible gateway.

Your application keeps using the familiar chat.completions.create() interface.

The only things you usually change are:

baseURL
apiKey
model

That means your app can switch between GPT, Claude, Gemini, Qwen, and other models while keeping most of your code unchanged.

Quickstart with Node.js

Install the OpenAI SDK:

npm install openai

Create a file called index.js:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL: "https://api.tokenbay.com/v1"
});

async function main() {
  const response = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: "You are a concise technical assistant."
      },
      {
        role: "user",
        content: "Explain model routing in one paragraph."
      }
    ]
  });

  console.log(response.choices[0].message.content);
}

main().catch(console.error);

Run it:

TOKENBAY_API_KEY=your_api_key_here node index.js

If your app already uses the OpenAI SDK, the main migration is usually just the client config:

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL: "https://api.tokenbay.com/v1"
});

That's the whole point.

You should not need a different SDK for every model family just to run a basic chat completion.

Routing by Task Type

Once your app can talk to multiple models through the same interface, you can make routing decisions in code.

Here's a simple example:

function selectModel(taskType) {
  switch (taskType) {
    case "reasoning":
      return "gpt-4o";

    case "cheap_summary":
      return "qwen-plus";

    case "long_context":
      return "claude-3-5-sonnet";

    default:
      return "gpt-4o-mini";
  }
}

Then call the selected model:

async function runTask(taskType, userInput) {
  const model = selectModel(taskType);

  const response = await client.chat.completions.create({
    model,
    messages: [
      {
        role: "user",
        content: userInput
      }
    ]
  });

  return response.choices[0].message.content;
}

This is intentionally boring code.

Boring is good here.

You want model routing to be understandable, testable, and easy to change.

Adding a Fallback

Provider outages happen. Rate limits happen. Weird transient API failures happen.

A basic fallback wrapper can save you from a lot of production pain:

async function completeWithFallback(messages) {
  const models = [
    "gpt-4o-mini",
    "qwen-plus",
    "gemini-1.5-flash"
  ];

  let lastError;

  for (const model of models) {
    try {
      const response = await client.chat.completions.create({
        model,
        messages
      });

      return {
        model,
        content: response.choices[0].message.content
      };
    } catch (error) {
      lastError = error;
      console.warn(`Model ${model} failed, trying next option...`);
    }
  }

  throw lastError;
}

Example usage:

const result = await completeWithFallback([
  {
    role: "user",
    content: "Summarize this customer support ticket in 3 bullet points."
  }
]);

console.log(`Used model: ${result.model}`);
console.log(result.content);

This is not a full production-grade retry system, but it gives you the shape:

Try your preferred model.
Fall back to another compatible model.
Keep the calling code simple.

Where This Helps Most

An OpenAI-compatible gateway is useful when you have multiple LLM workloads inside the same product.

Some common examples:

SaaS apps with user-facing AI features
Internal support tools
AI agents
Document summarization workflows
Batch extraction jobs
Coding assistants
Evaluation pipelines
Side projects where API cost actually matters

Not every task needs the most expensive model.

A lot of production AI work is routing the right request to the right model at the right cost.

Cost Control

This is where gateways become more than a convenience layer.

Once all requests pass through one API layer, you can start thinking about:

Per-task model selection
Usage tracking
Budget limits
Cheaper models for background jobs
Higher-quality models only where they matter
Fallbacks when a provider is unavailable

TokenBay is built around this workflow: one API key for major model families, OpenAI-compatible requests, and a usage dashboard so you can see what your app is actually spending.

In many cases, routing routine tasks to lower-cost models can reduce API spend without hurting the user experience.

The exact savings depend on your workload and model choices, so I would treat any blanket savings claim with suspicion. Test it against your own traffic.

A Simple Rule of Thumb

When I look at an AI feature, I usually split calls into three buckets:

Task	Model Strategy
User-facing reasoning	Use a stronger model
Summarization / tagging / extraction	Try cheaper models first
Background automation	Optimize heavily for cost and latency

You do not need a complicated ML routing system on day one.

A plain function like selectModel(taskType) is enough to start.

Final Thoughts

The best AI API architecture is usually the one that gives you room to change your mind.

Models change. Prices change. Latency changes. Your product changes.

If your code is tightly coupled to one provider, every model experiment becomes annoying.

If your app talks to one OpenAI-compatible interface, you can test and route across GPT, Claude, Gemini, Qwen, and others with much less friction.

That is the real win: not just saving money, but making model choice a runtime decision instead of a rewrite.

For this post, I used TokenBay as the OpenAI-compatible gateway example, but the same routing pattern applies to any gateway that supports the OpenAI API format.

I'd also be curious how other teams are handling model routing today. Are you using a gateway, building your own abstraction layer, or still calling each provider directly?