I Stopped Swapping LLM Providers Without a Smoke Test

For a while, my LLM provider migration test was embarrassingly simple:

1. Change the base URL
2. Send one normal prompt
3. Get a response
4. Call it done

Which is fine if your app only fails on days when it feels polite.

The problem is that a happy-path request does not tell you much. It tells you the provider can answer one boring prompt under normal conditions. It does not tell you what happens when the request times out, the stream cuts off, the rate limit response has a weird shape, or your app is about to retry something that may have already triggered a tool call.

So now, before I trust a new LLM provider in anything production-ish, I run a small smoke test first.

Not a full benchmark. Not a provider comparison. Just a quick "will this break my app in boring, predictable ways?" test.

The old test was basically useless

The old version looked something like this:

const response = await client.chat.completions.create({
  model: "some-model",
  messages: [
    { role: "user", content: "Say hello in one sentence." }
  ]
});

console.log(response.choices[0].message.content);

If that worked, I felt good.

That was the trap.

A normal request does not test the parts that usually hurt in production:

• timeout behavior
• retry safety
• streaming completion
• rate limit shape
• token limit behavior
• usage metadata
• provider request IDs
• tool call handling
• error classification
• whether the response envelope is actually complete

The provider can pass the hello-world test and still be a bad fit for your app.

What I actually want to know

Before swapping providers, I want answers to a few boring questions.

Boring is good here. Boring saves weekends.

I want to know:

• Can I classify failures without parsing random strings?
• Can I tell the difference between a retryable failure and a permanent one?
• Does streaming end with enough metadata for me to trust the turn is complete?
• Do timeouts cancel cleanly?
• Does the provider return useful request IDs?
• Does usage data exist where my billing and logging expect it?
• If the call could trigger a side effect, will my retry policy avoid replaying it blindly?

That is the point of the smoke test.

It is not trying to prove that one provider is "better." It is trying to prove that my app can understand what happened.

My 10-minute smoke test

I usually run five small checks.

They are simple on purpose.

1. A boring normal request

First, I still run the basic request.

But now I log more than the text.

const started = Date.now();

const response = await client.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    { role: "user", content: "Reply with exactly one short sentence." }
  ]
});

console.log({
  latency_ms: Date.now() - started,
  content: response.choices?.[0]?.message?.content,
  finish_reason: response.choices?.[0]?.finish_reason,
  usage: response.usage,
  id: response.id
});

I am not looking for a beautiful answer.

I am looking for the envelope:

• Did I get a response ID?
• Did I get a finish reason?
• Did I get usage?
• Is the response shape what my app expects?
• Is latency roughly sane?

If the normal path is already weird, I stop there.

2. A request that should timeout

Next, I force an aggressive timeout.

Not because I expect to run with a 200ms timeout in production, but because I want to see what failure looks like.

const controller = new AbortController();

const timeout = setTimeout(() => {
  controller.abort();
}, 200);

try {
  await client.chat.completions.create(
    {
      model: process.env.MODEL,
      messages: [
        {
          role: "user",
          content: "Write a detailed 1000-word explanation of database indexes."
        }
      ]
    },
    { signal: controller.signal }
  );
} catch (error) {
  console.log({
    name: error.name,
    message: error.message,
    code: error.code,
    status: error.status
  });
} finally {
  clearTimeout(timeout);
}

What I want to know:

• Does the request actually cancel?
• Does the SDK throw something I can classify?
• Is the error clearly a timeout, or just a generic network failure?
• Would my retry policy treat this correctly?

The retry decision depends on where the timeout happened.

A timeout before any response is usually retryable.

A timeout after partial output is much messier.

That difference matters a lot if the model is being used inside a workflow or agent.

3. A streaming request

Streaming is where a lot of "looks fine locally" bugs hide.

I want to know whether the stream gives me a clear ending.

const stream = await client.chat.completions.create({
  model: process.env.MODEL,
  stream: true,
  messages: [
    { role: "user", content: "Count from 1 to 5, one number per line." }
  ]
});

let text = "";
let finalChunkSeen = false;
let finishReason = null;

for await (const chunk of stream) {
  const delta = chunk.choices?.[0]?.delta?.content;
  if (delta) text += delta;

  const reason = chunk.choices?.[0]?.finish_reason;
  if (reason) {
    finishReason = reason;
    finalChunkSeen = true;
  }
}

console.log({
  text,
  finalChunkSeen,
  finishReason
});

For a normal app, partial text may be fine.

For a workflow, agent, or tool-calling system, partial text can be dangerous. The user may have already seen output. The app may have already started acting on it. The system may think the turn completed when it did not.

So I do not only ask, "Did I get tokens?"

I ask:

• Did the stream end cleanly?
• Did I get a finish reason?
• Did I get the final metadata I expected?
• Can my app distinguish "completed" from "stopped receiving chunks"?

A response can be useful and still not be complete enough to trust.

4. A rate limit or token limit check

I do not always try to actually hit a provider's rate limit. That can be annoying, noisy, and occasionally rude.

But I do want to know what rate limit and token limit failures look like.

If I can trigger one safely, great. If not, I test this through a mocked response in my wrapper.

The important part is classification.

function classifyLlmFailure(error) {
  const status = error.status;

  if (status === 429) {
    return {
      type: "rate_limit",
      retryable: true,
      action: "queue_or_backoff"
    };
  }

  if (status === 400 && /context|token/i.test(error.message || "")) {
    return {
      type: "token_limit",
      retryable: false,
      action: "shrink_or_fail"
    };
  }

  if (status >= 500) {
    return {
      type: "provider_error",
      retryable: true,
      action: "retry_or_fallback"
    };
  }

  return {
    type: "unknown",
    retryable: false,
    action: "fail_visibly"
  };
}

The detail I care about here is whether the failure is actionable.

"Too many requests" and "input too large" should not go through the same path.

Even token limits need a split:

• If the context is shrinkable, reduce optional context and retry.
• If the input is one large tool result, do not silently clip it.
• If clipping changes the meaning, fail visibly or queue the job.

Silently truncating important input is how you get a confident answer based on damaged context.

That is not a retry. That is a new incident wearing a fake mustache.

5. A tool-call-shaped request

This one matters if your LLM app can do anything outside the model call.

Send an email. Create a ticket. Write to a database. Call a webhook. Charge a card. Trigger a workflow.

Even if the smoke test does not execute a real tool, I still want to test the policy path.

function shouldRetry({ failureType, sideEffectRisk, streamStarted }) {
  if (sideEffectRisk === "none") {
    return failureType === "timeout" || failureType === "provider_error";
  }

  if (sideEffectRisk === "possible" && streamStarted) {
    return false;
  }

  if (sideEffectRisk === "confirmed") {
    return false;
  }

  return false;
}

console.log(
  shouldRetry({
    failureType: "timeout",
    sideEffectRisk: "possible",
    streamStarted: true
  })
);

This is where a lot of retry logic gets too cute.

If a request is just generating a summary, retrying is usually fine.

If a request may have already emitted a tool call, retrying blindly can duplicate the real-world action.

The smoke test should confirm that your app knows the difference.

What I log from the smoke test

I keep the output boring and structured:

{
  provider: "example-provider",
  model: "example-model",
  base_url: "https://api.example.com/v1",
  test_name: "streaming_completion",
  latency_ms: 842,
  status: 200,
  response_id: "abc123",
  finish_reason: "stop",
  usage_present: true,
  stream_completed: true,
  error_type: null,
  retryable: false,
  side_effect_risk: "none"
}

The exact fields depend on the app, but I usually want:

• provider
• model
• base URL
• test name
• latency
• status
• provider request ID
• finish reason
• usage presence
• stream completion
• classified error type
• retryable or not
• side-effect risk

If I cannot log the provider's behavior clearly, I probably cannot operate it clearly either.

The rule I use now

I do not need every provider to behave exactly the same.

That is unrealistic.

But I do need the differences to be visible.

If a provider has different rate limit semantics, fine. I can handle that.

If it has different streaming metadata, fine. I can adapt.

If it gives me failures I cannot classify, streams I cannot verify, or retry behavior that might duplicate side effects, I do not want to discover that from a user report.

The rule is simple:

Do not trust a provider because one normal prompt worked.

Trust it after you understand how it fails.

The smoke test is not about distrust

This might sound like paranoia, but I do not think of it that way.

LLM providers are complicated systems. So are the apps built on top of them. Weird edge cases are not a moral failure. They are just part of running software that depends on remote inference, streaming, rate limits, queues, SDKs, and user-facing workflows.

The smoke test is not there to prove that a provider is bad.

It is there to make sure my app and the provider agree on what "done," "failed," and "safe to retry" mean.

That is a much better thing to learn in 10 minutes than during an incident.

I work on TokenBay, so I spend a lot of time thinking about provider routing and OpenAI-compatible APIs. But this rule applies no matter what stack you use: swapping a base URL is easy. Trusting the new behavior should take a little more work.