My LLM Provider Returned 200. The Workflow Still Failed.

The request looked completely normal.

The server returned 200 OK. The response body was valid JSON. The latency was acceptable. Nothing appeared in the error logs.

So I marked the provider integration as working.

A few days later, the workflow failed in production.

The API request had succeeded, but the task had not.

That incident changed how I test LLM providers. I no longer treat a successful HTTP request as proof that an LLM operation succeeded.

The response that passed my first test

My original health check was embarrassingly simple:

const response = await fetch(`${baseUrl}/chat/completions`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model,
    messages: [
      {
        role: "user",
        content: "Reply with the word READY.",
      },
    ],
  }),
});

if (!response.ok) {
  throw new Error(`Provider returned ${response.status}`);
}

console.log("Provider is working");

It verified a few useful things:

• the endpoint was reachable
• the API key was accepted
• the model name was recognized
• the server returned a successful HTTP status

But it did not verify the thing I actually cared about:

Can this provider complete the operation my application depends on?

The answer, in my case, was no.

HTTP success and task success are different states

An LLM request can return 200 OK while still being unusable to the application.

I have learned to think about success in layers:

1. Transport success — Did the request reach the server?
2. Protocol success — Did the server return the expected response format?
3. Model success — Did the model produce a complete and valid result?
4. Workflow success — Can the application safely continue?
5. User success — Did the action the user requested actually happen?

A status code mostly tells me about the first two layers.

My product usually depends on the last three.

For example, all of these responses might arrive with a 200:

• an empty choices array
• a missing message.content
• truncated JSON
• a tool call with invalid arguments
• a stream that ends without a terminal event
• a refusal represented differently from what the application expects
• a response that is syntactically valid but violates the required schema

If my code checks only response.ok, every one of those cases can be mislabeled as success.

Validate the response you actually need

The first improvement was to validate the full response shape.

function validateChatCompletion(data) {
  if (!data || typeof data !== "object") {
    throw new Error("Response is not an object");
  }

  if (!Array.isArray(data.choices) || data.choices.length === 0) {
    throw new Error("Response contains no choices");
  }

  const choice = data.choices[0];

  if (!choice.message || typeof choice.message !== "object") {
    throw new Error("Response contains no assistant message");
  }

  if (
    typeof choice.message.content !== "string" ||
    choice.message.content.trim() === ""
  ) {
    throw new Error("Assistant message is empty");
  }

  if (!choice.finish_reason) {
    throw new Error("Response does not include a finish reason");
  }

  return choice.message.content;
}

Then the request check became:

async function runProviderCheck({ baseUrl, apiKey, model }) {
  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model,
      messages: [
        {
          role: "user",
          content: "Reply with exactly: READY",
        },
      ],
      temperature: 0,
    }),
  });

  if (!response.ok) {
    throw new Error(`HTTP failure: ${response.status}`);
  }

  const data = await response.json();
  const content = validateChatCompletion(data);

  if (content.trim() !== "READY") {
    throw new Error(`Unexpected model output: ${content}`);
  }

  return {
    status: "passed",
    model,
    finishReason: data.choices[0].finish_reason,
    usage: data.usage ?? null,
  };
}

This is still a small test, but it checks more than endpoint availability. It checks whether the provider can satisfy a precise application-level contract.

Tool calls need a different definition of success

Text validation is not enough when an agent can call tools.

Suppose the model is expected to create this tool call:

{
  "name": "create_support_ticket",
  "arguments": {
    "customer_id": "cus_123",
    "priority": "high",
    "summary": "Customer cannot access the dashboard"
  }
}

A provider could return a tool call with:

• malformed JSON arguments
• the wrong tool name
• required fields missing
• strings where the schema expects numbers
• duplicated tool calls
• a different tool-call structure than the SDK expects

The HTTP request still succeeded. Executing the tool call blindly would be the actual failure.

I now validate tool calls before allowing them anywhere near a real side effect:

function validateSupportTicketToolCall(toolCall) {
  if (toolCall?.function?.name !== "create_support_ticket") {
    throw new Error("Unexpected tool name");
  }

  let args;

  try {
    args = JSON.parse(toolCall.function.arguments);
  } catch {
    throw new Error("Tool arguments are not valid JSON");
  }

  if (typeof args.customer_id !== "string") {
    throw new Error("Missing or invalid customer_id");
  }

  if (!["low", "medium", "high"].includes(args.priority)) {
    throw new Error("Invalid priority");
  }

  if (
    typeof args.summary !== "string" ||
    args.summary.trim().length === 0
  ) {
    throw new Error("Missing ticket summary");
  }

  return args;
}

The model proposes an action. The application decides whether that action is valid and safe to execute.

Those should never be the same step.

Streaming has its own false-success cases

Streaming responses made this problem less obvious.

A stream can begin successfully, deliver several tokens, and then stop before the completion is usable. If the UI has already displayed partial output, both the user and the application may believe the task finished.

For streaming requests, I check for more than the first chunk:

• Was at least one content or tool-call delta received?
• Did the stream reach the expected terminal state?
• Was the final structured output complete?
• Were tool-call arguments fully assembled?
• Did the connection close normally?
• If the stream failed, was the partial state discarded or clearly marked?

The exact terminal signal depends on the API and SDK. The important part is not to interpret “some tokens arrived” as “the operation completed.”

A partial answer can be useful in a chat interface. A partial tool call can be dangerous in a production workflow.

My provider smoke test is now a small behavior suite

I used to test a provider with one prompt.

Now I run several small checks that represent the behaviors my application relies on:

• a deterministic text response
• structured JSON matching a schema
• a valid tool call
• a streaming completion
• timeout behavior
• rate-limit behavior
• malformed-request errors
• usage metadata, if billing or monitoring depends on it

I do not test every possible model behavior. I test the contracts that could break my application.

A simplified test runner looks like this:

const checks = [
  ["text completion", checkTextCompletion],
  ["structured output", checkStructuredOutput],
  ["tool calling", checkToolCalling],
  ["stream termination", checkStreaming],
];

async function runSmokeTests(provider) {
  const results = [];

  for (const [name, check] of checks) {
    const startedAt = Date.now();

    try {
      await check(provider);

      results.push({
        name,
        passed: true,
        durationMs: Date.now() - startedAt,
      });
    } catch (error) {
      results.push({
        name,
        passed: false,
        durationMs: Date.now() - startedAt,
        error: error instanceof Error ? error.message : String(error),
      });
    }
  }

  return results;
}

I run this before enabling a new provider or model in a production route. I also rerun the relevant checks after SDK changes or response-format changes.

Do not let “success” hide the evidence

Another mistake I made was logging only failed HTTP requests.

That meant the misleading 200 left almost no useful evidence.

For every important LLM operation, I now try to record:

• provider and model
• operation ID
• HTTP status
• latency
• finish reason
• response validation result
• tool name, without sensitive arguments
• token usage when available
• retry count
• final workflow outcome

I avoid logging raw prompts or sensitive model output by default. The goal is to preserve enough metadata to distinguish:

• request failed
• response was invalid
• model output was rejected
• tool execution failed
• workflow completed

Without that separation, several very different incidents become the same vague log message: “LLM call failed.”

The status code is evidence, not the verdict

I still check HTTP status codes. They are useful.

I just no longer let them make the final decision.

For an LLM application, success is a contract defined by the workflow:

• Was the response complete?
• Did it match the expected shape?
• Was the requested tool call valid?
• Is the proposed action safe?
• Did the workflow reach its intended state?

A clean 200 answers none of those questions on its own.

This matters even more when an application can switch between multiple models or providers. OpenAI-compatible endpoints can reduce integration work, but compatibility at the request level does not guarantee identical runtime behavior.

I work on TokenBay, so provider routing is part of what I think about day to day. The lesson I keep coming back to is simple:

Test provider behavior, not just provider availability.

The request can succeed while the workflow fails. Your success check needs to know the difference.