The Bug Only Happened After I Switched LLM Providers

The request succeeded.

The response was valid JSON.

The SDK did not throw an exception.

And the application still broke.

The bug appeared after I switched an LLM application to a different provider with an OpenAI-compatible endpoint. The integration looked almost too easy: change the base URL, replace the API key, keep the request body, and ship.

That worked for the simplest prompt.

It did not mean the providers behaved the same way.

The failure was not in the HTTP contract. It was in the assumptions my application had quietly built around one provider's behavior.

Compatible Requests Do Not Guarantee Compatible Behavior

When an API describes itself as OpenAI-compatible, it usually means that familiar requests are accepted:

{
  "model": "some-model",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this support ticket."
    }
  ]
}

That is useful compatibility. It lets us reuse clients, authentication patterns, and much of our integration code.

But production applications depend on more than whether the server accepts the request.

They may also depend on:

• whether content is a string, an array, an empty value, or null
• how tool-call arguments are serialized
• which finish_reason values can appear
• whether usage fields are always included
• how streaming completion is signaled
• whether structured-output constraints are enforced
• how errors, timeouts, and unsupported parameters are reported

Two providers can accept the same request and return responses that are syntactically valid but operationally different.

That is where the interesting bugs live.

The Assumption Hidden in My Parser

My original response handler effectively assumed this:

const text = response.choices[0].message.content.trim();

It worked until the selected model returned a tool call.

In that response, message.content was null, while the actual output was inside message.tool_calls.

The API had not failed. My parser had.

The first fix was straightforward:

function parseAssistantMessage(response) {
  const choice = response?.choices?.[0];

  if (!choice?.message) {
    throw new Error("Response is missing choices[0].message");
  }

  const { content, tool_calls: toolCalls } = choice.message;

  if (Array.isArray(toolCalls) && toolCalls.length > 0) {
    return {
      type: "tool_calls",
      toolCalls,
      finishReason: choice.finish_reason ?? null
    };
  }

  if (typeof content === "string" && content.trim() !== "") {
    return {
      type: "text",
      text: content,
      finishReason: choice.finish_reason ?? null
    };
  }

  throw new Error(
    `Assistant returned neither text nor tool calls. finish_reason=${choice.finish_reason ?? "missing"}`
  );
}

That fixed one symptom, but it exposed the larger problem: I had treated the response shape I usually saw as if it were the entire protocol.

Tool Calls Are an Especially Fragile Boundary

Tool calling creates at least three compatibility questions.

1. Are the arguments valid JSON?

A tool call may contain an argument string like this:

{
  "name": "create_ticket",
  "arguments": "{\"priority\":\"high\",\"summary\":\"Login failure\"}"
}

The outer response can be valid JSON while the nested arguments string is not.

Never pass it directly to a tool:

function parseToolArguments(toolCall) {
  const raw = toolCall?.function?.arguments;

  if (typeof raw !== "string") {
    throw new Error("Tool arguments are missing or are not a string");
  }

  try {
    return JSON.parse(raw);
  } catch (error) {
    throw new Error(
      `Invalid JSON in tool arguments for ${toolCall.function?.name ?? "unknown tool"}: ${error.message}`
    );
  }
}

2. Do the arguments satisfy the business contract?

Valid JSON is not the same as a valid action.

function validateCreateTicket(args) {
  const allowedPriorities = new Set(["low", "medium", "high"]);

  if (typeof args.summary !== "string" || args.summary.trim() === "") {
    throw new Error("create_ticket.summary must be a non-empty string");
  }

  if (!allowedPriorities.has(args.priority)) {
    throw new Error(
      `create_ticket.priority must be low, medium, or high`
    );
  }

  return {
    summary: args.summary.trim(),
    priority: args.priority
  };
}

The model proposes an action. Application code decides whether that action is allowed.

3. What tells the application that generation is complete?

Some integrations assume that a particular finish_reason always accompanies a tool call. That assumption should be tested, not inherited from a single provider.

I now treat the presence and validity of tool_calls as one signal, and finish_reason as separate diagnostic information. If they disagree, I log the response and stop before executing anything with side effects.

I Started Testing Provider Behavior, Not Just Availability

A normal health check asks:

Can this endpoint return a response?

That is necessary, but too weak for switching providers.

A compatibility check should ask:

Does this provider preserve the behaviors my application relies on?

Here is a small Node.js script I use as a starting point.

It requires Node.js 18 or later and no external packages.

// provider-compatibility-test.mjs

const config = {
  name: process.env.PROVIDER_NAME ?? "candidate-provider",
  baseUrl: process.env.LLM_BASE_URL,
  apiKey: process.env.LLM_API_KEY,
  model: process.env.LLM_MODEL
};

for (const [key, value] of Object.entries(config)) {
  if (!value) {
    throw new Error(`Missing configuration value: ${key}`);
  }
}

async function createChatCompletion(body) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 20_000);

  try {
    const response = await fetch(
      `${config.baseUrl.replace(/\/$/, "")}/chat/completions`,
      {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: `Bearer ${config.apiKey}`
        },
        body: JSON.stringify(body),
        signal: controller.signal
      }
    );

    const rawBody = await response.text();

    let data;
    try {
      data = JSON.parse(rawBody);
    } catch {
      throw new Error(
        `Provider returned non-JSON content with HTTP ${response.status}`
      );
    }

    if (!response.ok) {
      throw new Error(
        `Provider returned HTTP ${response.status}: ${JSON.stringify(data)}`
      );
    }

    return data;
  } finally {
    clearTimeout(timeout);
  }
}

function firstChoice(response) {
  const choice = response?.choices?.[0];

  if (!choice || !choice.message) {
    throw new Error("Missing choices[0].message");
  }

  return choice;
}

function inspectCommonFields(response) {
  const choice = firstChoice(response);

  return {
    hasId: typeof response.id === "string",
    hasModel: typeof response.model === "string",
    hasUsage: response.usage !== undefined,
    contentType:
      choice.message.content === null
        ? "null"
        : Array.isArray(choice.message.content)
          ? "array"
          : typeof choice.message.content,
    toolCallCount: Array.isArray(choice.message.tool_calls)
      ? choice.message.tool_calls.length
      : 0,
    finishReason: choice.finish_reason ?? "missing"
  };
}

async function testTextResponse() {
  const response = await createChatCompletion({
    model: config.model,
    temperature: 0,
    messages: [
      {
        role: "user",
        content: 'Reply with exactly: PROVIDER_TEST_OK'
      }
    ]
  });

  const choice = firstChoice(response);
  const content = choice.message.content;

  if (typeof content !== "string") {
    throw new Error(
      `Expected string content, received ${
        content === null ? "null" : typeof content
      }`
    );
  }

  if (!content.includes("PROVIDER_TEST_OK")) {
    throw new Error(`Unexpected text response: ${content}`);
  }

  return inspectCommonFields(response);
}

async function testToolCall() {
  const response = await createChatCompletion({
    model: config.model,
    temperature: 0,
    messages: [
      {
        role: "user",
        content:
          "Create a high-priority ticket with the summary Login failure. Use the provided tool."
      }
    ],
    tools: [
      {
        type: "function",
        function: {
          name: "create_ticket",
          description: "Create a support ticket",
          parameters: {
            type: "object",
            additionalProperties: false,
            properties: {
              summary: {
                type: "string"
              },
              priority: {
                type: "string",
                enum: ["low", "medium", "high"]
              }
            },
            required: ["summary", "priority"]
          }
        }
      }
    ],
    tool_choice: {
      type: "function",
      function: {
        name: "create_ticket"
      }
    }
  });

  const choice = firstChoice(response);
  const toolCall = choice.message.tool_calls?.[0];

  if (!toolCall) {
    throw new Error("Expected a tool call but none was returned");
  }

  if (toolCall.function?.name !== "create_ticket") {
    throw new Error(
      `Expected create_ticket, received ${toolCall.function?.name}`
    );
  }

  let args;
  try {
    args = JSON.parse(toolCall.function.arguments);
  } catch (error) {
    throw new Error(`Tool arguments are not valid JSON: ${error.message}`);
  }

  if (
    typeof args.summary !== "string" ||
    !["low", "medium", "high"].includes(args.priority)
  ) {
    throw new Error(
      `Tool arguments failed validation: ${JSON.stringify(args)}`
    );
  }

  return {
    ...inspectCommonFields(response),
    toolName: toolCall.function.name,
    argumentKeys: Object.keys(args).sort()
  };
}

async function runTest(name, test) {
  const startedAt = Date.now();

  try {
    const observations = await test();

    return {
      test: name,
      status: "passed",
      durationMs: Date.now() - startedAt,
      observations
    };
  } catch (error) {
    return {
      test: name,
      status: "failed",
      durationMs: Date.now() - startedAt,
      error: error.message
    };
  }
}

const results = [];

results.push(await runTest("text response", testTextResponse));
results.push(await runTest("tool call", testToolCall));

console.log(
  JSON.stringify(
    {
      provider: config.name,
      model: config.model,
      testedAt: new Date().toISOString(),
      results
    },
    null,
    2
  )
);

if (results.some((result) => result.status === "failed")) {
  process.exitCode = 1;
}

Run it against the provider currently in production:

PROVIDER_NAME=current \
LLM_BASE_URL=https://current-provider.example/v1 \
LLM_API_KEY=your-key \
LLM_MODEL=your-model \
node provider-compatibility-test.mjs

Then run the same script against the candidate:

PROVIDER_NAME=candidate \
LLM_BASE_URL=https://candidate-provider.example/v1 \
LLM_API_KEY=your-key \
LLM_MODEL=your-model \
node provider-compatibility-test.mjs

The goal is not merely to get two green results. Save and compare the observations.

A different contentType, missing usage information, unexpected finish reason, or changed tool-call shape may be harmless. It may also reveal an assumption that will break elsewhere in the application.

What I Test Before Switching Now

I keep the suite small enough to run during deployment, but specific enough to represent the application.

For a text-only application, that may include:

• a normal text response
• an intentionally invalid request
• a timeout or cancellation
• a response near the output-token limit

For an agent or workflow, I also test:

• a forced tool call
• multiple available tools
• malformed or incomplete tool arguments
• a tool result returned to the model
• a streaming tool call
• an action that must not execute twice

For structured data, I test the actual schema used by the application—not a toy object with two string fields.

The important question is not whether every provider produces identical JSON. They probably will not.

The important question is whether those differences remain inside the boundaries the application knows how to handle.

Normalize at the Boundary

Once I know which differences are intentional, I normalize them before the rest of the application sees them.

function normalizeCompletion(response) {
  const choice = response?.choices?.[0];

  if (!choice?.message) {
    return {
      status: "invalid",
      reason: "missing_assistant_message",
      raw: response
    };
  }

  const toolCalls = Array.isArray(choice.message.tool_calls)
    ? choice.message.tool_calls
    : [];

  if (toolCalls.length > 0) {
    return {
      status: "tool_calls",
      toolCalls,
      finishReason: choice.finish_reason ?? null,
      usage: response.usage ?? null
    };
  }

  if (typeof choice.message.content === "string") {
    return {
      status: "text",
      text: choice.message.content,
      finishReason: choice.finish_reason ?? null,
      usage: response.usage ?? null
    };
  }

  return {
    status: "invalid",
    reason: "unsupported_message_shape",
    finishReason: choice.finish_reason ?? null,
    raw: response
  };
}

The rest of the application consumes this internal contract instead of depending directly on every provider's response details.

That makes provider-specific behavior visible in one place. It also makes failures easier to log, test, and stop safely.

A Provider Switch Is a Dependency Upgrade

Changing an LLM provider may look like a configuration update:

- LLM_BASE_URL=https://provider-a.example/v1
+ LLM_BASE_URL=https://provider-b.example/v1

Operationally, I treat it more like upgrading a major dependency.

The endpoint may be compatible while the behavior at the edges is not. Those edges include streaming, tool calls, structured outputs, token limits, cancellation, error semantics, and usage reporting.

None of this makes OpenAI-compatible APIs a bad idea. Shared interfaces remove a lot of unnecessary integration work. I work on TokenBay, where routing requests across different models makes this compatibility layer especially useful.

But a shared interface should make switching testable—not invisible.

Now, before I change a base URL in production, I run the candidate provider through the same behavioral tests as the current one.

The request being accepted is only the first test.

It is not the definition of compatibility.