Raju Dandigam

Posted on Sep 22

Same Model, Three APIs: Contract-Testing with Docker Model Runner

#docker #ai #playwright #typescript

“OpenAI-compatible” is useful shorthand. It is not a portability guarantee.

Two APIs may accept similar chat messages while differing in streaming events, tool calls, token counts, error shapes, stop reasons, and supported parameters. Swapping a base URL can make a demo work while leaving the production abstraction full of assumptions.

Docker Model Runner is a useful local contract target because it exposes OpenAI-, Anthropic-, and Ollama-compatible formats over one local model environment. Instead of comparing three remote providers, we can test three protocol adapters against controlled local infrastructure.

Define the application contract first

Keep provider objects out of domain code:

type ChatRequest = {
  messages: Array<{ role: "user" | "assistant"; content: string }>;
  maxOutputTokens: number;
  stream?: boolean;
};

type ChatResult = {
  text: string;
  finishReason: "stop" | "length" | "tool" | "unknown";
  inputTokens?: number;
  outputTokens?: number;
};

interface ModelAdapter {
  id: string;
  chat(request: ChatRequest): Promise<ChatResult>;
}

Each adapter translates its wire format into this deliberately small result. Do not force unsupported provider details into fake equivalence. Optional usage fields are more honest than invented token counts.

Point the adapters at the local endpoints

For host processes, Docker's documented base URLs are:

Client shape	Base URL
OpenAI-compatible	`http://localhost:12434/engines/v1`
Anthropic-compatible	`http://localhost:12434`
Ollama-compatible	`http://localhost:12434`

From containers, use model-runner.docker.internal as documented for your Docker environment. Docker Engine setups may require an extra_hosts host-gateway mapping.

Create the matrix once:

const adapters: ModelAdapter[] = [
  openAiAdapter({ baseURL: "http://localhost:12434/engines/v1" }),
  anthropicAdapter({ baseURL: "http://localhost:12434" }),
  ollamaAdapter({ baseURL: "http://localhost:12434" }),
];

Test behavior, not identical prose

All three paths can reach the same configured model, but generated text may still vary because request translation and sampling surfaces differ. Contract-test properties your application actually needs:

describe.each(adapters)("$id adapter", (adapter) => {
  it("returns a bounded non-empty response", async () => {
    const result = await adapter.chat({
      messages: [{ role: "user", content: "Reply with one color." }],
      maxOutputTokens: 16,
    });

    expect(result.text.trim()).not.toBe("");
    expect(result.text.length).toBeLessThan(200);
    expect(["stop", "length", "tool", "unknown"])
      .toContain(result.finishReason);
  });
});

Then add separate contracts for the risky seams.

Streaming reconstruction

Concatenate provider-specific delta events and require the same normalized terminal result as the non-streaming path. Also test cancellation and an interrupted stream.

Error normalization

Send an invalid request and map each wire error into an application category such as invalid_request, unavailable, or rate_limited. Preserve the original error as diagnostic evidence, not as control flow.

Structured JSON

If the application depends on JSON output, validate the parsed result against your own schema. A protocol accepting response_format does not remove semantic validation.

Tool calls

Run tool contracts only for a model and inference engine that support them. Docker documents OpenAI-compatible function calling for compatible models under llama.cpp; a red test on an unsupported model is not evidence of a broken adapter.

Keep a capability manifest

Portability improves when unsupported behavior is explicit:

type AdapterCapabilities = {
  streaming: boolean;
  tools: boolean;
  jsonMode: boolean;
  tokenUsage: "reported" | "estimated" | "unavailable";
};

The application can then fail fast instead of discovering halfway through a run that the selected adapter cannot satisfy a required feature.

Version the capability observation rather than treating it as eternal configuration:

type TestedAdapter = {
  adapterVersion: string;
  protocol: "openai" | "anthropic" | "ollama";
  modelId: string;
  engine: string;
  observedAt: string;
  capabilities: AdapterCapabilities;
};

The same protocol adapter can behave differently with another engine or model. Cache the manifest only for the exact tuple you tested, and rerun the matrix when any member changes.

Include negative tests, not only successful prompts. Abort a stream, send an unsupported parameter, exceed the configured context, and request a tool from a model without tool support. A portable adapter should translate each response into an explicit application error while preserving enough provider detail for diagnosis. Silent parameter dropping is a failed contract even when text still arrives.

Docker Model Runner itself documents important differences. Its local OpenAI-compatible endpoint does not require an API key and ignores the authorization header. Token counting uses the model's native encoder and can differ from OpenAI. Supported features depend on the engine and model.

Those are exactly the reasons to contract-test rather than infer compatibility from the URL shape.

Portability is a property you measure

A compatible endpoint lowers migration cost. A provider abstraction becomes trustworthy only after you test the behaviors on which your application depends: stream completion, errors, tool arguments, structured output, cancellation, and usage accounting.

Running that matrix locally makes it fast and inexpensive. More importantly, it reveals where the abstraction is genuinely portable—and where the provider-specific capability must remain visible.

Use local compatibility to improve your adapter, not to erase provider identity. Production policies may still differ for credentials, data residency, rate limits, safety controls, and model lifecycle even when the request and response shapes look familiar.

DEV Community