Migrating an Internal Prompt Playground to a New Provider

#ai #machinelearning #llm #tutorial

A prompt playground is mostly UI, and the UI does not care which provider answered. The migration is one adapter — but the adapter has to make decisions the old single-provider code never had to, and the saved runs are where those decisions become visible.

What is actually provider-specific

Before writing anything, separate the tool into the part that is about prompts and the part that is about an API. The prompt part — the editor, variable substitution, the diff view, the run history, the share links — is provider-neutral and should not be touched. The API part is four things: listing available models, building a request, consuming a response or a stream, and reporting what the call cost.

The mistake that makes this migration expensive is that the fourth of those usually leaked. A playground built against one provider tends to render finish_reason straight into the UI, store the raw request body as the record of a run, and compute cost from usage.prompt_tokens in a component. Each of those is a provider-specific fact sitting outside the adapter, and each one has to come back inside before a second provider is possible.

The interface the UI already implies

Write the interface from what the screen needs, not from what either API offers. It is small:

type Adapter = {
  id: string;
  listModels(): Promise<ModelInfo[]>;   // id, context window, capabilities
  run(req: RunRequest, signal: AbortSignal): AsyncIterable<RunEvent>;
};

type RunRequest = {
  model: string;
  system?: string;
  messages: { role: "user" | "assistant"; content: string }[];
  maxOutputTokens: number;              // required; not every API defaults it
  temperature?: number;                 // in the tool's own 0..1 scale
  stopSequences?: string[];
  tools?: ToolDef[];
  jsonSchema?: object;
};

Two choices in that shape are worth defending. maxOutputTokens is required rather than optional, because one of the APIs you will target requires it and a tool that omits it works against one provider and 400s against the other. And temperature is declared in the tool’s own scale, not in a provider’s, for reasons the lossy section covers.

Normalising the stream

The UI wants a flat sequence of things to append. Define that union once and make every adapter produce it:

type RunEvent =
  | { type: "text"; delta: string }
  | { type: "tool_call"; index: number; id?: string; name?: string; argsDelta: string }
  | { type: "usage"; inputTokens: number; outputTokens: number; cachedInputTokens: number }
  | { type: "done"; stop: StopReason }
  | { type: "error"; message: string; retryable: boolean };

type StopReason = "complete" | "max_tokens" | "stop_sequence" | "tool_call" | "filtered" | "other";

The two source shapes it has to absorb are structurally different, and the difference is where playground bugs live. An OpenAI-style chat completions stream sends chunks whose choices[0].delta carries either a content string or a tool_calls array whose entries are identified by an index and whose function.arguments arrive as string fragments to be concatenated per index; the terminal reason appears as finish_reason on the last chunk and the stream ends with data: [DONE]. An Anthropic Messages stream sends named events instead — message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop — where tool arguments arrive as input_json_delta fragments inside a block that was opened with a tool-use type, and the terminal reason arrives as stop_reason on message_delta.

Both require you to accumulate partial JSON for tool arguments and both make it tempting to parse early. Do not: a fragment is not valid JSON and a parser that retries on every delta will burn CPU and log exceptions for the duration of every tool call. Accumulate, parse once at the block’s end, and show the raw accumulated string in the UI while it is incomplete. The library’s page on streaming JSON parsing covers the general problem, and testing partial JSON mid-stream covers pinning it.

What has no counterpart

These are the values that cannot be translated, only handled. A playground is the one place where getting this wrong is cheap to discover and therefore worth being strict about.

Temperature ranges differ. OpenAI’s chat completions documentation gives temperature a range of 0 to 2; Anthropic’s Messages documentation gives it 0 to 1. A saved run at 1.4 has no meaning on the second, and rescaling is a fiction — the two parameters are not the same function of the sampler. Declare the tool’s own scale, refuse to run a saved value that falls outside the target’s range, and say so in the UI rather than clamping silently.
Terminal reason vocabularies differ. OpenAI’s finish_reason takes values including stop, length, tool_calls and content_filter; Anthropic’s stop_reason takes values including end_turn, max_tokens, stop_sequence and tool_use. They overlap but are not a bijection, and both lists have grown over time. Map into your own enum at the adapter and never render the raw value, or your run history becomes a mixture of two vocabularies that no filter can query.
Some parameters exist on one side only. A top_k control, a seed for reproducibility, an n parameter returning several completions from one call. The right behaviour is to hide the control when the selected model reports no such capability, rather than to show it and drop it — a playground whose parameters silently do nothing is worse than useless because people draw conclusions from it. Where reproducibility is the goal and the target has no seed, the library’s fallback for a missing seed parameter is the honest workaround.
The system prompt sits somewhere else. A message with a system role in the array, against a top-level parameter alongside the array. The adapter owns this; the editor should keep showing one box.

Keeping old saved runs interpretable

A playground’s value accumulates in its history, and history is what a migration threatens. The record of a run has to be complete enough to explain itself years later, which means storing more than the prompt.

Store the provider id, the exact model string rather than a class or alias, the normalised request, the exact outbound body you actually sent, the normalised events, the raw usage object, and a schema version for the record itself. The exact body matters because it is the only thing that can answer “why did this behave differently” after a mapping change; the model string matters because aliases move under you, which the library covers under silent model updates.

Then decide what a saved run means when it cannot be replayed. Marking it unrunnable against the new provider, with the specific reason (“temperature 1.4 is outside this model’s range”), is better than either hiding it or quietly running something else. The run still has value as a record even where it cannot be reproduced.

Doing it

Pull the leaks back in. Find every place outside the existing adapter that touches a provider field name — usually the cost display, the stop-reason badge and the run serialiser — and route them through the normalised types. Do this before adding anything.
Write the new adapter against the union, not against the old adapter. Copying the old one and editing it reproduces its assumptions, which are exactly the things that differ.
Add a capability report to listModels and drive the parameter controls from it, so unsupported controls disappear rather than misleading.
Write a conformance test that runs one fixed prompt through every adapter and asserts the same shape: at least one text event, exactly one usage event with non-zero counts, exactly one done with a mapped stop reason, and no raw provider strings anywhere in the output.
Replay a sample of saved runs through the new adapter and classify each as reproducible, unrunnable with a reason, or errored. The third category is your bug list.
Ship the model picker last. Until the conformance test passes, the new provider should be reachable only by a flag, so nobody draws a conclusion about a model from an adapter that is still wrong.

The adapter above is the same object every service in your estate needs — the playground’s copy is simply the one people notice is wrong first, because they are staring at its output. That is the honest argument for a gateway like Multigrid: the playground calls one API and gets one event shape, and so does everything else, instead of each team maintaining its own mapping table. Building it by hand is entirely reasonable; building it five times is not.