Migrating an Internal Model Capability Matrix

#ai #machinelearning #tutorial #llm

The capability matrix is the table someone made in a wiki that says which models do vision, tool calling and structured output. It answers “can this model do X”. It is not the inventory of what you run in production, which is a different table with a different failure mode — that one is the model reference doc.

What belongs in the matrix

Rows are the models you can call. Columns are capabilities named in your vocabulary, not in either provider’s. This matters more than it sounds: if the column headings are copied from one vendor’s feature names, every cell for the other vendor requires a translation somebody performs from memory, and the translations disagree between the people who fill in the table.

Keep the row set small as well. Rows are models you can actually call with the credentials you hold, not every model the vendor publishes. A matrix listing models nobody has access to invites somebody to route traffic at one during an incident and discover the permission problem at the worst moment.

A workable column set for most teams: image input; tool calling; several tool calls in one response; schema-constrained output; prompt caching, with its minimum prefix; streaming of partial tool arguments; context window; maximum output tokens; and whatever else your product actually depends on. Add a column only when a routing decision or an adapter branch depends on it. A matrix with thirty columns is a matrix nobody refreshes.

Half of it can be fetched, half cannot

The migration-specific discovery is that the two provider families disagree about whether capability is part of the API at all.

Anthropic’s models endpoint returns, per model, an id, a display_name, a max_input_tokens (the context window), a max_tokens (the output cap), and a nested capabilities object with a supported boolean at each leaf — image input, structured outputs, thinking modes, effort levels and so on. That is a machine-readable matrix you can fetch on a schedule and diff. Note the absence of a field called context_window; the context window is max_input_tokens, and a script written against the intuitive name gets nothing.

OpenAI’s models endpoint returns an id, an object, a created timestamp and an owned_by string. There is no capability information in it at all, and there is no context-window field. Everything in your matrix for those rows has to come from documentation a human read, or from probing.

So the migration turns a table that could plausibly have been hand-maintained into two tables with different refresh mechanisms, and the honest thing to do is make that visible rather than pretend the whole thing is equally trustworthy.

Probing the half that cannot

A capability probe is a deliberately tiny request whose purpose is to observe the shape of the outcome, not the content. You are not asking whether the model is good at tool calling; you are asking whether the request is accepted and whether a tool call comes back at all.

# one probe per (model, capability); assert on shape, not on content
def probe_tools(client, model):
    r = client.chat.completions.create(
        model=model,
        max_completion_tokens=32,
        messages=[{"role": "user", "content": "What is the weather in Oslo?"}],
        tools=[{
            "type": "function",
            "function": {
                "name": "get_weather",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                    "additionalProperties": False,
                },
            },
        }],
    )
    msg = r.choices[0].message
    return "supported" if msg.tool_calls else "no-tool-call"
# a 400 from the call is its own result: record the error type and message

Three rules keep a probe suite useful. Catch the error rather than letting it fail the run, because a rejection is a result and its message is the most informative cell in the table. Keep the token budget small enough that a nightly full sweep is background noise on the bill. And never assert on model output text — a probe that checks the weather answer is a flaky test, while a probe that checks whether a tool_calls array is present is deterministic.

Supported is not a boolean

The cell value that causes the most trouble after a migration is a tick that should have been a caveat. Three examples, all of which will bite somebody on your team:

Schema-constrained output with schema restrictions. The capability is present, and the JSON Schema subset accepted is not the whole of JSON Schema. Recursion, numeric bounds and string length constraints are commonly unsupported, and additionalProperties: false is commonly required on every object. A schema that validated fine elsewhere is rejected here, and the matrix said “yes”.
Caching with a minimum that excludes you. Supported, documented, and inapplicable if your stable prefix is shorter than the model’s minimum cacheable length — in which case it silently does nothing, as covered in re-deriving caching projections.
Capability gated behind a mode. Available only when a particular parameter is set, or unavailable when another one is — a combination that returns a 400 rather than degrading. The cell needs to say which combination.

So each cell has three possible states, not two: supported, supported with a constraint, unsupported. In the constrained case the constraint text is the value of the cell. A matrix of ticks and crosses is a matrix that will be contradicted by production within a month.

Generating it, with provenance

Write the column vocabulary down first, in your own terms, with a one-sentence definition per column. Disagreements about what “tool calling” means are cheaper to settle here than in a cell.
Fetch what the provider will tell you. Where a models endpoint exposes capability flags, read them and record the response verbatim alongside the parsed value.
Write one probe per remaining cell, following the shape rules above, and run the suite in CI on a schedule rather than on every commit.
Emit the matrix as a generated file — the same artefact the wiki page renders — with two extra columns per cell: when it was last verified, and how (endpoint, probe, or documentation read by a human). A cell whose provenance is “documentation” and whose date is eight months old is the one to distrust first.
Fail the scheduled job loudly on a changed cell. The value of the matrix during a migration is not its current contents but its diff: a capability that disappeared between two nightly runs is a deprecation you would otherwise learn about from a customer.

A probe suite is per-capability, not per-service, so running it inside every service that calls a model duplicates both the code and the spend. Whatever holds the model bindings centrally — a gateway such as Multigrid, or a shared internal library — is the natural place for it, because that is also where the routing decisions the matrix informs are made.