An API for open-weight AI models: what to check, and how to call one with the OpenAI or Anthropic SDK

kaixiaocoding — Mon, 12 Oct 2026 03:47:37 +0000

Open-weight models — models whose trained weights are published for anyone to download — are now good enough for a lot of everyday work: coding agents, tool use, extraction, summarising. The hard part is no longer the model. It is running it. This post covers the three ways to use an open-weight model, what to check before you pick a hosted API, and the few lines of code it takes to call one from the SDKs you already use.

Open-weight is not the same as open source

"Open-weight" means the weights are public. It does not automatically mean the training data, the training code or a permissive licence are public too. Licences range from MIT and Apache 2.0 to custom licences with usage limits, so read the model card before you build a product on one. For calling a model through an API, what matters is that anyone can serve it — which is why the same open-weight model is often available from several providers at different prices and speeds.

Three ways to run an open-weight model

Run it yourself. Tools such as vLLM, SGLang, llama.cpp or Ollama expose an OpenAI-compatible endpoint on your own machine or GPU server. Full control, but you pay for the hardware whether you use it or not, and a 30B-class model needs a serious GPU.
Rent dedicated GPUs. Someone else runs the hardware; you still pay per hour, busy or idle.
Call a hosted, pay-per-token API. You send requests and pay for the tokens you use. Nothing to run, nothing to pay while idle. For most teams trying open models, this is the place to start.

What to check in a hosted API

Protocol compatibility. If the API speaks OpenAI Chat Completions (and ideally the Anthropic Messages protocol), your existing code and tools work after changing the base URL, the key and the model id.
Precision. Providers may serve the same model at full precision (BF16) or with quantized weights (FP8, 4-bit). Quantization makes serving cheaper and can change output quality slightly. A good provider tells you which precision each model runs at.
Context window and max output per model — they are not always the lab's maximum.
Tool calling and reasoning controls. Agents depend on reliable tool calls; reasoning models let you turn thinking down or off for simple tasks, which saves output tokens.
Prices per token for input, output and cached input. Cached input matters for agents, which resend long prompts on every step.
Data handling. Does the provider store request content? Use it for training? Read the privacy page, not the landing page.
Measured performance. Time to first token and tokens per second, measured and published, beat "blazing fast" in a headline.

Calling one: the code

The example below uses Tokener CA, a pay-as-you-go API for a short, curated list of open-weight models (I work on it — see the note at the end). The same pattern works with any provider that speaks these protocols. Keep the key and the model id in environment variables; model ids are listed on the models page.

export TOKENER_API_KEY="your-api-key"
export TOKENER_MODEL="a-model-id-from-the-models-page"

OpenAI SDK (Chat Completions)

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOKENER_API_KEY"],
    base_url="https://a.tokener.ca/v1",
)

resp = client.chat.completions.create(
    model=os.environ["TOKENER_MODEL"],
    messages=[{"role": "user", "content": "Explain open-weight models in one sentence."}],
)
print(resp.choices[0].message.content)
print(resp.usage)  # input, output and cached tokens: what you are billed for

Streaming (stream=True), function tools and JSON output carry over unchanged. For usage on a streamed response, pass stream_options={"include_usage": True}.

Anthropic SDK (Messages)

import os
import anthropic

client = anthropic.Anthropic(
    api_key=os.environ["TOKENER_API_KEY"],
    base_url="https://a.tokener.ca",  # no /v1: the SDK adds it
)

message = client.messages.create(
    model=os.environ["TOKENER_MODEL"],
    max_tokens=256,  # required by the Messages protocol
    messages=[{"role": "user", "content": "Explain open-weight models in one sentence."}],
)
print(message.content[0].text)

curl

curl https://a.tokener.ca/v1/chat/completions \
  -H "Authorization: Bearer $TOKENER_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"$TOKENER_MODEL\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}"

Using it from tools instead of code

Most AI tools have an "OpenAI-compatible provider" or "custom base URL" setting, and the same three values (base URL, key, model id) are all they need. Step-by-step guides: Claude Code (through the Anthropic-compatible endpoint), OpenCode, Cline, LiteLLM, Vercel AI SDK.

A sensible way to start

Do not move everything at once. Pick one well-understood task — a first-pass code review, ticket summaries, field extraction — and send it to an open-weight model for a week. Compare the outputs and the bill. If you do not want to write code first, a browser playground lets you try each model and see what a request costs (quickstart without code).

Disclosure: I work on Tokener CA, the API used in the examples. It is operated by ShowKey Information Systems Inc., a company incorporated in Ontario, Canada. Each model's price, context window, precision and measured performance are on its page at tokener.ca/models; the full API reference is at tokener.ca/docs/api-reference.

DEV Community: kaixiaocoding