Runcrate

Posted on Oct 21

Switch from OpenAI to open models in two lines (streaming, tools and JSON mode still work)

#ai #openai #llm #python

Most LLM code I see calls the OpenAI SDK. That's fine, because the SDK doesn't care whose server is on the other end. If the server speaks the same API, you can move a whole app to open models by changing two lines, and keep everything else: streaming, tool calls, JSON mode, retries.

This post shows the switch with Runcrate's inference API, which serves 223 models behind one OpenAI-compatible endpoint, with open-weight families like DeepSeek, Qwen, Llama, Kimi, GLM and gpt-oss among them. Every snippet below was run against the live API on 2026-10-21.

The two lines

Before:

from openai import OpenAI

client = OpenAI()

After:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.runcrate.ai/v1",
    api_key=os.environ["RUNCRATE_API_KEY"],
)

Then pick a model name from the catalog:

r = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2",
    messages=[{"role": "user", "content": "Summarize this stack trace: ..."}],
)
print(r.choices[0].message.content)

Keys start with rc_live_ and come from Dashboard > Keys > API Keys.

Or zero lines

Both official SDKs read the base URL and key from the environment, so for code that already does OpenAI() with no arguments you don't have to touch the code at all:

export OPENAI_BASE_URL=https://api.runcrate.ai/v1
export OPENAI_API_KEY=rc_live_...

I checked this with openai for Python and for Node. Same for TypeScript if you prefer it explicit:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.runcrate.ai/v1",
  apiKey: process.env.RUNCRATE_API_KEY,
});

const r = await client.chat.completions.create({
  model: "meta-llama/Llama-3.3-70B-Instruct-Turbo",
  messages: [{ role: "user", content: "Reply with the word ok." }],
});
console.log(r.choices[0].message.content);

And plain curl:

curl https://api.runcrate.ai/v1/chat/completions \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "Reply with the word ok."}]}'

Picking a model, with its price

models.list() returns every model with its per-token price, so you can choose from code instead of a pricing page. This prints the text models and what they cost per million tokens:

from openai import OpenAI

client = OpenAI()  # OPENAI_BASE_URL and OPENAI_API_KEY set as above
for m in client.models.list():
    if m.output_modalities != ["text"]:
        continue
    p = m.pricing
    print(f'{m.id:55} ${float(p["prompt"]) * 1e6:.3f} in / ${float(p["completion"]) * 1e6:.3f} out per 1M tokens')

On 2026-10-21 that printed 160 text models. A few of them:

Model	Input per 1M tokens	Output per 1M tokens
`openai/gpt-oss-120b`	$0.037	$0.17
`meta-llama/Llama-3.3-70B-Instruct-Turbo`	$0.10	$0.32
`deepseek-ai/DeepSeek-V3.2`	$0.26	$0.38
`Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo`	$0.30	$1.00
`moonshotai/Kimi-K2.5`	$0.49	$2.50

Prices change, so read them from the endpoint rather than from this post. The rest of the catalog is image, video and speech models (FLUX, Whisper and others), on the same key.

What carries over

I ran each of these against deepseek-ai/DeepSeek-V3.2 with the stock Python SDK, no changes:

Streaming

stream = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Tool calls. With a get_weather(city) tool and the prompt "What's the weather in Paris?", the response came back with finish_reason="tool_calls" and arguments {"city":"Paris"}, the same shape your existing tool loop already handles.

JSON mode. response_format={"type": "json_object"} returned a parseable object.

Reasoning models. Models that think out loud, such as openai/gpt-oss-120b, put the reasoning in a separate reasoning_content field next to content, so your app still reads content as the answer.

Support for tools and JSON mode depends on the model, as it does anywhere, so test the one you pick with your own prompts before you switch production traffic.

What doesn't carry over

Model names. gpt-4o doesn't exist here; an unknown name returns a 404 with code: "model_not_found". Keep the model name in config so the switch is one value.
Embeddings. There are no embedding models in the catalog right now, so leave your embedding calls where they are.
This post only covers chat completions. If you rely on other OpenAI endpoints, check them first.

A pattern I like: one env var per environment

Because only the base URL, key and model change, I keep all three in env vars and switch per environment without code changes:

OPENAI_BASE_URL=https://api.runcrate.ai/v1
OPENAI_API_KEY=rc_live_...
LLM_MODEL=deepseek-ai/DeepSeek-V3.2

That also makes A/B tests between models a config change, which is how most people end up comparing cost and quality on their own data instead of on a leaderboard.

DEV Community