How to check an OpenAI-compatible API relay against the official API (with a Python script)

Pat Z — Fri, 30 Oct 2026 03:41:16 +0000

Cheap OpenAI-compatible relays are everywhere: you swap base_url, keep your code, and pay less. What you can't see is what happens between your request and the model. A relay can be perfectly honest. It can also quietly do things that only show up weeks later as worse answers, a higher bill, or an agent that breaks on its third tool call.

This post lists what relays most often get wrong and how to check each one yourself in a few minutes, with one Python script you can run against your own relay. You don't need any tool of ours to use it.

The idea: same request, two endpoints

Every check below sends the same request to the relay and to the official API, then compares the two responses. A single response on its own tells you very little; the difference between the two is what you're after.

You need:

the relay's base URL and key
an official API key for the same model, so you have a baseline (a few cents of usage is enough)

pip install openai
export RELAY_BASE_URL=https://your-relay.example/v1
export RELAY_API_KEY=...
export OFFICIAL_API_KEY=...
export MODEL=gpt-4.1-mini
python relay_check.py

The full script is at the end. Here's what each check looks for.

1. Which model answered?

Every chat completion response has a model field. Ask both sides something trivial and compare.

r = client.chat.completions.create(model=MODEL, messages=[{"role": "user", "content": "Reply with the single word: ok"}], max_tokens=5)
print(r.model)

If you asked for gpt-4.1-mini and the relay's response says gpt-4o-mini-2024-07-18, that's worth a question to the relay. Two caveats:

The official API itself often returns a dated snapshot name (gpt-4.1-mini-2025-04-14), so a different suffix alone isn't suspicious.
A relay can write anything into this field. A matching name is weak evidence; a mismatched one is strong evidence.

Don't bother asking the model "which model are you?". Models are bad at answering that even when called directly.

2. Are token counts real?

You pay for tokens, and the relay reports them. Send an identical prompt to both sides with temperature=0 and compare usage.prompt_tokens.

For the same model and the same messages, the prompt token count should match the official one almost exactly; the prompt is fixed, so there's no randomness in it. A relay that consistently reports 20â€“30% more prompt tokens than the official API for the same input is overbilling you, or is wrapping your prompt in its own hidden system prompt, which you'd also want to know about.

Completion tokens will vary a bit between runs, so judge those over several requests rather than one.

3. Are your parameters passed through?

Some relays drop or rewrite parameters they don't support, or ones that cost them money. The easiest one to test is max_tokens, because the effect is unmistakable:

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Count from 1 to 200, separated by spaces."}],
    max_tokens=10,
)
print(r.choices[0].finish_reason, r.usage.completion_tokens)

The official API stops early and returns finish_reason="length". If the relay returns all 200 numbers and finish_reason="stop", it ignored your limit, and it probably ignores other parameters too. If your app depends on temperature, seed, response_format or logprobs, test those the same way: pick a request where the parameter visibly changes the output, and compare the two sides.

4. Do tool calls actually work?

This is where agents break. A relay that "supports tools" might:

return the tool call as plain text instead of a structured tool_calls array
return arguments that aren't valid JSON
handle the first call fine and then choke on the second turn, when you send back a tool message with the result

So test the whole round trip, not just the first response:

msgs = [{"role": "user", "content": "What's the weather in Lisbon? Use the tool."}]
r = client.chat.completions.create(model=MODEL, messages=msgs, tools=TOOLS, temperature=0)
call = r.choices[0].message.tool_calls[0]
args = json.loads(call.function.arguments)          # must parse
msgs += [r.choices[0].message,
         {"role": "tool", "tool_call_id": call.id, "content": json.dumps({"temp_c": 19, "sky": "clear"})}]
r2 = client.chat.completions.create(model=MODEL, messages=msgs, tools=TOOLS)
print(r2.choices[0].message.content)                 # should mention 19Â°C

If the official API completes this and the relay doesn't, the relay is not ready for agent workloads, whatever its landing page says. Chained calls (the model calls a tool, reads the result, then calls another) fail more often than single ones, so if you rely on them, extend the test to two tools.

5. Is long context really there?

A relay serving a cheaper model, or truncating long requests to save money, often gives itself away on long inputs. Hide a random code in a few thousand words of filler and ask for it back:

code = "".join(random.choices(string.ascii_uppercase + string.digits, k=8))
filler = ["Paragraph %d is ordinary filler text about nothing in particular." % i for i in range(600)]
filler.insert(random.randint(100, 500), "The secret code is %s. Remember it." % code)

Modern models find the code reliably at this length. If the official API gets it and the relay doesn't, something is being cut or swapped. Scale the filler up towards the context size you actually use.

6. How much slower is it?

Every relay adds a hop, so some extra latency is normal. Measure time to first token with streaming, take the median of a few runs, and compare:

t0 = time.perf_counter()
for chunk in client.chat.completions.create(model=MODEL, messages=msgs, stream=True):
    if chunk.choices and chunk.choices[0].delta.content:
        print(time.perf_counter() - t0); break

A few hundred milliseconds on top of the official time is a routing cost. Several seconds, or a time that swings wildly between runs, usually means queueing upstream or a route around the world. Run it at the time of day your users are active, not just once.

Reading the results

One failed check isn't a verdict; relays have bad minutes like everyone else. Run the script a few times, on different days. The patterns that matter:

Token counts consistently higher than official: you're paying for tokens you didn't send.
Parameters ignored: your app behaves differently in production than in testing against the official API.
Tool round trip fails: don't put agents on it.
Long-context recall fails while official passes: likely truncation or a different model.

And the obvious one: a relay sees your prompts and your key. Use a key you can revoke, with a spending limit, and rotate it if you stop using the relay.

The full script

Run it as shown at the top. It prints one line per check, relay next to official.

"""Compare an OpenAI-compatible relay against the official API, side by side.

pip install openai
RELAY_BASE_URL=https://your-relay.example/v1 RELAY_API_KEY=... \
OFFICIAL_API_KEY=... MODEL=gpt-4.1-mini python relay_check.py
"""
import json, os, random, string, time
from openai import OpenAI

MODEL = os.environ.get("MODEL", "gpt-4.1-mini")
relay = OpenAI(base_url=os.environ["RELAY_BASE_URL"], api_key=os.environ["RELAY_API_KEY"])
official = OpenAI(base_url=os.environ.get("OFFICIAL_BASE_URL", "https://api.openai.com/v1"),
                  api_key=os.environ["OFFICIAL_API_KEY"])
CLIENTS = {"relay": relay, "official": official}


def chat(client, **kw):
    t0 = time.perf_counter()
    r = client.chat.completions.create(model=MODEL, **kw)
    return r, time.perf_counter() - t0


def check_model_field():
    """Which model does the response say it came from?"""
    msgs = [{"role": "user", "content": "Reply with the single word: ok"}]
    return {n: chat(c, messages=msgs, max_tokens=5)[0].model for n, c in CLIENTS.items()}


def check_token_counts():
    """Same request on both sides should report (almost) the same prompt_tokens."""
    prompt = "Summarise in one sentence: " + ("The quick brown fox jumps over the lazy dog. " * 40)
    out = {}
    for n, c in CLIENTS.items():
        r, _ = chat(c, messages=[{"role": "user", "content": prompt}], max_tokens=60, temperature=0)
        out[n] = (r.usage.prompt_tokens, r.usage.completion_tokens) if r.usage else None
    return out


def check_max_tokens():
    """If max_tokens is passed through, a long answer must stop early with finish_reason=length."""
    msgs = [{"role": "user", "content": "Count from 1 to 200, separated by spaces."}]
    out = {}
    for n, c in CLIENTS.items():
        r, _ = chat(c, messages=msgs, max_tokens=10)
        out[n] = (r.choices[0].finish_reason, r.usage.completion_tokens if r.usage else None)
    return out


TOOLS = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]


def check_tool_call():
    """A real tool call: structured tool_calls, valid JSON arguments, then a second turn using the result."""
    out = {}
    for n, c in CLIENTS.items():
        msgs = [{"role": "user", "content": "What's the weather in Lisbon? Use the tool."}]
        r, _ = chat(c, messages=msgs, tools=TOOLS, tool_choice="auto", temperature=0)
        calls = r.choices[0].message.tool_calls or []
        if not calls:
            out[n] = "no tool_calls (text: %r)" % (r.choices[0].message.content or "")[:60]
            continue
        try:
            args = json.loads(calls[0].function.arguments)
        except json.JSONDecodeError:
            out[n] = "arguments are not valid JSON: %r" % calls[0].function.arguments[:60]
            continue
        msgs += [r.choices[0].message,
                 {"role": "tool", "tool_call_id": calls[0].id, "content": json.dumps({"temp_c": 19, "sky": "clear"})}]
        r2, _ = chat(c, messages=msgs, tools=TOOLS, temperature=0)
        answer = r2.choices[0].message.content or ""
        out[n] = "ok, args=%s, uses result: %s" % (args, "19" in answer)
    return out


def check_long_context_recall():
    """Hide one random code in ~6k words of filler; ask for it back."""
    code = "".join(random.choices(string.ascii_uppercase + string.digits, k=8))
    filler = ["Paragraph %d is ordinary filler text about nothing in particular." % i for i in range(600)]
    filler.insert(random.randint(100, 500), "The secret code is %s. Remember it." % code)
    msgs = [{"role": "user", "content": "\n".join(filler) + "\n\nWhat is the secret code? Reply with the code only."}]
    out = {}
    for n, c in CLIENTS.items():
        r, _ = chat(c, messages=msgs, max_tokens=20, temperature=0)
        out[n] = code in (r.choices[0].message.content or "")
    return out


def check_latency(runs=3):
    """Time to first token with streaming, median of a few runs."""
    msgs = [{"role": "user", "content": "Write two sentences about the sea."}]
    out = {}
    for n, c in CLIENTS.items():
        ttfts = []
        for _ in range(runs):
            t0 = time.perf_counter()
            stream = c.chat.completions.create(model=MODEL, messages=msgs, max_tokens=60, stream=True)
            for chunk in stream:
                if chunk.choices and chunk.choices[0].delta.content:
                    ttfts.append(time.perf_counter() - t0)
                    break
            for _ in stream:  # drain
                pass
        ttfts.sort()
        out[n] = "%.2fs" % ttfts[len(ttfts) // 2] if ttfts else "no content streamed"
    return out


if __name__ == "__main__":
    for check in (check_model_field, check_token_counts, check_max_tokens,
                  check_tool_call, check_long_context_recall, check_latency):
        try:
            res = check()
        except Exception as e:  # one failing check should not hide the others
            res = {"error": repr(e)[:120]}
        print("%-26s %s" % (check.__name__[6:], res))

Sample output against a misbehaving relay (this one is a mock I wrote to test the script, not a real provider):

model_field                {'relay': 'gpt-4o-mini-2024-07-18', 'official': 'gpt-4.1-mini'}
token_counts               {'relay': (473, 8), 'official': (364, 8)}
max_tokens                 {'relay': ('stop', 200), 'official': ('length', 10)}
tool_call                  {'relay': "ok, args={'city': 'Lisbon'}, uses result: True", 'official': "ok, args={'city': 'Lisbon'}, uses result: True"}
long_context_recall        {'relay': True, 'official': True}
latency                    {'relay': '0.40s', 'official': '0.00s'}

If you'd rather not run it yourself

Disclosure: I work on byok.icu, a free tool that runs a longer version of these checks (seven agent test cases including chained tool calls and error recovery) against your relay and the official API side by side, and gives a score out of 100 with the reasons behind it. Your key is used for that one run and not stored. It also supports Anthropic-compatible endpoints. The script above covers the basics; use whichever fits.

If you've run into other ways relays misbehave, I'd like to hear them in the comments.

DEV Community: Pat Z