Emery Chen

Posted on Aug 14

Smoke-Test Any Model API Before You Benchmark It

#ai #testing #api #python

A model that can't survive a five-minute API smoke test doesn't deserve a benchmark run.

Launch week makes everything look sharp.
Benchmarks won't tell you:

whether auth works without pain
whether JSON output is actually parseable
whether the model follows a one-line instruction
whether simple calls time out

So I run a smoke test first.

Why smoke, not benchmark

A benchmark answers: how good is this model?
A smoke test answers: can I build on it at all?

Four checks:

JSON shape
Exact format
Latency ceiling
Error shape

If any fail, I skip the heavy eval.

The script

Minimal Python. No SDK. This assumes a chat-completions style endpoint; if yours differs, only the call() function changes.

import os
import time
import httpx

ENDPOINT = os.environ["MODEL_ENDPOINT"]
KEY = os.environ["MODEL_KEY"]
LIMIT_S = float(os.environ.get("LATENCY_LIMIT_S", "8"))

def call(prompt, max_tokens=64, timeout=30):
    t0 = time.time()
    r = httpx.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {KEY}"},
        json={
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
            "stream": False,
        },
        timeout=timeout,
    )
    return r, time.time() - t0

def body_text(r):
    try:
        return r.json()["choices"][0]["message"]["content"]
    except Exception:
        return ""

def check(name, ok, note=""):
    print(('PASS ' if ok else 'FAIL ') + name + (' | ' + note if note else ''))

# 1. JSON shape
r, s = call('Return JSON only: {"ok": true, "count": 3}')
check('json-shape', r.status_code == 200 and '"ok"' in r.text and '"count"' in r.text, f'{s:.1f}s')

# 2. Exact format
r, s = call('Answer with exactly one word: yes or no. No punctuation.')
check('exact-format', r.status_code == 200 and body_text(r).strip().lower() in {'yes', 'no'}, repr(body_text(r)[:20]))

# 3. Latency ceiling
r, s = call('Repeat the word hello.')
check('latency', r.status_code == 200 and s < LIMIT_S, f'{s:.1f}s < {LIMIT_S}s')

# 4. Error shape
r, s = call('')
check('error-shape', r.status_code >= 400 and (r.text or '').strip() != '', f'status={r.status_code}')

Run it with:

export MODEL_ENDPOINT="https://your-endpoint/chat/completions"
export MODEL_KEY="your-key"
python smoke_test.py

Decision table

Check	Catches	Not a test of
JSON shape	structured-output breakage	output quality
Exact format	instruction drift	reasoning
Latency ceiling	unusable real-time behavior	throughput under load
Error shape	painful integration	security

Where free access fits

Heavy eval rigs burn quota on models that may not even be callable.
A smoke test should be nearly free.

When I want a fresh endpoint without touching a paid account, I use MonkeyCode's free model access and free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits

This is a filter, not a ranking.
It won't test accuracy, tool use, streaming, or large context.
Free tiers can change; don't build production on them.
Never send private code or data to an untrusted endpoint.

Who should skip this

You need production uptime or strict data controls.
You already maintain a full eval harness.
You're choosing a model by reasoning quality; this won't answer that.

Bottom line

Run four checks before you open another benchmark tab.
The five minutes will save more time than the thread you were about to read.

DEV Community