A Tiny Eval Harness for Picking Between GPT, Gemini and Claude

#ai #llm #testing #webdev

"Which LLM is best?" is the wrong question. The useful one is: which model performs best on my feature's real inputs, at a speed and token cost I can live with? You can answer that with a small harness and an afternoon.

The setup

You need three things:

A rubric. Three to five pass/fail or 1–5 criteria. Example for an extraction task: valid JSON, all fields present, numbers match the source.
A dataset. 30–50 real inputs (PII masked). Mix typical cases with edge cases: very long inputs, mixed languages, malformed text, requests that should be refused.
One interface. A single function your app and your harness both call.

Keep the model behind one function

The key design rule: the rest of your codebase never knows which provider it's talking to. With OROO Blue (one API, one OROO key for OpenAI, Google and Anthropic models), the model is just a string.

MODELS = ["oroo-chatgpt", "oroo-gemini", "oroo-claude"]

function generate(prompt, model):
    # same gateway, same key, only the model name changes
    request = {
        model: model,
        prompt: prompt,
        temperature: CONFIG.temperature,    # identical for every model
        max_tokens: CONFIG.max_tokens,
        idempotency_key: hash(prompt + model)  # safe retries
    }
    return oroo_gateway.call(request, key = env.OROO_KEY)

for case in dataset:
    for model in MODELS:
        start = now()
        out = generate(case.input, model)
        record(case.id, model, out.text, out.tokens, now() - start)

The idempotency key matters: if a request times out and your harness retries, OROO Blue's idempotency handling prevents a double charge.

Score blind

Export results with the model column hidden or shuffled, then score against the rubric. Unblind only after scoring. It's surprisingly easy to "see" quality in a model you already like.

Compare on three axes

For each model, look at:

Quality: average rubric score, and specifically how it did on edge cases
Latency: median and slowest responses
Tokens: average per request, which drives running cost

Pick based on the feature. High-volume, simple tasks often favour the faster, lighter option.

Guardrails while you experiment

Use a dedicated eval key with per-key model, token and rate limits.
Key secrets are shown once and stored hashed, so put the key straight into an env var or secrets manager.
Add an IP allowlist, and revoke the key when you're done.
Spend comes from OROO Mint, a prepaid INR wallet topped up via Razorpay, and you pay only for tokens used.
OROO doesn't store prompts or completions.

Re-run on every model update

Commit the dataset and rubric to the repo. When a model changes, re-run the harness. Because MODEL lives in config, switching in production is a one-line change you can roll back.

More at orooblue.com.

Originally published on the OROO Kids blog: https://orookids.blogspot.com/2026/10/gpt-gemini-or-claude-how-to-pick-right.html

DEV Community