Write the Validator Before the Prompt

Your free endpoint will rarely fail loudly. It fails politely — a fluent, well-punctuated answer in a shape your parser cannot use.

That is the failure I keep running into, and it is not really a model problem. It is a measurement problem. So I stopped tuning prompts first and started writing the validator first: the gate that output must pass before anything downstream touches it. Gate, then prompt, then harness. In that order the numbers mean something. In any other order you are grading vibes.

The workload I test with

Boring on purpose. One agent step: read a ticket title, decide one move, report confidence. Three keys. No nesting. No prose.

import json

REQUIRED = {"action": str, "target": str, "confidence": float}

def gate(raw: str):
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return False, None
    if set(obj) != set(REQUIRED):
        return False, None        # extra keys are failure, not a bonus
    if not all(isinstance(obj[k], t) for k, t in REQUIRED.items()):
        return False, None
    if not 0.0 <= obj["confidence"] <= 1.0:
        return False, None
    return True, obj

Look at that third check. set(obj) != set(REQUIRED) rejects surprises. Every helpful extra field — reasoning, notes, explanation — is schema drift you will be debugging next month. Be strict now, or be strict at 3 a.m. Your call, but the second one costs more.

And why reject instead of repair? Because a repair function is a second, untested parser sitting between you and the truth. If the gate cannot say yes cleanly, I want a no.

The harness

Six cells. Two schedules, three prompt sizes. Five requests per cell is enough to see a cliff and cheap enough to run twice.

import os, time, httpx

BASE  = os.environ["MODEL_BASE_URL"].rstrip("/")
PATH  = os.environ.get("MODEL_PATH", "/v1/chat/completions")  # copy from your endpoint's docs
KEY   = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_NAME"]

PROMPTS = {"short": "...", "medium": "... x200 ...", "long": "... x2000 ..."}

def call(prompt: str) -> str:
    r = httpx.post(
        f"{BASE}{PATH}",
        headers={"Authorization": f"Bearer {KEY}"},
        json={"model": MODEL,
              "messages": [{"role": "user", "content": prompt}],
              "temperature": 0},
        timeout=120,
    )
    return r.json()["choices"][0]["message"]["content"]   # adjust to your envelope

def run(gap: float, n: int = 5):
    rows = []
    for size, prompt in PROMPTS.items():
        for _ in range(n):
            t0 = time.perf_counter()
            raw = call(prompt)
            dt = time.perf_counter() - t0
            ok, _ = gate(raw)
            rows.append((size, ok, dt))
            time.sleep(gap)
    return rows

burst  = run(gap=0)     # warm path
spaced = run(gap=90)    # cold path

Only the extraction line is endpoint-specific. Everything else cares about one thing: the text that came back, and whether it cleared the gate.

Why burst and spaced are different experiments

Back-to-back requests measure a warm system. Ninety-second gaps measure whatever your agent actually does at 8 a.m. on a Monday. Those are rarely the same number, and the gap is where free tiers usually hide their cost.

So ask yourself honestly: does my workload look like a queue or a heartbeat? A cron job every ten minutes is a heartbeat. You cannot quote your burst p50 for it.

The table, and the thresholds you set first

Fill this in from your own run. I am deliberately not publishing my cells — they would be stale after the next deploy and you would be tuning to my network instead of yours.

Cell	Valid rate	p50 (s)	p95 (s)	Verdict
short / burst
short / spaced
medium / burst
medium / spaced
long / burst
long / spaced

Decide the verdicts before you look at the data, or you will rationalize whatever comes out:

• Valid rate ≥ 0.95 — keep it. Gate still runs in production, but it is a tripwire, not a crutch.
• 0.80 to 0.95 — gate plus exactly one retry, and log every retry with its cell. A retry is not free just because the token was.
• Below 0.80 — fall back, or shrink the task until it fits. Shrinking is usually the cheaper fix.

That last row is the whole point of the exercise. The gate does not make a weak cell good. It makes it visible.

Run it from where you deploy

A latency number measured from your laptop over hotel wifi is a story about your laptop. If your harness runs in the same place as your app, the numbers at least share a vantage point — same region, same egress, same container limits.

This is where MonkeyCode fits the workflow for me: the project offers free model access and a free server option, so the harness can run next to the workload it measures without a spend approval first. The operator also quotes a free token allowance (10M at the time of writing) and I am repeating their claim, not my measurement — check the current terms on the project page, because allowances move and my copy of the number will rot. I am not going to quote hardware specs or quotas I cannot verify myself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One honest limitation of this setup: a single free server is one vantage point. It tells you how the endpoint behaves for that node, not for your users in three regions. If geography matters, this experiment answers a narrower question than you asked.

Where this approach breaks

It breaks the moment the schema gets nested. Two levels of arrays inside a strict gate is a different experiment with a much bigger retry budget, and I would not sign off on the 0.80 threshold there.

It also breaks if you need a pinned model version for audits, or a p99 latency promise you can put in writing. Free tiers move. Any evaluation you run has a shelf life, and yours expires the next time the provider ships.

So who should not use this? Anyone whose next hop is a payment, a delete, or a migration. Gate-and-retry is fine for classification and routing. It is not fine as the last checkpoint before something irreversible. Keep a deterministic path there.

The takeaway

You cannot tell whether a free endpoint is good by reading its answers. You can tell by counting how many of them your code was willing to accept, and by admitting which schedule you measured.

Write the gate first. Then argue about the prompt. And if you want to run this without paying for the privilege of discovering your schema was too strict, the free model access and free server option are a reasonable place to start — do verify the current offer before you build anything on top of it.