Load-Test Oracles Do Not Belong on Free Inference

A latency SLO is a contract with numbers. A free inference endpoint is a courtesy with none. Teams that let a hosted model decide whether a k6 run looks healthy are not measuring the service. They are measuring whoever else sat on the shared pool that afternoon.

The confusion is easy to understand. Load tests produce ugly histograms. Models summarize ugly histograms in fluent English. Fluent English is not a p95. Once the summary is allowed to flip a CI job from red to green, the test has changed owners. The owner is no longer the threshold file. The owner is a sampler that can stall, return 429, or paraphrase a failure into a transient blip.

This is a field guide for when not to put that owner on a free path. It is not a ban on using a model to draft a scenario. Drafting and judging are different jobs. Mixing them is how a performance budget turns into a vibe.

Arithmetic does not need a prompt

Consider a frozen run. Each sample is a status code and a latency. The error ratio is a division. The 95th percentile is a sort and an index. Neither operation benefits from temperature. Neither operation should wait on a remote queue.

A small scorer makes that boring on purpose. The file below is a local oracle. It reads JSON, prints JSON, and exits 2 when the budget is missed. CI can read an exit code. CI cannot read a shrug.

# score_run.py — deterministic oracle, no network
import json, math, sys
from pathlib import Path

LIMITS = {"error_ratio": 0.01, "p95_ms": 250, "p99_ms": 400}

def pct(xs, p):
    ys = sorted(xs)
    if not ys:
        raise ValueError("empty series")
    k = min(len(ys), max(1, math.ceil(p / 100.0 * len(ys))))
    return ys[k - 1]

def score(samples):
    lat = [s["latency_ms"] for s in samples]
    n = len(samples)
    err = sum(1 for s in samples if s["status"] >= 400) / n
    p95, p99 = pct(lat, 95), pct(lat, 99)
    passed = (
        err <= LIMITS["error_ratio"]
        and p95 <= LIMITS["p95_ms"]
        and p99 <= LIMITS["p99_ms"]
    )
    return {
        "n": n,
        "error_ratio": round(err, 6),
        "p95_ms": p95,
        "p99_ms": p99,
        "pass": passed,
        "oracle": "local-arithmetic",
    }

if __name__ == "__main__":
    data = json.loads(Path(sys.argv[1]).read_text())
    verdict = score(data)
    json.dump(verdict, sys.stdout, indent=2)
    sys.stdout.write("\n")
    sys.exit(0 if verdict["pass"] else 2)

A generator that never talks to a model keeps the other half honest. Thresholds in the runner are documentation for humans. The offline scorer is the contract.

// load.js — k6 scenario; scoring happens after the process exits
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  vus: 20,
  duration: "90s",
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<250", "p(99)<400"],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/v1/health`);
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(0.05);
}

Capture raw points, then score them later. Two commands, one verdict, zero prompts.

k6 run --env BASE_URL=http://127.0.0.1:8080 --out json=raw.json load.js
python3 json_to_samples.py raw.json > samples.json
python3 score_run.py samples.json

A proposed converter for k6 JSON lines is mechanical. It does not interpret the run. It only reshapes points the scorer already knows how to judge.

# json_to_samples.py — proposed k6 --out json line converter
import json, sys

samples = []
for line in open(sys.argv[1]):
    line = line.strip()
    if not line:
        continue
    ev = json.loads(line)
    if ev.get("type") != "Point":
        continue
    if ev.get("metric") != "http_req_duration":
        continue
    data = ev.get("data", {})
    tags = data.get("tags") or {}
    samples.append({
        "latency_ms": data["value"],
        "status": int(tags.get("status", 0) or 0),
    })
json.dump(samples, sys.stdout)

If the last scoring command cannot be replayed from samples.json alone, the oracle has grown a network dependency. That is the first smell. A bathroom scale that texts a poet for the number is not a scale. It is a conversation. Conversations do not belong in the deploy graph.

Red flags on the write path

The first red flag is a model call inside the job that publishes pass or fail. A comment generator after the status is published is a report. A comment generator that is the status is a hidden coin. Free pools make the coin cheaper. They also make it noisier. Courtesy capacity is shared. Shared capacity is the wrong place to store a contract.

The second red flag is letting the model rewrite thresholds because a run looked like warmup. Warmup is a phase in the scenario file. It is not a paragraph the model is invited to invent after seeing a bad p99. Once the model can move the goalposts, every regression has a story. Stories do not page. Missed budgets do.

The third red flag is using free inference as the system under test and as the judge. That pairing is circular. The same congestion that wrecked the histogram can also delay or dilute the write-up. A delayed write-up that never fails the job is how a saturated pool becomes fine on the dashboard.

The fourth red flag is asking the model to read a trace dump and pick the bottleneck as a merge condition. Trace narratives are useful in a ticket. They are not a substitute for a recorded p95 against a frozen environment. If the environment cannot be frozen, the test is already lying. A fluent lie is still a lie.

Draft on courtesy compute, freeze the scenario, score at home

A model can draft load.js from a prose description of the API. That is authoring. Authoring can happen on a workstation, even on a free model path or a free server, because the artifact that leaves the session is text under review. The reviewed text is committed. The committed text is what CI runs. The model is not in the loop when the numbers appear.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option. Those are reasonable places to sketch a k6 script, a sample converter, or a threshold file. They are not reasonable places to host the oracle that flips the job. Remove the product names and the method still holds: draft with whatever is convenient, freeze the scenario, score with arithmetic on hardware the team controls.

After the scorer has spoken, a model may write a paragraph for humans. The paragraph must consume the JSON verdict, not replace it. Feed the model the already-computed object. Forbid it from emitting pass. A cheap guard is a schema that simply has no place for a vote.

# narrate.py — optional, after score_run.py, never the gate
import json, sys
from jsonschema import validate

NARRATION_SCHEMA = {
    "type": "object",
    "required": ["headline", "notes"],
    "additionalProperties": False,
    "properties": {
        "headline": {"type": "string", "maxLength": 120},
        "notes": {"type": "string", "maxLength": 2000},
    },
}

def main():
    verdict = json.loads(sys.stdin.read())
    assert "pass" in verdict and "p95_ms" in verdict
    draft = json.loads(sys.argv[1] if len(sys.argv) > 1 else "{}")
    validate(instance=draft, schema=NARRATION_SCHEMA)
    print(json.dumps({"verdict": verdict, "narration": draft}, indent=2))

if __name__ == "__main__":
    main()

The narration is furniture. The verdict is load-bearing. Furniture can live on a free path. Load-bearing walls cannot. A second alternative is to keep even the narration local: a template over the JSON. Templates do not 429.

jq -r '"p95=\(.p95_ms)ms p99=\(.p99_ms)ms errors=\(.error_ratio) pass=\(.pass)"' verdict.json

That line will look the same in a year. A chat completion will not.

Exit criteria

Stop using a free inference path the moment the output can change a deploy, a rollback, a page, or a customer-facing SLO report. That is the hard exit. Soft exits come earlier. Exit when two replays of the same samples.json can disagree. Exit when the scorer needs a network allow-list. Exit when a 429 from the model pool would leave the performance job yellow instead of red.

Exit when the test is the only evidence a capacity review will see. Reviewers will ask how the number was produced. A sentence about asking a model is not a method. Sorting 12,441 samples and comparing them to LIMITS in git is a method.

Stay off the free path when the system under test is itself a model endpoint the team does not provision. Courtesy capacity is a moving floor. A moving floor cannot prove a ceiling.

Who should ignore this, and what this is not

Throwaway laptop experiments with no budget, no CI job, and no audience can call whatever they want. They are not oracles. They are sketches. Teams that already compute percentiles in the load runner and only want a human-readable recap after the fact can use a model as a printer. Printers do not get a vote.

Do not use this article as an excuse to skip load tests. Deterministic scoring of an empty file is still empty. Do not use it as a claim about any vendor's uptime, quota, hardware, or model catalog. Those change, and this article does not measure them. It argues a placement rule: if the number is a contract, the contract cannot live on courtesy compute.

A practical check belongs in CI so the rule does not depend on memory. The grep is crude. Crude is fine. The point is a tripwire, not a parser.

# forbid_llm_oracle.sh — fail the pipeline if the scorer grows a client
set -euo pipefail
if grep -E 'requests\.|httpx|urllib\.request|openai|anthropic|chat\.completions' score_run.py; then
  echo "score_run.py must remain network-free" >&2
  exit 1
fi
python3 -m py_compile score_run.py

If the scorer imports an HTTP client next week, the tripwire should fire before the first looks-healthy merge. Placement is the skill. Free model access is a draft bench. A free server is a scratch pad. The oracle is a short Python file next to the thresholds. Keep the bench and the pad. Do not promote them into the courtroom.

If a team needs a scratch pad to draft the harness, MonkeyCode's free model access and free server option are one place to do that writing. Commit the script. Score it at home.