Bench-Testing a Free Model Endpoint in 90 Minutes

A nightly refactor job begins failing at 2 a.m., and nobody touched the code. The team had moved one summarization step onto a free model endpoint the week before, and the only thing that changed was the response shape and the latency tail. The details differ from team to team; the shape recurs. Free access answers whether a developer can try a model. It does not answer whether an automated loop can depend on one.

This is a workshop outline for that second question. It teaches participants to build a small probe, run it against whatever endpoint they actually have credentials for, and turn the results into a go/no-go decision. The whole thing fits in ninety minutes and produces an artifact the team keeps.

Why measure before wiring

Most fatigue around AI tooling is not fatigue with models. It is fatigue with unmeasured claims. A team adopts an endpoint because a demo looked fast, then discovers during an incident that timeouts arrive without a status code, that usage counters are sometimes missing, and that rate limits are signaled in a header nobody parsed.

The probe below does not benchmark a provider. It measures one account, one model, one network path, on one afternoon. That is a deliberately narrow claim, and it is the only claim the workshop defends.

The schedule

The first ten minutes are framing: the facilitator draws the failing-job story on a whiteboard and asks the room which property they would have wanted to know in advance. Latency, error taxonomy, and token accounting usually win.

Minutes ten through thirty are an inventory exercise. Participants list every place their own code calls a model, then mark which calls sit inside an automated loop rather than behind a human. Only the automated ones matter for this workshop.

Minutes thirty through fifty-five are the build. Two files, roughly ninety lines total, run from a terminal.

Minutes fifty-five through seventy are analysis: percentiles, error counts, and the awkward question of what happens when the usage object is absent. Minutes seventy through eighty-five turn the numbers into a decision with the table further down. The last five minutes are spent writing a stop rule into the repository, because a decision nobody wrote down is a preference.

Exercise 1: the probe

The probe sends the same fixed prompt twelve times, sequentially, and records one JSON line per run. Sequential matters. Concurrency looks impressive and destroys the signal, because a rate limiter will fold your latency distribution into a step function.

// probe.mjs — node probe.mjs > runs.jsonl
import { performance } from 'node:perf_hooks';

const BASE  = process.env.PROBE_BASE_URL;
const PATH  = process.env.PROBE_PATH ?? '/chat/completions';
const KEY   = process.env.PROBE_API_KEY;
const MODEL = process.env.PROBE_MODEL;
const N     = Number(process.env.PROBE_N ?? 12);

if (!BASE || !KEY || !MODEL) {
  console.error('set PROBE_BASE_URL, PROBE_API_KEY, PROBE_MODEL');
  process.exit(2);
}

The path above is the common OpenAI-compatible shape, not a guarantee. Participants confirm the base URL and path against the provider's current documentation before the first run, and write that link into the file header.

const PROMPT = [
  'Summarize this build log in exactly three bullet lines.',
  'No preamble. Do not invent line numbers.',
  'line 12: warning: unused import React',
  'line 44: error: TS2345 argument of type string',
  'line 91: warning: deprecated API call'
].join('\n');

async function once(i) {
  const started = performance.now();
  const ctrl = new AbortController();
  const timer = setTimeout(() => ctrl.abort(), 60_000);
  let attempts = 0, status = 'ok', http = 0, usage = null, text = '';

  while (attempts < 4) {
    attempts++;
    try {
      const res = await fetch(BASE + PATH, {
        method: 'POST',
        signal: ctrl.signal,
        headers: {
          'content-type': 'application/json',
          authorization: `Bearer ${KEY}`
        },
        body: JSON.stringify({
          model: MODEL,
          messages: [{ role: 'user', content: PROMPT }]
        })
      });
      http = res.status;
      if (res.status === 429 || res.status >= 500) {
        if (attempts < 4) {
          const ra = Number(res.headers.get('retry-after') ?? 0);
          const wait = ra > 0 ? ra * 1000 : 500 * 2 ** attempts + Math.random() * 250;
          await new Promise(r => setTimeout(r, wait));
          continue;
        }
        status = http === 429 ? 'rate_limited' : 'server_error';
        break;
      }
      if (!res.ok) { status = 'client_error'; break; }
      const body = await res.json();
      usage = body.usage ?? null;
      text = body.choices?.[0]?.message?.content ?? '';
      break;
    } catch (err) {
      status = err.name === 'AbortError' ? 'timeout' : 'network_error';
      if (attempts >= 4) break;
      await new Promise(r => setTimeout(r, 400 * 2 ** attempts));
    }
  }

  clearTimeout(timer);
  return {
    i, status, http, attempts,
    total_ms: Math.round(performance.now() - started),
    usage, chars: text.length, sample: text.slice(0, 120)
  };
}

for (let i = 0; i < N; i++) console.log(JSON.stringify(await once(i)));

Running it is one command. Participants export credentials into the shell rather than pasting them into the file, and the facilitator checks for that habit before the first execution.

PROBE_BASE_URL="$MC_BASE_URL" PROBE_PATH="/chat/completions" \
PROBE_API_KEY="$MC_API_KEY" PROBE_MODEL="$MC_MODEL" PROBE_N=12 \
node probe.mjs > runs.jsonl

Exercise 2: reading the numbers

The summarizer is short on purpose. Percentiles over twelve samples are descriptive, not inferential, and it should say so out loud when it prints.

// summarize.mjs — node summarize.mjs runs.jsonl
import { readFileSync } from 'node:fs';

const rows = readFileSync(process.argv[2], 'utf8')
  .trim().split('\n').map(l => JSON.parse(l));
const ok = rows.filter(r => r.status === 'ok');
const lat = ok.map(r => r.total_ms).sort((a, b) => a - b);
const q = p => lat.length ? lat[Math.floor(p * (lat.length - 1))] : NaN;
const usageSeen = ok.filter(r => r.usage).length;

console.log({
  runs: rows.length,
  ok: ok.length,
  errors: rows.length - ok.length,
  statuses: [...new Set(rows.map(r => r.status))],
  p50_ms: q(0.5),
  p95_ms: q(0.95),
  max_ms: lat.at(-1),
  usage_reported: `${usageSeen}/${ok.length}`
});

The interesting column is rarely the median. If usage_reported is not equal to ok, the accounting layer cannot be trusted, and any per-request cost model the team built becomes a guess. Participants who find that should stop tuning prompts and start reading the provider's terms.

Exercise 3: a collector that stays put

A laptop on office Wi-Fi is a bad measurement instrument, and a laptop on hotel Wi-Fi is worse. Participants who want the twelve runs to be comparable across weeks should place the collector somewhere stable.

MonkeyCode offers free model access and a free server option, which makes it a reasonable place to park this harness for a first pass: the probe runs from a fixed machine, writes runs.jsonl, and the team reads the file. Confirm the endpoint, the current limits, and the terms in the provider's documentation before the session rather than during it. No quota, hardware specification, or duration is assumed here, because those details change and a workshop should not teach stale numbers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same instruction applies to any free tier: the free server is a place to run a measurement, not a substitute for an availability guarantee. A team that needs one should be talking to a vendor, not reading a workshop outline.

Exercise 4: what the room usually gets wrong

One participant will want to add concurrency. Let them, in a separate file, and then compare. The step function that appears is the lesson. Another will want to tune the prompt until output quality looks good; that is a different workshop, and mixing it in here contaminates the latency data.

A third will ask whether twelve runs are enough. They are enough to catch a missing usage field, a 429 storm, or a two-second tail. They are not enough to estimate a p99, and the summarizer should print a warning when someone asks it to.

The decision table

The room fills this in together, then commits it next to the probe script.

Signal from the run	Reading	Action
p95 well under the loop's timeout, no errors, usage present	Predictable enough for a guarded trial	Wire behind a timeout and a kill switch, log every response
p50 fine, p95 close to or above the timeout	Tail will page someone at night	Add a timeout below the observed p95 and a fallback path first
usage missing on most successes	Cost and token budgets are unmeasurable	Do not build budget enforcement on this endpoint yet
Any 429s at concurrency one	Limits are tighter than the loop assumes	Batch, cache, or move the call out of the loop

Limitations and who should skip this

Twelve sequential calls from one machine measure one path at one moment. Provider behavior changes, free-tier conditions change, and a good afternoon proves nothing about next month. The probe also says nothing about output quality, data handling, or licensing, which are separate reviews with separate owners.

Teams under a service-level obligation, teams handling regulated data without a reviewed agreement, and teams planning production volume should not adopt this workflow as a substitute for procurement. The probe is a first filter, not a contract.

For groups that want to rerun the harness weekly without maintaining a box, standing the collector up on MonkeyCode's free server and pointing it at the free model access is the shortest path the organizers know — bring the current terms and read them in the room, out loud.