The Mean Looked Fine. Goodput Said No.

Mean latency is a liar. Goodput is the number I keep now.

I learned this the boring way. I pointed a small agent loop at a free model endpoint. The average looked fine. Users still complained.

Why? Because averages hide the tail. And free tiers live in the tail.

Two free things, one real question

The setup I wanted to test was simple. A free model endpoint, plus a free server to run the client from. That is a real offer now. MonkeyCode provides free model access and a free server option. (Disclosure: This article was prepared as part of MonkeyCode's product outreach.)

I will not quote quotas, model names, or box specs here. Those change weekly. A stale number is worse than no number, so check the current terms yourself before you plan around them.

The real question is not "is it free?". The real question is "does my workload survive the variance?".

The bottleneck was not speed

My first profile was useless. I measured mean round-trip time and got a small, happy number. I felt clever for about an hour.

Then I looked at the spread. A tight mean with a fat tail means a fraction of users wait far too long. Your dashboard says green. Your user says "this thing hangs".

So I stopped measuring speed. I started measuring goodput: the share of requests that land inside a latency budget I actually care about. Everything else counts as a miss, including failures.

The harness I keep

This is the artifact. It is a single file, no dependencies, Node 18+. Save it as goodput.mjs.

// goodput.mjs — measure the tail, not the vibe
const BASE   = process.env.BASE_URL;      // e.g. https://host/v1
const KEY    = process.env.API_KEY;
const MODEL  = process.env.MODEL;
const N      = Number(process.env.N ?? 100);
const BUDGET = Number(process.env.BUDGET_MS ?? 3000);
const PROMPT = 'Reply with exactly one word: pong';

const samples = [];
let attempts = 0;

async function one() {
  const t0 = performance.now();
  attempts++;
  const res = await fetch(`${BASE}/chat/completions`, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${KEY}`,
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: 'user', content: PROMPT }],
      max_tokens: 4,
    }),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  await res.arrayBuffer();               // drain the body
  return performance.now() - t0;
}

// one retry only, so retries cannot hide a broken tier
async function once() {
  try { return await one(); }
  catch { return await one(); }
}

const wallStart = performance.now();
await Promise.all(
  Array.from({ length: N }, async () => {
    try { samples.push(await once()); }
    catch { samples.push(Infinity); }   // failure is a miss, not a skip
  })
);
const wallMs = performance.now() - wallStart;

samples.sort((a, b) => a - b);
const q = (p) => samples[Math.min(samples.length - 1, Math.floor(p * samples.length))];
const hits = samples.filter((x) => x <= BUDGET).length;

console.log({
  n: samples.length,
  budget_ms: BUDGET,
  goodput: (hits / samples.length).toFixed(3),
  p50: Math.round(q(0.5)),
  p95: Math.round(q(0.95)),
  p99: Math.round(q(0.99)),
  attempts_per_hit: hits ? (attempts / hits).toFixed(2) : 'n/a',
  rps: (samples.length / (wallMs / 1000)).toFixed(2),
});

Run it like this. Keep N small at first.

BASE_URL="$BASE" API_KEY="$KEY" MODEL="$MODEL" \
  N=100 BUDGET_MS=3000 node goodput.mjs

The output below is illustrative only. Your numbers will differ, and they should.

{ n: 100, budget_ms: 3000, goodput: '0.910', p50: 640,
  p95: 2900, p99: 6100, attempts_per_hit: '1.04', rps: 3.10 }

Read attempts_per_hit before anything else. If it creeps past one, you are paying for reliability with wall time. A fast endpoint that needs three tries is a slow endpoint wearing a hat.

The experiment that actually settles it

One run proves nothing. So run the same file from two places. First from your laptop, then from the free server.

# on the free box
scp goodput.mjs box:~ && ssh box \
  'BASE_URL=$BASE API_KEY=$KEY MODEL=$MODEL N=100 node goodput.mjs'

Same code, same prompt, same N. If the free box's p95 is much worse, your problem is the box or the path. If both match, the endpoint is the variable. Now you know which lever to pull.

Then sweep the budget instead of picking one.

for B in 500 1000 2000 4000 8000; do
  BUDGET_MS=$B N=100 node goodput.mjs
 done

The graph I kept

I keep exactly one chart. X axis is the latency budget in milliseconds. Y axis is goodput. It is a curve, not a point.

Why a curve? Because a single number invites an argument. "Two seconds is fine." Maybe. The curve ends the argument — it shows what you buy at each budget.

And it survives a tier change. When the endpoint or the box shifts, re-run the sweep and overlay the line. If the new line sits under the old one, something regressed. You will see it before your users do.

Do the same after a cold period. Wait a while, then fire a single request and watch it. Cold starts are real on shared hosts. They belong in your budget, not in your excuses.

When free is the right call

Free is not a moral position. It is a fit question. Here is how I decide.

Workload	Shape	Free tier verdict
Interactive autocomplete	tight budget, tail hurts	measure hard, keep budget low
Overnight batch backfill	loose budget, volume high	usually fine, watch retries
CI test suite	bursty, must be predictable	risky, pin a fallback
Production p99 SLO	contractual	not here, not with this

Notice the pattern. Loose budgets forgive variance. Tight budgets do not. Free tiers are excellent when a miss is cheap and terrible when a miss is visible.

Limits, and who should walk away

This method measures a client's view. It will not tell you why the server stalled. It also assumes your provider exposes an OpenAI-shaped endpoint, and it skips streaming.

Two more honest caveats. A quiet laptop is not a benchmark rig. And a 100-request run is a smoke test, not a capacity plan.

Skip this if you need contractual latency. Skip it if your data cannot leave your network. Skip it if nobody will look at the curve again.

But if you just want to know whether a free endpoint and a free box can carry your workload, stop arguing about averages. Run the sweep from both places, plot goodput against budget, and keep one line.

If you want a place to start, MonkeyCode's free model access and free server option are enough to produce that first curve this afternoon. Then bring back your own graph, not mine.