Probe the Free Server Before You Promise a Response Time

Last month, a product team told me their free AI server was fine. They had dashboards, uptime alerts, and no idea when it actually got slow.

Their users knew. Support tickets knew. The retry button was doing a lot of quiet work.

I asked one question. What evidence did you collect before you promised a response time? Silence.

The consequence was not a crash. It was a thousand small retries. Users waited, wondered, and left. The point of reversibility — switching to their own key — came too late.

Free servers degrade. That is not a bug report. It is a design input. You cannot design the decision point until you measure the degradation.

I tested this workflow against MonkeyCode. It is an open-source project with a free server option and free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The current free tier includes a ten-million-token allowance. That number changes. Check the docs before you quote it. The server is shared. Shared means variable. Variable means you need a probe.

This tutorial builds a complete degradation probe from zero. Each stage has a verification step. If a stage fails, stop and fix it before moving on.

Stage 1: Measure the baseline

You need a real endpoint. Not a dashboard. Not a promise. A real request with a real token.

Here is the command. Replace the endpoint and key with your own.

ENDPOINT="https://your-free-server.example/v1/chat/completions"
for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{time_total} %{http_code}\n" \
    -X POST "$ENDPOINT" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $MC_FREE_KEY" \
    -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}'
  sleep 2
done

Run it once. Then run it again at a different hour. Then run it at 3am.

Verification: you now have thirty latency samples. Calculate the p50, the p95, and the error rate. If you cannot run this command, you are designing blind. Stop here.

Stage 2: Define stop conditions

A probe without thresholds is just noise. You need stop conditions. These are the lines where your design must change behavior.

Start with three. The p95 latency where the UI stops pretending. The error rate where the UI offers a fallback. The timeout where the UI stops retrying.

{
  "degraded": { "p95_latency_ms": 5000, "error_rate": 0.10 },
  "stop": { "timeout_ms": 10000, "consecutive_failures": 3 }
}

These numbers are hypotheses, not facts. Your Stage 1 baseline decides whether they make sense. If your p95 is already six seconds, a five-second threshold is a joke.

Verification: write the thresholds next to the baseline. Ask yourself which threshold you would want to know about first. That is your primary stop condition.

Stage 3: Automate the probe

Manual curls are fine for one afternoon. They are not fine for a product. Build a small script that logs results.

// probe.mjs — run with: node probe.mjs
const endpoint = process.env.MC_FREE_ENDPOINT;
const key = process.env.MC_FREE_KEY;
const samples = 30;

const results = [];
for (let i = 0; i < samples; i++) {
  const start = Date.now();
  try {
    const res = await fetch(endpoint, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${key}`
      },
      body: JSON.stringify({
        messages: [{ role: "user", content: "ping" }],
        max_tokens: 5
      })
    });
    results.push({ latency_ms: Date.now() - start, status: res.status });
  } catch {
    results.push({ latency_ms: Date.now() - start, status: 0 });
  }
  await new Promise(r => setTimeout(r, 2000));
}

const ok = results.filter(r => r.status === 200).map(r => r.latency_ms).sort((a, b) => a - b);
const p95 = ok[Math.floor(ok.length * 0.95)] ?? null;
const errorRate = results.filter(r => r.status !== 200).length / results.length;
console.log(JSON.stringify({ p95, errorRate, samples: results.length }, null, 2));

Run it with environment variables. Log the output to a file. Then schedule it.

MC_FREE_ENDPOINT="https://your-free-server.example/v1/chat/completions" \
MC_FREE_KEY="$MC_FREE_KEY" node probe.mjs >> probe-log.jsonl

Verification: you can point the probe at any endpoint and get one JSON line. If the log stays empty, the script failed. Fix that first.

Stage 4: Design the hand-back moment

Now the interesting part. The probe crossed a threshold. What does the user see?

Not a spinner. A spinner promises resolution. A degraded free server may not resolve. Show a decision instead.

probe → thresholds → state → decision card → user choice

I have argued elsewhere that a degraded server deserves a decision, not a spinner. This is the evidence pipeline that makes that decision honest.

The decision card needs three things. What is happening. What the user can do. What each option costs.

// UI state derived from the probe, not from vibes
const state = {
  mode: p95 > 5000 ? "degraded" : "normal",
  options: p95 > 5000
    ? ["wait", "bring your own key", "come back later"]
    : ["continue"]
};

Notice what this state object does not contain. No spinner. No fake progress bar. Just a mode and a set of options. That is the whole design.

The "wait" option needs a stop condition. Never let a user wait forever. Give them a countdown and a way out. The "bring your own key" option needs a clear hand-back. The user should know exactly what changes when they switch.

Verification: walk through each option. Can the user exit every state? Can they recover their original context? If not, the design is incomplete.

Stage 5: Verify with a fake slow server

You cannot wait for real degradation to test your design. Build a fake one.

// slow-server.mjs — simulates degradation for testing
import http from "node:http";
http.createServer((req, res) => {
  const delay = Math.random() > 0.7 ? 8000 : 300;
  setTimeout(() => { res.writeHead(200); res.end("ok"); }, delay);
}).listen(9999);

Point the probe at it. Confirm the p95 crosses your threshold. Confirm the UI flips to degraded mode.

Then check accessibility. Screen readers must announce the decision card. Focus must move to the card. The countdown must not be the only signal. Color alone is never enough.

Verification: run the probe against the fake server and capture the state change. If the state does not change, your threshold logic is broken. If the card is not announced, your accessibility work is broken.

Limitations

This probe measures the endpoint you test. It does not measure your user's network path. It will not catch regional issues. It is a design tool, not an observability platform.

Do not use this approach if you have a paid SLA. Use real monitoring, real tracing, and real on-call. A thirty-line script is not that.

Free tiers change. The token allowance and the server behavior are not permanent. I quoted the allowance as of this writing. Verify it against the current docs.

Do not hardcode thresholds from one run. One afternoon of samples is a hypothesis. A week of samples is evidence.

The point

A free server is a gift with a hidden cost. The cost is variability. The cure is measurement.

Run the probe. Define the stop conditions. Design the hand-back. Then you can honestly promise a response time.

Your users will not thank you for the probe. They will thank you for the honest decision it enables.

If you want a free endpoint to practice on, MonkeyCode's free server is one option. Its docs list the current limits. Measure first. Then design.