Which Budget Binds: Designing a Time-Boxed Spike Around Free Tokens and a Free Server

Consider an illustrative, hypothetical case: a three-person team wants to know whether handing small legacy-module fixes to an AI assistant shortens the time to a first correct patch, and it has only a finite free token allowance plus a shared free server instead of a funded evaluation budget. Nothing in this article was executed for publication, and every number below appears as a decision rule rather than as an observation.

The tempting move is to copy a benchmark design from a funded lab and start collecting samples. Two days later the allowance is gone, the hypothesis is still untested, and the team owns a folder of half-labelled transcripts that nobody can re-read. A cheaper design begins by deciding which budget will bind first, because a spike limited by prompt count must be measured differently from one limited by wall-clock time on shared hardware.

The binding budget sets the unit of evidence

When tokens are the scarce input, the useful unit of evidence is one falsification attempt together with the prompt cost that produced it. The spike should therefore be designed to spend its smallest prompts on the cases most likely to kill the hypothesis, not on the cases most likely to look impressive in a summary. Each run writes one ledger row, and the stop rule is expressed in tokens rather than in hours, which keeps the time-box honest even when a single slow response eats an afternoon.

When the free server is the scarce or unpredictable input, the unit of evidence shifts toward repeated measurement of the same fixed task. Shared machines produce cold starts, noisy neighbours, and occasional timeouts, so a single wall-clock reading carries little information. A short noise probe, run before any real case, tells the team whether timing differences of a few seconds are even distinguishable from the platform's own variance.

MonkeyCode, which the operator describes as an open-source project offering free model access and a free server option, is one place such a spike could run, and the operator reports a free allowance on the order of ten million tokens. That figure is operator-supplied and unverified here, so it should be confirmed on the project's own page before any budget arithmetic depends on it, and the same caution applies to how long free access lasts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Artifact: a ledger-first harness

The harness below is illustrative and has not been run; endpoint paths, request shapes, and usage fields differ between providers and must be adapted from the official API documentation before use. It wraps a single model call, records wall time and whatever token counts the response reports, and appends one JSON line per attempt so that the whole spike can be recomputed from disk.

# spike_harness.py — illustrative only; adapt to the provider's documented API
import json, os, time, urllib.request

LEDGER = os.environ.get("SPIKE_LEDGER", "spike_ledger.jsonl")

def call_model(prompt, timeout=180):
    base = os.environ["SPIKE_BASE_URL"].rstrip("/")
    key = os.environ["SPIKE_API_KEY"]
    model = os.environ["SPIKE_MODEL_ID"]
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    req = urllib.request.Request(
        f"{base}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"},
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    wall = time.monotonic() - started
    usage = payload.get("usage") or {}
    return {
        "wall_s": round(wall, 3),
        "prompt_tokens": usage.get("prompt_tokens"),      # may be null
        "completion_tokens": usage.get("completion_tokens"),
        "text": payload["choices"][0]["message"]["content"],
    }

def record(arm, case_id, result, outcome, note=""):
    row = {
        "arm": arm,                # "baseline" or "assisted"
        "case_id": case_id,
        "wall_s": result["wall_s"],
        "prompt_tokens": result["prompt_tokens"],
        "completion_tokens": result["completion_tokens"],
        "outcome": outcome,        # "support" | "disproof" | "inconclusive"
        "note": note,
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }
    with open(LEDGER, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")

Arm assignment has to be written before the run, not inferred afterwards, otherwise the ledger quietly becomes a story about what the team hoped to see. The stop rule should also be fixed in advance, for example that the spike halts when forty percent of the reported allowance has been consumed without a single disproof. A short shell sequence keeps that discipline visible in the repository rather than in someone's memory of the conversation.

export SPIKE_BASE_URL="https://example.invalid/v1"   # from the provider's docs
export SPIKE_API_KEY="..."                            # never commit this
export SPIKE_MODEL_ID="..."                           # whatever the project lists today
export SPIKE_LEDGER="spike_ledger.jsonl"

# step 1: noise probe, same trivial prompt, five times
for i in 1 2 3 4 5; do
  echo "probe $i"
done > probe_plan.txt

# step 2: real cases, ordered so the most falsifying cases run first
python spike_harness.py --plan probe_plan.txt --arm baseline
python spike_harness.py --plan falsify_first.txt --arm assisted

# step 3: recompute everything from the ledger
python summarize_ledger.py spike_ledger.jsonl

Recomputation is the part that makes a spike survivable, since a summary produced from memory cannot be audited by a reviewer who arrives a week later. The script below derives token spend per disproof, median and high-percentile wall time per arm, and refuses to divide by zero when no disproof occurred. It treats missing usage fields as zero rather than crashing, which is itself a finding worth writing down.

# summarize_ledger.py — illustrative only
import json, statistics, sys

rows = [json.loads(l) for l in open(sys.argv[1], encoding="utf-8") if l.strip()]
arms = {}
for r in rows:
    arms.setdefault(r["arm"], []).append(r)

for arm, rs in sorted(arms.items()):
    walls = sorted(r["wall_s"] for r in rs if r["wall_s"] is not None)
    tokens = [(r["prompt_tokens"] or 0) + (r["completion_tokens"] or 0) for r in rs]
    disproofs = sum(1 for r in rs if r["outcome"] == "disproof")
    p95 = walls[max(0, int(round(0.95 * len(walls))) - 1)] if walls else float("nan")
    per_disproof = (sum(tokens) / disproofs) if disproofs else float("inf")
    print(f"{arm:9s} n={len(rs):3d} tokens={sum(tokens):7d} "
          f"tok/disproof={per_disproof:8.1f} "
          f"median={statistics.median(walls):6.2f}s p95={p95:6.2f}s")

Reading the ledger without fooling yourself

The decision table below converts a first pass over the ledger into a next action, and it deliberately favours narrowing the hypothesis over extending the run. Rows assume that the noise probe was executed first, so the spread between median and high-percentile wall time is already known.

Observation after the pre-registered number of cases	Reading	Next move
Token spend past the stop threshold with zero disproofs	The hypothesis was never exposed to a hard case	Rewrite the case list around the smallest failing input, then restart the clock
High-percentile wall time far above median on the same trivial prompt	Platform variance dominates the timing signal	Switch the primary metric to prompt count and keep wall time as context
Usage fields null on most responses	The provider reports no accounting data	Count prompts and characters instead, and stop claiming token cost
Fix rate improves while spend per disproof stays flat	The gain may be real but not cheaper	Keep the effect, drop the cost claim, and say so in the write-up

A ledger that shows nothing is still a result, and reporting it as such protects the next spike from inheriting a false premise. The honest phrasing is that under the stated conditions and within the remaining allowance, the hypothesis survived or failed, with the raw rows attached.

Boundaries and who should skip this

The approach assumes that response-level usage data exists, that cases can be ordered by falsifying power, and that a reviewer accepts a small sample as decision input rather than as proof. It does not survive when the underlying free access changes mid-run, so the ledger should record the date and the stated conditions of each session alongside the results. Rate limiting deserves the same respect as a failing test: a 429 response means stop and record, not retry until the limiter relents.

Teams that owe anyone an SLO, a regulated audit trail, or a comparison across model versions should not run this design at all, because free tiers rarely offer the stability or the retention those obligations require. The same caution applies to anyone who needs a number they can publish as a benchmark, since a spike with a dozen cases produces a direction, not a measurement. What it produces cheaply is a defensible decision about whether to spend more.

If the two-budget framing fits a problem already on your board, the harness in this article is small enough to adapt in an afternoon, and the free tier that prompted it is worth checking against the project's current documentation before the arithmetic begins.