Make Batch Composition an Invariant Before Free-Tier Token Savings Rewrite Your Eval Verdicts

Opening violation (numbers below are illustrative; the failure mode is not)

Last Saturday I re-ran a 400-item evaluation sweep on a free-tier inference endpoint and the pass rate moved from roughly 71% to roughly 62%. Nothing in the model name had changed, the prompts were committed in git, and temperature was pinned to zero. What had changed was my own optimization: the earlier run sent one item per request, and the new run packed sixteen items into each call to stretch a limited free token allowance.

So which one was the experiment, and which one was the transport? My honest answer at the time was that I had no idea, and that is a bad place to be when a product decision rides on the delta. This article is the architecture review I did afterwards, and the artifact is a small probe you can run in an afternoon.

1. State the claim before you measure it

Here is what I am actually claiming, in falsifiable form. Composition invariance: for a fixed item set, fixed prompt template, fixed model, and fixed decoding settings, the extracted verdict for an item must not depend on which other items share its request.

If that property fails, then batch size is an independent variable in your eval, and every cross-run comparison silently compares two experiments. Let me be explicit about assumptions, because this argument collapses without them:

• A1. The provider does not guarantee per-item isolation inside a batched prompt. (Most chat-style endpoints do not; you are sending one concatenated user message.)
• A2. Retries re-send a modified chunk, because the failed chunk is not the original chunk.
• A3. Output is truncated at a shared max_tokens budget, so a verbose neighbour steals tokens from a terse item.
• A4. temperature=0 reduces variance but does not make a hosted endpoint deterministic across versions.

A1 through A3 are enough to break the invariant on their own. A4 tells you why a passing probe is evidence, not proof.

2. Data flow: where composition leaks in

 golden.jsonl
      |
      v
 [ shard by batch_size ] --(item order, neighbours)--> prompt builder
                                                             |
                                                             v
                                              single user message, N items
                                                             |
                             +-------------------------------+----------------+
                             |                                                |
                        429 / 5xx                                       200 OK
                             |                                                |
                             v                                                v
                 retry with N-1 items                        shared max_tokens budget
                 (different neighbours!)                                  |
                             |                                                v
                             +----------------------------------> line-oriented parse
                                                                              |
                                                                              v
                                                              UNPARSED / label drift

The interesting edge is the retry loop, not the happy path. A retry does not merely repeat work; it changes the experimental condition for every item that survives into the retried chunk. That is the counterexample I want you to hold in your head: item 7 passes at batch size 16, fails at batch size 16 with a retry, and passes again at batch size 1.

3. The minimal probe

Five numbered steps, one file. No framework, because the point is to make the failure visible, not to build a platform.

Step 1 — Freeze a golden set you are willing to re-run

Forty to a hundred items is enough to catch composition sensitivity; it is not enough to estimate your true pass rate. Each line is an id, a prompt, and an expected label.

{"id": "c001", "prompt": "Does this retry policy terminate? ...", "expected": "pass"}
{"id": "c002", "prompt": "Is this ledger update idempotent? ...", "expected": "fail"}

Step 2 — Build the batched request explicitly

# batch_probe.py - is your verdict a function of the item, or of its neighbours?
import json, time, urllib.request

SYSTEM = 'Answer with exactly one label per line: [id] pass or [id] fail. No prose.'

def call_batch(items, *, endpoint, key, model, temperature=0, max_tokens=1024):
    user = '\n'.join('[{}] {}'.format(it['id'], it['prompt']) for it in items)
    payload = {'model': model, 'temperature': temperature,
               'max_tokens': max_tokens,
               'messages': [{'role': 'system', 'content': SYSTEM},
                            {'role': 'user', 'content': user}]}
    req = urllib.request.Request(
        endpoint, data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + key})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.loads(r.read())
    return body, time.time() - t0

Step 3 — Parse strictly, and let unparsable count as a distinct outcome

def parse_labels(body, expected_ids):
    text = body['choices'][0]['message']['content']
    out = {}
    for line in text.splitlines():
        line = line.strip()
        if not line.startswith('['):
            continue
        head, _, tail = line.partition(']')
        iid = head.strip('[').strip()
        if iid in expected_ids:
            out[iid] = tail.strip().lower() or 'EMPTY'
    return out

Do not be generous here. A parser that repairs truncated lines hides exactly the failure you are hunting, and UNPARSED is a real measurement, not noise to be cleaned away.

Step 4 — Sweep batch size and diff per item

def run_sweep(items, batch_size, *, endpoint, key, model):
    verdicts = {}
    for start in range(0, len(items), batch_size):
        chunk = items[start:start + batch_size]
        body, _ = call_batch(chunk, endpoint=endpoint, key=key, model=model)
        labels = parse_labels(body, {it['id'] for it in chunk})
        for it in chunk:
            verdicts[it['id']] = labels.get(it['id'], 'UNPARSED')
    return verdicts

def flips(runs):
    ids = sorted({iid for r in runs.values() for iid in r})
    return [(iid, {bs: r.get(iid, 'MISSING') for bs, r in runs.items()})
            for iid in ids
            if len({r.get(iid, 'MISSING') for r in runs.values()}) > 1]

Step 5 — Run it, and publish the denominator

python batch_probe.py --items golden.jsonl --batch 1,4,16 --model "$MODEL" \
  --endpoint "$BASE_URL" --out runs/$(date +%F)
# report: items=60 batch_sizes=[1,4,16] flips=7 flip_rate=0.117 calls=85

Seven flips out of sixty items is not a rounding error; it means your previous comparison was measuring packing efficiency. The flip_rate denominator is the golden set size, and it belongs in the same paragraph as your pass rate.

4. Failure injection matrix

Run these deliberately, one at a time, against a small slice. If a row produces no observable change, that is a finding worth recording too.

Injected fault	Mechanism	What should break	Signal to watch
Mid-batch 429	Retry shrinks the chunk	Item neighbours change	Verdict flip on surviving items
Shared max_tokens exhausted	Last items truncated	Parse failure, not wrong label	UNPARSED count rises with batch size
Duplicate id echo	Model repeats an id	Last-write-wins overwrite	Two labels for one id in a single response
Reordered output	Model groups by label	Position-based parsing	Parser assigns labels to wrong ids
Runner restart mid-sweep	Chunk boundary lost	Mixing of two batch sizes in one report	Call count inconsistent with item count
Allowance exhausted mid-sweep	Partial run	Cherry-picked prefix of the corpus	Missing tail ids, unreported

That last row is the one that embarrasses people in review. A sweep that dies at 60% and reports the 60% is not a smaller experiment, it is a biased one, because your golden set is rarely randomly ordered.

5. Tradeoff table (template; fill the numbers from your own run)

Batch size	Tokens per item	Wall clock	Observed flip_rate	Acceptable when
1	Baseline	Highest	Expected ~0	You need defensible numbers
4	Measure it	Measure it	Measure it	Flip rate is zero across 3 repeats
16	Measure it	Measure it	Measure it	Exploratory triage only
Dynamic (retry-shrunk)	Lowest	Lowest	Usually worst	Never for a reported metric

I refuse to give you multipliers here, because tokens-per-item depends on your prompts, your output verbosity, and the provider's accounting. The method is the deliverable; the numbers are yours to produce.

6. Acceptance rule

A run is comparable to another run only if all three hold. Same golden set hash, same declared batch_size, and flip_rate measured at or near zero for that batch size on that model version. If any of the three changes, you do not have a regression; you have a re-baseline, and the report should say so in the first sentence rather than the appendix.

A pragmatic gate I now use: if flip_rate > 0 at the batch size I want, I either fall back to batch size 1 for the reported number, or I keep batching but re-run the full corpus three times and report the spread. The second option costs tokens, which brings us to where a free tier actually belongs.

7. Where a free tier belongs in this workflow

A composition sweep is a high-volume, low-stakes, disposable workload, which is the one shape of work that fits a free allowance well. I run the multi-batch probe against MonkeyCode's free model access, and I run the harness itself on the free server option so the sweep never competes with anything I care about. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance is reported by the project's outreach material (10M tokens as of this writing) and I have not independently verified its size, duration, or what happens when it is spent. Treat it as a budget variable, not a constant: log tokens per call, and stop the sweep at a declared ceiling so a partial run is visibly partial. If you want to reproduce the probe end to end, the free model access plus the free server option is enough to get from golden.jsonl to a flip_rate number today, with current terms on the project's site as your source of truth.

8. Limitations and who should not use this

• Model version drift. The probe controls composition, not the provider's weights. A probe that passes today can fail next month for reasons outside your repository, so pin the model identifier you actually called in the run output.
• Underpowered golden sets. Forty items detect gross contamination; they cannot detect a two-point regression. Scale the set before you scale the claim.
• Determinism is not promised. Many hosted endpoints treat temperature=0 as best-effort. Three repeats, not one, before you write zero in the table.
• Do not use this for latency-critical paths. If each request serves one user, batching is irrelevant and the probe is wasted effort.
• Do not use a third-party free endpoint as your audit trail if you need contractual reproducibility. Run the probe there, then re-baseline the reported number on infrastructure you control.

9. What I would change next

Two things. First, I would make batch_size a field in the result record rather than a flag in my shell history, so no future reader can mistake a repacking for a regression. Second, I would add a shuffle repeat: same items, same batch size, different order, and diff again. If order alone flips verdicts, the contamination is positional and you can often fix it by randomizing once and freezing that order forever.

Here is my counterexample question for you. Which event order in your own harness breaks composition invariance first: a retry that shrinks the chunk, a shared output budget that truncates the tail, or a runner restart that splices two batch sizes into one report? And once you find it, should the system reject the item, replay the original chunk unchanged, or compensate by re-running the affected items at batch size 1 before publishing a number?