Treat Free Inference as a Removable Part

A free inference lane looks like capacity you gained.
It is also a dependency you accepted.

If you cannot unplug it in one commit, review it as architecture.
That framing changes which questions you ask first.

Most teams adopt a free lane during a spike.
The spike ends, the lane stays, and nobody writes the exit test.
Six months later the exit cost lands on release day.

Review against four constraints

A removable part has a documented boundary.
A free lane usually has none of that written down.

Constraint	What you can assume	What to verify before relying on it
Availability	Best effort, shared capacity	Whether any SLA exists in writing
Permanence	Terms and limits can change	The current published terms, on the day you ship
Quota	Some allowance exists	The exact allowance and what happens at zero
Data handling	Your prompt leaves your machine	Retention, logging, and training policy

None of these are reasons to avoid a free lane.
They are reasons to keep the lane behind a seam.

Trace the data flow first

Draw the hops before you write the adapter.
Here is the flow I would expect to find.

job spec -> lane adapter -> network egress -> provider
         -> response -> artifact -> receipt -> your repo

Each arrow is a failure domain you own.
The provider owns only the middle one, and only partly.

Your adapter is the only hop you fully control.
That is why the seam belongs there, not in your business logic.

Artifact 1: one seam for every lane

The scaffold below is a proposal, not a tested release.
It exists to make lane swapping a config change, not a refactor.

# lane.py - one seam for every inference lane
import os, time, json, hashlib, urllib.request

class Lane:
    name = "abstract"
    model = "unset"

    def complete(self, prompt: str) -> str:
        raise NotImplementedError

class HttpLane(Lane):
    def __init__(self, name, model, url, key_env):
        self.name, self.model, self.url = name, model, url
        self.key = os.environ.get(key_env, "")

    def complete(self, prompt: str) -> str:
        payload = {"model": self.model,
                   "messages": [{"role": "user", "content": prompt}]}
        req = urllib.request.Request(
            self.url,
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json",
                     "Authorization": "Bearer " + self.key})
        with urllib.request.urlopen(req, timeout=30) as resp:
            data = json.loads(resp.read())
        return data["choices"][0]["message"]["content"]

def receipt(lane, prompt, out, ms, attempt, ok):
    return {"lane": lane.name, "model": lane.model,
            "prompt_sha256": hashlib.sha256(prompt.encode()).hexdigest()[:16],
            "out_len": len(out), "ms": ms,
            "attempt": attempt, "ok": ok, "at": int(time.time())}

Two details matter more than the code.
The model string is recorded, never assumed.
And the receipt stores a prompt hash, not the prompt.

A hash proves you ran the same input twice.
Storing the prompt would move your egress problem into your logs.

Name the failure domains

A free lane fails differently from a paid one.
Build the detection path before the retry path.

Failure	Detection	Degradation you should define
Quota is exhausted	Non-success status on first attempt	Stop the batch, keep the queue
Provider returns errors	Timeout or 5xx in the receipt	Two retries, then switch lanes
Free server is unreachable	Connection error before any response	Fall back, never fake output
Terms or limits changed	Weekly config check fails	Fail the build, not the user
Output quality drifts	Swap test agreement drops	Route that job type to review

Retries are where free lanes quietly become expensive.
Cap them at two per job and count them in the receipt.

Artifact 2: the swap test

The only honest way to judge a lane is to run your jobs on two.
Here is a small harness that does one job set through both.

# swap_test.py - run one job set through two lanes and diff
import json, time, difflib, pathlib
from lane import HttpLane, receipt

LANES = [HttpLane("free", "MODEL_A", "https://free.example/v1/chat", "FREE_KEY"),
         HttpLane("paid", "MODEL_B", "https://paid.example/v1/chat", "PAID_KEY")]

raw = pathlib.Path("jobs.txt").read_text()
prompts = [p.strip() for p in raw.split("\n---\n") if p.strip()]
rows = []

for p in prompts:
    outs = {}
    for lane in LANES:
        t0, ok, out = time.time(), False, ""
        try:
            out = lane.complete(p)
            ok = True
        except Exception as err:
            out = "ERROR: " + type(err).__name__
        ms = int((time.time() - t0) * 1000)
        outs[lane.name] = out
        rows.append(receipt(lane, p, out, ms, 1, ok))
    ratio = difflib.SequenceMatcher(None, outs["free"], outs["paid"]).ratio()
    rows.append({"kind": "agreement", "ratio": round(ratio, 3)})

print(json.dumps(rows, indent=2))

Run it and keep the output as your baseline.

printf 'job one\n---\njob two\n---\njob three\n' > jobs.txt
python swap_test.py > receipts.json
python -c "import json;d=json.load(open('receipts.json'));print([r['ratio'] for r in d if 'ratio' in r])"

Agreement is a routing signal, not a quality score.
Read it as a heuristic and re-run it after every lane change.

My working rule, still untested on real workloads:
run the set three times, and treat a job type as lane-portable only if agreement stays high across all three.
Below that line, the free lane still runs, but a human reads the output.

What I would change next

Cap retries at two and record each attempt separately.
A single "failed" flag hides a lane that burned your quota slowly.

Keep a fallback lane configured but disabled.
An enabled fallback becomes the primary lane within a month.

Hash prompts in receipts and keep the raw text in your own store.
That keeps the audit trail inside your boundary.

Pin the model string you sent, not the model you intended.
Aliases move, and your receipts will not tell you.

Add one config check per week that reads the provider's current terms page.
Fail the check loudly when the text changes.

Re-run the swap test whenever the lane, the model, or the job set changes.
Cheap tests age badly, and so do free lanes.

Where a free lane fits in this design

A free lane is a good fit when the seam exists and the exit is tested.
That is the whole point of building the adapter first.

The operator of this account has supplied two availability claims for MonkeyCode, an open-source coding project: free model access, and a free server option.
I have not benchmarked either, and I have not verified the current allowance.
The allowance has been described to me as a fixed token amount with a free server, and that description should be re-checked against the published terms on the day you decide.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where it genuinely participates in this workflow is as a lane behind the seam.
You point one HttpLane at it, run the swap test, and read the agreement numbers for your own jobs.
The free server option matters for the same reason: a box you do not manage is a failure domain you must still name.

That is a testable claim, which is the only kind worth repeating.
If the seam works, the free lane is an easy addition and an easy removal.

When this approach is wrong

Do not send regulated or licensed data to any lane you cannot audit.
No free tier changes that constraint, and no adapter hides it.

Do not use this if your job needs a hard latency target.
Shared capacity and retries do not mix with tight SLOs.

Do not use it in a no-egress environment, even for drafts.
The first prompt is already an egress event.

And do not adopt a free lane without a fallback you have actually exercised.
A fallback you never ran is a comment, not a plan.

A free lane you cannot remove is not capacity.
It is a deadline you scheduled for someone else.