Pin the Model, Log the Tokens: An FAQ for Free-Tier Agent Runs

Every week ships a new agent demo and a new free tier. The demos look identical. The invoices and the incident reports do not. So which claims survive contact with your own terminal?

I keep a run ledger for every model call I make. It is boring. It is also the only reason I trust any free-tier claim.

Here is the FAQ I wish someone had handed me before I burned a week on mystery failures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access, a free server option, and a free grant of ten million tokens. Treat those as vendor-stated terms and confirm the current numbers yourself before you plan around them.

Myth 1: "A free model is a fixed thing"

It is not. A free endpoint is a routing decision someone else can change on a Tuesday.

The model string you typed last month may now resolve to a different backend, a different quantization, or a different context limit. Your test suite is then comparing apples to a rumor.

The fix is boring: pin and timestamp. Log the exact model identifier, the base URL, and the clock time on every call.

No pin, no comparison. No timestamp, no bisect.

Myth 2: "The run succeeded, so the model is fine"

Success means one thing. The HTTP layer returned 200 and the JSON parsed.

It says nothing about whether the answer was conformant, complete, or even on topic. I have shipped "green" runs that were quietly wrong for six days.

Ask yourself two questions. Did my parser accept it? Did my assertion accept it?

Those are different gates. Keep them separate in the ledger.

Myth 3: "The token grant is a budget"

A grant is a ceiling. A budget is a plan. They are not the same object.

Worse, usage reporting is optional in practice. Some responses carry a usage block. Some carry nothing, or zeros, or numbers that only count half the request.

If usage is missing, you cannot reconcile anything. You can only guess. So log both: the reported numbers, and a clearly labeled estimate.

Never mix the two columns. Never present an estimate as a measurement.

Myth 4: "A free server is a deploy target"

This is the expensive one. A free server is often ephemeral. Disk resets, processes sleep, and cold starts eat your latency budget.

That does not make it useless. It makes it a runner, not a host of record.

Good fits: stateless sidecars, cron-ish jobs, log shippers, a scratch environment for the ledger itself. Bad fits: anything holding the only copy of state.

Would you run your database there? Then do not run your only artifact there either.

Myth 5: "Logging is overhead I can add later"

Later never comes, because later is when you need the history you did not record.

The ledger has to exist before the first interesting failure. It is ten lines of code. Write it now.

The artifact: a 60-line call wrapper that logs itself

Two files. Standard library only. No SDK required.

ask.py calls an OpenAI-compatible endpoint and appends one JSON line per attempt.

#!/usr/bin/env python3
import json, os, sys, time, hashlib, datetime, urllib.request, urllib.error

BASE_URL = os.environ["BASE_URL"].rstrip("/")   # e.g. https://host/v1
API_KEY  = os.environ["API_KEY"]
MODEL    = os.environ["MODEL_ID"]               # pin it, do not default it
LEDGER   = os.environ.get("RUNLOG", "runs.jsonl")

def log(rec):
    rec["ts"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
    with open(LEDGER, "a", encoding="utf-8") as f:
        f.write(json.dumps(rec) + "\n")

def ask(prompt, max_tokens=400):
    body = json.dumps({"model": MODEL,
                       "messages": [{"role": "user", "content": prompt}],
                       "max_tokens": max_tokens}).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"})
    t0 = time.monotonic()
    status, err, payload = None, None, {}
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            status = r.status
            payload = json.loads(r.read())
    except urllib.error.HTTPError as e:
        status, err = e.code, e.read()[:200].decode("utf-8", "replace")
    except Exception as e:
        err = repr(e)
    usage = payload.get("usage") or {}
    log({"model": MODEL, "base_url": BASE_URL, "status": status,
         "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
         "prompt_chars": len(prompt),
         "usage_reported": bool(usage),
         "tokens_in": usage.get("prompt_tokens"),
         "tokens_out": usage.get("completion_tokens"),
         "latency_ms": round((time.monotonic() - t0) * 1000),
         "error": err})
    return payload

if __name__ == "__main__":
    out = ask(sys.stdin.read())
    msg = (out.get("choices") or [{}])[0].get("message", {}).get("content", "")
    print(msg)

Run it, then look at the ledger instead of your memory.

export BASE_URL="https://your-endpoint/v1"
export API_KEY="your-key"
export MODEL_ID="your-pinned-model-id"
echo "Reply with the word READY and nothing else." | python3 ask.py
tail -n 1 runs.jsonl | python3 -m json.tool

If usage_reported is false, you just learned something important. Your grant is unobservable from inside the client.

The audit: which claims are actually checkable

audit.py rolls the ledger up per model. Reported and estimated columns stay apart on purpose.

#!/usr/bin/env python3
import json, sys, collections

rows = [json.loads(l) for l in open(sys.argv[1] if len(sys.argv) > 1 else "runs.jsonl", encoding="utf-8") if l.strip()]
agg = collections.defaultdict(lambda: {"runs": 0, "ok": 0, "no_usage": 0,
                                      "tin": 0, "tout": 0, "est_tin": 0})
for r in rows:
    m = agg[r["model"]]
    m["runs"] += 1
    m["ok"] += 1 if r.get("status") == 200 else 0
    m["no_usage"] += 0 if r.get("usage_reported") else 1
    m["tin"] += r.get("tokens_in") or 0
    m["tout"] += r.get("tokens_out") or 0
    if not r.get("usage_reported"):
        m["est_tin"] += round(r.get("prompt_chars", 0) / 4)  # rough estimate only

print(f"{'model':<28}{'runs':>6}{'200':>6}{'no_usage':>10}{'tin':>9}{'tout':>9}{'est_tin':>9}")
for k, v in sorted(agg.items()):
    print(f"{k:<28}{v['runs']:>6}{v['ok']:>6}{v['no_usage']:>10}"
          f"{v['tin']:>9}{v['tout']:>9}{v['est_tin']:>9}")

The no_usage column is the honest one. A high count means you cannot audit spend at all.

A 12-minute drill you can run today

1. Pin MODEL_ID and BASE_URL in your shell profile. No fallbacks.
2. Send the same 3 prompts, 5 times each. Vary nothing.
3. Run audit.py. Read no_usage first, tokens second.
4. Re-run tomorrow. Diff the model string and the latency band.
5. If anything changed, you found the real variable. Not the model.

Do this before you write a single benchmark post. Please.

Decision table: when this setup fits

Your workload	Free model access	Free server as runner	Verdict
Throwaway prototypes, prompt shaping	Yes	Yes	Good fit. Log everything.
CI linting, JSON shape checks	Yes	Yes, if stateless	Good fit. Pin the model per job.
Anything with an SLA or a paying user	No	No	Pay for a contract you can point at.
The only copy of your data	No	Never	Ephemeral disk is not a backup.
Regulated or audited token spend	Only if usage is reported	Only with external logging	You must supply the ledger.
Long-running stateful services	Maybe	No	Cold starts and resets will find you.

Read the middle column as "technically possible", not "supported forever".

Limitations, stated plainly

The ledger only records what your client can see. Server-side throttling stays invisible. If a provider strips the usage block, your numbers are estimates and should be labeled that way.

The prompt_chars / 4 estimate is a rough heuristic for English text. It will be wrong on code, JSON, and non-Latin scripts. Treat it as a shape, not a measurement.

Free grants change. A ten-million-token allowance today says nothing about next quarter. Snapshot the terms you were given, with a date, and re-read them monthly.

And a passing run proves one sample passed. Nothing more. Sample size is a separate article.

Who should skip this

If you already have a paid contract with a usage dashboard you trust, your provider's billing UI beats my JSONL file. Use theirs.

If your workload is stateful, or your users are paying, the free server is the wrong tool. Stop reading and provision a real one.

If you cannot afford to pin a model identifier because you need automatic fallback, this workflow will fight you. Logging is cheap. Pinning has a cost.

What I actually take away

The interesting question is never "is the free tier good?". It is "what did I observe, when, against which model string?".

That question has a cheap answer. Sixty lines, a JSONL file, and a habit.

If you want to try the workflow against MonkeyCode's free model access and free server option, run the drill above on your first ten calls. Then decide, with your own ledger in hand, whether the grant is enough for your workload.