Price a Free Model Pilot With a Token Burn Probe, Not a Demo

#ai #productivity #serverless #developertools

Thirty million tokens is enough to run thousands of evaluation calls. It is also enough to hide a bad prompt or an output-heavy agent from view. A team that treats a free allowance as a demo budget learns almost nothing; a team that treats it as a cost instrument learns its token mix, retry rate, and break-even sample size before a paid tier is involved.

Current developer threads are debating whether agent tool calls should be gated and whether model output can be reliably watermarked. Both discussions are downstream of a smaller decision: how many tokens a single action consumes, and how quickly the evaluation budget disappears.

The MonkeyCode open-source project offers operator-supplied free model access and a free server option. The stated allowance at the time of writing is 30 million free tokens; confirm the current terms before relying on the figure. The free server option is useful for running an isolated probe against that allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a free allowance should be a cost instrument

A demo answers whether a model can produce a plausible response. It does not answer whether a planned AI pilot will consume a predictable number of tokens per task, how often retries multiply cost, or how many evaluation rounds the free tier can support.

The probe records five fields per call:

prompt_tokens
completion_tokens
status
latency_ms
finish_reason

A high output-token share or a truncated finish reason changes the earlier cost model even when the response looks acceptable.

Reproducible token burn probe

The following is a runnable template, not an executed benchmark. Replace the endpoint and model values with the ones shown in the provider console.

import json
import os
import statistics
import time

import requests

BASE_URL = os.environ['MONKEYCODE_BASE_URL'].rstrip('/')
API_KEY = os.environ['MONKEYCODE_API_KEY']
MODEL_NAME = os.environ['MODEL_NAME']  # from the provider console

CASES = {
    'short_extract': [
        {'role': 'system', 'content': 'Return one plain sentence.'},
        {'role': 'user', 'content': 'Summarize: The pilot has a 14-day gate and an explicit owner.'},
    ],
    'tool_pick': [
        {'role': 'system', 'content': 'Choose one tool for the task. Reply with only the tool name.'},
        {'role': 'user', 'content': 'Task: rename config.yml to config.yaml. Tools: shell, git, file.'},
    ],
}

def call(messages):
    start = time.perf_counter()
    resp = requests.post(
        f'{BASE_URL}/v1/chat/completions',
        headers={'Authorization': f'Bearer {API_KEY}'},
        json={
            'model': MODEL_NAME,
            'messages': messages,
            'temperature': 0,
            'max_tokens': 128,
        },
        timeout=60,
    )
    elapsed_ms = (time.perf_counter() - start) * 1000
    if resp.status_code != 200:
        return {
            'status': resp.status_code,
            'latency_ms': round(elapsed_ms, 1),
            'prompt_tokens': None,
            'completion_tokens': None,
            'total_tokens': None,
            'finish_reason': resp.text[:120],
        }
    body = resp.json()
    usage = body.get('usage', {})
    choice = body.get('choices', [{}])[0]
    return {
        'status': resp.status_code,
        'latency_ms': round(elapsed_ms, 1),
        'prompt_tokens': usage.get('prompt_tokens'),
        'completion_tokens': usage.get('completion_tokens'),
        'total_tokens': usage.get('total_tokens'),
        'finish_reason': choice.get('finish_reason'),
    }

results = []
for case_name, messages in CASES.items():
    for _ in range(3):
        row = {'case': case_name}
        row.update(call(messages))
        results.append(row)
        time.sleep(1)

print(json.dumps(results, indent=2))

def mean(key):
    values = [r[key] for r in results if isinstance(r.get(key), (int, float))]
    return round(statistics.mean(values), 1) if values else None

total_input = sum(r['prompt_tokens'] or 0 for r in results)
total_output = sum(r['completion_tokens'] or 0 for r in results)
failed = sum(1 for r in results if r.get('status') != 200)

print('attempts', len(results))
print('failed', failed)
print('total_input_tokens', total_input)
print('total_output_tokens', total_output)
print('avg_latency_ms', mean('latency_ms'))
print('avg_prompt_tokens', mean('prompt_tokens'))
print('avg_completion_tokens', mean('completion_tokens'))

Run it from an isolated environment:

python3 -m venv .venv && source .venv/bin/activate
pip install requests
export MONKEYCODE_BASE_URL='https://your-endpoint/v1'
export MONKEYCODE_API_KEY='replace-me'
export MODEL_NAME='replace-me'
python token_burn_probe.py

If the endpoint does not return usage counts, count the prompt locally with the provider's tokenizer or ask for usage inclusion before drawing conclusions from the totals.

Scorecard with a worked example

The table below is a conversation tool, not an objective measure. An owner can change a threshold, but the change should be written down.

Field	Example probe value	Threshold	Action
Average prompt tokens	620	Above 1,200 in a 5,000-call run	Shorten or refactor the prompt before scaling
Average completion tokens	180	Above 500	Check whether the agent is over-explaining or doing hidden work
Failure rate	1 in 100	Above 2%	Stop and inspect error class, timeout, and retry behavior
p95 latency	2.3 seconds	Above 30 seconds	Do not use for synchronous human-in-the-loop UX
Retry amplification	1.0x	Any automatic retry	Add idempotency and backoff, then rerun

Worked break-even: if the probe records 620 prompt tokens and 180 completion tokens per call, each call consumes 800 tokens. A planned 5,000-call evaluation consumes 4,000,000 tokens. Under a 30,000,000-token allowance, the team can run 7.5 complete rounds before the allowance is gone.

The formula is:

rounds = floor(allowance / (planned_calls * tokens_per_call))

If input prompts grow to 1,200 tokens, the per-call total becomes 1,380 and the same 5,000 calls consume 6,900,000 tokens. The team now gets 4.3 full rounds instead of 7.5. That is the kind of change that should reverse a decision or force a prompt revision.

Exit criteria: stop or rework the pilot if the average prompt token count exceeds 1,200, p95 latency exceeds 30 seconds, or the failure rate exceeds 2% in a 200-call sample. Assign one owner and set an expiry, such as 14 days or 2,000 recorded calls, whichever comes first.

Limitations and who should not use this

The probe is not a production benchmark. A free endpoint can have rate limits, shared capacity, and different data-retention rules, so a good result here does not prove production reliability.

Do not use the free lane for:

regulated, sensitive, or personal data
user-facing production traffic
workloads with a hard latency SLA
deterministic agent behavior that cannot tolerate model variability

The free server option is for isolated evaluation. It should not become an unlicensed production host or a substitute for a security review.

The one useful next step

Check the current signup page for the live token allowance and free server option before running the harness. If the probe changes a threshold, the free allowance has done something a demo cannot. If it does not, the team has spent a small fraction of a free allowance to avoid an expensive pilot.

DEV Community