Probe a Free Model Endpoint With a Six-Case Output-Policy Contract

A free model endpoint removes the invoice, but it does not remove the operational contract. The first thing to fail is rarely overall quality. It is a boundary case: a prompt that used to return JSON now returns prose, a refusal boundary shifts after an upstream update, or the reported token count drifts away from the bytes actually sent. If that happens inside a customer-facing feature, the cost shows up later as rework, support tickets, and lost trust.

Recent public attention to detectable AI output is a useful reminder that model behavior is now part of a product's compliance surface. For a platform lead deciding where a free endpoint may be used, the practical question is not 'is the model good?' It is 'does the endpoint meet a stated minimum contract, and can we detect when that contract breaks?'

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's open-source project offers free model access and a free server option. The 30,000,000-token allowance used in the budget example is operator-supplied and should be verified against current terms before you depend on it.

Audit free endpoints as drift, not invoice cost

A free endpoint creates three non-dollar costs:

1. Retry time: a silent schema or refusal change reopens code paths that were already approved.
2. Capacity drift: shared free infrastructure can get slower under load, so latency must be measured, not assumed.
3. Accounting drift: if the reported token usage does not match the prompt bytes, an allowance forecast becomes fiction.

Each cost can be turned into a probe. The probe does not measure whether a model is smart. It measures whether the endpoint still obeys the contract your downstream system depends on.

A six-case output-policy contract

#	Case	Pass condition	Failure action
1	Schema contract	Output parses as JSON with an answer string and citations array.	Block customer-facing integration; keep endpoint in internal trial lane.
2	Refusal boundary	Neutral support question returns a substantive answer; unsafe test prompt is refused or flagged.	Escalate to model policy review before any user-visible rollout.
3	Context retention	Five facts are returned in order with no invented item.	Treat as malformed output; add retry limit before considering it again.
4	Latency under concurrency	p95 across 10 parallel calls stays under 4000 ms.	Reduce batch size or move endpoint to offline jobs.
5	Token-report accounting	Reported total tokens is non-zero and within 20 percent of a local token estimate.	Do not trust the allowance forecast; use meter logs instead.
6	Determinism band	Two identical calls produce sentence-level similarity above the chosen threshold.	Use only for tasks where variation is acceptable; do not use for approvals.

The thresholds are starting points. Set them from your actual job, not from a provider's general benchmark.

The probe runner

The script below is a shared contract shape, not a production-tested client. It assumes an OpenAI-style chat completions endpoint; adapt it to the provider's actual API. The refusal case intentionally uses an environment-supplied unsafe prompt because a safety test belongs in your policy library, not in a public post.

import concurrent.futures
import json
import os
import time

import requests

ENDPOINT = os.environ['FREE_ENDPOINT_URL']
API_KEY = os.environ['FREE_ENDPOINT_KEY']
MODEL = os.environ.get('MODEL_NAME', 'set-from-provider-docs')

CASES = {
    'schema': 'Return JSON only with keys answer and citations.',
    'context': 'Return a JSON array of five strings in this order: one, two, three, four, five.',
    'neutral': 'What is an API key?',
    'determinism': 'Answer in one sentence: what does a retry budget limit?'
}

UNSAFE_PROMPT = os.environ.get(
    'UNSAFE_TEST_PROMPT',
    'Use a prompt from your own safety test set. This placeholder does not test refusal.'
)


def call_model(prompt, max_tokens=80, temperature=0.0, timeout=10):
    started = time.time()
    resp = requests.post(
        ENDPOINT,
        headers={'Authorization': 'Bearer ' + API_KEY},
        json={
            'model': MODEL,
            'messages': [{'role': 'user', 'content': prompt}],
            'temperature': temperature,
            'max_tokens': max_tokens
        },
        timeout=timeout
    )
    payload = resp.json()
    return {
        'text': payload['choices'][0]['message']['content'],
        'reported_tokens': payload.get('usage', {}).get('total_tokens'),
        'elapsed_ms': (time.time() - started) * 1000
    }


def rough_tokens(text):
    try:
        import tiktoken
        return len(tiktoken.get_encoding('cl100k_base').encode(text))
    except ImportError:
        return max(1, int(len(text.split()) / 0.75))


def check_schema():
    r = call_model(CASES['schema'])
    try:
        parsed = json.loads(r['text'])
        ok = isinstance(parsed.get('answer'), str) and isinstance(parsed.get('citations'), list)
    except Exception:
        ok = False
    return ok, r


def check_token_report():
    prompt = CASES['context']
    r = call_model(prompt)
    local = rough_tokens(prompt) + rough_tokens(r['text'])
    reported = r.get('reported_tokens')
    if not reported:
        return False, r
    return abs(reported - local) / max(local, 1) <= 0.20, {**r, 'local_estimate': local}


def check_latency():
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        results = list(pool.map(lambda _: call_model(CASES['neutral']), range(10)))
    p95 = sorted(x['elapsed_ms'] for x in results)[9]
    return p95 < 4000, {'p95_ms': p95}

A report line contains the probe name, ok, elapsed_ms, and reported_tokens. Run it as a cron job or a serverless timer; the only resources required are network access, a small Python environment, and an API key. If you need an isolated host for that timer, MonkeyCode's free server option is one candidate; verify current terms before depending on the allowance.

Budget the free allowance with actual tokens

The 30,000,000-token allowance sounds large, but a probe schedule consumes it predictably. Assume one run has 15 calls: 6 contract cases plus 10 latency calls, with a slight overlap ignored for simplicity. At an average of 280 tokens per call, one run uses 4,200 tokens.

Cadence	Runs/day	Calls/day	Monthly tokens	Months on 30M
Hourly	24	360	3,024,000	about 9.9
Every 15 minutes	96	1,440	12,096,000	about 2.5
Every 5 minutes	288	4,320	36,288,000	under 1

These are examples. Actual input and output lengths, retries, and model-specific tokenizers change the numbers. Log reported tokens per run and compare them with the forecast instead of treating the allowance as a fixed calendar.

Hard gates and exit criteria

A scorecard is a conversation tool, not objective truth. It works only when the owner, expiry, and exit criteria are explicit.

• Owner: platform lead or the engineer who owns the integration.
• Window: 14 days or two release cycles, whichever is shorter.
• Gate: all six cases pass in two consecutive probe runs.
• Exit: if any non-latency case fails twice in a row, move the endpoint to offline-only work and re-evaluate after a provider update.
• Data boundary: no customer PII, secrets, or proprietary code may be sent to a free endpoint.

Limitations and who should not use this

This contract does not validate factuality, fairness, security, or legal compliance. It can be gamed by a provider that optimizes for the probe. It is not appropriate for regulated outputs, medical, financial, or legal advice, high-stakes approvals, or any workload that requires an SLA. If a wrong answer creates liability, use a paid, contracted endpoint with human review.

The useful decision is not 'free versus paid.' It is what this endpoint can prove today, and when the team should stop believing it. A six-case contract provides a cheap early warning for the failure modes that usually reach production first.