You wire a small internal tool to a free model endpoint. The first call returns clean JSON. The second wraps the same object in a Markdown fence. The third times out at 30 seconds. The fourth reports 900 completion tokens for a 40-token answer.
The demo still looks fine. Your CI job does not.
That gap is why a free endpoint needs a contract bench. You are not asking whether the model is the smartest one available. You are asking whether the endpoint returns the shape your code needs, repeatedly, inside the time and token budget you can afford. Those are different questions. Only the second one protects a build.
MonkeyCode is an open-source project. Its current outreach advertises free model access, a free server option, and 10 million free tokens. Treat all three as time-bound commercial terms, not technical guarantees; verify the current offer before you depend on it.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
For this workflow, the free access is useful as a probe target, not as proof of quality. The bench below stays useful if you swap in another endpoint.
1. Write the contract before the prompt
A leaderboard asks which model wins. A contract bench asks which promises survive repetition. Start with the promises your code already makes.
| Contract | Probe | Pass evidence | Failure class |
|---|---|---|---|
| JSON object | Ask for one object with two keys | Whole response parses; no fence stripping | format_drift |
| Schema | Fixed key set and value | Keys match exactly | schema_miss |
| Truncation | Small max_tokens with a long request |
finish_reason is present and plausible |
truncation_ambiguity |
| Token accounting | Known short prompt | 0 < completion_tokens <= max_tokens |
accounting_anomaly |
| Timeout | Client timeout enforced | Request ends within budget | timeout |
| Error contract | Malformed request | Non-2xx with a structured body | error_contract |
| Streaming, if used | Streamed response | Final event arrives | stream_hang |
The table is deliberately narrow. It does not test reasoning, factuality, or code quality. Those need a different dataset. This bench tests whether the endpoint is a stable dependency for one client.
2. Build a small, frozen probe set
Start with four cases. Expand to at least 20 before you make a team decision. The shape matters more than the count.
Use four groups:
- Strict JSON extraction, with and without a code-fence instruction.
- Truncation pressure, where the answer cannot fit in the token budget.
- Unicode and whitespace, because transport layers normalize text in surprising ways.
- A malformed request control, sent outside the main run.
Store the cases in JSONL with an id, group, prompt, and expect field. Hash the file. Put the hash in your notes. If the fixture changes, the old numbers are a different experiment.
Do not put secrets, customer text, or production prompts in the fixture. A free service may log requests. Treat every probe as public.
3. Run it with controls
Controls are what separate a bench from a screenshot.
- Set
temperatureto 0 when the endpoint accepts it. If it ignores the value, record that fact. - Fix
max_tokens, client timeout, and client version for the whole run. - Run three attempts per case. Do not retry inside the measurement; record the retry as a new attempt.
- Repeat in two windows at least two hours apart. Free capacity is not stationary.
- Shuffle case order with a fixed seed. Do not let one slow request poison every later sample.
- Log status, latency, usage, finish reason, and a hash of the raw body. Do not log the API key.
- Send one malformed request. A 200 response is a contract failure, not a quirky success.
With 12 attempts, a p95 latency is a range check, not an SLA estimate. That is fine. You are looking for failure classes and gross instability, not a precise capacity number.
4. A harness you can adapt
The template below uses only the Python standard library. It is an unexecuted template: it assumes a chat-completions-shaped payload and a BASE_URL. Replace call_endpoint for a different API. Keep the record shape and scoring logic.
import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.request
from pathlib import Path
BASE_URL = os.environ['BASE_URL'].rstrip('/')
API_KEY = os.environ['API_KEY']
MODEL = os.environ.get('MODEL', 'default')
TIMEOUT_S = 25.0
MAX_TOKENS = 64
CASES = [
{
'id': 'json-01',
'group': 'strict_json',
'prompt': 'Return only a JSON object with keys ok and count. count must be 2.',
'kind': 'json',
},
{
'id': 'json-02',
'group': 'strict_json',
'prompt': 'Return only a JSON object with keys ok and count. count must be 2. Do not use a code fence.',
'kind': 'json',
},
{
'id': 'long-01',
'group': 'truncation',
'prompt': 'Write a 1000-word explanation of HTTP caching.',
'kind': 'text',
},
{
'id': 'unicode-01',
'group': 'unicode',
'prompt': 'Return the exact text: café / 第二行 / end',
'kind': 'text',
},
]
def call_endpoint(prompt: str) -> dict:
payload = {
'model': MODEL,
'messages': [{'role': 'user', 'content': prompt}],
'temperature': 0,
'max_tokens': MAX_TOKENS,
}
req = urllib.request.Request(
f'{BASE_URL}/chat/completions',
data=json.dumps(payload).encode('utf-8'),
headers={
'Authorization': f'Bearer {API_KEY}',
'Content-Type': 'application/json',
},
method='POST',
)
started = time.perf_counter()
status = None
raw = ''
try:
with urllib.request.urlopen(req, timeout=TIMEOUT_S) as resp:
status = resp.status
raw = resp.read().decode('utf-8')
except urllib.error.HTTPError as exc:
status = exc.code
raw = exc.read().decode('utf-8')
except Exception as exc:
return {
'status': None,
'text': '',
'usage': {},
'finish_reason': None,
'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
'raw_sha256': None,
'transport_error': type(exc).__name__,
}
body = {}
try:
parsed = json.loads(raw)
if isinstance(parsed, dict):
body = parsed
except json.JSONDecodeError:
pass
choice = (body.get('choices') or [{}])[0]
message = choice.get('message') or {}
return {
'status': status,
'text': message.get('content') or '',
'usage': body.get('usage') or {},
'finish_reason': choice.get('finish_reason'),
'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
'raw_sha256': hashlib.sha256(raw.encode('utf-8')).hexdigest(),
'transport_error': None,
}
def score(case: dict, rec: dict) -> str:
if rec['transport_error']:
return 'transport_error'
if rec['status'] != 200:
return 'http_error'
if rec['elapsed_ms'] > TIMEOUT_S * 1000:
return 'timeout'
if case['kind'] == 'json':
try:
value = json.loads(rec['text'])
except json.JSONDecodeError:
return 'format_drift'
if not isinstance(value, dict):
return 'schema_miss'
if set(value) != {'ok', 'count'}:
return 'schema_miss'
if value['count'] != 2:
return 'content_miss'
completion = rec['usage'].get('completion_tokens')
if isinstance(completion, int) and completion > MAX_TOKENS:
return 'accounting_anomaly'
return 'pass'
def main() -> None:
random.Random(7).shuffle(CASES)
rows = []
for repetition in range(3):
for case in CASES:
rec = call_endpoint(case['prompt'])
rec.update({
'case_id': case['id'],
'group': case['group'],
'repetition': repetition,
'result': score(case, rec),
})
rows.append(rec)
Path('results.jsonl').write_text(
chr(10).join(json.dumps(row, ensure_ascii=False) for row in rows),
encoding='utf-8',
)
counts = {}
for row in rows:
key = '{}:{}'.format(row['group'], row['result'])
counts[key] = counts.get(key, 0) + 1
latencies = sorted(row['elapsed_ms'] for row in rows)
p95 = latencies[min(len(latencies) - 1, int(0.95 * (len(latencies) - 1)))]
print(json.dumps({'counts': counts, 'p95_ms': p95}, indent=2))
if __name__ == '__main__':
main()
Save it as bench.py and run it with BASE_URL='https://your-endpoint.example' API_KEY='...' python bench.py. Run the malformed-request control separately. Change the payload to invalid JSON and assert that the status is 4xx and the body is parseable. Do not fold that result into the pass count.
5. Read counts, not a composite score
A single score hides the failure you need to fix. Report a small set of separate numbers.
| Metric | How to report it | Why it matters |
|---|---|---|
| Strict JSON |
passed / attempts by group |
Shows format drift directly |
| Timeout | count / attempts |
Exposes client and capacity failures |
| Token anomaly | count |
Catches impossible usage accounting |
| Latency | p50 and p95, plus max | Shows budget risk without hiding outliers |
| Error contract | valid 4xx / malformed requests |
Shows whether failures are machine-readable |
| Raw evidence | fixture hash and body hashes | Lets another person reproduce the run |
Use counts when the denominator is small. 9/12 strict-JSON passes, 3 format_drift is useful. 75% quality is not. The second sentence invites a comparison the bench cannot support.
6. Why these numbers are not marketing
Marketing numbers compress time, sample size, and failure detail. They report one score, one date, one model label. That is easy to read and easy to misuse.
A contract bench reports a fixture hash, a client version, a denominator, a failure histogram, and the time windows. Those details are boring. They are also what lets another developer reproduce your result or falsify it.
Do not turn the bench into a ranking. If you test two endpoints, keep the fixture and client fixed, report each failure class separately, and say how many attempts you made. If the free offer changes, your old number is a historical observation, not a current capability claim.
That is the difference between evidence and a screenshot. A screenshot shows a good moment. A bench shows the conditions under which the moment happened.
7. Decision table
| Workload | Minimum evidence | Do not use the free endpoint when |
|---|---|---|
| Throwaway prototype | 12 attempts, malformed control, classified failures | You cannot send the data under the current terms |
| CI job | 30+ attempts across two windows, p95 under job timeout, fallback path | A timeout blocks every merge |
| Internal tool | Contract pass counts plus error-budget monitoring | You need a hard latency promise |
| Production dependency | Contract bench plus fallback, monitoring, and data policy | You need an SLA the free offer does not provide |
The table is a gate, not a guarantee. A pass means the endpoint is worth another test. It does not mean the model is correct.
8. Limitations and who should skip this
This bench does not measure reasoning quality, factuality, or code correctness. It will not tell you which model writes better patches. Use a separate evaluation for that.
The sample is small. Twelve or thirty attempts can reveal gross instability, but it cannot prove steady-state capacity. The free offer is also time-bound. The 10 million token figure and the free server option are commercial terms, not protocol guarantees; they can change without notice.
Do not send secrets, regulated data, or customer content to a free endpoint unless the current terms clearly allow it. Do not use this workflow as your only production control. Teams that need deterministic latency, strict data residency, or a contractual SLA should not treat a free endpoint as a production dependency.
9. Where the free server fits
Use the free server option as a disposable probe runner. Keep the fixture, the results JSONL, and the summary in storage you control. If the server resets, you should lose a run, not the evidence.
Inject the API key at runtime. Do not put it in the fixture, the repository, or the result log. Treat the runner as cattle, not a database.
The useful output is not a slogan about a model. It is a sentence like this: under fixture hash abc123, with client version 0.4.1, 7 of 12 strict-JSON attempts passed and 5 failed as format drift across two windows. That sentence survives a demo, a handoff, and a skeptical reviewer.
If you want a low-stakes place to start, the project's current free access and free server offer are enough for one run. Verify the current terms, keep your raw counts, and let the bench decide what happens next.
Top comments (0)