48-Hour Field Notes: Every Free-Tier Request Returned 200. My Acceptance Probes Still Failed.

Two days ago I moved a small batch job onto a free model endpoint and a free server box, and then I spent most of the next 48 hours asking a much less flattering question than "does it answer?" The question I actually cared about was this: does it answer the same way twice, with the model I asked for, on a box that still has my evidence afterwards? A 200 OK only tells you that a socket accepted bytes, and honestly, that is all it tells you.

Why a status code is not an acceptance test

Most of us treat the first successful response as the end of the integration work, because the integration did work — once, on our laptop, with our one prompt. But a free endpoint is a shared, moving target, and the failures that actually hurt are the quiet ones: a different model behind the same alias, a cached answer served as fresh, a stream that got cut in half and still exited cleanly. None of those raise an exception you can catch.

So instead of a checklist that passes quietly, I wrote four probes that fail loudly.

The setup I kept deliberately small

I did not port a real product onto the free tier. I kept the job tiny on purpose, because the point of the 48 hours was to measure the tier, not to ship anything on it. MonkeyCode offers access to a set of free models and a free server option, and that availability is what the operator told me; the specific models and limits you see in your own account may differ from mine, so check the current list before you plan around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran the probe harness on that free server rather than on my laptop, for one boring reason: my laptop sleeps, and a measurement that stops when a lid closes is not a measurement. Everything below is a workflow you can reproduce against any endpoint that speaks an OpenAI-compatible chat shape — just adjust the payload to whatever your provider documents.

The artifact: four probes in one file

The harness does one thing per probe and appends a JSON line, so a failed run leaves a trail instead of a mystery. It uses only the standard library, which means you can drop it on a fresh box without a dependency install.

#!/usr/bin/env python3
"""
Free-tier endpoint acceptance probes.
Appends one JSON line per probe to PROBE_OUT.
Assumes an OpenAI-compatible chat endpoint; adjust if yours differs.
"""
import hashlib
import json
import os
import time
import urllib.error
import urllib.request

BASE = os.environ['ENDPOINT_BASE'].rstrip('/')
KEY = os.environ['ENDPOINT_KEY']
MODEL = os.environ['ENDPOINT_MODEL']
OUT = os.environ.get('PROBE_OUT', 'probe_report.jsonl')


def post(payload, timeout=60):
    req = urllib.request.Request(
        f'{BASE}/chat/completions',
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': f'Bearer {KEY}'},
    )
    started = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            raw, status = resp.read().decode(), resp.status
    except urllib.error.HTTPError as exc:
        raw, status = exc.read().decode(), exc.code
    except Exception as exc:  # transport failures are data too
        raw, status = json.dumps({'transport_error': repr(exc)}), 0
    return {
        'status': status,
        'elapsed_ms': round((time.monotonic() - started) * 1000),
        'body_sha256': hashlib.sha256(raw.encode()).hexdigest(),
        'raw': raw[:2000],
    }


def ask(prompt, nonce, timeout=60):
    return post({
        'model': MODEL,
        'temperature': 0,
        'messages': [
            {'role': 'system',
             'content': 'Echo the nonce verbatim first, then answer in one sentence.'},
            {'role': 'user', 'content': f'nonce={nonce}\n{prompt}'},
        ],
    }, timeout)


def body_text(raw):
    try:
        return json.loads(raw)['choices'][0]['message']['content']
    except Exception:
        return ''


def answer_of(raw):
    text = body_text(raw)
    return '\n'.join(ln for ln in text.splitlines() if 'nonce=' not in ln).strip()


def reported_model(raw):
    try:
        return json.loads(raw).get('model', '')
    except Exception:
        return ''


def record(probe, verdict, **evidence):
    line = {'probe': probe, 'verdict': verdict, 'ts': round(time.time(), 3)}
    line.update(evidence)
    with open(OUT, 'a', encoding='utf-8') as handle:
        handle.write(json.dumps(line, sort_keys=True) + '\n')
    print(json.dumps(line, sort_keys=True))

Then the probes themselves, each one short enough to read in a code review.

Probe 1 — is the alias pointing where you think?

A fresh nonce goes into the prompt, and I check that it comes back in the answer. If it does not, I am probably reading a cached or canned response rather than a live call. I also log the model field the response reports, next to the alias I requested, because those two strings are the whole basis of every later assumption I make.

def probe_identity():
    nonce = os.urandom(4).hex()
    result = ask('Name the model you are.', nonce)
    text = body_text(result['raw'])
    record('identity',
           'pass' if result['status'] == 200 and nonce in text else 'fail',
           status=result['status'],
           requested=MODEL,
           reported=reported_model(result['raw']),
           nonce_echoed=nonce in text)

Probe 2 — the same prompt, twice, compared

I send one deterministic prompt twice with different nonces and compare the answers after stripping the nonce line. I am not testing whether the model is "correct" here; I am testing whether my pipeline can expect a stable answer at all. If two runs diverge on a question this small, every diff-based review I build on top of it is noise.

def probe_repeatability():
    prompt = 'State the number of days in a leap year, in five words or fewer.'
    first, second = ask(prompt, 'a' * 8), ask(prompt, 'b' * 8)
    a, b = answer_of(first['raw']), answer_of(second['raw'])
    record('repeatability', 'pass' if a and a == b else 'review',
           first_sha256=first['body_sha256'],
           second_sha256=second['body_sha256'])

Probe 3 — does the box still have my report after a restart?

This is the part people skip, and it is the part that decides whether a free server is usable for scheduled work. Write the report, hash it, restart the box through whatever control your provider gives you, then verify the hash again.

python probe_endpoint.py
sha256sum probe_report.jsonl | tee report.hash
# restart the free server here
sha256sum -c report.hash

If it prints OK, my evidence survived. If it prints No such file or directory, then my evidence lives exactly as long as the process did — which is a perfectly good answer as long as I learned it before the job ran unattended, and not after.

Probe 4 — the stream that ends early

I set a ten-second client timeout on a deliberately unbounded request and record whether the call came back as a clean completion or as something I would have to treat as partial. This is the probe that changed my mind about silent truncation being theoretical: a cut stream can look like a short, confident answer.

def probe_timeout():
    result = ask('Count upwards until told to stop.', os.urandom(4).hex(), timeout=10)
    record('timeout', 'pass' if result['status'] == 200 else 'review',
           status=result['status'], elapsed_ms=result['elapsed_ms'])


if __name__ == '__main__':
    probe_identity()
    probe_repeatability()
    probe_timeout()

What broke during the 48 hours

Failures, in the order I hit them, and none of them were exotic:

1. A verification script that verified nothing. My first repeatability comparison mutated the prompt list in place between the two calls, so I was comparing two different questions and calling the difference "nondeterminism". The probe was wrong, not the endpoint.
2. Evidence written outside the directory I downloaded. The report existed, the download missed it, and I almost concluded that the box had wiped my file. Path hygiene is part of the experiment.
3. A pass verdict from a response I never parsed. I marked a run as passing because the HTTP status was 200, while the body was an error-shaped payload. Status checks belong next to body checks, not instead of them.
4. A restart I did not perform. I scheduled the verification step and then forgot to actually restart anything, which produced a green result that proved nothing except my own timing.

None of those are endpoint bugs. All of them are the reason I now keep the probe output as a file that a second person can read without me in the room.

The decision table I actually use

Workload	After a probe run, acceptable on a free tier?	Reasoning
Drafting commit messages, renaming variables, explaining a stack trace	Yes, with a human reading the output	Failures are small, visible, and cheap to redo
CI steps that must produce the same artifact twice	Only with the probe report stored next to the build	An alias resolving differently is invisible in logs
A scheduled batch job with nobody watching	Not until it checks the reported model and alerts	A silent model change is a silent behavior change
Anything containing customer data, secrets, or regulated content	No	I cannot audit retention or isolation on infrastructure I do not control
Deadline work that needs predictable capacity	No	Shared free capacity is not a capacity plan

Who should not use this approach

If you need an availability guarantee, a data-processing agreement, or a per-request billing model you can forecast, this workflow will not give you any of those, and a probe report will not substitute for them. If you will not read a JSONL file before trusting a number, the harness just adds a step. And if your prompts contain anything you would not paste into a public issue, do not send them to a tier you have not audited — probe the endpoint with throwaway text instead.

What I would repeat

I would repeat the ordering more than the code: measure the endpoint before writing the feature, on the machine that will run the job, with the verdict written to disk. I would also repeat the four-probe split, because each probe maps to exactly one decision — trust the response, expect stable output, expect persistence, expect clean endings. And I would repeat the small scope: two days of probing a batch job was enough to change how I schedule it.

The one habit worth copying, whether or not you use MonkeyCode at all, is keeping a probe report in the repository beside the code that calls the endpoint. That single file is the difference between "the free tier worked when I tried it" and a claim you can re-check next month.