Probe the Endpoint Before You Probe the Model

Free model endpoints fail in ways that have nothing to do with model quality: cold starts, rate limits, empty completions, and timeout spikes. A benchmark that only scores answers will mix those failures into the model's score and make a reliable model look bad. This article describes a small probe that separates endpoint reliability from output quality. It uses MonkeyCode's operator-supplied free model access and free server option as the concrete target, but the script is provider-neutral. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Separate the two questions

When you test a free model, you are really asking two questions at once:

1. Does the endpoint respond consistently?
2. Does the model produce useful output under that endpoint?

Most eval harnesses answer only the second question, then silently let the first question contaminate the result. A model that would have answered correctly still scores zero if the request times out, returns 429, or produces an empty completion. This probe answers the first question on its own so you can diagnose the second accurately later.

The probe

The script below is a design, not a reported benchmark. It reads plain text prompts, calls an OpenAI-style chat/completions endpoint, retries on 429, and appends one CSV row per attempt. Install the only dependency with pip install requests.

#!/usr/bin/env python3
'''Failure-mode probe for a free model endpoint.

This is a design, not a reported benchmark. Run it with:

    export MODEL_ENDPOINT='https://example.invalid/v1/chat/completions'
    export API_KEY='...'
    python endpoint_probe.py --prompts prompts.txt --out runs.csv
'''
import argparse
import csv
import os
import sys
import time

import requests


def call(prompt, timeout):
    headers = {}
    api_key = os.getenv('API_KEY')
    if api_key:
        headers['Authorization'] = 'Bearer ' + api_key
    payload = {
        'model': os.getenv('MODEL_NAME', 'replace-me'),
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
        'stream': False,
    }
    return requests.post(
        os.environ['MODEL_ENDPOINT'],
        headers=headers,
        json=payload,
        timeout=timeout,
    )


def extract_text(data):
    try:
        return data['choices'][0]['message']['content']
    except Exception:
        return ''


def probe(prompt, max_retries, backoff, timeout):
    last = ('error', None, 0.0, 'unknown', 0)
    for attempt in range(1, max_retries + 2):
        started = time.monotonic()
        try:
            response = call(prompt, timeout)
            latency = time.monotonic() - started
            if response.status_code == 429:
                last = ('rate_limited', response.status_code, latency, '', attempt)
                time.sleep(backoff * attempt)
                continue
            response.raise_for_status()
            text = extract_text(response.json())
            if not text.strip():
                return ('empty_text', response.status_code, latency, '', attempt)
            preview = text[:80].replace('\n', '\\n')
            return ('ok', response.status_code, latency, preview, attempt)
        except requests.exceptions.Timeout:
            last = ('timeout', None, time.monotonic() - started, '', attempt)
        except Exception as exc:
            last = ('error', None, time.monotonic() - started, type(exc).__name__, attempt)
    return last


def load_prompts(path):
    prompts = []
    with open(path, encoding='utf-8') as handle:
        for lineno, raw in enumerate(handle, 1):
            line = raw.strip()
            if not line or line.startswith('#'):
                continue
            prompts.append((lineno, line))
    return prompts


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--prompts', default='prompts.txt')
    parser.add_argument('--out', default='runs.csv')
    parser.add_argument('--max-retries', type=int, default=3)
    parser.add_argument('--backoff', type=float, default=5.0)
    parser.add_argument('--timeout', type=float, default=20.0)
    parser.add_argument('--delay', type=float, default=0.5)
    args = parser.parse_args()

    if not os.getenv('MODEL_ENDPOINT'):
        sys.exit('Missing MODEL_ENDPOINT environment variable')
    prompts = load_prompts(args.prompts)
    if not prompts:
        sys.exit('No prompts found in ' + args.prompts)

    fieldnames = ['timestamp', 'id', 'status', 'http_status', 'latency_s', 'preview', 'attempts']
    new_file = not os.path.exists(args.out)
    with open(args.out, 'a', newline='', encoding='utf-8') as handle:
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        if new_file:
            writer.writeheader()
        for lineno, prompt in prompts:
            status, http_status, latency, preview, attempts = probe(
                prompt, args.max_retries, args.backoff, args.timeout
            )
            writer.writerow({
                'timestamp': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
                'id': str(lineno),
                'status': status,
                'http_status': http_status or '',
                'latency_s': f'{latency:.2f}',
                'preview': preview,
                'attempts': str(attempts),
            })
            time.sleep(args.delay)


if __name__ == '__main__':
    main()

Create a small prompt file with one prompt per line:

# prompts.txt
Return the word ok
Explain HTTP 429 in one sentence
Write a two-column markdown table of three fruits

The point of these prompts is not to judge quality. They are simple enough that a working endpoint should return non-empty text quickly, so failures stand out as infrastructure problems.

What the rows mean

status	what it tells you	next step
ok	endpoint responded with non-empty text	continue to deeper quality eval
rate_limited	quota or throttling was hit	back off or check account limits
timeout	cold start or overloaded endpoint	increase timeout, retry later
empty_text	response shape may not match the extractor	inspect response JSON
error	transport or provider error	log the exception and payload

A single rate_limited may be noise. A cluster of timeouts at the same prompt, or an empty_text on every prompt, is a signal that the endpoint is not stable enough for your next test.

Running it on a free server

A laptop run gives you one observation. A free server lets you collect many observations without keeping your machine awake. A scheduled task like this is enough:

*/15 * * * * cd /path/to/probe && /usr/bin/env python3 endpoint_probe.py --prompts prompts.txt --out runs.csv

Treat the free server itself as unreliable. It may lose local storage when it restarts, so copy runs.csv somewhere durable if you need a history. MonkeyCode's free server option is the concrete availability claim here, but I am not assuming storage persistence, uptime guarantees, or cron support; those details should come from the operator's current documentation.

Interpret an illustrative run

An illustrative clean run, not captured from a live endpoint:

timestamp,id,status,http_status,latency_s,preview,attempts
2026-08-13T04:00:00Z,1,ok,200,1.42,Return the word ok,1
2026-08-13T04:00:01Z,2,ok,200,1.61,Explain HTTP 429 in one sentence,1
2026-08-13T04:00:02Z,3,ok,200,1.58,Write a two-column markdown table of three fruits,1

A messier run might look like this:

timestamp,id,status,http_status,latency_s,preview,attempts
2026-08-13T05:00:00Z,1,rate_limited,429,0.31,,3
2026-08-13T05:00:01Z,2,empty_text,200,2.10,,1
2026-08-13T05:00:02Z,3,timeout,,20.00,,1

That second pattern is not a model-quality failure. It is an availability and response-shape failure. Fixing the endpoint or the extractor must happen before any quality benchmark can be trusted.

Limitations

• This probe does not score correctness, reasoning, or style. It only measures whether the endpoint returns non-empty text without transport failures.
• It assumes an OpenAI-like chat/completions shape. Providers with tool calls, streaming-only responses, or different JSON paths need changes to call() and extract_text().
• Free endpoints change quotas, model names, auth requirements, and rate limits. Treat every run as point-in-time evidence, not a permanent result.
• A scheduled task on a free server can miss observations if the server restarts, sleeps, or clears storage.

Who should not use this approach

Do not use this if you need a production SLA, a guaranteed uptime percentage, or a legal compliance review. Do not use it as a replacement for a quality benchmark. Do not send sensitive or regulated prompts to an unvetted free endpoint just because the probe accepts arbitrary text. This workflow is useful for early triage, not for final model selection.

If you already have free model access and a free server, run the probe for 24 hours before you trust the endpoint for a weekend project. The resulting CSV will tell you more about availability than any single demo prompt.