Probe a Free AI Endpoint Before You Make It a Production Dependency

#ai #python #performance #webdev

Probe a Free AI Endpoint Before You Make It a Production Dependency

The current AI debate keeps asking which model is “better.” Most teams I work with have a cheaper problem first: they are about to build a feature on an endpoint they have never pushed past single-digit concurrency. A model can look great in a chat box and fail at eight parallel requests because of rate limits, queueing, or a client timeout. Before you compare weights, measure the boundary. You can do that without a production credit card if the endpoint exposes an OpenAI-compatible HTTP API.

I tested this workflow against MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The 30M token allowance and free server slot mentioned here are operator-supplied claims, not something I benchmarked; verify the current dashboard before you depend on them. The probe script is generic, so you can run the same ladder against any compatible endpoint.

The layer that matters is the transport contract

Don't start with an SDK. SDKs hide the parts that fail first: HTTP statuses, timeouts, retries, and concurrency ceilings. For an OpenAI-compatible endpoint, the interesting contract is a small JSON POST to /chat/completions. If that contract buckles under concurrency, every wrapper on top of it will buckle too.

A free tier is useful for exactly this test because failure is cheap. You want the endpoint to reject you before you have built the rest of the feature. The output you need is not a clever answer; it is the first non-200 response and the latency spread around it.

A small concurrency ladder

The following Python script is a reproducibility artifact, not a load tester. It steps through low concurrency levels and records status codes and latency without retrying. Not retrying is deliberate: you want to see the raw failure shape, not hide it behind backoff.

# probe_endpoint.py
import asyncio
import os
import statistics
import time

import httpx

ENDPOINT = os.environ['ENDPOINT_URL']
API_KEY = os.environ.get('API_KEY', '')
MODEL = os.environ.get('MODEL', '')
STEPS = [int(x) for x in os.environ.get('STEPS', '1,2,4,8').split(',')]
REQUESTS_PER_STEP = int(os.environ.get('REQUESTS_PER_STEP', '10'))

PROMPT = 'Reply with exactly: ok'

def quantile(values, q):
    ordered = sorted(values)
    idx = max(0, min(len(ordered) - 1, int(len(ordered) * q)))
    return ordered[idx]

async def call(session, semaphore):
    async with semaphore:
        started = time.perf_counter()
        payload = {'messages': [{'role': 'user', 'content': PROMPT}]}
        if MODEL:
            payload['model'] = MODEL
        headers = {'Authorization': 'Bearer ' + API_KEY} if API_KEY else {}
        try:
            response = await session.post(
                ENDPOINT,
                json=payload,
                headers=headers,
                timeout=30,
            )
            return {
                'status': response.status_code,
                'latency_ms': round((time.perf_counter() - started) * 1000, 1),
            }
        except Exception as exc:
            return {
                'status': 0,
                'latency_ms': round((time.perf_counter() - started) * 1000, 1),
                'error': type(exc).__name__,
            }

async def main():
    async with httpx.AsyncClient() as session:
        for concurrency in STEPS:
            semaphore = asyncio.Semaphore(concurrency)
            results = await asyncio.gather(
                *(call(session, semaphore) for _ in range(REQUESTS_PER_STEP))
            )
            ok_latencies = [
                r['latency_ms'] for r in results if 200 <= r['status'] < 300
            ]
            statuses = {}
            for result in results:
                statuses[result['status']] = statuses.get(result['status'], 0) + 1
            if ok_latencies:
                print(
                    f'step={concurrency} ok={len(ok_latencies)}/{len(results)} '
                    f'statuses={statuses} p50={statistics.median(ok_latencies):.0f}ms '
                    f'p95={quantile(ok_latencies, 0.95):.0f}ms'
                )
            else:
                print(f'step={concurrency} ok=0/{len(results)} statuses={statuses}')

if __name__ == '__main__':
    asyncio.run(main())

Run it first with a single request:

export ENDPOINT_URL='https://your-endpoint.example/chat/completions'
export API_KEY='your-key-if-required'
export MODEL=''  # only if the endpoint requires a model field
python probe_endpoint.py

Then read the ladder from left to right. In a healthy setup, the first step returns 200s with a low p95. As concurrency rises, one of three things usually appears: a 429 rate-limit response, a 5xx, or timeouts (status=0). That is the boundary you should design around, not the model's benchmark score.

Decide before you build

Use the output against your actual call pattern.

Use case	Call pattern	Free endpoint acceptable?
Read-only classification job	Batched, async, idempotent	Yes, behind a queue with retries and durable state
Interactive chat in the request path	User waits, synchronous	Only after the p95 and 429 rate hold under peak traffic
Writing state or modifying data	Needs idempotency and authorization	Free tier for evaluation only, never production writes
Long-context document review	Long latency and high token use	Check token allowance and context limits before committing

The decision should be about failure recovery, not about whether the model answers impressively. If a request can fail once and the user sees nothing, the free tier may work. If a user is waiting synchronously, the first 429 in your ladder is a product outage.

What the probe does and does not prove

The artifact proves:

whether the endpoint accepts the request shape;
the approximate concurrency step where failures start;
whether failures surface as rate limits, server errors, or timeouts;
how latency spreads at small load.

It does not prove:

model quality;
output safety;
token accounting accuracy;
that a free quota will exist tomorrow;
that the endpoint can scale to your production peak.

Those require a separate evaluation loop and a written plan for quota and support. A free server slot is useful for reducing cost while you build that loop. It is not a substitute for an availability decision.

A checklist for the handoff

Pin ENDPOINT_URL and the key in environment variables, not inside the code.
Send one request manually and confirm the response body is valid JSON.
Run the concurrency ladder and write down the first non-200 status.
Set a max concurrency below the step where failures became common.
If you keep the free endpoint, route only retryable or non-real-time work through it.
Add idempotency keys before any write path touches the endpoint.
Re-run the probe with a production budget or a real quota decision before launch.

The fastest way to know whether a free tier can carry your feature is to make it fail on purpose under small concurrency. The result will usually be a 429, not a model quality problem. If you run the ladder against a MonkeyCode free endpoint, I would be interested in which step produces your first failure.