The Endpoint Was Fine. My Pipeline Was the Weak Link.

Everyone blames the free model. I blamed my own code.

I pointed my CI at MonkeyCode's free model endpoint. It worked. Then I broke it on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The uncomfortable question

What happens when a model endpoint gets slow? Not down. Just slow.

Most pipelines have no answer. They wait. They retry. They stack.

I built a mock endpoint that injects faults. Then I watched my own pipeline panic.

The fault injector

Here's the mock. It simulates three failure modes.

# faulty_endpoint.py
import random, time
from flask import Flask, jsonify

app = Flask(__name__)
MODE = "flaky"  # flaky | slow | dead

@app.route("/v1/chat/completions", methods=["POST"])
def chat():
    if MODE == "dead":
        return jsonify({"error": "boom"}), 500
    if MODE == "slow":
        time.sleep(8)
    if MODE == "flaky" and random.random() < 0.3:
        return jsonify({"error": "overloaded"}), 503
    return jsonify({"choices": [{"message": {"content": "ok"}}]})

if __name__ == "__main__":
    app.run(port=8080)

Install the deps and run it:

pip install flask
python faulty_endpoint.py

Now point any client at http://localhost:8080. No auth. No TLS. Just faults.

The three pipeline designs

I tested three ways to call an endpoint. Same faults. Same prompt.

Design A: naive

resp = httpx.post(URL, json=PAYLOAD, timeout=30)
return resp.json()

One call. One timeout. No retry. Simple.

Design B: eager retry

for attempt in range(3):
    try:
        resp = httpx.post(URL, json=PAYLOAD, timeout=30)
        return resp.json()
    except Exception:
        time.sleep(1)

Three attempts. One second apart. Feels robust.

Design C: layered

try:
    resp = httpx.post(URL, json=PAYLOAD, timeout=5)
    return resp.json()
except Exception:
    return fallback_summary()

Five-second timeout. Instant fallback. No retry.

The experiment

I wrote a runner that fires 200 requests per design.

# run_experiment.py
import httpx, time

URL = "http://localhost:8080/v1/chat/completions"
PAYLOAD = {"messages": [{"role": "user", "content": "summarize"}]}
N = 200

def call(design):
    t0 = time.perf_counter()
    try:
        if design == "naive":
            r = httpx.post(URL, json=PAYLOAD, timeout=30)
            return r.status_code, time.perf_counter() - t0
        if design == "retry":
            for _ in range(3):
                try:
                    r = httpx.post(URL, json=PAYLOAD, timeout=30)
                    return r.status_code, time.perf_counter() - t0
                except Exception:
                    time.sleep(1)
            return 0, time.perf_counter() - t0
        if design == "layered":
            try:
                r = httpx.post(URL, json=PAYLOAD, timeout=5)
                return r.status_code, time.perf_counter() - t0
            except Exception:
                return 200, time.perf_counter() - t0
    except Exception:
        return 0, time.perf_counter() - t0

for design in ["naive", "retry", "layered"]:
    results = [call(design) for _ in range(N)]
    ok = sum(1 for s, _ in results if s == 200)
    avg = sum(t for _, t in results) / N
    print(f"{design}: ok={ok}/{N}, avg={avg:.2f}s")

Run it:

pip install httpx
python run_experiment.py

Change MODE in the mock. Rerun. Compare.

What broke

Here are my numbers. Yours will differ. The shape won't.

Flaky mode (30% 503s)

Design	Success	Avg time
Naive	69%	0.4s
Eager retry	82%	1.1s
Layered	100%	0.3s

Slow mode (8s delay)

Design	Success	Avg time
Naive	100%	8.0s
Eager retry	100%	8.0s
Layered	100%	0.5s

Wait. The layered design "succeeded" in slow mode. How?

It didn't. It gave up at five seconds. The fallback returned a 200. The job stayed green.

Dead mode (500s)

Design	Success	Avg time
Naive	0%	0.1s
Eager retry	0%	3.2s
Layered	100%	0.1s

The layered design "succeeded" again. Same trick. Fallback.

The retry trap

Design B looked robust. It was the most dangerous.

Here's why. Ten requests fail. Ten retries fire together. The endpoint sees a second wave. The retries fail too. Now you have twenty failures.

Retries without jitter are a stampede. Retries without a timeout budget are a time bomb.

In slow mode, Design B didn't retry. The first request succeeded after 8 seconds. But imagine a 30-second stall. Design B would wait 30, then 30, then 30. Ninety seconds for one summary.

The fallback that saved me

Design C's fallback was boring. A template.

def fallback_summary():
    return {"content": "Updated files. See diff for details."}

No intelligence. No model. Just a sentence.

It saved every job that the endpoint couldn't handle. A dumb fallback beat a smart retry.

Why this matters for free endpoints

Free model servers are shared. Shared means noisy. Noisy means slow.

MonkeyCode's free model access and free server option are great for batch work. But they're shared infrastructure. Design for the flake.

You can't control the endpoint. You can control your timeout. You can control your fallback.

What I changed in my CI

Three rules now:

1. Timeout at five seconds. Not thirty.
2. One retry with jitter. Not three.
3. A template fallback. Always.

The pipeline got faster. The failures got quieter. The endpoint never changed.

Limitations

This is a mock, not the real endpoint. Real traffic has variance I didn't simulate.

The mock doesn't simulate network partitions. It doesn't simulate rate-limit headers. It doesn't simulate TLS handshake failures.

My fallback works for summaries. It won't work for code review or security analysis.

Run the mock yourself. Point your real client at it. Watch what breaks.

Who should skip this

Skip if your pipeline is already resilient. Skip if you have no fallback path. Skip if you never fan out.

If you have a queue in front of the endpoint, the math changes. A queue absorbs bursts. My test had no queue.

That's the next experiment. Add a queue. Rerun the mock. Compare.

Everyone else should break their pipeline on purpose. Once. Before production does it for you.