48-Hour Field Notes: What Broke on a Free AI Stack and What I'd Repeat

I gave myself 48 hours to turn a pile of support tickets into structured summaries using free model access and a free server. The model was smart enough; the operational envelope around it was not. These are the field notes from that window: what broke, what I tried, and what I would repeat. If you are planning to build anything on a free tier, read the failure list before you write the happy path.

The Setup

The pipeline was deliberately boring: a Python service reads a ticket from a queue, sends it to a model, and writes a JSON summary back. I ran the whole thing on MonkeyCode's free model access and their free server option because the budget was zero and the deadline was real. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The first thing I should have written was a probe, a tiny script that fires a short request ten times and records latency and failure types. I wrote it after the first outage instead of before it, and that ordering cost me an evening. Run the probe, plot the results, and set your timeouts and retries from data rather than hope.

# probe.py — map the tier before you trust it
import asyncio
import time

async def probe(client, prompt: str, n: int = 10):
    latencies, failures = [], []
    for i in range(n):
        start = time.perf_counter()
        try:
            await client.chat(prompt, max_tokens=20)
            latencies.append(time.perf_counter() - start)
        except Exception as exc:
            failures.append(type(exc).__name__)
        await asyncio.sleep(1)
    return latencies, failures

The first version of the real pipeline was 40 lines and embarrassingly optimistic:

# naive.py — the happy path that stopped being happy
import requests

def summarize(ticket: str) -> dict:
    resp = requests.post(API_URL, json={"prompt": ticket}, timeout=30)
    return resp.json()["summary"]

It worked for the first 40 tickets. Then the 429s started, and the happy path became a debugging session.

Hour 0–6: The Naive Path

I assumed the free tier would behave like a paid one, just slower, and that assumption held for exactly one batch. The first failure mode was rate limiting, and my code treated every non-200 as a crash instead of a signal. The fix was a retry wrapper with exponential backoff and jitter:

# retry.py — backoff with jitter, because plain backoff thunders at the same time
import random
import time
from functools import wraps

def retry(max_attempts=5, base_delay=1.0):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return fn(*args, **kwargs)
                except RateLimitError:
                    delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
                    time.sleep(delay)
            raise
        return wrapper
    return decorator

What broke next was my own design: retrying every failure made the batch take three times longer, and some failures were never going to succeed on retry. I needed a decision table, not a blanket policy.

Hour 6–18: The Decision Table

I stopped treating all errors the same and sorted them into three buckets:

Failure	First move	Second move	Repeat?
429 rate limit	backoff with jitter	degrade to a smaller batch	Yes
5xx gateway	retry once	queue for the next run	Yes
Truncated output	re-prompt with a smaller budget	fall back to a shorter summary	No — fix the prompt

The table became the core of the pipeline, and the retry wrapper shrank to a thin layer that only handled the first column. Everything else went to a dead-letter queue for manual inspection. That single table saved more time than any library I added that weekend.

Hour 18–30: Cold Starts

The free server went idle after a few minutes of silence, and the first call after idle took over twenty seconds. My client timeout was ten, so every cold start looked like a failure. Was the server broken, or was my timeout the real bug? It was the timeout.

I tried a keep-alive ping every three minutes:

# keepalive.py — the trade-off I later regretted
async def keepalive(client, interval=180):
    while True:
        try:
            await client.chat("ping", max_tokens=1)
        except Exception:
            pass
        await asyncio.sleep(interval)

It worked, but it burned requests and tokens around the clock. The better move was to accept the cold start, raise the client timeout, and let the queue absorb the latency. Fighting a free tier's lifecycle is usually more expensive than tolerating it.

Hour 30–42: Truncated JSON

Long tickets produced responses that ended mid-JSON, and my parser raised exceptions that looked like model failures. They were not model failures; they were budget failures, because the model had run out of output room. The fix was a validation gate that checked the response before trusting it:

# validate.py — grade the output before the pipeline does
import json

def parse_summary(raw: str) -> dict | None:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return None
    required = {"category", "priority", "summary"}
    if not required.issubset(data):
        return None
    return data

When the gate failed, I re-prompted with a smaller summary budget instead of retrying the same oversized request. That single change cut my failure rate more than every retry policy combined.

Hour 42–48: The Full Run and What I'd Repeat

The final run processed 500 tickets in about four hours, with twelve failures and zero silent corruptions. Every failure was either a gateway error that retried cleanly or a truncated response that the validation gate caught. The pipeline was boring by design, and boring is exactly what I wanted.

One more thing I would repeat is the worker pool, even though I only added it at hour forty. The synchronous loop was the biggest time waste in the whole experiment.

# worker.py — the concurrency I should have started with
import asyncio

async def worker(queue, client):
    while ticket := await queue.get():
        await process_with_retry(ticket, client)
        queue.task_done()

What I would repeat:

• Probe before building. A one-hour probe that maps the free tier's actual behavior is worth more than any documentation.
• Degrade instead of retry forever. A decision table beats a retry loop every single time.
• Keep state in the queue. The process died twice, and the queue was the only memory that mattered.

What I would not repeat:

• Keep-alive pings. Tolerating the cold start was cheaper than preventing it.
• Synchronous processing. A worker pool would have finished the batch in half the time.

Limitations

This is a field note, not a production recipe. Free tiers change quotas and behavior without notice, so the probe script is a snapshot, not a contract. Do not use this approach for latency-critical services, high-volume production traffic, or anything with a real SLA. If your pipeline must be predictable, pay for predictability.

If you are about to build on a free tier, run a probe for an hour first. It is the closest thing to a contract a free tier will give you.