48 Hours of Field Notes on a Free AI Stack: What Broke, What Held, and What I'd Repeat

Load tests tell you how a system behaves when you push it. They rarely tell you how it behaves at 3 AM on a Tuesday when nobody is pushing anything, and that gap bothered me enough to run a different kind of experiment. For 48 hours I kept field notes on a free AI stack: one small job, one log file, and a promise to leave the code alone.

The setup was deliberately boring. A free server (MonkeyCode's free server option) ran a digest job every 15 minutes, calling a free model (MonkeyCode's free model access) to summarize the latest batch of commits. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Every run appended one JSONL line to a log file, and the only thing I allowed myself to change was the log itself.

Why field notes instead of a benchmark? Because a benchmark is a controlled lie, and I wanted the uncontrolled truth. What does a free stack actually do when nobody is watching? A load test asks how much a system can take, while field notes ask what really happens, and those turned out to be very different questions. After 48 hours I had 192 runs, 14 failures, and three lessons I would happily repeat.

What I tried first: a hash gate that was useless

My first instinct was to hash the model output and flag any run where the hash changed. It flagged every run, because a free model never returns the same text twice, even for the same input, and the gate became a noise generator instead of a drift detector. Exact matching is the wrong tool for generative output, and I learned that lesson within two hours.

I replaced the hash with a shape signature: the set of JSON keys, the length bucket of the summary, and a crude marker for whether the output started as a list. This caught the drift that mattered, like when the model switched from prose to bullet lists, while ignoring the harmless rewording that happens on every single call.

What broke at hour 3: the cold start I had designed around

The first call after twenty idle minutes took 34 seconds, while my client timeout was set to 10 seconds, so the job failed and the retry fired immediately. The retry hit the same cold start, and one slow call became three failed calls in the log. My first reaction was to blame the retry logic, but the fix was not a better retry; it was a warmer expectation. I logged the latency, accepted the slow first call as a predictable cost of being free, and moved on.

What broke at hour 11: the rate limit had a personality

At a fixed minute past the hour, the provider started returning 429s in a burst and then went silent, and my exponential backoff made it worse. Every retry landed inside the same window because the backoff clock had no idea what time it was, which is the moment I stopped looking at error codes and started looking at the wall clock. Once I plotted failures against time of day, the pattern was obvious, and I shifted the job's schedule by a few minutes with random jitter. The burst still happened, but my job no longer attended it.

What broke at hour 22: drift that no exception could catch

The model returned valid JSON with correct keys, and the summary was subtly wrong in structure because it had switched from prose to a terse list. My pipeline stored the list happily, since the schema still validated, and nothing threw an error, which is the failure that scares me most. The shape signature caught it, the fix was to pin two examples in the prompt, and I learned to schedule a human review at hour 24. That review is how I noticed the drift had started at hour 19, three runs before my gate fired.

The artifact: a field log that tells you which failure you're in

Here is the core of the log writer I used, stripped to the essentials:

# field_log.py — append-only JSONL notes for a free-tier pipeline
import json, time
from pathlib import Path

LOG = Path('field_notes.jsonl')

def shape_signature(output) -> dict:
    # not a hash: a coarse, forgiving fingerprint of output shape
    if isinstance(output, dict):
        keys = sorted(output.keys())
        body = json.dumps(output, sort_keys=True)
    else:
        keys = []
        body = str(output)
    return {
        'keys': keys,
        'length_bucket': len(body) // 200,
        'starts_with_list': body.lstrip().startswith(('-', '*', '1.')),
    }

def record(run_id, stage, outcome, payload, note=''):
    entry = {
        'run_id': run_id,
        'stage': stage,
        'outcome': outcome,            # ok | retried | failed | skipped
        'wall': time.time(),           # what the calendar says
        'mono': time.monotonic(),      # what the stopwatch says
        'latency_ms': round(payload.get('latency_ms', 0), 1),
        'shape': shape_signature(payload.get('text', '')),
        'note': note,
    }
    with LOG.open('a') as fh:
        fh.write(json.dumps(entry) + '\n')

The dual clocks matter more than they look. Wall time tells you when the world thinks an event happened, while monotonic time tells you how long things actually took, and on a free server the two disagreed after every restart. If I had logged only wall time, the rate-limit pattern would have stayed invisible.

The triage table I used at hour 24

Symptom in the log	Evidence	Verdict	Action
First call after idle is slow	latency_ms far above p95, mono gap large	Transient	Warm-up call, longer timeout, don't retry blindly
429s burst at a fixed minute	outcome=retried, wall shows same minute	Systemic	Shift schedule, add jitter, stop backoff inside the window
Valid JSON, wrong shape	shape signature changed, no code change	Drift	Pin examples, re-prompt, re-run the digest
Log order wrong after restart	wall and mono disagree	Environmental	Reconcile by run_id, never by arrival order

What I would repeat, and what I would not

I would repeat the dual clocks, the shape signature, and the fixed cadence, because those three things turned 192 messy runs into a readable story. I would also repeat the mid-experiment human review, which caught the drift that my own gate had missed by three runs. I would not repeat the hash gate, and I would not pretend this experiment proves anything about production readiness. A 48-hour field log is a snapshot of one workload, one week, and one provider's mood, not a guarantee of anything.

Who should not copy this approach

If you need a response in under two seconds, or if your pipeline feeds regulated decisions, this workflow is the wrong tool. Free servers and free models trade money for variance, and variance is exactly what you cannot afford in those places. But if you are running a background job that can tolerate a slow first call and a little drift, a 48-hour field log will teach you more than another load test. If you have kept field notes like these and caught something I missed, I would genuinely like to read them.