Pairing on a Flaky Free-LLM Pipeline: Three Dead Ends and the Decision That Survived

A three-hour pairing session on a flaky free-LLM pipeline ended with one durable decision: record every model decision before touching another prompt. The session produced three dead ends that looked promising and one small artifact that caught real drift within a day. This post reconstructs the questions, the failures, and the exact trace format the pair kept. The session is a representative reconstruction, and the code is runnable as written.

The pipeline under review was a small triage bot that read an issue title and returned a bug, feature, or docs label. It ran on the free server option from MonkeyCode, an open-source project that provides free model access and a free server tier. At the time of writing, the free tier includes a 10-million-token allowance, and the project repository remains the source of truth for current limits. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The bot's answers were mostly correct, but the same issue occasionally produced a different decision on retry. The wider debate about untested AI reviewers is the backdrop here, but the session treated the problem as a debugging exercise. The senior engineer opened with a question about which failure mode actually cost the team, and the answer changed the plan.

The Three Questions

The session settled on three questions that separated signal from noise. The first was which failure mode actually cost the team, and the answer was silent nondeterminism rather than latency. The second was what changed between two runs of the same prompt, and the answer was retry count and cache state. The third was whether the next prompt edit could be proven as an improvement, and the answer was no, because no baseline existed.

Dead End One: More Prompt Constraints

The first attempt added explicit consistency instructions and a few-shot example to the prompt. Ten repeated runs showed that outputs became more verbose but not more stable, and the failure rate stayed flat. The extra constraints also increased token usage and made the prompt harder to maintain. The pair discarded the change after one hour.

Dead End Two: Temperature Zero

The second attempt set temperature to zero, which looked like a clean fix in local tests. The free endpoint's gateway retries and cache states still produced drift because a retried request could return a different completion. The pair learned that temperature controls sampling, not transport-level nondeterminism, and the setting added nothing once the pipeline ran remotely. This dead end took ninety minutes to confirm.

Dead End Three: A Second Model Call

The third attempt added a second model call to cross-check the first decision. The two models disagreed on borderline cases in ways that required a third opinion, and the adjudication logic became a second bug factory. Latency doubled and token consumption climbed, which hurt on a free allowance. The pair abandoned the design after two hours.

The Decision That Survived: A Decision Ledger

The decision the pair kept was to record every call in a JSONL ledger and replay a pinned regression set after any change. Each ledger entry stores the prompt hash, retry count, cache state, latency, and the raw decision text. The replay script diffs new decisions against the stored baseline and fails loudly on drift. The artifact is small enough to review in one sitting, which is why it survived.

The Trace Wrapper

# trace.py — records one decision per free-LLM call
import hashlib
import json
import time
import uuid

LEDGER_PATH = "decisions.jsonl"
MODEL_NAME = "your-free-model-id"  # set from the provider config


def prompt_hash(prompt: str) -> str:
    return hashlib.sha256(prompt.encode()).hexdigest()[:12]


def record(entry: dict) -> None:
    with open(LEDGER_PATH, "a") as fh:
        fh.write(json.dumps(entry) + "\n")


def traced_decision(client, prompt: str, *, retries: int = 0, cache_hit: bool = False) -> str:
    started = time.time()
    message = client.chat.completions.create(
        model=MODEL_NAME,
        messages=[{"role": "user", "content": prompt}],
    )
    decision = message.choices[0].message.content.strip()
    record(
        {
            "trace_id": str(uuid.uuid4()),
            "ts": time.time(),
            "prompt_hash": prompt_hash(prompt),
            "retries": retries,
            "cache_hit": cache_hit,
            "latency_ms": int((time.time() - started) * 1000),
            "decision": decision,
        }
    )
    return decision

The wrapper does not care which model or server hosts the call, because the model name and client come from the provider configuration. That portability let the pair run the same ledger against MonkeyCode's free model access and against a local mock. The ledger file stays append-only so the baseline can never be silently rewritten.

The Replay Script

# replay.py — replays the pinned regression set and diffs against the baseline
import json
import sys

from trace import prompt_hash, traced_decision

REGRESSION_PROMPTS = [
    "Classify this issue as bug, feature, or docs: 'Login page crashes on Safari 17'",
    "Classify this issue as bug, feature, or docs: 'Add dark mode support'",
    "Classify this issue as bug, feature, or docs: 'README typo in installation section'",
]


def load_baseline(path: str) -> dict:
    baseline = {}
    for line in open(path):
        entry = json.loads(line)
        baseline.setdefault(entry["prompt_hash"], []).append(entry["decision"])
    return baseline


def main() -> int:
    client = get_provider_client()  # provider-specific setup
    baseline = load_baseline("baseline.jsonl")
    failures = 0
    for prompt in REGRESSION_PROMPTS:
        decision = traced_decision(client, prompt)
        expected = baseline.get(prompt_hash(prompt), [])
        if expected and decision not in expected:
            failures += 1
            print(f"DRIFT prompt={prompt_hash(prompt)} expected={expected} got={decision}")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

The script exits non-zero when any pinned prompt drifts from every stored decision for that prompt hash. A prompt edit changes the hash and immediately looks like a new prompt, which forces a deliberate baseline decision. The pair ran this script manually during the session and then scheduled it on the free server.

Scheduling the Replay on the Free Server

# /etc/systemd/system/llm-replay.timer
[Unit]
Description=Run the free-LLM decision replay every six hours

[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00

[Install]
WantedBy=timers.target

# /etc/systemd/system/llm-replay.service
[Unit]
Description=Free-LLM decision replay

[Service]
WorkingDirectory=/opt/llm-ledger
ExecStart=/usr/bin/python3 replay.py

The pair enabled the timer with systemctl enable --now llm-replay.timer and let the ledger accumulate for a day. The first scheduled run caught a drift that manual testing had missed, which validated the whole approach. A cron line would work the same way on a server without systemd.

What the Ledger Caught

1. Prompt drift: an edit changed the prompt hash, and the baseline comparison flagged the new prompt as unverified.
2. Retry nondeterminism: the same prompt hash produced two different decisions when the gateway retried a timed-out request.
3. Cache-state dependence: a cold cache returned a different completion than a warm cache for the same input.

None of these failures were visible in the application logs, because the application only stored the final answer. The ledger made the invisible failure mode measurable, which was exactly the cost the senior engineer had identified. That single property justified the artifact's existence.

What the Ledger Does Not Catch

The ledger records decisions, but it does not judge whether a stable decision is correct. A consistently wrong label will replay cleanly and still ship, so the regression set needs human-reviewed ground truth. The ledger also does not detect prompt injection, data leakage, or cost overruns, and token metering must come from the provider dashboard.

Limitations and Who Should Skip This

The approach is a tripwire, not an oracle, and it assumes the team can maintain a small pinned prompt set. Teams with strict SLA guarantees or regulated data should not build on a free endpoint, because the free allowance and server option can change without notice. The pair kept the artifact because it made every future prompt change measurable, not because it made the pipeline perfect.

The Takeaway

The pairing session ended with a one-line conclusion: record the decision, not just the answer. The trace format and replay script in this post are the complete artifact, and they work with any endpoint that returns text. If you want to run the same replay against a free endpoint, MonkeyCode's free model access and free server option are a reasonable place to start, and the project repository has the current details.