DEV Community: Morgan Xu

Postmortem: The Stale-Response Incident That Broke Our Review Bot

Morgan Xu — Tue, 01 Sep 2026 13:12:14 +0000

A broken PR slipped through. The model was fine. Our cache was not. This postmortem covers timeline, root causes, and the durable fix we now apply to every AI-backed job.

We run our review bot on MonkeyCode's free server using its free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Timeline

All times UTC.

01:10 Deployed v1.8 of the review bot to staging.
01:15 Added a Redis cache for all model responses. TTL was one hour.
03:22 Production received a security-sensitive PR.
03:23 Bot returned a cached approval. The cached answer was two hours old.
03:25 Auto-merge rule approved the PR.
04:47 Integration tests failed. The change broke a payment edge case.
06:15 Rolled back the PR and disabled auto-merge.
09:30 Root cause confirmed: stale cache poisoning.

Contributing Factors

Cache Key Missed the Environment

The cache key was just the prompt string. Staging and production share the same model prompts. A staging request with a truncated context populated the cache. Production later hit that same key.

The fix was obvious in hindsight. Cache keys must include the environment, model version, and a context hash.

Retry Loop Amplified the Damage

When the model provider rate-limited us, the bot retried every five seconds. On failure, it fell back to whatever was in the cache. That makes sense in a simple design. In practice it converted transient rate limits into silent stale outputs.

The retry loop had no circuit breaker state. The free server we used has an ephemeral process lifecycle. Any restart wiped the in-memory breaker. The bot charged ahead after every restart.

No Cache-To-Cache Parity Tests

We tested the model, not the cache. All green tests used fresh responses. The stale path only appeared under a specific key collision. We never wrote a test for that.

The Durable Fix

We replaced the naive cache with three layers.

1. Context-Aware Cache Keys

import hashlib

def cache_key(env: str, model: str, context: str) -> str:
    payload = f"{env}:{model}:{hashlib.sha256(context.encode()).hexdigest()}"
    return f"ai-response:{payload}"

Now staging cannot poison production. The prompt alone no longer decides the key.

2. Bounded TTL With Refresh Jitter

import random

def ttl_seconds(base_ttl: int = 3600) -> int:
    jitter = random.randint(-300, 300)
    return max(60, base_ttl + jitter)

Jitter prevents thundering herd at TTL expiry. We also cap TTL at 30 minutes for review bots. Old code reviews are worse than none.

3. Circuit Breaker With Persistent State

class AIBreaker:
    def __init__(self, redis):
        self.redis = redis
        self.key = "ai:breaker:state"

    def is_open(self) -> bool:
        return self.redis.get(self.key) == b"open"

    def open(self, ttl=120) -> None:
        self.redis.set(self.key, b"open", ex=ttl)

The breaker state lives in Redis. A server restart no longer resets it. We stop calling the model and stop reading the cache when the breaker is open.

Testing the New Behavior

We added a regression test that simulates the entire incident. You can reproduce it with this plan:

Start a local Redis instance.
Populate a cache entry with a fake model response.
Change the environment key from staging to production.
Confirm the production call misses the cache.
Set the breaker to open.
Restart the worker process.
Confirm the breaker is still open.

Here is the core test in Python:

def test_staging_does_not_poison_production():
    staging_key = cache_key("staging", "gpt-x", "patch context")
    prod_key = cache_key("production", "gpt-x", "patch context")
    assert staging_key != prod_key

def test_breaker_survives_restart():
    breaker = AIBreaker(redis)
    breaker.open(ttl=120)
    new_breaker = AIBreaker(redis)
    assert new_breaker.is_open()

The test suite now catches any regression in key isolation or persistent state.

What We Changed in Operations

We added a cache miss metric and a stale-hit alert. Every cached response now carries a generated_at timestamp. Alerts fire when the timestamp is older than 15 minutes.

We also removed the fallback-to-cache behavior on retry. If the model call fails and the breaker is closed, the job fails loudly. A failed job is safer than a confident stale answer.

Who Should Not Use This Approach

Do not copy this fix if you have no observability around cache hits. A cache with no metrics is a black box. You need at least a hit-rate counter and a stale-response log.

Do not use cache fallback for security-critical decisions. Approval gates should never silently reuse old model outputs. If you cannot tolerate a wrong answer, skip the cache entirely.

Lessons

Caching AI responses is not free. It trades compute cost for consistency risk. You must treat cache keys as security boundaries.

Free model tiers and free servers are useful. They force you to think about quotas and ephemeral state. But they do not excuse sloppy cache design.

Our review bot is now boring. It misses cache keys correctly. It opens breakers persistently. It fails loudly when it should.

That boringness is the best outcome of a painful postmortem.

If you run AI-assisted code review, think first about your cache key. The model is the easy part. The cache is where trust goes to die.

Postmortem: The Free Tier Wasn't the Problem — Our Guardrails Were

Morgan Xu — Mon, 31 Aug 2026 12:19:25 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier failed. The servers didn't. The tokens didn't. Our guardrails did.

This postmortem covers a real incident from our CI pipeline. We relied on MonkeyCode's free model access and free server option. We skipped the operational discipline. That mistake cost us a release night.

The lesson is not "free tiers are unreliable." The lesson is: free access needs stricter guardrails than paid access. Here is what broke, why it broke, and how to fix it.

The Incident Timeline

All times are UTC.

21:47

The nightly batch started. The pipeline sent 1,200 code review requests to MonkeyCode's free endpoint. No throttling was configured.

22:13

The first rate limit responses arrived. HTTP 429 mixed with HTTP 200 in the logs.

The client treated 429 as a retryable error. No backoff. No cap on attempts.

22:18

Retry storm started. Each 429 spawned 15 new requests. The queue grew exponentially.

22:36

The free server hit its memory ceiling. The process started swapping.

22:41

OOM killer terminated the server. All in-flight requests failed.

22:47

The retry loop restarted the server. Then killed it again. This loop lasted 18 minutes.

23:05

Someone disabled the retry flag manually. The pipeline drained. 30% of the batch had failed permanently.

The release went out without those reviews. We found a broken import at 02:14.

Contributing Factors

We identified five root causes:

No endpoint separation. Free and paid endpoints shared the same client. We could not apply different policies.
Unbounded retries. The default retry was infinite. It amplified every transient error.
No concurrency cap. The client fired 200 parallel requests. That is fine for a paid SLA. It is deadly for a free server.
No budget alarm. We had no token or request budget. No alert, no dashboard.
No fallback path. When free failed, the pipeline did not degrade to a smaller batch or a paid endpoint.

We treated a free resource like an infinite one. That is the real mistake.

The Durable Fix

We built a small Python client. It wraps any OpenAI-compatible endpoint with three guardrails:

A fixed-size token bucket for concurrency.
Exponential backoff with a hard retry cap.
A daily request budget.

Here is the core implementation:

import time
import threading
from datetime import date

class GuardedClient:
    def __init__(self, max_concurrency=8, max_retries=3, daily_budget=500):
        self.semaphore = threading.Semaphore(max_concurrency)
        self.max_retries = max_retries
        self.daily_budget = daily_budget
        self.used_today = 0
        self.lock = threading.Lock()
        self.today = date.today()

    def _check_budget(self):
        with self.lock:
            if date.today() != self.today:
                self.today = date.today()
                self.used_today = 0
            if self.used_today >= self.daily_budget:
                raise RuntimeError("Daily budget exhausted")
            self.used_today += 1

    def call(self, fn, *args, **kwargs):
        self._check_budget()
        with self.semaphore:
            for attempt in range(self.max_retries + 1):
                try:
                    return fn(*args, **kwargs)
                except Exception as e:
                    if attempt == self.max_retries:
                        raise
                    time.sleep(2 ** attempt)

Use max_concurrency=4 for free servers. Use max_retries=2 for free endpoints. Keep the budget low enough to survive a full day.

Decision Matrix

We now route requests by task weight. This table shows what goes to the free tier and what must go elsewhere.

Task	Use free endpoint?	Reason
Single file review	Yes	Low volume, low blast radius
Repo-wide scan	No	Too many calls, likely to throttle
One-off code explanation	Yes	Latency tolerant
CI blocking check	No	Must not depend on free tier
Batch summarization	Yes, chunked	Use small batches and long delays

If the task is on the critical path, pay for it. If it is async and idempotent, free is fine.

Validation

We benchmarked this client against MonkeyCode's free server for one week.

The test used 50 synthetic review tasks. We measured success rate, p95 latency, and budget usage.

Run the test yourself with this script:

for i in $(seq 1 50); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST "$MONKEY_FREE_ENDPOINT" \
    -H "Authorization: Bearer $MONKEY_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Review this function","max_tokens":200}' \
    --max-time 30
  sleep 2
done

Our results after this change:

Success rate: 94% (up from 61%)
p95 latency: 4.2s (down from 18s)
No OOM or retry storm

Your numbers will differ. The point is to measure before you trust.

Limitations

Free tiers change. Quotas shift. Servers get recycled. Our fix does not guarantee uptime.

The 10M token promotion and free server are real today. We verified that this week. But we do not assume they will exist tomorrow.

Design your pipeline so that a broken free tier degrades gracefully. Use a queue. Log failures. Alert on budget exhaustion.

Who Should Not Use This Approach

This approach is wrong for you if:

Your traffic spikes without warning
You need a strict SLA
You cannot tolerate occasional data loss

In those cases, use a paid endpoint with a guarantee. The guardrail client still helps, but it is not a replacement for a contract.

The Final Survivor

The free tier was never the enemy. Our assumptions were.

MonkeyCode's free access is a useful sandbox. It fits experimentation, low-risk tasks, and side projects. It does not fit release gates without proper constraints.

Add guardrails first. Then enjoy the free tokens.

Postmortem: The Review Gate That Approved Everything

Morgan Xu — Sun, 30 Aug 2026 11:26:11 +0000

A bad default can turn a review gate into a rubber stamp.

When a rule fails to parse, many gates fall back to approve. That one-liner converts a guardrail into a deployment pipeline. This post reconstructs that failure, walks a timeline, and ends with a negative-test fix.

The scenario below is a reconstruction. The code is runnable, and the failure is reproducible from the snippet alone.

The Failure Pattern

Production broke 90 minutes after merge. The patch touched auth/session.go. The review gate was configured to deny auth/**. It did not stop the merge. The config file still contained the rule.

The real bug lived in the loader, not in policy.

rules:
  - id: protect-auth
    paths: "['auth/**']"
    action: deny

YAML produced a JSON array turned into a string. The paths value was a string, not a list. The parser rejected it, and the loader converted corruption into an empty rule set.

import json

def parse_paths(raw: str) -> list[str]:
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return []   # BUG: corruption becomes "no rules"

def decide(files: list[str], rules: dict) -> str:
    patterns = parse_paths(rules["paths"])
    for path in files:
        for pattern in patterns:
            if pattern in path:
                return "deny"
    return "approve"

Parse errors returned an empty list. The gate evaluated no patterns. Every path reached the approve default.

Silent corruption. No log. No error. Just a green check.

Repro

Run the snippet below to see silent approval:

python - <<'PY'
import json

def parse_paths(raw):
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return []

def decide(files, rules):
    patterns = parse_paths(rules["paths"])
    for path in files:
        for pattern in patterns:
            if pattern in path:
                return "deny"
    return "approve"

rules = {"paths": "['auth/**']"}
print(decide(["auth/session.go"], rules))
# "approve" -- wrong. Expected: "deny"
PY

The gate denied nothing and still reported success. One config change did that.

Timeline (reconstruction)

Time	Event
00:00	Config migration writes the list as a JSON string.
00:01	`json.loads` fails; loader returns `[]`.
00:02	Gate evaluates no rules and returns `approve`.
01:30	CI reports the gate as green.
01:31	Merge queue completes.
03:10	Production shows a 500 spike after session changes.
03:40	Rollback happens.
04:15	Team discovers every patch has been approved for hours.

The failure was not a flaky model. It was an error-handling decision made long before merge.

Contributing Factors

Default-approve semantics. The gate treats "no rule hit" as safe.
Parse errors fail silent.
The gate had zero negative tests.
Merge treats missing feedback as a green check.

Any one of those factors is tolerable. Together they form a deploy permit.

Fix 1: Fail Loud

Parse failure must crash the gate, not become an empty list.

def parse_paths(raw: str) -> list[str]:
    parsed = json.loads(raw)
    if not isinstance(parsed, list):
        raise ValueError("paths must be a JSON array")
    return [p for p in parsed if isinstance(p, str)]

Errors now fail the gate. No data, no approval.

Fix 2: A Negative Test Matrix

The gate needs known-denied paths. Run this in CI.

# test_gate.py
import json
import pytest
from gate import decide

CASES = [
    (["controllers/user.go"], "approve"),
    (["db/migrations/0042.sql"], "approve"),
    (["auth/session.go"], "deny"),      # negative case
    (["auth/middleware.go"], "deny"),   # negative case
]

@pytest.mark.parametrize("files,expected", CASES)
def test_gate_matrix(files: list[str], expected: str):
    rules = {"paths": json.dumps(["auth/**"])}
    assert decide(files, rules) == expected

The matrix knows that deny must exist. It turns review-gate expectations from prose into assertions.

Fix 3: Merge Must Block on Missing Checks

Some merge systems treat a missing required check as success. Disable that behavior.

A silent gate needs to leave the queue blocked, not completed.

required_status_checks:
  - review-gate

If the review gate cannot report, do not merge. Missing is not green.

Cheap Infrastructure for Probing

Gate tests need disposable runtimes. MonkeyCode offers free model access and a free server option; both fit throwaway probe workloads well. Spin up, run the matrix, tear down. Keep production decisions on infrastructure with contractual availability, not an unpaid hobby box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness itself is vendor-neutral. Any compatible endpoint can run it. The fix is the matrix, not the backend.

Who Should Not Use This

If review rules rarely change, the lightest fix is still a small negative matrix.
If an AI agent rewrites config daily, the matrix must run on every change.
The free server option is not a production firewall. Availability changes. Keep it for probes, not auth paths.

The Durable Fix

Parse errors fail the gate.
Negative tests run in CI.
The merge queue stops when the gate cannot report.
A green gate is not a tested gate.

Run the self-test before the next patch. The assumption that kept the rubber stamp alive is gone.

Postmortem: The 93% Green Eval Was Statistical Noise

Morgan Xu — Sat, 29 Aug 2026 10:34:02 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The team tested the model. Nobody tested the test. The eval went 93% green. The merge broke production. This postmortem explains how the number lied. It also ships a durable fix.

The timeline below is a composite. It models a failure pattern, not one real incident.

Timeline

09:00 — Merge window opens. A refactor patch enters review.
09:12 — The eval runs once. It passes 14 of 15 cases. The gate turns green.
09:40 — The patch merges. Staging gets the build.
11:05 — Production error rate climbs. The team rolls back at 11:20.
13:30 — Investigation starts. The team re-runs the exact same eval.
13:31 — It scores 13 of 15.
13:35 — It scores 12 of 15.
13:40 — It scores 15 of 15.

One patch. Four different pass rates. The single green run was luck, not evidence.

Contributing factors

The eval ran once per patch. One draw from a random distribution became a fact.
The gate compared a point estimate. Nobody measured variance.
The free model endpoint had variable latency. Timeouts truncated eval contexts.
The timeout handler counted a truncated run as a pass.
Nobody tested the eval itself. The reviewer was never reviewed.

Factor five is the expensive one. Evals look like code. They behave like experiments.

Why free tiers amplify this

Free model access bends incentives quietly. A team on a metered endpoint runs the eval twice. A team on a free tier runs it once, to save tokens. Single-run evals hide nondeterminism completely.

MonkeyCode is an open-source project with free model access and a free server option. Its free tier currently includes 10 million tokens, per the project's terms. The team used it to reproduce this failure at scale. Repetition finally cost zero dollars. The free server option kept the long repeat-N runs off metered CI workers. Quotas move. Check the repository README before you depend on the numbers.

The durable fix: a repeat-N gate

A point estimate cannot judge a nondeterministic system. A confidence interval can. The new gate checks two properties: rate and stability.

# repeat_n_gate.py — fail the gate on instability, not just on low pass rate
import argparse
import math

def wilson(passes: int, runs: int, z: float = 1.96):
    if runs == 0:
        return 0.0, 0.0
    p = passes / runs
    denom = 1 + (z * z) / runs
    center = (p + (z * z) / (2 * runs)) / denom
    margin = z * math.sqrt(
        (p * (1 - p)) / runs + (z * z) / (4 * runs * runs)
    ) / denom
    return max(0.0, center - margin), min(1.0, center + margin)

def run_eval_once() -> bool:
    # Replace with your real eval invocation.
    # Never count a timeout or truncated run as a pass.
    raise NotImplementedError

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", type=int, default=20)
    ap.add_argument("--min-rate", type=float, default=0.90)
    ap.add_argument("--max-width", type=float, default=0.20)
    args = ap.parse_args()

    results = [run_eval_once() for _ in range(args.runs)]
    passes = sum(results)
    rate = passes / args.runs
    lo, hi = wilson(passes, args.runs)
    width = hi - lo

    print(f"passes={passes}/{args.runs} rate={rate:.2f} CI=[{lo:.2f}, {hi:.2f}] width={width:.2f}")

    if rate < args.min_rate:
        print("FAIL: pass rate below threshold")
        return 1
    if width > args.max_width:
        print("FAIL: confidence interval too wide — rerun or fix the eval")
        return 2
    print("PASS: rate and stability are acceptable")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The gate fails in two ways:

Exit 1 — pass rate below threshold.
Exit 2 — confidence interval too wide.

Width is the new signal. A wide interval means the eval needs more runs. It does not mean the model needs more prompts.

What the numbers look like

Runs	Passes	Rate	95% Wilson CI
1	1	1.00	[0.21, 1.00]
5	4	0.80	[0.38, 0.96]
20	18	0.90	[0.70, 0.97]
100	90	0.90	[0.83, 0.95]

One green run carries a huge error band. Twenty runs narrow the call. One hundred runs make it defensible. Pick N from the width you can tolerate. Do not pick N from your token budget alone.

Choosing N

The width formula is simple. For a pass rate near 0.90, the half-width is roughly 1.96 * sqrt(0.9 * 0.1 / N). A target half-width of 0.10 needs about 35 runs. A half-width of 0.05 needs about 140 runs.

Start with 20 runs. Measure the actual width. Raise N only when the gate tells you to.

Wiring it into CI

Run the gate on merge candidates. Not on every keystroke.
Store the last N results in a JSON file. Compare width over time.
Treat every timeout as a failed run. Never count truncation as a pass.
Lock temperature to 0 where the API allows it. Where it does not, expect noise.
Watch the width trend. Rising width means the endpoint changed, not your code.

Limitations

This gate tests stability. It does not test correctness. A stable wrong answer passes the repeat-N gate. Keep a held-out set that never touches tuning.

Repeat-N multiplies token use. Free tiers carry quotas. A 20-run gate over a 1,000-case suite will exhaust a small budget fast. Teams with deterministic endpoints can skip this entirely. Teams with tiny evals should spend budget on better cases first.

The composite failure above has one root cause. Variance was treated as truth. The remedy is boring — measure the spread before trusting the average. To reproduce this pattern cheaply, MonkeyCode's free tier or free server is a good start. Read their docs first. Quotas move fast.

Postmortem: The Agent Loop That Died Quietly While the Eval Said Green

Morgan Xu — Fri, 28 Aug 2026 04:01:48 +0000

Postmortem: The Agent Loop That Died Quietly While the Eval Said Green

A green eval is not a green light for the pipeline. This article walks through a synthetic incident reconstructed in a local harness. No production outage is claimed. The goal is to expose a common failure mode and a durable fix.

At 03:47 UTC, batch job 8447 stopped producing results. The review harness reported 41/41 checks passed. No alert fired. Six hours passed before a human noticed.

This postmortem examines that silent failure. It covers the timeline, the contributing factors, and the durable fix. The pattern applies to any AI agent pipeline, including free-tier setups.

MonkeyCode provides free model access and a free server option for experiments. Limits and terms change. Check the current docs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same reliability rules apply whether the worker runs on a free server or a paid cluster.

Timeline

All timestamps are UTC in the reproduction scenario.

Time	Event
02:10	Batch job 8447 starts.
02:10–03:45	Worker processes 212 tasks.
03:45	Queue reaches capacity. Worker requests more tasks.
03:46	Worker receives an empty page. Marks itself idle.
03:47	A socket timeout kills the worker's heartbeat.
03:47	Queue has 1,890 tasks. Nobody is consuming them.
08:12	A developer checks the dashboard.
08:14	Operator restarts the worker.
08:15	Worker resumes processing from checkpoint.

The failure produced no exception. The log contained one line: heartbeat_interval exceeded. The monitoring dashboard showed the worker as "idle." Idle looked healthy.

What Actually Broke

Three components failed together.

The model review harness measured outputs only.
The free server option has limited resources, but nobody checked the worker's liveness.
The worker treated "idle" as a valid end state.

The review harness compared generated patches against a contract. Every patch was valid. The harness had nothing to say about the queue, the disk, or the heartbeat.

Contributing Factors

The harness measured the model, not the system

The harness scored model outputs. It did not score the delivery path. A model can generate a perfect patch and the pipeline can still lose it.

The worker had no liveness contract

The handler loop had no heartbeat. When the socket closed, the process stayed alive. It just stopped pulling work.

Retries were not idempotent

The job state lived only in memory. A restart lost the cursor. The operator had to guess which tasks were already complete.

The Durable Fix

The fix has three layers.

1. Infrastructure contract test

Before every batch, run a test that verifies the environment.

def infrastructure_check(settings):
    checks = {
        "queue_connect": ping_queue(settings.queues.url),
        "disk_headroom": disk_free_pct(settings.worker.path) > 20,
        "model_endpoint": model_ready(settings.model.url),
        "clock_delta": abs(clock_skew()) < 5,
    }
    failed = [k for k, ok in checks.items() if not ok]
    if failed:
        raise SystemExit(f"infra check failed: {failed}")

This test runs in the orchestrator. It does not run inside the model's prompt. It checks the actual environment the worker will use.

2. Heartbeat dead-man switch

The worker writes a heartbeat file every 15 seconds. A supervisor watches it.

import time, pathlib

HEARTBEAT = pathlib.Path("/var/run/agent/heartbeat")

def beat():
    HEARTBEAT.write_text(str(time.time()))

def heartbeat_fresh(timeout=45):
    return time.time() - float(HEARTBEAT.read_text()) < timeout

If the heartbeat goes stale, the supervisor kills the process. Then the orchestrator starts a fresh worker. Idle is no longer a valid silent state.

3. Idempotent checkpointing

The worker persists task state to SQLite before applying a side effect.

CREATE TABLE jobs (
    task_id   TEXT PRIMARY KEY,
    status    TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

SELECT task_id FROM jobs WHERE status = 'queued' LIMIT 1;

A task is safe to process only after a transaction marks it in_progress. On restart, the worker scans for tasks in that state. It can safely re-run them because the application is idempotent.

def reserve_next(conn):
    conn.execute("BEGIN")
    row = conn.execute(
        "SELECT task_id FROM jobs "
        "WHERE status = 'queued' LIMIT 1"
    ).fetchone()
    if row:
        conn.execute(
            "UPDATE jobs SET status = 'in_progress', "
            "updated_at = datetime('now') "
            "WHERE task_id = ?", (row[0],))
    conn.commit()
    return row

Validation

The reproduction harness verified the fix.

Infrastructure test passed before the worker started.
The heartbeat monitor killed a simulated dead worker within one timeout window.
Checkpoint resume restored the full task queue.
Two forced restarts lost zero tasks.

The fix converted a six-hour outage into a 36-second restart.

Limitations

This approach does not fix everything.

It does not detect degraded model quality.
It does not replace integration tests run by a human.
It assumes the side effect is idempotent. Non-idempotent operations need external idempotency keys.
The free server option is suitable for experiments and low-stakes workloads. Critical production traffic needs a stronger resource contract.

Who Should Not Use This Pattern

Teams that cannot tolerate duplicate side effects should not adopt this pattern as-is. Payment operations and message sends need idempotency keys before retry. Teams that have no monitoring system should still implement the heartbeat. It is the cheapest alarm you can build.

Takeaway

The review harness answered the wrong question. It asked: "Does the model output meet the contract?" The real question was: "Will the output reach the user?" Those are different claims.

Run an infrastructure contract test before every batch. Give each worker a heartbeat. Persist the state. A green eval then means only one thing: the model was correct. It says nothing about the delivery path.

Measure the whole loop, not just the model. That is the only way an exhaustive review cannot hide an empty queue.

Postmortem: The Free AI Server That Passed Every Test, Then Failed at 02:14

Morgan Xu — Wed, 26 Aug 2026 11:24:17 +0000

A model can pass every functional test and still break your pipeline. The postmortem below is a composite reconstruction. It represents a common failure pattern with free-tier AI servers. Use it as a template for your own incident review.

Incident summary

At 02:14 UTC, a batch summarization job started receiving 503 errors. At 02:31, the retry queue held 12,000 messages. Impact was limited to staging. No customer data was involved. Severity: SEV-2. Total duration: 47 minutes.

The endpoint was a free-tier AI server. It had passed all functional contract tests the day before. Response shape, tone, and format were correct. The team routed traffic and went to sleep.

Timeline

01:27 — Batch job starts routing traffic to the new free-tier server.
02:14 — First 503 responses appear.
02:16 — The retry loop amplifies request volume by 6x.
02:23 — The provider's rate limit triggers.
02:31 — Queue backlog reaches 12,000 messages.
02:34 — An on-call engineer disables the routing flag.
02:47 — The queue drains. The service recovers.

Contributing factors

The model passed functional contract tests. Nobody tested the server under sustained load.
The retry policy had no backoff and no cap. Every failure created six more requests.
The health endpoint was never probed before routing.
The free server had no SLO. The team assumed it behaved like a paid endpoint.
The alert threshold was set to 'queue non-empty.' It fired 17 minutes after the first error.

Root cause

The failure was a missing operational contract. The team tested what the model returned. It never tested how the server behaved under load, latency, or failure. The model was fine. The server was an unverified dependency.

The durable fix: a server admission test

Before routing traffic to any new endpoint, run an admission test. It checks five properties. Uptime. Latency. Stability. Failure mode. Recovery.

The script below is a minimal version. It runs a health check, a latency probe, an error-rate burst, and a recovery check. Point it at a health or ping endpoint.

#!/usr/bin/env python3
# Server admission test: gate a new AI endpoint before routing traffic.
import json
import sys
import time
import urllib.error
import urllib.request

ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else 'http://localhost:8000/health'
TIMEOUT = 5.0

def probe(payload: dict) -> tuple[float, int]:
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'},
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
            return time.perf_counter() - start, resp.status
    except urllib.error.HTTPError as e:
        return time.perf_counter() - start, e.code
    except Exception:
        return time.perf_counter() - start, 0

# 1. Health check
lat, status = probe({'text': 'ping'})
assert status == 200, f'health check failed: HTTP {status}'

# 2. Latency probe: 50 sequential requests
latencies = [probe({'text': f'probe {i}'})[0] for i in range(50)]
p95 = sorted(latencies)[int(len(latencies) * 0.95)]
print(f'p95 latency: {p95 * 1000:.0f} ms')
assert p95 < 3.0, 'p95 latency exceeds the 3s budget'

# 3. Error rate under a small burst
errors = sum(1 for _ in range(20) if probe({'text': 'burst'})[1] != 200)
print(f'error rate: {errors / 20:.0%}')
assert errors / 20 < 0.05, 'error rate exceeds 5%'

# 4. Recovery: force a timeout, then verify the next request succeeds
try:
    urllib.request.urlopen(urllib.request.Request(ENDPOINT), timeout=0.1)
except Exception:
    pass  # expected: the endpoint should not hang forever
lat, status = probe({'text': 'after timeout'})
assert status == 200, 'server did not recover after a forced timeout'

print('admission test passed')

Run this before every routing change. Put it in CI or a pre-deploy step. It takes about 90 seconds.

The second fix: a retry budget

The admission test prevents the first mistake. A retry budget prevents the second. The old job retried forever with no backoff. The new policy allows three attempts, then dead-letters the message.

for attempt in 1 2 3; do
  curl -sS -f $ENDPOINT && break
  sleep $((2 ** attempt))
done

Pair it with a circuit breaker. Open the circuit after a 5% error rate over two minutes. Close it only after a clean admission test.

What the admission test would have caught

The admission test would have caught this incident. The latency probe would have flagged the degradation. The error-rate burst would have exceeded the 5% threshold. The recovery check would have failed after the forced timeout. The team would have kept the old endpoint and opened a ticket instead. That is the point of a gate. It fails before your users do.

Changes made after the incident

The team made five changes after the incident.

Added the admission test to the deployment pipeline.
Replaced the infinite retry loop with a three-attempt budget.
Added a circuit breaker to the routing layer.
Changed the alert to fire on error rate, not queue length.
Documented the free-server decision table in the runbook.

When a free server is acceptable

Free servers are not bad. They are unguaranteed. The decision table below is the rule the team now follows. The table is conservative. It errs on the side of caution.

Use case	Free server OK?	Why
Prototyping and eval	Yes	Data loss is cheap
Batch jobs with dead-letter	Yes	Retries absorb hiccups
User-facing real-time	No	No SLO or capacity guarantee
SLA-bound pipelines	No	Recovery is best-effort

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The open-source MonkeyCode project offers free model access and a free server option. That combination is useful for exactly this workflow. The team re-ran the admission test on a fresh MonkeyCode free server. This happened before any traffic was routed. The free token allowance (10 million at the time of writing) covered hundreds of probe runs. Verify the current terms and quotas before relying on them.

The contract test checked the model. The admission test checks the server. MonkeyCode's free tier made both cheap to run repeatedly. If you want a cheap place to run this experiment, the free tier is a good starting point.

Limitations

This approach does not fit every team. Teams with hard SLAs should not route production traffic through free servers. The admission test is a snapshot, not a guarantee. It cannot predict behavior after hours of sustained load. Treat the results as a floor, not a ceiling. Free tiers change. Quotas, rate limits, and availability can shift without notice.

The lesson

The model was never the problem. The server was an unverified dependency. Treat every free endpoint like an external service with no SLO. Test its behavior before you trust it with traffic. The next time you wire up a free endpoint, run the admission script first. It is cheaper than a 47-minute incident.

Free AI Servers Break in Seven Patterns: A Field Guide

Morgan Xu — Sun, 23 Aug 2026 17:44:51 +0000

Free AI servers break in seven patterns. These patterns are predictable. Each pattern has a specific workaround. This field guide classifies them. It applies to shared free-tier services. MonkeyCode's free model access and free server option fit this category. The operator-supplied quota is 10 million tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Classify Failures

Error messages describe what failed. They rarely explain why. A timeout can mean overload. It can mean a hung request. It can mean network trouble. Each cause needs a different fix.

Classification replaces guesswork with diagnosis. It builds a shared vocabulary. It makes failures reproducible. It converts "it broke" into "mode C, use workaround C."

The Seven Modes

Mode 1: Silent Truncation

The response stops mid-sentence. No error appears. The output looks complete at first glance.

The server hit a token or time limit. It returned partial output.

Check response length. Check for an unclosed code block.

Split the request. Request a summary instead. Set explicit length expectations.

Mode 2: Queue Black Hole

The request takes ten times longer. Then it times out. No response arrives.

The server is saturated. Requests queue behind other users.

Send a trivial request. If that is slow too, the server is saturated.

Retry with backoff. Work off-peak. Use batch mode, not interactive mode.

Mode 3: Context Amnesiac

The model forgets earlier instructions. It repeats answered questions.

The server truncated the conversation. It dropped older messages.

Ask for a fact from the first message. If it fails, context was dropped.

Keep conversations short. Restate constraints each time. Use one-shot prompts.

Mode 4: Confident Hallucination

The model gives a plausible wrong answer. The code compiles. The logic is broken.

The model is outside its training distribution. It cannot detect its own error.

Run the output. Compare against a known-good example.

Add a verification step. Never trust unexecuted code. Use drafts for exploration.

Mode 5: Syntax Slip

The model invents an API. The function does not exist. The parameter is wrong.

The training data is outdated. The model knows an older SDK.

Check against current documentation. Look for renamed functions.

Provide the API signature in the prompt. Include a working example. Pin the SDK version.

Mode 6: Load Degrader

The same prompt gives different quality. Morning output is worse. Afternoon output is better.

The server reduces quality under load. It trades accuracy for throughput.

Run the same prompt at different hours. Compare the outputs.

Schedule critical work off-peak. Keep a fallback for important output.

Mode 7: Random Drop

Requests fail intermittently. No pattern exists. Retry works immediately.

Transient network issues. Server restarts. Unknown causes.

Log every failure. Look for hidden patterns.

Implement retry with backoff. Treat the first failure as a warning.

The Diagnosis Table

Symptom	Mode	First action
Mid-sentence stop	Silent Truncation	Check output length
10x latency, timeout	Queue Black Hole	Send a ping
Forgets instructions	Context Amnesiac	Ask for message 1
Plausible but wrong	Confident Hallucination	Run the output
Nonexistent API	Syntax Slip	Check docs
Quality varies by hour	Load Degrader	Compare 9 AM vs 2 PM
Random failures	Random Drop	Log and retry

The Diagnostic Sequence

Follow this order when a failure appears.

Check for truncation. Verify output completeness.
Check for saturation. Test a trivial request.
Check for context loss. Ask for a fact from message 1.
Check for hallucination. Run the output.
Check for API drift. Compare against current docs.
Check for time patterns. Compare quality across hours.
Check for randomness. Retry the same request.

This sequence takes two minutes. It eliminates modes in likelihood order.

Two Companion Utilities

The Failure Log

# failure_log.py
import csv
import time

def log_failure(mode, symptom, prompt_hash, latency_s, retry_success):
    with open("failures.csv", "a", newline="") as f:
        w = csv.writer(f)
        w.writerow([
            time.strftime("%Y-%m-%d %H:%M:%S"),
            mode,
            symptom,
            prompt_hash,
            latency_s,
            retry_success,
        ])

The log reveals three patterns: dominant modes, worst hours, and triggering prompts. Without logs, classification is memory. With logs, it is evidence.

The Retry Wrapper

# retry.py
import time

def retry_with_backoff(func, max_attempts=3, base_delay=1.0):
    for attempt in range(max_attempts):
        try:
            return func()
        except Exception:
            if attempt == max_attempts - 1:
                raise
            time.sleep(base_delay * (2 ** attempt))

This wrapper handles Mode 7. It also softens Mode 2. It does not fix the root cause. It buys time for diagnosis.

Limitations

This taxonomy has boundaries.

It reflects common patterns, not a formal study.
Free-tier behavior changes with infrastructure.
Some failures are genuinely random.
The 10M token quota may change. Verify it.
This is a starting point, not a complete theory.

Teams with SLO requirements should not rely on this guide. Teams needing deterministic behavior should not either. Free tiers are not production infrastructure.

The 15-Minute Adoption Plan

Copy the diagnosis table into the team wiki. (2 minutes)
Add the logger to the integration. (5 minutes)
Add the retry wrapper to the client. (3 minutes)
Log every failure for one week. (passive)
Review the log. Identify the top three modes. (5 minutes)
Apply the matching workarounds. (ongoing)

Conclusion

Free AI servers fail in patterns. Patterns are classifiable. Classification enables targeted fixes. MonkeyCode's free tier is a useful tool. The taxonomy makes its failures predictable.

Start logging. Classify your failures. Share your field notes. The community needs observed patterns, not vendor promises.

MiniMax H3: Run This 30-Minute Gate Before You Trust It

Morgan Xu — Fri, 14 Aug 2026 13:58:58 +0000

Before you trust MiniMax H3 with real work, run a 30-minute gate: four private, repeatable checks against your own code tasks. This gives you a defensible decision faster than any leaderboard screenshot.

Why a 30-Minute Gate Beats a Leaderboard Screenshot

When a new model like MiniMax H3 starts circulating, you will see a clean benchmark number and a thread arguing about it. Neither tells you whether the model will survive your actual work. Public leaderboards such as the Hugging Face Open LLM Leaderboard and Stanford HELM are useful aggregate signals, but they are also contaminated by training data, prompt leakage, and selection bias. The HELM paper makes a strong case for multi-metric evaluation, but even that does not replace private tasks.

A 30-minute gate forces you to evaluate models against failures you have already experienced, not someone else's highlight reel. You are not testing the vendor's claimed score. You are testing four things that matter for your own work.

The Four Checks That Matter for My Work

The gate is intentionally narrow. It catches most early failures before you invest hours.

Can it follow a constrained instruction with an unfamiliar input?
Can it generate a small, correct code change?
Can it debug a failing test without being handed the answer?
Can it say 'I don't know' instead of hallucinating an API?

Each check is tied to a task I have actually hit: a function that broke last month, a prompt that made a previous model invent a requests.post signature, or a code review comment I wrote in anger.

A Reproducible Harness You Can Run on Demand

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Before you touch H3, put the harness somewhere you can run it on demand. I use a free server because it makes the evaluation reproducible for other people instead of being a screenshot on my machine. MonkeyCode's free model access and free server option are useful here for running the same task set against a baseline model and against the new release without pulling your own GPU.

1. Keep a private task set

Create a directory of tasks that are not in public benchmark datasets.

git clone git@github.com:yourname/private-eval-30m.git
cd private-eval-30m
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Use tasks from your actual projects. These are the failures you do not want to repeat.

2. Run the same harness against a baseline

Pick a model you already use and trust for this task, and run it first.

export EVAL_BASE_URL='https://api.monkeycode.example/v1'
export EVAL_MODEL='your-current-model'
python run_gate.py --suite private_set --output baseline.json

The URL and model names above are placeholders. Use the endpoint and model identifier shown in your server's API settings; the point is to have a comparable baseline, not a specific vendor.

3. Point the same harness at H3

export EVAL_MODEL='h3'
python run_gate.py --suite private_set --output h3.json

Run both passes with the same timeout, system prompt, and temperature. Do not tune the prompt between runs. If you tune for H3, you are testing your tuning, not the model.

4. Compare with a decision table

Do not dump the raw score. Write the result into a table that forces a decision.

Check	Threshold	Baseline	H3	Action
Follow unfamiliar instructions	>= 80%	84%	79%	Investigate prompt
Generate a small correct diff	>= 70%	72%	68%	Borderline
Debug without hinted answer	>= 60%	63%	71%	Pass
Say 'I don't know' instead of invent	>= 90%	88%	94%	Pass

The table is the artifact. It tells you which specific behavior improved or regressed, so your next step is small: rerun only the borderline case, or inspect the H3 failure log.

Why This Is an Open-Source Habit

Open source is not just a license. It is the practice of making results reproducible, auditable, and cheap enough for another person to check. When you run a gate on a free server and keep the task set in version control, you are applying that practice to model evaluation. You are saying: here is the exact command, here is the exact input set, and here is where the model failed. Somebody else can rerun it without paying for a cluster or trusting your screenshot.

MonkeyCode's free tier fits here because it lowers the cost of the control run. The free server path is not a replacement for production serving; it is a place to run honest comparisons before you commit.

Limitations and When to Skip This Gate

This is a smoke test, not a production qualification. A model can pass all four checks and still fail under concurrent load or long context, leak sensitive data if you send private code to an external endpoint, change behavior between API revisions without warning, or look good on your private tasks but terrible on tasks from another team.

If you are handling customer data or regulated workloads, do not send it to any third-party server, including a free one. Keep a human review step before any generated patch reaches main, especially when the model is new.

Skip this exact workflow if you need a dedicated SLA and cannot tolerate queue delays, must keep your code on your own infrastructure for legal reasons, only need a model to chat, or want a single leaderboard number to make the decision for you.

Your Next Step

When MiniMax H3 starts circulating, do not forward the benchmark image. Run your own four checks first. The point of the gate is not to be cynical about new models; it is to make sure you are testing them against your work, not someone else's highlight reel.

Here is your action: clone the harness, pick one real failure from last month, and run it as your baseline task this week. Then point the same task at H3 and post the decision table for your team. If you need a free control environment for the baseline, MonkeyCode's free tier is one option—just keep private code out of any external endpoint. If you already run a similar gate, share your threshold table in the comments so others can compare.

A Free-Tier Triage Harness for AI Patches: Make the Model Earn Your Attention

Morgan Xu — Fri, 14 Aug 2026 12:27:02 +0000

Why this is worth your time: You do not have a model shortage. You have a triage problem. Every few days another release claims to be cheaper and better, but 'cheaper and better' does not tell you whether a generated patch should waste an hour of your review time. This guide gives you a small, repeatable first-pass filter you can run on a free tier. It treats a free model endpoint and a free server as a disposable lab, not as a truth machine. You will leave with a decision table, a runnable Python harness, three failure cases that green tests usually miss, and a clear list of teams that should not use this approach.

You should read this if you already review AI-generated patches and want a cheaper way to reject the bad ones before they reach a human. You should skip it if you need a production merge gate, sub-second latency, or a privacy guarantee.

Start with a decision table, not a benchmark

You are not evaluating 'the model'. You are evaluating one candidate patch against the few constraints that matter for your repository. Write those constraints before you run anything.

Check	What it catches	What it still misses
Syntax compile	Broken Python/TypeScript/Go files	Undefined names, bad imports, bad logic
Pytest/unit suite	Behavior regressions covered by tests	Gaps in test coverage
Linter or static checker	Undefined names, unused imports, obvious misuse	Runtime behavior and semantic drift
Wall-clock timeout	Slow loops, accidental network calls	Complexity hidden behind small inputs
Diff size cap	Massive rewrites that are too risky to review	Small but toxic changes

You can run the first four checks on almost any free Linux box. The free tier is the right place for this because the work is bursty, short, and failure-tolerant: you run a command, wait a few seconds, and throw away the container.

The example setup: a free model endpoint plus a free server

The example provider used here is MonkeyCode, because it offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness does not depend on MonkeyCode-specific APIs; it treats the model as an HTTP endpoint and the server as a place to run shell commands.

You should verify the current free-tier limits yourself before you run anything. Do not copy a quota from a blog post, including this one, because quotas change. This article intentionally avoids naming a specific model or token limit.

A minimal setup looks like this:

[your branch] -> patch file
          |
          v
[free server] -> triage_patch.py
          |
          +-- apply patch to a temp copy
          +-- run syntax, lint, unit, smoke checks
          +-- call the free model endpoint only for review annotations
          +-- print reject / review_slow / human_review

You do not need a GPU box for this. You need a small Linux instance, a checkout of the relevant code, and enough disk for a test environment.

A runnable harness you can steal as a starting point

The script below is a proposal, not a product. It assumes you have pytest and ruff installed, and it does not send your source code to any model. The model call is deliberately kept out of the critical path; the deterministic checks run first.

#!/usr/bin/env python3
'''First-pass AI patch triage for a free-tier server.

Use as a starting point. Adjust PATCH_CHECKS to match your stack.
'''
import argparse
import subprocess
import sys
import time
from pathlib import Path


def run_check(args, timeout=20):
    start = time.monotonic()
    try:
        proc = subprocess.run(
            args,
            text=True,
            capture_output=True,
            timeout=timeout,
        )
        return {
            'cmd': ' '.join(args),
            'returncode': proc.returncode,
            'stdout_tail': proc.stdout[-400:],
            'stderr_tail': proc.stderr[-400:],
            'wall_seconds': round(time.monotonic() - start, 2),
        }
    except subprocess.TimeoutExpired:
        return {
            'cmd': ' '.join(args),
            'timed_out': True,
            'wall_seconds': timeout,
        }


def evaluate_patch(repo_dir: Path, timeout=20):
    checks = [
        {'name': 'syntax', 'run': ['python', '-m', 'py_compile', 'candidate.py']},
        {'name': 'lint', 'run': ['ruff', 'check', 'candidate.py']},
        {'name': 'unit', 'run': ['pytest', '-q', 'tests/']},
        {'name': 'smoke', 'run': ['python', 'candidate.py', '--smoke-test']},
    ]

    results = []
    for check in checks:
        result = run_check(check['run'], timeout=timeout)
        result['name'] = check['name']
        results.append(result)

    return results


def decide(results, max_wall_seconds=15):
    for result in results:
        if result.get('timed_out') or result['returncode'] != 0:
            return 'reject'

    slowest = max(result['wall_seconds'] for result in results)
    if slowest > max_wall_seconds:
        return 'review_slow'

    return 'human_review'


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--repo', default='.')
    parser.add_argument('--timeout', type=int, default=20)
    args = parser.parse_args()

    results = evaluate_patch(Path(args.repo), timeout=args.timeout)
    verdict = decide(results)

    for result in results:
        name = result['name']
        print(f'{name:<8} {result}')

    print(f'verdict: {verdict}')
    sys.exit(0 if verdict != 'reject' else 1)


if __name__ == '__main__':
    main()

This is not a complete product. Before you use it, replace candidate.py and tests/ with the actual entry points in your repository, add a diff-size cap, and decide whether you want the model call to be inside or outside this script. Run it once on a known-good patch and once on a known-bad patch to make sure the checks are actually differentiating.

Three failure cases green tests will not catch

The value of this harness is not that it runs tests; it is that it makes a few failure modes impossible to ignore.

Case 1: The off-by-one patch that passes existing tests. A model changes a loop boundary from range(len(items)) to range(len(items) - 1). Your unit tests pass because they only test small lists. Add one boundary test for empty input, single-item input, and the exact size where the old code failed. If that test is not in the suite, the harness can still reject the patch when the smoke test fails on a representative payload.

Case 2: The hallucinated API call. A patch calls os.set_immutable(path) or list.unique() because the model mixed up another language or library. py_compile will not always catch this. ruff or pyflakes can flag an undefined name, but only if you run it. This is why the linter step is not optional.

Case 3: The silent resource regression. A patch adds a nested loop that looks fine on three test items and then crawls on a 10,000-item payload. The wall-clock timeout is your backstop. You can also add a small benchmark command such as python -m timeit -n 1 -r 1 -s 'from candidate import process' 'process(large_input)' and reject anything above your threshold.

Where a free tier should not be the decision point

Free tiers are good for triage, not for authority. Do not use this workflow when:

The patch touches authentication, payments, personal data, or secrets.
You need a fixed latency or a contractual SLA.
The test suite is too slow to finish within free-tier timeouts.
The model output is trained or cached in a way you cannot audit.
Your review process requires every decision to be reproducible for compliance.

If you are in one of those situations, use the same checks but run them on infrastructure you control, with an approved model endpoint and a real data handling policy.

Use the free tier as a first filter, not as a belief system

You do not need to accept or reject every model release. You need a consistent way to ask: 'Does this patch fail my cheapest available checks?' If it does, stop. If it passes, send it to a human with the timings and tails attached.

If you already have access to MonkeyCode's free model and free server, wire this harness to a draft branch or a draft pull request and let it be the first triage step, not the merge decision. Then keep the reports. A month of rejections tells you more about your test gaps than any model card.

The model name you saw this morning will be replaced by another one next week. The checks above will still be useful.

Before You Plug a Model Into Your Pipeline, Make It Pass a Contract Test

Morgan Xu — Fri, 14 Aug 2026 10:52:57 +0000

Why this is worth reading

A new model release often claims strong JSON or tool-calling behavior, but your task fails because the output is shaped wrong, not because the model lacks "intelligence". You will learn to define a small output contract, run it against any OpenAI-compatible endpoint with a repeatable Python script, and interpret the results without trusting a leaderboard. This is a smoke test for the exact failure that breaks pipelines: drift in structure and assertions. It runs cheaply, and the free-infrastructure section below mentions one availability option.

The failure mode most evals do not catch

Most benchmark numbers summarize average performance across someone else's distribution. They do not tell you whether a model will emit the correct JSON shape for your incident summarizer, keep duplicate action IDs out, or reject extra keys that your types ignore. When that contract breaks, you notice as a runtime error after deployment, usually as NoneType or a schema validation exception in code you did not write. The fix is to stop comparing models as interchangeable generators and instead treat each one as a candidate implementation of a contract.

A contract has two parts: a JSON Schema and a small set of task-specific assertions. The schema checks shape; the assertions check meaning and invariants. Neither is enough alone. A model can return valid JSON that says nothing useful; or it can return useful text that crashes your parser. The harness below makes both checks repeatable.

Start with the contract, not the prompt

Create contracts/incident.json:

{
  "schema": {
    "type": "object",
    "required": ["summary", "actions", "confidence"],
    "additionalProperties": false,
    "properties": {
      "summary": {"type": "string", "minLength": 10},
      "actions": {
        "type": "array",
        "items": {
          "type": "object",
          "required": ["id", "command"],
          "additionalProperties": false,
          "properties": {
            "id": {"type": "string"},
            "command": {"type": "string"}
          }
        }
      },
      "confidence": {"type": "number", "minimum": 0.0, "maximum": 1.0}
    }
  }
}

Then define assertions in contracts/incident_checks.py:

def summary_is_not_a_refusal(doc):
    return doc['summary'].strip().lower() not in {
        'as an ai language model',
        'i cannot help',
        'i am unable to'
    }


def action_ids_are_unique(doc):
    ids = [a['id'] for a in doc['actions']]
    return len(ids) == len(set(ids))


def confidence_is_usable(doc):
    return doc['confidence'] >= 0.0 and doc['confidence'] <= 1.0

Notice you are not asking the model to "try harder" in the prompt. You are defining the failure condition first, which makes a pass less ambiguous.

A minimal contract probe you can run anywhere

The script accepts an endpoint, API key, model name, and input file. It sends a single request, parses the first JSON object in the completion, validates schema, runs assertions, and prints a compact result plus the first failure. It is deliberately small so you can read every line before sending your data anywhere.

Save as contract_probe.py:

#!/usr/bin/env python3
import argparse
import json
import os
import sys
import time
import requests
from jsonschema import Draft202012Validator


def first_json_object(text):
    decoder = json.JSONDecoder()
    for i, ch in enumerate(text):
        if ch == '{':
            try:
                obj, _ = decoder.raw_decode(text[i:])
                return obj
            except json.JSONDecodeError:
                continue
    raise ValueError('no JSON object found')


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--endpoint', default=os.environ.get('MODEL_ENDPOINT'))
    p.add_argument('--key', default=os.environ.get('MODEL_KEY', 'not-used'))
    p.add_argument('--model', default=os.environ.get('MODEL_NAME'))
    p.add_argument('--prompt', required=True)
    p.add_argument('--contract', required=True)
    p.add_argument('--temperature', type=float, default=0.0)
    args = p.parse_args()

    contract = json.load(open(args.contract))
    schema = contract['schema']
    prompt = open(args.prompt).read()

    checks = {}
    checks_file = args.contract.replace('.json', '_checks.py')
    if os.path.exists(checks_file):
        import importlib.util
        spec = importlib.util.spec_from_file_location('checks', checks_file)
        mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(mod)
        checks = {
            name: getattr(mod, name)
            for name in dir(mod)
            if callable(getattr(mod, name)) and not name.startswith('_')
        }

    start = time.time()
    if not args.endpoint or not args.model:
        print('MODEL_ENDPOINT and MODEL_NAME are required')
        sys.exit(2)

    payload = {
        'model': args.model,
        'temperature': args.temperature,
        'messages': [
            {
                'role': 'system',
                'content': 'Return exactly one JSON object matching the provided requirement and no prose.',
            },
            {
                'role': 'user',
                'content': prompt,
            },
        ],
    }

    try:
        r = requests.post(
            args.endpoint,
            headers={'Authorization': f'Bearer {args.key}'},
            json=payload,
            timeout=60,
        )
        r.raise_for_status()
        content = r.json()['choices'][0]['message']['content']
        doc = first_json_object(content)
        elapsed = time.time() - start

        validator = Draft202012Validator(schema)
        schema_errors = sorted(e.message for e in validator.iter_errors(doc))
        assertion_failures = [
            name for name, fn in checks.items() if not fn(doc)
        ]
        outcome = 'PASS' if not schema_errors and not assertion_failures else 'FAIL'
        print(json.dumps({
            'outcome': outcome,
            'latency_s': round(elapsed, 2),
            'schema_errors': schema_errors[:3],
            'assertion_failures': assertion_failures,
            'token_or_choice_count': len(content),
        }, indent=2))
    except Exception as e:
        print(json.dumps({
            'outcome': 'ERROR',
            'error_type': type(e).__name__,
            'error': str(e)[:400],
            'latency_s': round(time.time() - start, 2),
        }, indent=2))


if __name__ == '__main__':
    main()

Run it locally:

python -m venv .venv && source .venv/bin/activate
pip install requests jsonschema
mkdir -p contracts
# add contracts/incident.json and contracts/incident_checks.py from above
cat > incident.txt <<'EOF'
A cron job on db-3 failed with OOM at 04:11. It was restarted once,
the second run completed at 04:26, and no rows were lost.
EOF

python contract_probe.py --endpoint "$MODEL_ENDPOINT" --model "$MODEL_NAME" --prompt incident.txt --contract contracts/incident.json

The result is a JSON line, so you can redirect to a file and diff across repeated runs. Repeat the probe three to five times because a single lucky parse is not evidence. Count a model as passable only when every run returns PASS and the failure mode is stable when it does not.

Interpret failures without guessing

Use this table when a run falls over, because the failure category tells you where to spend time:

If you see	What it usually means	What to do next
`schema_errors` includes `additionalProperties`	The model returns extra fields you did not ask for	Tighten `additionalProperties: false` and add those fields only if they should be in the contract
`schema_errors` on `confidence`	Number emitted as string or outside 0-1	Normalize or reject at the model boundary; do not silently patch downstream
`summary_is_not_a_refusal`	Prompt/safety refusal without useful output	Treat as a capability miss for this task, not a reason to lower the threshold
`action_ids_are_unique`	Duplicate identifiers	Your contract needs a uniqueness invariant; schema cannot express it
`ERROR` with `HTTPError`	Endpoint, auth, or quota problem	Check the dashboard limits and retry with a cool-off, not in a tight loop

The useful conclusion from a probe is not "model A is better than model B." It is "model A keeps violating the additionalProperties rule, so I can either fix my prompt, add a repair step, or exclude it from this extraction path."

Where a free endpoint and free server fit

You do not need to send every probe to an expensive hosted account. The script above is small enough to run from a free CI runner or a low-cost server. One option the operator of this content is making available is MonkeyCode, which advertises free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Verify current quotas, model names, endpoint shape, and retention rules in the dashboard before you commit any data; the contract harness itself does not depend on MonkeyCode and works with any OpenAI-compatible endpoint.

For this kind of smoke test, use the free tier only for non-sensitive prompts and public fixtures. Do not log API keys, and do not treat a free endpoint as an availability guarantee for production.

Limitations you should be honest about

The contract harness is not an eval suite. JSON Schema can confirm shape and a few invariants, but it cannot tell you whether the summary is true or whether the command is safe. It also cannot measure factuality, tone, latency under load, privacy, or cost. If a model passes the probe, you have cleared only the cheapest bar: the output is parseable and obeys the contract you wrote. You still need golden examples, a human review for risky commands, and production monitoring.

The artifacts are a plan, not a claim that any specific model or free tier will behave the same for your traffic or stay free tomorrow. Product limits change, and a public model endpoint may rotate models or alter routing without notice.

Who should skip this

Skip or adapt this approach if your output is long-form prose, creative copy, or open-ended advice: a JSON schema adds clutter without improving quality. Skip it also if you are sending sensitive customer data to an external endpoint; run a local model or a private deployment instead. And skip it if you are trying to replace a full acceptance suite: this is a gate, not a green light.

Put it in front of the next model you try

Write the contract before you read the model card. Run the probe from a free server, repeat it enough to separate bad luck from bad outputs, and keep the failures where you can see them. The model is not ready until your parser can trust the shape, and the most useful model evaluation you can run today is the one you write around the exact JSON your pipeline expects.

If you try it, the comment worth leaving is which assertion caught the first real break: shape, uniqueness, or number type. That tells the next reader more than any leaderboard table will.

A New Open-Weight Model Drops Every Week. Here's the 30-Minute Eval Harness I Run Before Believing Any Benchmark

Morgan Xu — Mon, 10 Aug 2026 10:59:39 +0000

If your feed looks anything like mine this week, it's wall-to-wall hot takes about the latest open-weight coding model release — right now it's MiniMax's H3 getting the treatment, last month it was something else, next month it'll be something newer. The pattern is always the same: a launch post, a set of vendor benchmark tables, a wave of "this changes everything" threads, and then, quietly, a trickle of "actually it's mediocre at X" follow-ups.

I'm not going to tell you whether H3 is good. I haven't run it through enough of my own workloads to say, and honestly, neither have most of the people posting about it. What I can give you is something more durable: a small, reproducible evaluation harness that takes about 30 minutes to set up and tells you whether any newly released model is worth switching to for your code, not for a benchmark suite.

Why vendor benchmarks don't answer your actual question

The question a launch benchmark answers is: "How does this model perform on a standardized set of problems under controlled prompting?" The question you actually have is: "Will this model waste less of my time than the one I'm already using, on the kind of work I actually do?"

These diverge fast. Your work has a specific language mix, a specific repo style, specific tolerance for verbosity, specific failure modes you can't stand (mine: confidently inventing API methods that don't exist). A two-point swing on a public leaderboard says almost nothing about any of that.

So the harness below is built around your tasks, scored automatically where possible and by checklist where not.

The harness: 12 tasks, 3 scoring tiers

The whole thing is a directory of task prompts plus a runner script. Here's the structure:

model-eval/
├── tasks/
│   ├── 01_fix_failing_test.md
│   ├── 02_refactor_for_readability.md
│   ├── 03_explain_unfamiliar_code.md
│   ├── ...
│   └── 12_migration_snippet.md
├── runner.py
├── score.py
└── results/

Each task file is a prompt plus an expected-output specification. Three tiers of tasks, four each:

Tier 1 — mechanically verifiable. Tasks where the output can be checked by a script: "fix this failing pytest," "write a function passing these assertions," "produce valid JSON matching this schema." These get scored by actually executing the output.

Tier 2 — checklist verifiable. "Refactor this function" tasks scored against an explicit rubric: no behavior change (run the existing tests), no new dependencies, cyclomatic complexity not increased. Partially automatable.

Tier 3 — judgment tasks. "Explain what this regex does," "review this diff for security issues." Scored by you, but against a written answer key you make before looking at model output, so you're not grading on vibes.

Here's the runner — deliberately boring, provider-agnostic, ~40 lines:

import json, subprocess, sys, time
from pathlib import Path

def run_task(client, model: str, task_file: Path) -> dict:
    prompt = task_file.read_text()
    start = time.time()
    response = client.complete(model=model, prompt=prompt, max_tokens=2048)
    elapsed = time.time() - start
    return {
        "task": task_file.stem,
        "model": model,
        "output": response.text,
        "latency_s": round(elapsed, 2),
        "tokens_out": response.usage.completion_tokens,
    }

def score_tier1(result: dict, task_dir: Path) -> bool:
    """Write model output to a file, run the task's checker script."""
    candidate = task_dir / "_candidate.py"
    candidate.write_text(result["output"])
    checker = task_dir / "check.sh"
    proc = subprocess.run(["bash", str(checker)], capture_output=True)
    return proc.returncode == 0

if __name__ == "__main__":
    model = sys.argv[1]
    client = get_client()  # whatever provider you're testing
    results = []
    for task in sorted(Path("tasks").glob("*.md")):
        r = run_task(client, model, task)
        r["tier1_pass"] = score_tier1(r, Path("tasks")) if is_tier1(task) else None
        results.append(r)
    Path(f"results/{model.replace('/', '_')}.json").write_text(
        json.dumps(results, indent=2)
    )

Twelve tasks feels small. It is small — deliberately. You'll actually finish a 12-task eval for every new model that drops. You will not finish a 200-task eval more than once, and an eval you run once is a demo, not a tool.

Where to run it without torching your budget

The annoying part of doing this properly used to be access: to compare a new open-weight model against your incumbent, you need both models available through something you can script against, and paying per-token across several providers for eval runs adds up fast, especially when you're iterating on the harness itself.

This is where I've been using MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The relevant bit for this workflow is that it offers free access to a set of models plus a free server option, which means the "spin up an endpoint, point the runner at it, tear it down" loop costs me nothing while I'm debugging the harness — and harness debugging is where most of the token spend actually goes, not the final eval run. I'd rather burn my mistakes on a free tier and reserve paid API calls for the comparisons that matter.

What I appreciate beyond the pricing is the posture: the project leans into open source — the tooling is out in the open, you can read what the client is actually sending, and you're not reverse-engineering a black box to figure out why your eval numbers look weird. For an eval harness specifically, that transparency matters more than for casual use, because "I don't know what the wrapper did to my prompt" invalidates your results.

If you want to try this workflow yourself, the free server option is enough to get the runner above working end-to-end; grab a task from your own recent git history and start there.

What I learned running this against recent releases

Without naming specific scores (your tasks will differ, and that's the whole point), the pattern across the last few open-weight releases I've put through this harness:

Tier 1 gaps are smaller than marketing implies. Most current-generation open coding models pass the mechanically-verifiable tasks at similar rates. The differentiation has moved to Tier 2 and 3.
Verbosity is a hidden cost. One model I tested passed tasks at the same rate as my incumbent but produced 40% more output tokens per task — more to review, more latency, more cost per accepted suggestion. My harness counts tokens_out for exactly this reason.
Failure style matters more than failure rate. I keep a one-line note per failed task: "wrong but obviously wrong" vs "wrong and confident-looking." Models in the second category get rejected even with decent pass rates, because review time is my real bottleneck.
Latency on your hardware/endpoint is not the benchmark's latency. A model that looks great in a datacenter demo can be unusable if the endpoint you're actually hitting is slow. Measure it yourself; it's two lines of code, as above.

Limitations, and who shouldn't bother

Twelve tasks can't measure everything. This harness is a smoke test, not a certification. It's good at answering "is this worth a week-long trial," bad at answering "is this safe for production code review."
Task selection bias is real. If your tasks are all Python bugfixes, you'll pick the model that's best at Python bugfixes. That's fine if that's your job; it's a trap if it isn't. Rebuild the task set when your work changes.
Judgment-tier scoring is still you. The pre-written answer key reduces but doesn't eliminate grader drift.
If you only use a model for autocomplete-style completions, this harness is overkill — you're better off measuring accept-rate in your editor for a week.
Free tiers change. Any free model access or free server option — MonkeyCode's included — is an availability claim, not a permanent guarantee. Design your harness so the client is swappable and you're never locked into one endpoint.

The actual takeaway

The next time a model drops and the feed explodes — H3 today, something else in three weeks — the correct response isn't hype or dismissal. It's: pull the release, run your twelve tasks, compare against your incumbent on your rubric, and make a boring evidence-based decision in an afternoon. The teams that get value from the current pace of model releases aren't the ones with the strongest opinions; they're the ones with the cheapest evaluation loop.

Build the harness once. Every future launch post becomes a 30-minute chore instead of a week of discourse.

Green Tests Lie: How I Gate AI-Generated Patches Before They Touch Main

Morgan Xu — Mon, 10 Aug 2026 08:36:35 +0000

Every team chat I'm in has landed on the same uncomfortable question: an agent can write a plausible patch in ninety seconds, but who vouches for it? Reading the diff by eye feels like due diligence. In practice, it mostly rewards fluency — and generated code is fluent by construction.

The stance I settled on: treat agent output like a pull request from a stranger with no commit history. You wouldn't merge that because it read nicely. You'd merge it because it survived your gauntlet. Below is the gauntlet I run, built entirely on compute that costs nothing, followed by an honest list of where it falls apart.

Why eyeballing diffs keeps letting bad patches through

When I audit agent patches that got a thumbs-up and later caused trouble, three defects recur:

Prompt-shaped correctness. The code nails the scenario in the request and crumbles on the adjacent ones — empty input, DST boundaries, a record with ten thousand rows.
Scope creep in the edit set. You asked for a fix in one module; the patch also nudges a lint config or a lockfile you never mentioned.
Gates quietly lowered. The suite is green because a strict assertion became a loose one, or a test file got an early return.

None of these announce themselves in a casual read. So the process below makes each one impossible to skip.

The gauntlet, stage by stage

Four stages, each producing an artifact you can revisit months later:

Quarantine — run the patch in an environment with no network and no secrets.
Scope audit — diff the patch against what was actually requested.
Adversarial probes — tests you write before looking at the implementation.
Decision record — one file capturing inputs, outputs, and verdict.

Stage 1: Quarantine

The test environment should cost less to destroy than to disinfect. Mine is a scratch clone executed inside a container with networking disabled:

#!/usr/bin/env bash
# quarantine.sh — run an untrusted patch with no way in or out
set -euo pipefail

REPO=$1
PATCHFILE=$2
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

git clone -q "$REPO" "$WORKDIR/tree"

docker run --rm \
  --network none \
  --cap-drop ALL \
  --read-only \
  --tmpfs /tmp \
  -v "$WORKDIR/tree":/repo \
  -v "$PATCHFILE":/queued.patch:ro \
  -w /repo \
  local/gate-runner:py312 \
  sh -c 'git apply /queued.patch && python -m pytest -x -q'

Two properties carry all the weight: traffic cannot escape (--network none), and nothing sensitive can wander in (no credential mounts, no capabilities, read-only root). A useful mental model: you're grading the artifact, not the agent. It doesn't matter where or how the model executed — only whether the patch survives this box.

Stage 2: Scope audit

Before reading a single implementation line, get the full inventory of what the patch touches:

git apply --stat /queued.patch
git apply --check /queued.patch   # a patch that won't apply cleanly gets rejected on the spot

Then classify every path relative to the original request:

Path in patch	Asked for?	Verdict
`src/parse_date.py`	yes	normal review
`tests/test_parse_date.py`	no	manual line-by-line
`pyproject.toml`	no	reject pending justification

The one heuristic I'd tattoo on the process: unrequested modifications to tests, CI configs, or dependency pins start with a presumption of guilt. Softening a gate is the cheapest way to make a wrong patch look right.

Stage 3: Adversarial probes

A passing project suite only tells you the patch preserved behaviors someone previously thought to test. Everything else is on you. I write a small set of hostile probes aimed at the change's probable weak spots. Say the patch rewrote a date-string parser:

# probes/test_date_hostile.py — written by me, never shown to the model
import pytest
from src.parse_date import parse_date

def test_rejects_feb_29_on_non_leap_year():
    with pytest.raises(ValueError):
        parse_date("2025-02-29")

def test_no_silent_rollover_on_month_13():
    with pytest.raises(ValueError):
        parse_date("2025-13-01")

def test_whitespace_and_unicode_padding_not_accepted():
    with pytest.raises(ValueError):
        parse_date("\u20032025-01-01")

def test_epoch_boundary_does_not_wrap_negative():
    assert parse_date("1969-12-31").year == 1969

One discipline makes this stage honest: write the probes before you open the diff. Once you've seen the implementation, your supposedly independent tests drift toward the cases the model already covers. Attack first, read second.

Stage 4: Decision record

Every accepted patch leaves one plain-text bundle:

ask: "make parse_date strict about malformed input"
date: 2026-08-09
compute: free model access + free server via MonkeyCode
scope: 1 requested file, 0 surprise files
project suite: 312 passed
hostile probes: 4 passed (attempt 1 failed 2, regenerated, then passed)
verdict: merged; human pass over parse_date.py:8-41

This feels like paperwork until a September merge quietly regresses in November. Then the record turns a forensic afternoon into a ten-minute lookup.

Doing this on zero budget

The gauntlet consumes two things: repeated model calls (hostile probes regularly send a patch back — budget two or three regeneration rounds per change) and sacrificial compute to run them in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My current loop runs on MonkeyCode's free model access together with its free server option, which suits exactly this pattern: frequent, low-stakes regeneration where per-call pricing would otherwise push you to ration attempts. That said, nothing in the four stages depends on any specific vendor — any model endpoint plus any disposable machine (a spare laptop, a CI free tier) runs the identical gauntlet. If you're assembling this from zero, rehearsing the workflow on a free tier is a reasonable first step before committing budget anywhere.

Where the gauntlet breaks

It proves behavior, not intent. Hostile probes won't surface a deliberately concealed flaw. Human diff review remains required — this process feeds it evidence, it never replaces it.
A container is hygiene, not a vault. No-network, dropped-capability defaults are prudent, not a promise against a determined exploit. Never quarantine untrusted patches on a host with privileges you'd miss.
Weak probes, false confidence. Stage 3 carries the rigor. If you phone it in, the remaining stages merely document your complacency.
Skip the ceremony where it doesn't pay: one-token typo fixes don't justify four stages. Conversely, irreversible domains — payments, signing, authorization — need a domain expert reading every line regardless of what any suite says.
Free tiers shift. The model and server availability described here are operator-supplied claims at the time of writing; re-verify before baking them into a team process.

The actual takeaway

The meaningful change is one of phrasing: stop asking "does this generated patch look correct?" and start asking "what can I demonstrate about it?" An offline quarantine, a scoped file audit, probes authored before the reveal, and a saved verdict — that chain is a demonstration, and today it costs essentially nothing to run.