DEV Community: Jordan Huang

A Restart Is Not a Reset: Five Persistence Myths and a Boot-ID Probe

Jordan Huang — Fri, 06 Nov 2026 02:30:14 +0000

A Restart Is Not a Reset: Five Persistence Myths and a Boot-ID Probe

My worker wrote a state file at 23:40. By 06:00 the file was gone. No crash log. No alert. Just an empty directory and a new boot ID.

I had tested that job. It passed. But I tested it without ever restarting the machine. That is not a test. That is a demo.

What I actually wanted to know

Three questions. Does my data survive a restart? Does my process come back on its own? Does my health check tell the truth?

I run small jobs on a free server option, and I call free models from the same place. Both are operator-supplied conveniences, not guarantees. So I stopped assuming and started measuring.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe: one file, two runs, one restart

Save this as probe.py. Run it. Restart the box. Run it again. Then diff the two JSON files.

# probe.py - run before and after a restart, then diff the outputs
import json, os, pathlib, sqlite3, time

def boot_id():
    p = pathlib.Path('/proc/sys/kernel/random/boot_id')
    return p.read_text().strip() if p.exists() else 'unknown'

def write_probe(path):
    try:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps({'at': time.time()}))
        return {'writable': True, 'bytes': path.stat().st_size}
    except OSError as e:
        return {'writable': False, 'error': str(e)}

def sqlite_probe(path):
    try:
        con = sqlite3.connect(path)
        con.execute('CREATE TABLE IF NOT EXISTS t (v TEXT)')
        con.execute('INSERT INTO t VALUES (?)', (str(time.time()),))
        con.commit()
        rows = con.execute('SELECT count(*) FROM t').fetchone()[0]
        con.close()
        return {'rows': rows}
    except sqlite3.Error as e:
        return {'error': str(e)}

report = {
    'boot_id': boot_id(),
    'hostname': os.uname().nodename,
    'cwd': os.getcwd(),
    'deploy_id': os.environ.get('DEPLOY_ID', 'unset'),
    'tmp_file': write_probe(pathlib.Path('/tmp/probe.json')),
    'home_file': write_probe(pathlib.Path.home() / 'probe.json'),
    'sqlite_tmp': sqlite_probe(pathlib.Path('/tmp/probe.db')),
    'sqlite_home': sqlite_probe(pathlib.Path.home() / 'probe.db'),
}
print(json.dumps(report, indent=2))

Now the shell side:

python3 probe.py > run-1.json
# restart the server now, without redeploying
python3 probe.py > run-2.json
diff <(jq -S . run-1.json) <(jq -S . run-2.json)

A changed boot_id means the machine rebooted. Rows going from 3 to 1 means the disk did not survive. That single diff answers more than an hour of guessing.

FAQ: five claims I used to repeat

1. 'If it survived a restart, it is persistent.'

Surviving a restart proves the image was reused. It does not prove the storage is durable.

Many free setups keep the root image and drop instance storage. My /tmp write kept succeeding every time. It also kept vanishing. Write success is not durability.

2. 'It is the same box, so it is the same disk.'

Hostname is not identity. boot_id is. Check both, and log them early.

If your deploy script keys a lock or a cache on hostname, two reboots can hand you the same name and an empty disk. That is a bad combination.

3. 'The model remembers our conversation.'

No. Each model call is stateless. The transcript is your state, not the model's.

If your history file lives in /tmp, your memory dies with the process. Here is the shape I use. Pseudocode, simplified:

history = store.read_json('history.json', default=[])
history.append({'role': 'user', 'content': prompt})
reply = call_model(history)   # stateless call, no server-side memory
history.append({'role': 'assistant', 'content': reply})
store.write_atomic('history.json', history)

Two rules. Persist before you render. Write atomically, or a crash leaves half a turn.

4. 'It is only a cache, so losing it is fine.'

A cache is only safe to lose if it rebuilds without a human. Can yours?

Ask three questions. What rebuilds it? How long does that take? What breaks during the rebuild? If any answer is 'me, by hand', it is not a cache.

5. 'A 200 from /healthz means my state is fine.'

Liveness says the process answers. Readiness says it can serve. Those are different checks.

A process with an empty database still returns 200. Add one readiness probe that touches real state.

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/readyz

A durability decision table

Data	Where it goes	Rebuild cost	Acceptable loss
Session transcript	external store, atomic write	high	none
Model output cache	`/tmp` with a TTL	low, one call	all of it
SQLite job queue	home dir plus export	medium	last minute
Secrets	env at deploy time	none	n/a
Logs	stdout, shipped off-box	n/a	small window

Fill the table in before you pick a directory. The table is the design. The paths are just consequences.

The restart drill I run before shipping

Deploy. Let the job write something real.
Restart the server. Do not redeploy. Redeploying hides the bug.
Run the probe. Diff boot_id, row counts, file sizes.
Kill the process mid-write. Check the store is not corrupt.
Point /readyz at real state. Confirm it fails when state is missing.
Write down what you lost. An empty list means you learned nothing.

Steps 2 and 4 catch most of the surprises. They cost about ten minutes.

Limitations, and who should skip this

This probe tests your box, not your provider's contract. It cannot prove durability across regions or disasters.

It assumes a Linux host with /proc. It covers a single node only.

Skip it if you need strong durability guarantees, multi-node consistency, or audited storage. Use managed storage and document that choice. Skip it too if restarts are forbidden in your environment; use a canary node instead.

The probe is cheap. The wrong assumption is not.

If you want somewhere to run the two-run diff before paying for anything, the free server option is where I ran mine. Bring the table with you.

Your Free Tier Is Not a Capacity Plan: Five Throughput Myths and a Concurrency Ramp

Jordan Huang — Thu, 05 Nov 2026 23:26:48 +0000

A demo handles one request at a time. A launch sends ten at once. The endpoint falls over.

Was the model worse today? Or did nobody measure the shape of the curve?

I keep seeing the same five claims about free models and free servers. Each one sounds reasonable. Each one falls apart under a ten-minute test.

I run these probes against MonkeyCode's free model access and its free server option while prototyping.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Nothing below depends on that relationship. Swap in any endpoint you control. The method is the point.

Myth 1: "Latency grows linearly with concurrency"

If one call takes two seconds, ten calls take twenty. Right?

No. That is a queue, not a multiplier. Little's Law says concurrency equals arrival rate multiplied by time in system. When arrivals outrun service capacity, waiting time stops being bounded.

The curve has a knee. Left of it, latency is flat. Right of it, latency goes vertical and errors appear.

You cannot read the knee in a doc page. You find it by ramping.

Myth 2: "Retries add capacity"

Retries feel like resilience. Under saturation they are a load multiplier.

Say a third of calls fail and you retry twice. Offered load can nearly double. At the knee, that turns a brownout into an outage.

A corrected model: retry only classified transient failures. Use exponential backoff with jitter. Keep a retry budget, something like ten percent of successful calls.

Primary sources worth reading: AWS on backoff and jitter and the SRE book chapter on handling overload.

Myth 3: "It is free, so the numbers do not matter"

Free tiers make people skip measurement. "It is just a prototype."

But the shape transfers. The knee moves. The existence of a knee does not.

A prototype that dies at ten users teaches you nothing. A prototype with a written envelope teaches you where the queue belongs.

Myth 4: "A 429 or a timeout means the service is broken"

A 429 is admission control talking. It says your offered load exceeded what the server will accept.

A timeout says your deadline was shorter than time to first byte.

Neither one is a bug report. Both are signals to slow down, queue, or shed.

Make 429 a first-class state in the client. Do not swallow it as an exception.

Myth 5: "The model is the slow part"

Can you prove that? Most people cannot.

Split each request into a time budget:

Client prep: serialize the prompt, sign the request.
Connection: DNS, TCP, TLS.
Time to first byte: queue wait plus prefill.
Generation: first token to last token.
Client parse: decode, validate, store.

Without all five numbers, "the model is slow" is a guess with good manners.

The artifact: a concurrency ramp

Here is the probe. It is a template, not a run. I am not publishing numbers from it.

Stop rule first. Stop a level when either condition fires:

p95 exceeds three times the level-1 baseline.
Error rate passes five percent.

# ramp.py - template only. Point it at an endpoint you control.
import asyncio, time
import httpx

URL = 'https://your-endpoint.example/v1/chat'
LEVELS = [1, 2, 4, 8, 16, 32]
PER_LEVEL = 24
P95_STOP = 3.0
ERR_STOP = 0.05

async def call(client, sem, rows):
    async with sem:
        t0, ttfb = time.perf_counter(), None
        try:
            async with client.stream('POST', URL, json={'input': 'ping'}) as r:
                async for _ in r.aiter_bytes():
                    if ttfb is None:
                        ttfb = time.perf_counter()
                rows.append({'ok': r.status_code < 400, 'code': r.status_code,
                             't0': t0, 'ttfb': ttfb, 'end': time.perf_counter()})
        except Exception as e:
            rows.append({'ok': False, 'code': type(e).__name__,
                         't0': t0, 'ttfb': None, 'end': time.perf_counter()})

def summarize(rows, wall):
    lat = sorted((r['end'] - r['t0']) * 1000 for r in rows)
    q = lambda p: lat[min(len(lat) - 1, int(p * len(lat)))]
    errs = sum(1 for r in rows if not r['ok'])
    return {'rps': round(len(rows) / wall, 2),
            'p50_ms': round(q(0.50)), 'p95_ms': round(q(0.95)),
            'err': round(errs / len(rows), 3)}

async def main():
    async with httpx.AsyncClient(timeout=60) as client:
        base = None
        for level in LEVELS:
            sem, rows = asyncio.Semaphore(level), []
            start = time.perf_counter()
            await asyncio.gather(*[call(client, sem, rows) for _ in range(PER_LEVEL)])
            s = summarize(rows, time.perf_counter() - start)
            base = base or s['p95_ms']
            print(level, s)
            if s['p95_ms'] > P95_STOP * base or s['err'] > ERR_STOP:
                print('stop: knee near level', level)
                break

asyncio.run(main())

Reading the output

Three fields matter more than the rest:

rps tells you whether more concurrency bought throughput.
p95_ms tells you where waiting started.
err tells you where the server pushed back.

If rps flattens while p95 climbs, you are past the knee. More concurrency is now just more waiting.

The decision table

Zone	What you see	What to do
Left of the knee	p95 flat, errors near zero	Raise the cap slowly, keep the ramp in CI
At the knee	p95 jumps 2-3x	Set your cap at half the knee
Right of the knee	p95 vertical, 429s or timeouts	Queue, shed, or return a fast 503
No ramp at all	no baseline	You have no capacity envelope, only hope

Half the knee, not the knee. Headroom is for the day your prompt gets longer.

The gate you ship

Now encode what you measured. Do not leave it in a notebook.

# gate.py - cap below the knee you measured.
import asyncio, random

class Retryable(Exception):
    pass

class Envelope:
    def __init__(self, knee, headroom=0.5):
        self.cap = max(1, int(knee * headroom))
        self.sem = asyncio.Semaphore(self.cap)

    async def run(self, fn, *args, **kwargs):
        async with self.sem:
            for attempt in range(3):
                try:
                    return await fn(*args, **kwargs)
                except Retryable:
                    delay = (2 ** attempt) * 0.2 + random.random() * 0.2
                    await asyncio.sleep(delay)
            raise

Retryable is yours to define. Map 429, 503, and connect timeouts to it. Map everything else to a hard failure.

What this does not tell you

The generator shares a network with the target. Measure from a second host when you can.
TTFB mixes queue wait with prefill. The client cannot split them apart.
A ramp is a snapshot. Provider-side changes move the knee without warning.
One prompt shape is not all prompt shapes. Long prompts bend the curve down.
Free tiers can change. That is a reason to re-run the ramp, not a reason to skip it.

Who should not use this

One-shot scripts and nightly batch jobs. Nobody is waiting on them.
Teams with real load tests and SLO dashboards. Reuse those numbers instead.
Anyone who cannot label synthetic traffic as synthetic. Ask first.

So why ship a cap you never measured? You cannot defend a number you never took.

Run one ramp. Write the knee in your README. If you are prototyping on a budget, run the same script against MonkeyCode's free model access and free server option before you commit to a paid plan. The script does not care whose endpoint answers.

Streaming Feels Faster, but Your Model Isn't: A Wire-Level Timestamp FAQ

Jordan Huang — Thu, 05 Nov 2026 20:20:32 +0000

Does streaming make a model answer faster?

Every time I hear that, I ask a boring follow-up. Faster for whom, and measured where?

I keep seeing the same four claims repeated in review threads and README files. Each one is testable in about a minute. None of them survives contact with a clock.

Here is the FAQ, the probe I use, and the corrected mental model.

The claim that keeps coming back

A developer adds stream: true. The UI paints tokens sooner. Everyone agrees the endpoint got faster.

But the model did not change. The queue did not change. Only the delivery schedule changed.

So what is actually being measured? And what is being hidden?

Two runs, one clock

You need two numbers per run: time to first byte, and total wall clock.

You also need event timestamps, not network read timestamps. Those differ more than you think.

Here is the probe. Node 20+, no dependencies, one file.

// probe.mjs — two runs, one monotonic clock.
import { performance } from 'node:perf_hooks';

const ENDPOINT = process.env.ENDPOINT;   // full chat-completions URL
const KEY = process.env.API_KEY;
const PROMPT = 'List five two-word color names. No other text.';

async function run(stream) {
  const t0 = performance.now();
  const res = await fetch(ENDPOINT, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${KEY}`,
      accept: stream ? 'text/event-stream' : 'application/json',
    },
    body: JSON.stringify({ stream, messages: [{ role: 'user', content: PROMPT }] }),
  });

  if (!stream) {
    const text = await res.text();
    const t1 = performance.now();
    return { stream, status: res.status, ttfb: t1 - t0, total: t1 - t0, events: 1, bytes: text.length };
  }

  const reader = res.body.getReader();
  const dec = new TextDecoder();
  const marks = [];
  let buf = '';
  let bytes = 0;
  let firstByteAt = null;

  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    const now = performance.now();
    firstByteAt ??= now;
    bytes += value.byteLength;
    buf += dec.decode(value, { stream: true });

    let i;
    while ((i = buf.indexOf('\n\n')) !== -1) {
      const evt = buf.slice(0, i);
      buf = buf.slice(i + 2);
      if (!evt.startsWith(':')) marks.push(now - t0); // skip keep-alive comments
    }
  }

  const t1 = performance.now();
  return { stream, status: res.status, ttfb: (firstByteAt ?? t1) - t0, total: t1 - t0, events: marks.length, bytes, marks };
}

const batch = await run(false);
const live = await run(true);
console.table([batch, live].map(({ marks, ...rest }) => rest));

// Burst check: share of SSE events landing in the last 10% of the clock.
if (live.marks?.length) {
  const span = live.total - live.ttfb;
  const tail = live.marks.filter((m) => m > live.ttfb + span * 0.9).length;
  console.log(`tail share: ${((tail / live.marks.length) * 100).toFixed(0)}%`);
}

Run it twice against the same prompt and the same endpoint. Keep the terminal output.

I have not run this against your endpoint. Treat it as the instrument, not the result. Run it, and the numbers below will be yours, not mine.

FAQ: four claims, four corrections

1. "Streaming makes generation faster."

It does not touch generation at all. It changes when bytes leave the server.

A streamed run often ends slightly later in wall clock, because every event carries framing and header overhead.

Corrected model: streaming trades total efficiency for earlier partial value. That is a UX decision, not a speed decision.

2. "Time to first byte is a model latency number."

TTFB is a pipeline number. It bundles DNS, TLS, queueing, admission control, prompt processing, tokenizer warm-up, and proxy buffering.

Compare two providers and you are comparing pipelines, not models.

Corrected model: label the metric honestly. Call it client-observed first-byte time, and keep the endpoint and region in the same note.

3. "If chunks are arriving, nothing is buffering."

Buffering can be bursty. A proxy flushes every N bytes, or on a timer.

Look at the tail share. A high value means most events landed in a narrow window near the end.

That is a buffering signature, not a slow model.

If you terminate on nginx, one response header usually settles it:

# Disable proxy buffering for the SSE route.
proxy_buffering off;
add_header X-Accel-Buffering no;

Also check content-encoding. Compression on an event stream tends to delay flushes.

4. "Streaming is the safer default on a free tier."

Not always. A stream holds one connection open for the entire generation.

Long-held connections collide with idle timeouts and with whatever concurrency cap you were granted. Batch jobs usually prefer one request and one generous client timeout.

Corrected model: streaming is for humans watching output. Non-streaming is for queues, retries, and anything that runs at 3 a.m.

Cross-check in ten seconds

Before you trust the script, confirm with curl. Two tools disagreeing is information.

curl -sS -N -o /dev/null \
  -w 'starttransfer=%{time_starttransfer} total=%{time_total}\n' \
  -H "authorization: Bearer $API_KEY" \
  -H 'content-type: application/json' \
  -d '{"stream":true,"messages":[{"role":"user","content":"List five two-word color names."}]}' \
  "$ENDPOINT"

-N turns off curl's own output buffering, so you can watch events land. The -w timings are recorded either way.

If curl shows a fast first byte and your app shows a slow one, stop blaming the endpoint. Your client is buffering.

Decision table

Use the shape of the workload, not the shape of the demo.

Workload	Recommend	Why
Chat UI a human watches	Stream	Partial value arrives early
Background summarization	Non-stream	One request, one timeout, simpler retries
Mobile on flaky networks	Non-stream	Fewer half-open connections to reconcile
Anything behind an unverified proxy	Measure first	Buffering can erase the benefit
Strict total-latency budget	Non-stream	Streaming adds framing overhead

Where this bites on a free tier

Free capacity makes the pipeline longer and less predictable. That is exactly when a bad metric hurts most.

I run these probes against the free model access and free server option that MonkeyCode provides, mostly because the pipeline in front of a free endpoint is the interesting part. The product is the environment here, not the lesson.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free environment changes is variance. Shared capacity means your TTFB depends on neighbours, not just your prompt. So record the time of day with every run.

What this test does not tell you

It does not measure model quality. You need an eval set for that.
It does not separate network RTT from server queueing. You need a second vantage point for that.
One run is noise. Five runs minimum, ideally spread across an hour.
It says nothing about cost. Free-to-paid transitions are a separate question.
It cannot prove your proxy config. Only the tail share hints at it.

Who should skip this

If your users never see partial output, skip streaming entirely and skip this probe.

If you already log server-side timings per request, you likely have better data than any client probe.

And if you are choosing between providers on vibes, a timestamp test will not fix the decision process.

The model that survives the clock

Streaming is a delivery choice. It says nothing about model speed.

First-byte time is a pipeline fact, not a model fact. Name it that way.

Bursts near the end mean buffering, not a slow generator. Check the proxy before the prompt.

So: run the probe once. What is your tail share, and does it match what you assumed?

If the two numbers surprise you, that is the article doing its job.

Valid JSON, Wrong Order: Five Structured-Output Myths and a Three-Gate Replay Test

Jordan Huang — Thu, 05 Nov 2026 16:16:51 +0000

A pipeline of mine broke last week. Not on a timeout. Not on a 500.

It broke because the model returned 'quantity': '3'. A string. Twelve downstream orders went out with a quantity of zero.

So let me ask the boring question. Would your tests have caught that? Mine did not, and I wrote them.

Here are five claims I keep hearing about structured output. Each one comes with evidence you can collect yourself, and a corrected mental model.

Myth 1: JSON mode means my schema is enforced

No. It means the bytes parse as JSON. That is a syntax promise, not a contract.

I keep seeing this confusion in reviews. Someone enables a JSON flag and deletes the validator. Type drift then walks straight into the database.

Evidence you can collect in ten minutes: save 50 raw responses, then run them through json.loads. Count the parses. Then run the parses through a real schema validator. The two numbers will differ.

Corrected model: JSON mode protects the parser. Your schema protects the domain.

Myth 2: If it parses, the model was right

Parsing is the cheapest gate. It is also the least informative.

The output can parse, match the schema, and still be business-wrong. Totals that do not add up. A status the state machine never allows. An order id from another tenant.

Corrected model: validation has three layers, not one.

Gate A: does it parse?
Gate B: does it match the schema?
Gate C: does it survive your cross-field rules?

Most teams ship gate A and call it done. That is where my string '3' came from.

Myth 3: Retrying fixes bad output

Retrying with the same prompt is a coin flip. Sometimes it lands. You learn nothing either way.

Ask yourself what the retry actually changed. If the answer is nothing, you bought variance, not reliability.

Before you add a retry loop, measure your gate failure rate. If gate B fails 4% of the time, a retry hides it. It does not remove it. And you still cannot explain the 4%.

Corrected model: retries are a mitigation. The gate is the fix.

Myth 4: Strict mode enforces my constraints

Strict modes vary by provider. Some enforce types well. Fewer enforce regex patterns, ranges, or cross-field arithmetic.

Can your validator tell the difference between absent and null? Does it reject booleans where you expect integers? In Python, isinstance(True, int) is true. That bites people.

Corrected model: treat generated constraints as hints. Treat your validator as law.

Myth 5: Schema drift is a model problem

It is a contract problem. You version your API. Version your output schema too.

When a prompt changes and the field list changes with it, you have a silent breaking change. Nothing alerts. Nothing fails loudly.

Corrected model: the schema lives in version control, next to the prompt. It has a version number. The gate logs that version with every rejection.

The three-gate check

Here is the gate I use. It is a template, not a benchmark. I have run it on my own traffic; your failure mix will differ.

# gate.py
import json
from collections import Counter
from jsonschema import Draft202012Validator

SCHEMA = {
    '$schema': 'https://json-schema.org/draft/2020-12/schema',
    'type': 'object',
    'required': ['order_id', 'quantity', 'unit_cents', 'total_cents', 'status'],
    'additionalProperties': False,
    'properties': {
        'order_id':    {'type': 'string', 'pattern': '^ord_[0-9a-f]{8}$'},
        'quantity':    {'type': 'integer', 'minimum': 1, 'maximum': 999},
        'unit_cents':  {'type': 'integer', 'minimum': 0},
        'total_cents': {'type': 'integer', 'minimum': 0},
        'status':      {'type': 'string', 'enum': ['new', 'paid', 'shipped']},
    },
}

VALIDATOR = Draft202012Validator(SCHEMA)
COUNT = Counter()

def gate_a(raw):
    try:
        return json.loads(raw), None
    except json.JSONDecodeError as e:
        COUNT['gateA'] += 1
        return None, 'gateA:not-json@%d' % e.pos

def gate_b(obj):
    errors = sorted(VALIDATOR.iter_errors(obj), key=lambda e: list(e.path))
    if errors:
        path = '/'.join(str(p) for p in errors[0].path) or '$'
        COUNT['gateB'] += 1
        return None, 'gateB:schema:' + path
    return obj, None

def gate_c(obj):
    if obj['total_cents'] != obj['quantity'] * obj['unit_cents']:
        COUNT['gateC:arithmetic'] += 1
        return None, 'gateC:total-mismatch'
    return obj, None

Three functions. Three failure counters. That is the whole idea.

One nuance: coerce before you reject. '3' can become 3. But log the coercion. A rising coercion rate is a prompt regression, and you want to see it.

The replay harness

You do not need a live model to test this. Save raw responses, then replay them.

# replay.py  — expects JSONL where each line is {'content': '<raw model output>'}
import json, sys
from gate import gate_a, gate_b, gate_c, COUNT

kept = 0
with open(sys.argv[1], encoding='utf-8') as fh:
    for line in fh:
        raw = json.loads(line)['content']
        obj, err = gate_a(raw)
        if err:
            continue
        obj, err = gate_b(obj)
        if err:
            continue
        obj, err = gate_c(obj)
        if err:
            continue
        kept += 1

print('kept', kept, dict(COUNT))

Run it like this:

python -m venv .venv && . .venv/bin/activate
pip install jsonschema
python replay.py saved_responses.jsonl

Now the output is a number you can argue about. Not a vibe.

Decision table

Failed gate	Likely cause	Action
A (parse)	truncation, markdown fence	raise output budget, strip fences
B (schema)	type drift, extra field	tighten prompt, pin schema version
B (schema)	missing field	make it required, give a default in code
C (semantics)	model did arithmetic	compute totals in code
C (semantics)	guessed enum value	ask for inputs, derive the state

That last row matters. Stop asking the model to do arithmetic. Ask for quantity and unit_cents. Multiply in code.

Running the gate next to the model

Here is the practical part. The gate should sit in the same process as the model call. Network hops cost latency. Latency costs retries.

That is why I keep the gate script on the same box as the model call. If you are trying this with free model access, MonkeyCode describes a free server option that lets you co-locate them. The offer described to me is free model access with a token allowance reported as 10 million tokens, plus that free server option. Quotas change, so check the live page before you plan around a number.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier is fine for the replay loop. It is a poor place for a p99 latency claim. Do not confuse the two.

Limitations you should hear

The gate catches contract violations, not hallucinations. A field can be valid and still wrong.
Gate C is domain code. It will never be generic, and it should not be.
My replay numbers are not published here because they are my traffic. Replay your own.
Schema libraries differ on format support. Test the validator, not just the model.

Who should not use this

Skip the three-gate design if your SDK already enforces a typed contract end to end. Skip it if your output is prose for humans, with no downstream consumer. Skip it if you cannot store raw responses, because then replay is impossible.

Also skip it if you plan to hide gate failures behind retries. That defeats the point.

A five-minute checklist

Save 50 raw model responses today.
Write the schema. Add additionalProperties: False.
Run the replay. Write down three numbers.
Add one gate C rule. Just one.
Re-run. Compare.

So what does your schema actually guarantee? If the answer is only that the bytes parse, you have a gate A pipeline. That is a start, not a contract.

Your Cache Key Is Not Your Prompt: Four Response-Cache Myths and a Collision Test

Jordan Huang — Thu, 05 Nov 2026 13:51:44 +0000

You put a cache in front of your model calls. The first week looks great. Then a ticket lands: same question, different answer. Sometimes.

Your prompt did not change. Your key did not change either. That is the bug.

Caching a model response is a claim about equivalence. Most teams never write that claim down. They hash a string and hope.

Here are four myths I keep hearing. Each one comes with a corrected mental model and a test you can run today.

Myth 1: "The prompt text is the cache key"

This is the most expensive myth. It feels obvious, so nobody reviews it.

A prompt string is only the input. The answer also depends on:

the model identifier your endpoint actually resolves to
decoding parameters such as temperature, top_p, max_tokens, stop sequences
the system prompt template, which changes far more often than the version number
the tool or function schema, even when no tool gets called
the tenant, user, or document scope the answer was allowed to see

Hash the prompt alone and you get collisions between calls that must not share an answer. You will not see them in staging. You will see them in a support thread.

The corrected model: a key is a written equivalence claim

Two requests may share a cache entry only if a reviewer would say "these must produce the same answer." Write that sentence down before you write the hash.

A key builder you can actually test

The two small modules below are the whole harness. Copy them, then adapt the field names.

# cache_key.py
import hashlib, json

KEY_VERSION = "v3"

def template_sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]

def _canonical(obj) -> str:
    # sort_keys makes dict ordering irrelevant at every depth
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def cache_key(req: dict) -> str:
    material = {
        "key_version": KEY_VERSION,
        "tenant": req["tenant"],
        "model": req["model"],
        "template_sha": req["template_sha"],
        "params": req["params"],
        "tools": req.get("tools", []),
        "input": req["input"],
    }
    return hashlib.sha256(_canonical(material).encode("utf-8")).hexdigest()

Note the key_version field. When you change what counts as equivalent, bump it. Old entries then become unreachable instead of wrong.

hashlib and deterministic JSON serialization are the only primitives you need. If you prefer a standard for freshness and invalidation vocabulary, HTTP caching already has one; application-level response caches usually do not.

Four tests that catch real collisions

# test_cache_key.py
from cache_key import cache_key, template_sha

def base_request():
    return {
        "tenant": "acme",
        "model": "model-id-from-your-endpoint",
        "template_sha": template_sha("You are a terse assistant."),
        "params": {"temperature": 0, "max_tokens": 300},
        "tools": [],
        "input": "How do I rotate a log file?",
    }

def test_params_are_part_of_the_key():
    a = cache_key(base_request())
    r = base_request(); r["params"]["temperature"] = 0.9
    assert a != cache_key(r)

def test_template_edit_changes_key_without_a_version_bump():
    a = cache_key(base_request())
    r = base_request(); r["template_sha"] = template_sha("You are a verbose assistant.")
    assert a != cache_key(r)

def test_tenant_isolation():
    a = cache_key(base_request())
    r = base_request(); r["tenant"] = "other-customer"
    assert a != cache_key(r)

def test_key_is_stable_across_dict_order():
    a = cache_key(base_request())
    r = base_request(); r["params"] = {"max_tokens": 300, "temperature": 0}
    assert a == cache_key(r)

That last test matters more than it looks. Nondeterministic key order gives you a cache that silently never hits.

Myth 2: "A hit means the answer is still correct"

A TTL is a guess about staleness. It is not a proof of correctness.

Ask three questions before you trust a hit:

Did the input data change, or only the prompt text?
Did the model behind that identifier change?
Did a policy, price, or schema that the answer quotes change?

If the answer depends on live data, a time-based expiry is a coin flip. Prefer content-addressed invalidation: put the data version into the key, not just into the clock.

Myth 3: "Caching always cuts cost"

Caching moves cost around. It rarely deletes it.

Call shape	Cache it?	Why
Fixed template, temperature 0, stable inputs	Yes	Key is easy to defend
High temperature, creative output	No	No two answers are equivalent
Tool-using agent loop	Key per step, not per turn	Step inputs change constantly
Long-tail user prompts	Usually no	Hit rate near zero, review cost stays
Answers that quote live prices or quotas	Only with data version in key	Stale answer is a wrong answer

Count the misses too. A cache you must debug is not free.

Myth 4: "The cache is on my server, so it is mine"

An in-process dictionary on an ephemeral instance is not a cache. It is a coincidence.

A restart empties it. Your hit rate becomes a function of uptime.
Two instances hold two different views of the world.
Stored payloads can outlive the request that created them unless you set a bound.

If the key omits tenant, the fastest path to a data leak is a cache hit.

A 20-minute verification workflow

Do this on a scratch deployment before touching production.

Write the equivalence sentence for one call path. One sentence, no hedging.
Implement cache_key with the fields that sentence names.
Run the four tests above. Add one test per new field you argued about.
Send the same request twice. Log hit / miss for both.
Change only the system template text. Confirm the second request misses.
Restart the instance. Measure how far hit rate falls.
Read one stored payload. If reading it makes you uncomfortable, encrypt or drop it.

For step 4 you need a place to send real calls cheaply. One option for that scratch environment is MonkeyCode: it offers free model access and a free server option, both as operator-supplied availability claims, not guarantees. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Measure elapsed time with a monotonic clock, never with wall time. Log these fields on every call:

cache_key, truncated to 12 chars
key_version and template_sha
model as resolved, not as requested
hit or miss
elapsed_ms from time.monotonic()
tenant

Six fields. That log answers almost every "why did it say that" question.

Limitations and who should skip this

This approach is not universal. Skip it, or get review first, if:

you store raw prompts that contain personal or regulated data
your answers must reflect live state with no version marker available
you cannot name the model version your endpoint resolves to
you run unsampled creative output and someone will still call a hit "wrong"
your team has no owner for cache invalidation; an unowned cache rots quietly

And do not cache to win a latency argument you have not measured. Measure first. Then cache one path.

What to take away

A cache is a promise that two requests deserve one answer. Write the promise. Test the promise. Version the promise.

If your key is a hash of the prompt text, you have not made a promise. You have made a guess with a very fast lookup.

Run the four tests. If one fails, you just found a bug before a customer did.

Four Idle Myths About Free Servers, and the Wake Probe That Settles Them

Jordan Huang — Thu, 05 Nov 2026 10:44:52 +0000

Someone pings your free server every five minutes. They swear it stays fast. Do you believe them?

I stopped believing. Not because the claim is obviously wrong. Because nobody in the thread had measured it.

So I built a small probe instead. It answers the question with timestamps, not folklore. Here are four idle myths I keep hearing, and the check that ends each one.

The claims I keep reading

"Just hit /health every few minutes. The box never sleeps."
"If /health returns 200, the app is ready."
"Idle costs nothing, so keep-alive is free."
"My scheduled job runs at 03:00. It's in the config, right?"

Each one sounds reasonable. Each one is a guess about a system you do not control.

Sleep policy is platform-specific. It can change without a changelog you read. Read your provider's docs first, then measure your own deployment.

Myth 1: A ping every five minutes keeps the instance warm

What is "warm," exactly? Warm for HTTP is not warm for your database pool. Warm for the pool is not warm for the model call path.

A keep-alive ping proves one narrow thing. Something accepted a TCP connection recently. It does not prove your cold path is gone.

And the ping is not free of side effects. It can consume request quota. It can keep logs noisy. It can even hold the instance in a state your provider bills differently.

Corrected model: a ping is a heartbeat, not a thermometer. Measure the latency you actually serve, not the fact that a socket answered.

Myth 2: If `/health` returns 200, the app is ready

I have shipped this mistake more than once. /health returned 200 while the first real request took far longer.

Why? The health route was cheap. It touched no dependency. It proved the process was alive, and nothing more.

Split liveness from readiness. Liveness says "the process is up." Readiness says "the next request can be served."

// Illustrative example. Not executed here. Adapt paths to your app.
app.get('/live', (req, res) => res.status(200).send('ok'));

app.get('/ready', async (req, res) => {
  const checks = {};
  checks.db = await pingDb();        // your own helper
  checks.downstream = await pingModel(); // your own helper
  const ok = Object.values(checks).every(Boolean);
  res.status(ok ? 200 : 503).json({ ok, checks });
});

Point your keep-alive at /ready, not /live. A 503 from /ready is information. A 200 from /live is a shrug.

Myth 3: Idle time is free

Free tiers rarely bill idle in currency. They bill it in other ways.

Request quotas burned by pings you never needed.
Log lines that bury the one error you care about.
A warm instance that hides a cold-start bug until real users arrive.
Cron runs that overlap because the previous one never finished.

Ask a blunt question. Is the keep-alive protecting users, or protecting you from seeing a number?

Corrected model: idle has a cost. It is just paid in quota, observability, and surprise.

Myth 4: Scheduled jobs run on schedule on a sleeping instance

Your config says 03:00. Your instance may not be listening at 03:00.

In-process schedulers need a running process. If the process is asleep, the timer is not pending anywhere. It is just gone.

Check three things before you trust a schedule:

Where does the timer live? In the process, or in an external scheduler?
What happens to a missed run? Retry, skip, or silent drop?
Can the job start twice? Make it idempotent either way.

The wake probe

Here is the check I run. It is boring on purpose.

It sends a request, waits a controlled idle window, sends another, and records both. Run it from a machine outside your host. localhost timing proves nothing about the network.

#!/usr/bin/env bash
# Illustrative. Not executed here. Point URL at a read-only endpoint.
URL="https://your-app.example/ready"
OUT="wake-probe.jsonl"

hit() {
  local mode="$1" idle="$2"
  local ttfb status
  read -r ttfb status < <(curl -s -o /dev/null \
    -w '%{time_starttransfer} %{http_code}\n' \
    --max-time 60 "$URL")
  printf '{"ts":"%s","mode":"%s","idle_s":%s,"ttfb_s":%s,"status":%s}\n' \
    "$(date -u +%FT%TZ)" "$mode" "$idle" "$ttfb" "$status" >> "$OUT"
}

# Warm baseline: five back-to-back requests
for i in 1 2 3 4 5; do hit warm 0; sleep 2; done

# Wake cycles: increase the idle window, one request each
for idle in 300 900 1800; do
  echo "sleeping ${idle}s"
  sleep "$idle"
  hit wake "$idle"
done

Then summarize. Do not eyeball the raw lines.

// Illustrative. Not executed here. Node 18+, no dependencies.
import { readFileSync } from 'node:fs';

const rows = readFileSync('wake-probe.jsonl', 'utf8')
  .trim().split('\n').map((l) => JSON.parse(l));

const median = (xs) => xs.sort((a, b) => a - b)[Math.floor(xs.length / 2)];

const warm = rows.filter((r) => r.mode === 'warm').map((r) => r.ttfb_s);
const base = median(warm);

console.log('warm median ttfb:', base.toFixed(3), 's');
for (const r of rows.filter((r) => r.mode === 'wake')) {
  const ratio = r.ttfb_s / base;
  console.log(`idle ${r.idle_s}s -> ${r.ttfb_s.toFixed(3)}s (${ratio.toFixed(1)}x) status ${r.status}`);
}

Run one cycle first. Do not hammer a free endpoint. If you see 429, stop and wait. That is the platform telling you something useful.

Reading the result

The numbers are yours, not mine. I will not hand you a benchmark. My wake penalty says nothing about your app.

What you are looking for is shape, not precision:

Does the penalty grow with the idle window, or stay flat?
Does it appear after one minute, or after thirty?
Does /ready return 503 during the wake, or 200 with a slow body?
Does the penalty disappear on the second request in a row?

That last one matters most. A penalty that vanishes on request two is a cold-start story. A penalty that persists is something else.

What to do with the answer

Wake penalty	Traffic shape	Reasonable move
Small, flat	Bursty	Accept it. Do not add a keep-alive.
Small, flat	Steady	Measure again under real load before deciding
Large, first request only	Steady	Warm the specific cold path, not the socket
Large, every request	Any	Stop; this is not a cold-start problem
429 or 5xx during wake	Any	Back off, then read the docs

Notice what is missing. No row says "ping every five minutes." Keep-alive is one option, not the default.

Corrected mental model: you do not warm an instance. You warm the specific path a user is about to walk.

Where MonkeyCode actually fits here

My probe checks a downstream dependency, usually a model call. That call has its own cold path and its own failure modes.

MonkeyCode's free model access and free server option are useful for exactly this kind of throwaway measurement rig. I get a place to run the probe, and a downstream endpoint to put in the /ready check. Operator-supplied availability, not a benchmark promise. Your quotas and limits are whatever your account shows.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One honest caveat. If you need guaranteed warm capacity, a free tier is the wrong tool. Buy the guarantee. Do not build a ping farm around a policy you cannot see.

Limitations, and who should skip this

This measures one vantage point at one time, not your users' median.
Provider sleep policy can change. Re-run the probe after any platform notice.
A single probe conflates DNS, TLS, proxy, and app startup time.
Read-only endpoints only. Do not wake a box that writes on GET.

Skip this approach if you are already on paid always-on capacity. Skip it if your endpoint is not safe to call repeatedly. Skip it if you cannot stop when you see 429.

The short version

A ping is not a thermometer. A 200 from /health is not readiness. Idle is not free. A sleeping box does not run your cron.

Four myths, one probe, and about an hour of waiting. That is the whole trick.

What is your wake penalty? You cannot know until you time it. Run one cycle, read four lines, and let the graph argue for you next time.

Your Timeout Is Local: Five Myths About Model Calls You Retry

Jordan Huang — Wed, 28 Oct 2026 01:14:22 +0000

Your Timeout Is Local: Five Myths About Model Calls You Retry

My batch job died at 03:40. The box was up. The model was reachable. My retry logic was wrong.

I keep seeing the same five claims in code review. Each one sounds sensible. Each one quietly produces duplicate work, empty output, or a worker stuck forever.

So I stopped arguing and built a probe. One script, one verdict table, one ledger. No benchmarks here. Free-tier conditions change too fast for my numbers to help you.

Myth 1: "A timeout means the call did not happen"

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

A local clock expiring tells you one thing only. You stopped listening. It says nothing about remote execution.

Corrected model: DEADLINE means unknown, not failed.

outcome = with_deadline(lambda: call_model(model, prompt), seconds=20)
if outcome.verdict is Verdict.DEADLINE:
    # remote state unknown: do not assume "no work done"
    log.warning("stopped listening fp=%s", fp)

This matters when the call has side effects. A tool call that writes a row, or sends a message, may have landed. Retry blind and you write twice.

Myth 2: "HTTP 200 means I got an answer"

A 200 says the transport worked. It does not say the text is usable.

I have seen three failure shapes arrive with a healthy status code:

Empty — whitespace only, or a blank field where text should be.
Truncated — ends mid-phrase because a cap was hit.
Off-target — fluent text that ignores the instruction.

Only the first two are cheap to detect. The third needs a check you wrote yourself, like a required field or a schema parse.

Corrected model: success has two gates. Transport success, then content success. Your retry policy should see both.

Myth 3: "Retries are free"

Every retry is another call. Another unit of time. Another chance to hit a limit.

Ask yourself a question. What is your worst case, attempts times deadline? If the answer is forty minutes, you do not have a retry policy. You have an outage.

Cap two numbers, not one:

Attempts — a small integer, not a loop.
Total wall clock — the budget the whole job may spend.

And treat THROTTLED differently from TRANSPORT. Backpressure wants a longer pause. A dropped connection wants a short jitter.

Myth 4: "Same prompt, same answer, so I can cache by prompt"

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

That is fine for chat. It is a bug for a cache key.

Corrected model: the key is a fingerprint of everything that shapes the output.

def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

Model id, prompt, and parameters all belong in the hash. Swap the model and your cache must miss. Otherwise you serve yesterday's answer under a new label.

Myth 5: "Streaming means progress, so I can drop the timeout"

Streaming gives you bytes sooner. It gives you no guarantee of finishing.

A stalled stream is worse than a slow response. It holds a worker slot and looks alive. Your total timeout never fires because data keeps trickling.

Use two clocks:

Idle timeout — no bytes for N seconds, give up.
Total timeout — hard ceiling for the whole call.

If you only have one, you have the wrong one.

Build the probe

Here is the harness I actually run. It is provider-agnostic on purpose. Inject your client and keep your own numbers.

"""probe_call.py - classify model-call outcomes before you retry them."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import threading
import time
from dataclasses import dataclass
from enum import Enum
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER", "calls.jsonl"))


class Verdict(str, Enum):
    OK = "ok"
    TRANSPORT = "transport"
    DEADLINE = "deadline"
    THROTTLED = "throttled"
    EMPTY = "empty"
    TRUNCATED = "truncated"


@dataclass
class Outcome:
    verdict: Verdict
    text: str = ""
    detail: str = ""
    seconds: float = 0.0


def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]


def with_deadline(fn, seconds: float) -> Outcome:
    """Run fn in a thread. Return DEADLINE on overrun. Never block forever."""
    box: dict = {}

    def runner() -> None:
        try:
            box["text"] = fn()
        except Exception as exc:  # classify, do not swallow
            box["error"] = exc

    t = threading.Thread(target=runner, daemon=True)
    started = time.monotonic()
    t.start()
    t.join(seconds)
    elapsed = time.monotonic() - started

    if t.is_alive():
        return Outcome(Verdict.DEADLINE, detail="local clock expired", seconds=elapsed)
    if "error" in box:
        return Outcome(Verdict.TRANSPORT, detail=repr(box["error"]), seconds=elapsed)

    text = (box.get("text") or "").strip()
    if not text:
        return Outcome(Verdict.EMPTY, detail="status ok, no usable text", seconds=elapsed)
    if text.endswith((",", " and", " the")):
        return Outcome(Verdict.TRUNCATED, text=text, detail="ends mid-phrase", seconds=elapsed)
    return Outcome(Verdict.OK, text=text, seconds=elapsed)


def record(fp: str, out: Outcome) -> None:
    row = {"fp": fp, "verdict": out.verdict.value, "seconds": round(out.seconds, 3)}
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")


def seen(fp: str) -> str | None:
    if not LEDGER.exists():
        return None
    for line in LEDGER.read_text(encoding="utf-8").splitlines():
        row = json.loads(line)
        if row["fp"] == fp:
            return row["verdict"]
    return None


RETRYABLE = {Verdict.TRANSPORT, Verdict.THROTTLED, Verdict.DEADLINE}


def call_model(model: str, prompt: str) -> str:
    """Wire your client here. Return raw text. Raise on transport errors."""
    raise NotImplementedError("plug in your client, then rerun")


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--model", default=os.environ.get("MODEL_ID", "unset"))
    ap.add_argument("--attempts", type=int, default=1)
    ap.add_argument("--deadline", type=float, default=20.0)
    args = ap.parse_args()

    fp = fingerprint(args.model, args.prompt, {})
    if seen(fp) == Verdict.OK.value:
        print(f"fp={fp} already OK; skipping on ledger hit")
        return

    for attempt in range(1, args.attempts + 1):
        out = with_deadline(lambda: call_model(args.model, args.prompt), args.deadline)
        record(fp, out)
        print(f"attempt={attempt} verdict={out.verdict.value} t={out.seconds:.2f}s")
        if out.verdict is Verdict.OK:
            print(out.text)
            return
        if out.verdict not in RETRYABLE:
            break
        time.sleep(min(2 ** attempt, 10) * 0.5)
    raise SystemExit(f"fp={fp} gave up; read {LEDGER}")


if __name__ == "__main__":
    main()

Two things make this useful beyond a toy. The ledger is append-only, so a restarted job can skip finished work. And the verdict is written before any retry decision.

Run the kill tests first

Do not test your retry path on a good day. Force the bad ones.

# 1) force the local-clock path
python probe_call.py --prompt "reply with: ok" --deadline 0.01

# 2) force empty and truncated shapes
#    point call_model at a stub that returns "" and "the"
python probe_call.py --prompt "reply with: ok" --attempts 1

# 3) force transport errors, then restore
#    drop your network briefly, or point at a closed port
python probe_call.py --prompt "reply with: ok" --attempts 3

# 4) prove the ledger gate
python probe_call.py --prompt "reply with: ok" --attempts 1

Step 4 is the one people skip. If you run the same fingerprint twice, you should see a skip, not a second call.

The decision table

Verdict	Retry?	Why	What to log
`OK`	No	You have text	Fingerprint, seconds
`TRANSPORT`	Yes, short jitter	You never got a reply	Error class
`THROTTLED`	Yes, long pause	The remote asked you to slow down	Pause length
`DEADLINE`	Only if side-effect free	Remote state is unknown	Fingerprint, seconds
`EMPTY`	No, inspect first	Often a prompt or filter problem	Raw body
`TRUNCATED`	Yes, with a bigger cap	Output hit a ceiling	Last 40 chars

The DEADLINE row is the one that needs judgement. Retrying the same fingerprint is safe when the call only reads. It is not safe when the call writes.

How I run this on a free host

I run the probe matrix on resources that cost me nothing, so a bad loop cannot turn into a bill. That is where MonkeyCode comes in for me: it offers free model access and a free server option, which is enough to run the kill tests above end to end.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server is exactly the kind of environment that exposes these myths. It can be slow, it can sleep, and it can be reclaimed. That is a feature for this experiment. It is not a latency baseline.

So read the verdicts. Ignore the wall-clock numbers. Rerun on your own host before you trust any of them.

What this harness does not prove

Be honest about the edges. Mine are sharp.

It does not measure model quality. OK means text arrived, not that the text is right.
It does not make calls idempotent. The ledger records attempts. Only your code can prevent double writes.
The truncation check is a heuristic. Three suffix patterns is a demo, not a parser.
It does not model your rate limits. Any specific attempt count I print would be a guess about your account.
Threads are not cancellation. The DEADLINE branch abandons a thread, it does not kill the remote work.

That last one is the whole article in a sentence. You can stop waiting. You cannot un-ask.

Who should skip this

Some teams should not adopt this pattern. I would rather say so up front.

Hard latency SLOs. A free host and a sleeping box make latency claims meaningless.
Strict data residency. Read the terms of whatever host and model you pick. Do not take my word.
High fan-out pipelines. A per-call thread plus ledger append gets clumsy fast. Use a real queue.
Anything with money attached. If a duplicate write costs real money, build an idempotency key at the API layer first.

The one question to keep

Before you add a retry, ask it. If this call already ran, what breaks?

If you cannot answer, the retry is not the fix. The missing ledger is. Start with the probe, run the kill tests, and let your own verdicts shape the policy.

If you want a zero-cost place to run those kill tests, the free models and free server on MonkeyCode will do the job.

Your App Is Up, Your Model Call Is Down: Five Coupling Myths and a Two-Probe Check

Jordan Huang — Tue, 27 Oct 2026 22:09:05 +0000

Your App Is Up, Your Model Call Is Down: Five Coupling Myths and a Two-Probe Check

A 200 from your own health endpoint proves one thing. Your process is alive. It says nothing about the network path to your model provider.

I keep seeing the same five claims in code review. Each one quietly merges two failure domains into one. Then a green dashboard hides a red dependency.

Here is the myth list, the evidence you can collect yourself, and a small probe script that keeps the domains separate.

Myth 1: "The app is up, so the model path is up."

Your process health and your outbound dependency health are different signals. They fail on different clocks, for different reasons.

A sleeping or restarted host can still answer a local health route. A DNS blip can break outbound calls while every local route returns 200.

Corrected model: you have two probes, not one. One asks "is my process serving?" The other asks "can I complete one minimal model round trip right now?"

If you only have budget for one dashboard row, split it into two rows. Merging them back is how you get the classic "everything is green and nothing works" screen.

Myth 2: "Retrying a failed call is free and safe."

Retries cost wall-clock time and quota. That part is obvious. The non-obvious part is side effects.

If your model call sits inside a loop that writes files, posts comments, or opens merge requests, a blind retry can do the work twice. The second write looks like a new event to everything downstream.

Corrected model: every attempt gets a key, and every side effect checks that key.

import hashlib, json, uuid

def attempt_key(prompt: str, params: dict, job_id: str) -> str:
    # Stable for a logical unit of work, not for a single network attempt.
    blob = json.dumps({"p": prompt, "params": params, "job": job_id}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

Log the key before the call. Log the outcome after. If the key already has a success row, skip the side effect.

Is that extra bookkeeping annoying? Yes. Is it cheaper than explaining a duplicate write? Almost always.

Myth 3: "temperature=0 gives me the same answer twice."

It gives you a lower-variance sample. It does not give you a contract.

Even with fixed parameters, other things move: the served model revision, batching, hardware assignment, and any routing in front of the endpoint. Your prompt hash did not change. The thing answering it might have.

Corrected model: treat a model response as an observation with metadata, not a pure function output.

Record at minimum:

The model identifier string you sent
Any identifier string the response returned
Parameter set (temperature, max tokens, stop sequences)
Prompt hash
Timestamp, attempt number, latency

If those five fields are missing, you cannot reproduce the bug. You can only re-enact it and hope.

Myth 4: "The model name is the model."

A name in a config file is a pointer. Pointers get repointed.

I am not claiming a specific provider moved a specific alias on a specific date. I am claiming you cannot prove they did not, because you never logged the response-side identifier.

Corrected model: the name you send is intent. The identifier you receive is evidence. Store both, and diff them across runs.

A one-line check catches this cheaply:

# Two probes an hour apart, same config, different response-side id?
grep -o '"served_id": *"[^"]*"' probe.jsonl | sort -u | wc -l

If that count is greater than one and you did not change anything, you learned something real about your dependency.

Myth 5: "A client timeout means no work happened upstream."

A timeout is a statement about your patience. It is not a statement about their side.

The request may have been queued, started, or finished after you stopped listening. Retrying immediately is how you pay twice for one answer you never read.

Corrected model: a timeout is unknown, not failed. Route it to a separate branch with a longer backoff and a duplicate-suppression check.

What you observed	What it proves	Safe next action
Connection refused	Path is down now	Backoff, then re-probe
429	You are over a limit	Stop, wait, do not hammer
5xx	Their side errored	Bounded retry with the same key
Client timeout	Unknown	Same key, longer wait, check ledger first
200 with odd output	Call succeeded	Validate output, log metadata

Note what is missing from that table: "retry five times fast." That row does not exist for a reason.

The artifact: a two-probe check you can run today

This is a reference implementation. I have not run it against your account, so adapt the endpoint shape before trusting it.

MonkeyCode's free model access covers the model probe, and the free server option is enough to host the script itself. Both are operator-supplied availability claims, not measured numbers from me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# probe.py - two independent checks, one JSONL ledger
import datetime, json, os, time, uuid

MODEL_ID = os.environ["MODEL_ID"]        # from your account, never hardcoded
BASE_URL = os.environ["MODEL_BASE_URL"]
API_KEY  = os.environ["MODEL_API_KEY"]

def now():
    return datetime.datetime.now(datetime.timezone.utc).isoformat()

def record(kind, ok, detail, **extra):
    row = {"ts": now(), "kind": kind, "ok": ok, "detail": detail, **extra}
    with open("probe.jsonl", "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

def probe_local():
    path = "probe_local.tmp"
    token = str(uuid.uuid4())
    with open(path, "w") as f:
        f.write(token)
    with open(path) as f:
        readback = f.read()
    return record("local", readback == token, "disk round trip")

def probe_model():
    started = time.time()
    prompt = "Reply with the single word: ok"
    payload = {
        "model": MODEL_ID,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 8,
    }
    req = __import__("urllib.request").request.Request(
        BASE_URL,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"},
        method="POST",
    )
    try:
        with __import__("urllib.request").request.urlopen(req, timeout=30) as r:
            body = json.loads(r.read().decode())
        return record("model", True, "call completed",
                      latency=round(time.time() - started, 3),
                      served_id=body.get("model"),
                      attempt=1)
    except Exception as e:
        # Timeout lands here too. Treat it as unknown, not as failure.
        return record("model", False, type(e).__name__,
                      latency=round(time.time() - started, 3), attempt=1)

if __name__ == "__main__":
    print(probe_local())
    print(probe_model())

Run it twice a day for a week. Then read probe.jsonl and count the rows where local.ok is true and model.ok is false.

That count is the number of times a single health check would have lied to you.

What this does not prove

It does not measure provider reliability. Two probes a day is a smoke test, not a study.
It does not catch slow degradation. A 30-second call that succeeds still hurts.
It does not validate output quality. A round trip can succeed and return nonsense.
It does not survive a full host wipe. The ledger lives on the same disk as the script.

Ship the ledger somewhere else before you depend on it. A file you can lose is a file you will lose.

Who should skip this

If you have a real queue, real tracing, and an SLO on dependency latency, you already own this problem in a better tool. Do not build a probe script to duplicate it.

If your work is regulated, or needs a contractual uptime number, a free tier is the wrong foundation. Use it to learn the failure shape, then move.

And if you only run the model call interactively, in a chat window, you do not need a two-probe check. You have a human watching every call. That human is your monitor.

The short version

Five claims, one fix: stop letting one green light speak for two systems.

Process health is not dependency health
Retries need keys, not courage
Fixed parameters are not a reproducibility contract
A model name is intent, a served id is evidence
A timeout is unknown, not failed

If you want to watch the two domains split on your own box, the free model access and free server option are enough to run probe.py for a week. Then count the mismatched rows and decide what your dashboard has been hiding.

A 200 OK Is Not a Finished Answer: Classifying Model-Call Failures Before You Retry

Jordan Huang — Tue, 27 Oct 2026 19:03:43 +0000

Last week one generation came back with a 200. My parser died at character 812. So was that a failure? Or a good answer that stopped early?

I keep finding the same bug in client code. Every non-200 becomes 'retry'. Every 200 becomes 'done'. Both assumptions burn tokens and hide the real problem.

Here is the mental model I use now. A model call has three outcomes, not two. It fails, it succeeds, or it stops partway. Most clients only have buckets for the first two.

Myth 1: A 200 means the model answered

A 200 means the server accepted the request. That is all. The body can be empty. The body can be an error envelope dressed up as a 200.

Streaming makes this worse. The 200 arrives before the first token exists. The status line says nothing about the content.

So what does your code check after the status line? If the answer is 'nothing', your parser is doing a validator's job.

Myth 2: The stream ended, so the answer is complete

A stream ends when the connection closes. That is not a finished generation.

The evidence you want is finish_reason. Missing means truncated. A value of length means a token cap cut the answer mid-sentence.

Partial JSON often still parses, if you slice carefully. That is the nasty case. You get a valid object with a missing field and zero errors anywhere.

Myth 3: Retrying a call is free

Retries cost tokens, wall time, and sometimes duplicate side effects. If the call also runs a tool, a retry can run that tool twice.

Retry storms also make rate limits worse. A tight loop on a 429 is you building your own outage. Stop the run instead.

Myth 4: If the JSON parses, the schema is fine

Parsing is not validation. Parsing asks 'is this a JSON document?' Validation asks 'does it have the fields I need?'

Check the keys you depend on. Check their types. Reject early, loudly, with the raw text attached.

Myth 5: A slow call is a broken call

Slow and failed are two different axes. A nine-second success is still a success. Collapsing both into one 'unhealthy' gauge hides which one you actually have.

Record elapsed time per call. Decide about it later, with data instead of vibes.

A small probe that classifies outcomes

Here is the artifact I run before trusting a new endpoint. One file, one log, one row per call.

#!/usr/bin/env python3
"""probe.py - classify model-call outcomes instead of guessing.

    python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl

Keep --n small. This is an evidence log, not a load test.
The request body is a chat-completions example. Adapt it to your API.
"""
import argparse
import json
import time
import urllib.error
import urllib.request
from collections import Counter

PROMPT = 'Reply with a small JSON object that has keys ok and n.'
REQUIRED = {'ok', 'n'}


def extract(raw):
    """Pull text and finish_reason out of an unknown envelope."""
    try:
        env = json.loads(raw)
    except json.JSONDecodeError:
        return '', None
    choice = (env.get('choices') or [{}])[0]
    msg = choice.get('message') or {}
    text = msg.get('content') or choice.get('text') or ''
    return text, choice.get('finish_reason')


def classify(status, text, finish_reason, err):
    if err:
        return 'TIMEOUT' if err == 'timeout' else 'NETWORK'
    if status == 429:
        return 'RATE_LIMITED'
    if status >= 500:
        return 'SERVER_ERROR'
    if 400 <= status < 500:
        return 'CLIENT_ERROR'
    if not text.strip():
        return 'EMPTY_CONTENT'
    if finish_reason in (None, '', 'length'):
        return 'TRUNCATED'
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return 'NOT_JSON'
    if not REQUIRED.issubset(obj):
        return 'SCHEMA_MISSING_KEYS'
    return 'OK'


def one_call(url, key, timeout):
    body = json.dumps({
        'messages': [{'role': 'user', 'content': PROMPT}],
        'stream': False,
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, method='POST')
    req.add_header('Content-Type', 'application/json')
    if key:
        req.add_header('Authorization', 'Bearer ' + key)

    start, status, raw, err = time.monotonic(), None, '', None
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status = resp.status
            raw = resp.read().decode('utf-8', 'replace')
    except urllib.error.HTTPError as e:
        status = e.code
        raw = e.read().decode('utf-8', 'replace')
    except (urllib.error.URLError, OSError) as e:
        err = 'timeout' if 'timed out' in str(e) else 'network'

    text, finish = extract(raw)
    return {
        'status': status,
        'elapsed_s': round(time.monotonic() - start, 3),
        'finish_reason': finish,
        'chars': len(text),
        'outcome': classify(status, text, finish, err),
        'raw_head': raw[:200],
    }


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--url', required=True)
    p.add_argument('--key', default='')
    p.add_argument('--n', type=int, default=10)
    p.add_argument('--sleep', type=float, default=2.0)
    p.add_argument('--timeout', type=float, default=60.0)
    p.add_argument('--out', default='probe.jsonl')
    args = p.parse_args()

    rows = []
    with open(args.out, 'w', encoding='utf-8') as fh:
        for i in range(args.n):
            row = one_call(args.url, args.key, args.timeout)
            row['i'] = i
            fh.write(json.dumps(row) + '\n')
            fh.flush()
            rows.append(row)
            print(i, row['outcome'], row['status'], row['elapsed_s'])
            if row['outcome'] == 'RATE_LIMITED':
                print('rate limited: stop, do not tighten the loop')
                break
            time.sleep(args.sleep)

    print('tally:', dict(Counter(r['outcome'] for r in rows)))
    print('log:', args.out)


if __name__ == '__main__':
    main()

Run it, then summarize without eyeballing twelve rows.

python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl
jq -r '.outcome' probe.jsonl | sort | uniq -c | sort -rn
jq -r 'select(.outcome=="TRUNCATED") | [.i,.status,.chars,.finish_reason] | @tsv' probe.jsonl

Turn each outcome into a retry rule

Do not write one except. Write one rule per class.

Outcome	Retry?	Wait	Log
`OK`	no	none	elapsed, chars
`EMPTY_CONTENT`	once	2s	raw_head
`TRUNCATED`	once, smaller cap	2s	finish_reason, chars
`NOT_JSON`	no auto-retry	none	full text
`SCHEMA_MISSING_KEYS`	no auto-retry	none	parsed keys
`RATE_LIMITED`	no	stop the run	status, time
`SERVER_ERROR`	once	5s + jitter	status, raw_head
`TIMEOUT`	once	5s	elapsed, timeout value
`NETWORK`	once	5s	error string
`CLIENT_ERROR`	never	none	status, raw_head

Two rules matter more than the rest. Never retry a 4xx that came from your own bad request. Never auto-retry a schema failure, because it usually repeats.

What the tally actually tells you

Twelve calls in under a minute is a smoke test. It does not measure p50, and you should not report it as one.

It answers a smaller question. Which failure classes does this endpoint produce, for my request shape, right now? That is exactly what your retry code needs to know.

If you see four classes in twelve calls, your old except was hiding all of them. If you see only OK, you learned the happy path and nothing else.

Where I run the probe

My laptop lid closes. That kills a one-minute run halfway through.

For this I want a small box that stays awake for the run and has its own network path. MonkeyCode offers a free server option, and free model access for the model side, so I can keep the whole loop off my own machine. Both are availability claims from the product, not benchmarks from me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat that box as 'somewhere that is not my laptop'. Not as a scheduler, not as durable storage. The run writes probe.jsonl, and I copy the log out when it finishes.

Keep runs short and spaced. Free access is not a licence to hammer an endpoint, and I do not assume it stays identical forever.

What this probe does not prove

It is not a benchmark. Twelve calls, one hour, one network path.
It does not test long prompts, tool calls, or image inputs.
It cannot tell you an endpoint changed; it only shows what happened today.
It says nothing about cost, quota, or whether free access continues.

Who should skip this

If you need an SLA number, this is the wrong tool. Use a real load test with a large sample, a fixed schedule, and a proper percentile report.

If you are running at volume, add tracing and per-class metrics in production instead. A probe script is for the day before you trust an endpoint.

If a duplicate tool call would hurt you, fix idempotency first. Retry policy comes second.

The corrected mental model

A response is not one bit. It is a status, a body, a finish reason, and a duration.

Read all four before you decide anything. Then pick a retry rule per class, not per exception type.

If you run the probe, keep --n small, keep the log, and change REQUIRED to match your own schema.

Same Prompt, Different Answer: Five Reproducibility Myths in Free-Model Pipelines

Jordan Huang — Tue, 27 Oct 2026 15:58:38 +0000

Two runs. Same prompt. Different output. Is that a bug, or is that just how the endpoint works?

I keep seeing the same claims in review threads. The model is deterministic. A retry is harmless. A 200 means it ran once. Let's test those claims instead of repeating them.

Below is a small harness you can run in ten minutes. It uses stdlib Python only. It gives you evidence, not vibes.

The setup: a stub you control

First, a local server that is deliberately nondeterministic. Real endpoints vary too. This one just makes the variation obvious.

# stub_model_server.py
import json, random
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(n) or b"{}")
        prompt = body.get("prompt", "")
        answer = f"answer-{random.randint(1000, 9999)} for: {prompt[:20]}"
        payload = json.dumps({"output": answer}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

HTTPServer(("127.0.0.1", 8765), H).serve_forever()

Run it in one terminal: python3 stub_model_server.py

Now the runner. It writes a two-phase receipt: one row before the call, one row after the side effect.

# job_runner.py
import hashlib, json, os, time, urllib.request, uuid

BASE_URL = os.environ.get("MODEL_BASE_URL", "http://127.0.0.1:8765/generate")
LEDGER = os.environ.get("LEDGER_PATH", "receipts.jsonl")

def key_for(prompt, params):
    blob = json.dumps({"prompt": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:32]

def call_model(prompt, params):
    req = urllib.request.Request(
        BASE_URL,
        data=json.dumps({"prompt": prompt, **params}).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.loads(r.read())["output"]

def record(row):
    with open(LEDGER, "a") as f:
        f.write(json.dumps(row) + "\n")

def applied(key):
    if not os.path.exists(LEDGER):
        return False
    with open(LEDGER) as f:
        return any(
            json.loads(l)["key"] == key and json.loads(l)["status"] == "applied"
            for l in f if l.strip()
        )

def run(prompt, params):
    key = key_for(prompt, params)
    if applied(key):
        print("replay detected, skipping side effect:", key)
        return key
    record({"key": key, "status": "started", "ts": time.time()})
    out = call_model(prompt, params)
    record({
        "key": key,
        "status": "applied",
        "attempt": str(uuid.uuid4()),
        "output_sha": hashlib.sha256(out.encode()).hexdigest()[:16],
        "ts": time.time(),
    })
    print("output_sha:", hashlib.sha256(out.encode()).hexdigest()[:16])
    return key

if __name__ == "__main__":
    run("summarize ticket 42", {"temperature": 0})

Run it twice: python3 job_runner.py. Compare the two output_sha values. That is myth one, dead.

Myth 1: "Temperature 0 makes it repeatable"

Temperature 0 narrows sampling. It does not promise an identical string.

My two runs above differ. Same prompt, same params, same server. Different hash.

So what do you actually need? A fingerprint of the output, stored next to the request. Not a promise of sameness. Then you can detect drift instead of arguing about it.

What to record instead

Request key: hash of prompt plus params, sorted.
Output hash: short SHA of the returned text.
Attempt id: a UUID per call, so duplicates are visible.
Status: started and applied, never one row.
Timestamp, in UTC, from the runner.

Five fields. That's the whole trick.

Myth 2: "A retry only costs time"

Does it, though? A retry can be a second side effect.

Picture this. Your worker calls the model, gets a timeout, retries. Both calls actually succeeded. Now you have two summaries, two emails, two tickets.

The guard is an idempotency key. Mine is key_for(prompt, params). Same input, same key. The ledger refuses the second apply.

But look at my code. The side effect runs after the model call. If the process dies in between, you get an orphan: started, never applied. That's myth two's real cost — not the call, the ambiguity.

# audit.py
import json
rows = [json.loads(l) for l in open("receipts.jsonl") if l.strip()]
started = {r["key"] for r in rows if r["status"] == "started"}
applied = {r["key"] for r in rows if r["status"] == "applied"}
print("orphaned keys:", sorted(started - applied))

Run python3 audit.py and kill the runner mid-flight. You'll see the gap. An orphan is not a failure. It's a question your logs now ask out loud.

Myth 3: "HTTP 200 means it ran exactly once"

A 200 tells you one response arrived. It says nothing about how many attempts happened upstream.

Timeouts, proxy retries, and client reconnects all break the one-to-one story. So treat every response as at least once. Design your write path for duplicates.

Ask yourself: if this job ran twice, what breaks? If the answer is "nothing," good. If it's "the invoice," you have work to do.

Myth 4: "The model name pins the behavior"

Model names are labels. Endpoints get updated behind labels.

I'm not claiming a specific provider changed anything on a specific date — check their changelog today, not my memory. The general lesson holds: a name is not a hash.

So log a response fingerprint on a fixed canary prompt. Run it daily. When the hash shifts, you learn about it from your ledger, not from a user.

This is where free access gets useful. MonkeyCode's free model access gives you a low-pressure place to run that canary (operator-supplied availability; confirm current terms yourself). Cheap enough to test daily, which is the point.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 5: "Local logs are a ledger"

The ledger must outlive the process that wrote it. Files on ephemeral disk are not a ledger. They are a rumor.

MonkeyCode also offers a free server option. If you run the runner there, keep LEDGER_PATH on durable external storage — object storage, a managed database, or a volume the provider documents as persistent. Otherwise audit.py finds nothing after a restart.

I learned this the boring way, so you don't have to.

Who should not use this

Be honest about the fit.

Throwaway scripts. If a duplicate costs nothing, skip the ledger. It's overhead.
Strict exactly-once needs. A file ledger gives at-least-once plus detection. It is not a transaction. Use a database with unique constraints.
Multi-writer queues. Two writers appending to one file will interleave rows. Move to SQL first.
Teams without cleanup. Orphans pile up. If nobody runs the audit, you built a diary, not a control.

Run it in ten minutes

Start the stub. Run the job twice. Diff the hashes. Kill it once. Run the audit.

That's four commands and one honest conclusion: reproducibility is something you measure, not something you assume.

Which myth breaks first in your logs? I'd bet on the retry.

Your Free Server Forgot Your Files: Four Restart Myths and a Survival Check

Jordan Huang — Tue, 27 Oct 2026 13:54:16 +0000

Your job wrote the file. You redeployed. The file was gone.

So which part failed: the write, or your assumption about the disk?

The bug shape you will hit

A scheduled job appends rows to state.jsonl.
The container restarts for an unrelated reason.
The file comes back empty, missing, or half-written.
The job re-runs and double-sends, double-charges, or double-builds.

Nothing in that list is exotic. Free compute just makes it routine.

Myth 1: "A restart is a clean reset"

Restarting a process clears memory. It does not clear disk.

That cuts both ways. Your cache and in-memory locks disappear, which is fine. Your stale worker.lock, your half-finished .part file, and yesterday's build artifact may not.

A restart is a partial reset you never specified. Good to know before you debug it at 2am.

Evidence: write a marker file, restart, read it back. That is the whole test. We automate it below.

Myth 2: "/tmp is temporary, the app dir is permanent"

Folder names are not durability policies.

On plenty of container hosts, the app directory is an image layer. A redeploy rebuilds it. Your "permanent" file was never on a volume; it was on a layer that gets replaced.

/tmp may be a tmpfs. It may also be the same overlay as everything else. The name tells you nothing.

Corrected model: ask where the bytes physically land, not what the folder is called.

Myth 3: "If the write returned, the data is safe"

Three boundaries exist here, and they fail separately.

write() plus flush() moves bytes from your process to the OS page cache.
os.fsync() pushes them toward the storage device.
The volume itself can be discarded on the next deploy or scale event.

Steps 1 and 2 protect you from a crash. Nothing in your code protects you from step 3.

Myth 4: "Free compute is only for demos"

Wrong in one direction, right in another.

Free compute can run real jobs, as long as state lives outside it and every job is idempotent. What it cannot be is your database, your queue, or your source of truth.

Treat it as disposable compute and it behaves predictably. Treat it as a server and it will surprise you.

The probe: a survival check

Two modes. write appends a checksummed record. verify re-reads and re-hashes every line.

#!/usr/bin/env python3
"""survival_check.py -- does this path survive a restart?"""
import argparse, hashlib, json, os, time, uuid
from pathlib import Path


def target(state_dir: Path) -> Path:
    return state_dir / "survival.jsonl"


def digest(rec: dict) -> str:
    body = json.dumps(rec, sort_keys=True)
    return hashlib.sha256(body.encode("utf-8")).hexdigest()


def cmd_write(state_dir: Path, note: str) -> None:
    state_dir.mkdir(parents=True, exist_ok=True)
    rec = {"id": uuid.uuid4().hex[:12], "note": note, "ts": round(time.time(), 3)}
    rec["sha256"] = digest(rec)
    with open(target(state_dir), "a", encoding="utf-8") as f:
        f.write(json.dumps(rec, sort_keys=True) + "\n")
        f.flush()
        os.fsync(f.fileno())   # survives a crash, not a redeploy
    print(f"wrote id={rec['id']} to {target(state_dir)}")


def cmd_verify(state_dir: Path) -> int:
    path = target(state_dir)
    if not path.exists():
        print(f"MISSING {path}")          # the path did not survive
        return 2
    bad, n = 0, 0
    for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        rec = json.loads(line)
        claimed = rec.pop("sha256", None)
        ok = claimed == digest(rec)
        bad += 0 if ok else 1
        print(f"line={n} id={rec['id']} checksum={'ok' if ok else 'MISMATCH'}")
    print(f"lines_checked={n} mismatches={bad}")
    return 0 if bad == 0 else 1


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("mode", choices=["write", "verify"])
    ap.add_argument("--dir", required=True, type=Path)
    ap.add_argument("--note", default="probe")
    a = ap.parse_args()
    raise SystemExit(cmd_verify(a.dir) if a.mode == "verify" else cmd_write(a.dir, a.note))

Write to three candidate paths, then restart the process or container:

for d in /tmp/survival ./state /data/survival; do
  python3 survival_check.py write --dir "$d" --note before-restart \
    || echo "WRITE FAILED $d"
done

# restart the process, container, or instance here

for d in /tmp/survival ./state /data/survival; do
  echo "== $d"
  python3 survival_check.py verify --dir "$d"
done

Reading the results

verify output	what it means	what to do
`MISSING`	the path vanished	move state to a real volume or external store
`MISSING` but `write` succeeded	write went to a layer that was discarded	stop trusting that path
`mismatches > 0`	bytes changed or truncated	suspect the storage layer, not your hash
all `ok`	survived this restart	repeat after a redeploy before believing it

One clean pass proves very little. A restart is not a redeploy. A redeploy is not a scale event. Run the probe after each one you can trigger on purpose.

Then make the job idempotent

Persistence is half the fix. The other half is a key.

def run_once(job_key: str, store) -> bool:
    if store.seen(job_key):      # store must live outside the free box
        return False
    store.mark(job_key)
    do_the_work(job_key)
    return True

If store is a file on the ephemeral disk, you renamed the bug. You did not fix it.

Where free model access fits

I use a model for two narrow jobs here: drafting the checksum logic, and reviewing idempotency keys for collisions. Both are text problems with testable answers, which is where a model earns its keep.

MonkeyCode is an open-source project offering free model access and a free server option. That is enough to run the write → restart → verify loop above without paid infrastructure. The exact free allowance changes over time, so read the current terms on the project page instead of trusting a number quoted in any article, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not read "it ran on free compute" as a durability claim. Read it as evidence about one boundary, on one day.

Who should skip this

You need a written durability or uptime guarantee. Use a managed database and a paid runtime.
Your data carries residency or audit requirements a shared free host cannot meet.
Your job moves money or sends messages. Get a real queue first.
You cannot redeploy on demand. Then you cannot test the boundary that matters.

The one-line version

A restart is a memory reset, not a disk reset. Probe the disk, externalize the state, key the job. Then trust it, after you have re-run the probe once more.

Want to try the loop yourself? The free server option is a reasonable place to start.

Same URL, Different Machine: Four State Myths on a Free Server

Jordan Huang — Tue, 27 Oct 2026 12:53:33 +0000

You deploy to a free server. The first request works. The second one forgets everything.

Is your code wrong? Was it a restart? Or is the URL just a label in front of a machine you never met?

I keep seeing the same three guesses in issue threads. All three are myths. Here is how I check, with a probe you can run in ten minutes.

The setup: one route, two questions

I want two answers from any free box. Which process am I talking to? And what state can that process honestly promise?

So I add a fingerprint route. It is boring on purpose. Stdlib only, no dependencies.

# app.py - stdlib only, no frameworks
import json, os, time, uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

NONCE = uuid.uuid4().hex[:12]   # new value per process
COUNT = 0                        # a lie we are about to test

def boot_id():
    try:
        return open("/proc/sys/kernel/random/boot_id").read().strip()
    except OSError:
        return "unavailable"

def proc_start_ticks():
    # /proc/self/stat field 22 = starttime, in clock ticks since boot
    try:
        raw = open("/proc/self/stat").read().rsplit(") ", 1)[1].split()
        return int(raw[19])
    except (OSError, IndexError, ValueError):
        return -1

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        global COUNT
        COUNT += 1
        body = json.dumps({
            "nonce": NONCE,
            "pid": os.getpid(),
            "boot_id": boot_id(),
            "proc_start": proc_start_ticks(),
            "count": COUNT,
            "uptime_s": round(time.monotonic(), 1),
            "host": os.uname().nodename,
        }).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *a):
        pass

ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever()

Run it, expose port 8080, then poll it from your laptop:

# probe.py  ->  python probe.py https://your-app.example.com
import json, sys, time, urllib.request

url = sys.argv[1].rstrip("/") + "/"
prev = None
for i in range(12):
    with urllib.request.urlopen(url, timeout=10) as r:
        d = json.load(r)
    key = (d["nonce"], d["boot_id"], d["proc_start"])
    print(f"{i:02d} {'same' if key == prev else 'NEW':4s} "
          f"count={d['count']:3d} up={d['uptime_s']:7.1f}s "
          f"pid={d['pid']} host={d['host']}")
    prev = key
    time.sleep(5)

That is the whole harness. Twelve requests, five seconds apart. Cheap enough that you will not fight a rate limit.

Myth 1: "One URL means one process."

This is the most expensive myth on the list. A URL is routing, not identity.

Look at the nonce. If it survives every poll, you have one process. If it changes, you are talking to someone new.

Now compare boot_id. Same nonce, same boot, same process. Different boot means a different kernel booted, so a different machine answered you.

Same boot but a new proc_start? Your process restarted on the same host. That is a crash, a redeploy, or an idle shutdown waking back up.

Illustrative output, not a measurement from your box:

00 same count=  1 up=    0.4s pid=41 host=box-a
01 same count=  2 up=   5.4s pid=41 host=box-a
02 NEW  count=  1 up=    0.4s pid=77 host=box-a

That jump at index 02 is the whole story. Same host, new PID, counter back to one.

Myth 2: "In-memory state is cheap, so keep it."

In-memory state is cheap. It is also the first thing you lose.

Ask a sharper question. What does this counter actually guarantee? Nothing beyond the life of the process.

So I sort state by what breaks when it disappears:

Liveness flags, caches, warm handles: fine in memory. Losing them costs latency, not truth.
Job progress, retry budgets, "already emailed": never in memory. Losing them costs correctness.
Counters a user paid for: never on the box at all. The box is not your ledger.

On a free server the suspend/resume behaviour is the thing to verify, not assume. I would rather measure it once than argue about it in a thread.

Myth 3: "A 200 with a body means the model finished."

This one hides inside free model access. You get a clean response, a tidy JSON body, and a stop reason you never read.

Then someone asks why the last paragraph is cut. The response was never complete. It was just delivered.

So I check the reason field on every call in the path I care about. Field names vary by provider, so read yours:

# shape varies - adapt, do not copy blindly
choice = resp.json()["choices"][0]
reason = choice.get("finish_reason")
if reason == "length":
    raise RuntimeError(f"truncated, not complete: {reason}")

Now the failure is loud. A truncated answer becomes an exception, not a mystery.

This is the workflow I actually run: the fingerprint probe on a free server to see what my process is, and a free model call to check whether my completion is real. That pairing is why I use MonkeyCode's free model access and free server option, both operator-supplied availability rather than anything I can promise for you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 4: "If the write returned, the data landed."

A 200 on open(...).write() means the kernel accepted bytes. It does not mean the disk kept them past the next restart.

On ephemeral filesystems, files vanish with the instance. So I test the claim instead of trusting it. Write a marker, force a restart, poll again.

curl -s $URL >/dev/null; echo "marker-$(date +%s)" > /tmp/marker
# restart the instance from your dashboard, then:
curl -s $URL/marker   # 404 here means: treat /tmp as cache, not storage

If that 404 happens, you learned something real in one restart instead of one incident.

The decision table I keep next to the probe

Signal you saw	Likely cause	Correct response
`nonce` changed, `boot_id` same	Process restart or idle recycle	Move durable state out of memory
`boot_id` changed	Different host or fresh boot	Never assume local disk persists
`count` resets between polls	New process answered	Check your retry and dedup keys
`count` jumps by more than your polls	More than one instance served	Read replicas cannot own a counter
`finish_reason == "length"`	Output cap hit	Treat as failure, retry with a plan

Five rows. That is the entire mental model. State belongs where instance identity does not matter.

Limitations, and who should skip this

This probe measures identity, not performance. It says nothing about latency, throughput, or cost.

Skip it if all four of these are true for you: paid always-on compute, one instance, a real database, and no idle suspend. You already have the guarantees you need.

The free-server route also is not for everything. If you need predictable cold start under a second, a suspend-on-idle box will fight you. If you need a signed uptime guarantee, read the terms before writing code around it.

One more warning. Do not loop the probe forever. Twelve requests is a test. Twelve thousand is a quota incident you caused yourself.

The corrected mental model

A URL is a promise to route bytes. It is not a promise about which process receives them.

A 200 is a delivery receipt. It is not a completion proof. A file write is a syscall. It is not a durability guarantee.

So fingerprint the process, read the stop reason, and move durable state somewhere that survives a reboot. Which of those three are you skipping right now?

DEV Community: Jordan Huang

A Restart Is Not a Reset: Five Persistence Myths and a Boot-ID Probe

A Restart Is Not a Reset: Five Persistence Myths and a Boot-ID Probe

What I actually wanted to know

The probe: one file, two runs, one restart

FAQ: five claims I used to repeat

1. 'If it survived a restart, it is persistent.'

2. 'It is the same box, so it is the same disk.'

3. 'The model remembers our conversation.'

4. 'It is only a cache, so losing it is fine.'

5. 'A 200 from /healthz means my state is fine.'

A durability decision table

The restart drill I run before shipping

Limitations, and who should skip this

Your Free Tier Is Not a Capacity Plan: Five Throughput Myths and a Concurrency Ramp

Myth 1: "Latency grows linearly with concurrency"

Myth 2: "Retries add capacity"

Myth 3: "It is free, so the numbers do not matter"

Myth 4: "A 429 or a timeout means the service is broken"

Myth 5: "The model is the slow part"

The artifact: a concurrency ramp

Reading the output

The decision table

The gate you ship

What this does not tell you

Who should not use this

Streaming Feels Faster, but Your Model Isn't: A Wire-Level Timestamp FAQ

The claim that keeps coming back

Two runs, one clock

FAQ: four claims, four corrections

1. "Streaming makes generation faster."

2. "Time to first byte is a model latency number."

3. "If chunks are arriving, nothing is buffering."

4. "Streaming is the safer default on a free tier."

Cross-check in ten seconds

Decision table

Where this bites on a free tier

What this test does not tell you

Who should skip this

The model that survives the clock

Valid JSON, Wrong Order: Five Structured-Output Myths and a Three-Gate Replay Test

Myth 1: JSON mode means my schema is enforced

Myth 2: If it parses, the model was right

Myth 3: Retrying fixes bad output

Myth 4: Strict mode enforces my constraints

Myth 5: Schema drift is a model problem

The three-gate check

The replay harness

Decision table

Running the gate next to the model

Limitations you should hear

Who should not use this

A five-minute checklist

Your Cache Key Is Not Your Prompt: Four Response-Cache Myths and a Collision Test

Myth 1: "The prompt text is the cache key"

The corrected model: a key is a written equivalence claim

A key builder you can actually test

Four tests that catch real collisions

Myth 2: "A hit means the answer is still correct"

Myth 3: "Caching always cuts cost"

Myth 4: "The cache is on my server, so it is mine"

A 20-minute verification workflow

Limitations and who should skip this

What to take away

Four Idle Myths About Free Servers, and the Wake Probe That Settles Them

The claims I keep reading

Myth 1: A ping every five minutes keeps the instance warm

Myth 2: If /health returns 200, the app is ready

Myth 3: Idle time is free

Myth 4: Scheduled jobs run on schedule on a sleeping instance

The wake probe

Reading the result

What to do with the answer

Where MonkeyCode actually fits here

Limitations, and who should skip this

The short version

Your Timeout Is Local: Five Myths About Model Calls You Retry

Your Timeout Is Local: Five Myths About Model Calls You Retry

Myth 1: "A timeout means the call did not happen"

Myth 2: "HTTP 200 means I got an answer"

Myth 2: If `/health` returns 200, the app is ready