DEV Community: Maxim

I treated my LLM like a machine on a factory floor — and it stopped lying

Maxim — Fri, 16 Oct 2026 10:02:39 +0000

I didn't start in software. I spent years around real production lines.

A part goes out against a drawing. Someone checks it. Scrap doesn't move to the next station.
Nobody says "hopefully it's fine".

Then I started building systems with LLMs — and watched teams do exactly that.
Prompt. Hope. Print the output. Ship.

The shift that changed everything

Stop treating the model like a colleague who tries their best.
Treat it like a noisy machine:

Draw a contract (what the output must look like)
Inspect every unit (schema validation)
If scrap: rework with the defect note (repair loop)
If the machine is broken: turn it off (circuit breaker)

That's it. Same as QC. Just for tokens instead of steel.

Why LLM reliability is its own engineering

Code fails loudly. An LLM fails politely.

Wrong type? Sometimes valid JSON.
Missing field? Model says "Sure!" and you're happy until QA.
Truncated? Might look complete.

So you need more than try/except. You need gates.

What I actually shipped

Small open-source layers:

structured-llm-output-guard — text → typed JSON
guards-and-laws — quality gates + domain laws with HARD VETO

Nothing fancy. Boring layers that make models usable in production.

Practical tips (if you want the factory mindset)

Write the drawing first. If you can't say what "good" looks like, the model can't either.
Inspect every unit. One shared validator. Not "we'll see in prod".
Rework beats recreate. Feed the defect note back. Cheaper than a new prompt.
Scrap is allowed. Better to reject output than ship junk and apologize later.
Measure scrap rate. If 20% of outputs fail schema, fix the prompt or the model tier.

Takeaway

If you can feel where a factory line breaks,
you can feel where an LLM pipeline breaks.

Same ear. Different material.

How I stopped losing 3 hours of GPU work every time a job crashed

Maxim — Fri, 16 Oct 2026 09:57:19 +0000

We had a pipeline that could run for hours. GPU time is expensive. A crash at 95%
was the same as a crash at 0% — except we were angrier.

I got tired of it. Not in the "we need a framework" way. In the "this is stupid" way.

What we had before

A long script:

research → script → render (forever) → assemble → done

If the process died in render, we started over. Every time.
People say "use a job queue". We had workers. We still lost time, because
progress lived only in RAM.

The fix (boring on purpose)

After every stage, write down that you finished it.

research   [done]
render     [done]   ← crash here? next run skips research+render
assemble   [pending]

Then:

run() executes pending stages
resume() skips done, retries failed
state is written atomically (tmp file → rename), so a kill mid-write doesn't corrupt everything
workers send a heartbeat; watchdog notices when they go silent

That's it. No magic scheduler. Just refusing to forget finished work.

Why not "just checkpoint tensors"?

We don't need mid-layer optimizer state for every job. Most of our cost is
stage-level work: research, render, package. Stage checkpoints cover 95% of the pain
with 5% of the complexity.

Trade-off on purpose: a stage still re-runs from its start. That's acceptable.
Losing the whole pipeline is not.

What broke along the way

Non-atomic state writes. Kill during save → corrupted JSON → resume lied to us. Fixed with write-to-temp + os.replace.
Retries that duplicated side effects. Stage contract now says: re-entry must be safe (idempotent), or the stage owns its own local temp dir.
"Busy" hung workers. Looked alive in the process list. Heartbeat file + staleness check.

Numbers

Long jobs: tens of minutes to hours
Before: crash = full re-run
After: crash = re-run from last finished stage
Tests: 12 behavioral contracts (resume, fail-fast, atomic state, stale heartbeat)

Practical checklist (copy/paste)

[ ] Checkpoint after each stage, not "sometime later"
[ ] Atomic write (tmp + rename). Kill mid-write must not corrupt state
[ ] resume() skips done, retries failed
[ ] Stages should be idempotent on re-entry
[ ] Heartbeat file + staleness check for hung workers
[ ] Fail-fast on poison stages
[ ] Test: crash after stage 1 → resume does stage 2 only

Rule of thumb

If re-running the job costs more than 5 minutes — checkpoint it.
If it costs GPU-hours — checkpoint it today.

Takeaways

Checkpoint I/O is cheap. Re-running 3 hours of GPU is not.
Atomic writes matter more than fancy progress bars.
Resume is a feature, not an ops afterthought.
Heartbeat ≠ "it works". It's only "it's alive".
Fail-fast on poison stages beats "keep going and hope".

Open-sourced here: checkpointed-task-pipeline

Related reliability pieces:

How I fixed silent JSON failures in my LLM pipeline (spoiler: json.loads was the trap)

Maxim — Fri, 16 Oct 2026 09:27:00 +0000

I'm an engineer. I build pipelines where an LLM is not a chat toy — it's a part in a machine.

One evening I was staring at logs. Half the videos went out with broken titles. JSON looked fine. json.loads didn't crash. The data was garbage anyway.

That was the first time we fell into this hole. Not the last.

The trap

The code looks innocent:

data = json.loads(llm_response)
title = data["title"]

Nice and short. And fragile.

An LLM does not return JSON. It returns text that looks like JSON. Like a person you asked to fill a form, and they answered: "Well, you see…" — maybe useful, maybe not.

What we actually got in production:

1. Markdown fences

The model wraps JSON in triple-backtick code fences. json.loads dies immediately. At least that failure is loud.

2. Politeness

Sure! Here is the JSON you asked for:
{"title": "Hello"}

Dies again. The model was helpful. The pipeline wasn't.

3. Trailing commas, smart quotes

{"title": "Hello",}

4. Truncation

{"title": "Hello", "score":

5. The worst one — silent junk

{"ok": true}

instead of {"title": "...", "score": 42}.

json.loads is happy. Valid JSON. Wrong shape. Garbage flows downstream. You notice an hour later. Or never.

Two failure modes, not one

Type	What happens	When you see it
Loud failure	Exception, job dies	immediately
Silent failure	wrong fields / wrong types	too late

try/except handles the first. The second is the one that burns you.

What we tried (and what didn't work)

"Just tell the model to always return clean JSON." Sometimes helps. A model is not a compiler.
Regex between { and }. Breaks on nested objects and braces inside strings.
Another pile of if-statements. Every new response shape = more ifs. Snowball.

We stopped begging the model and started inspecting it.

The fix

LLM text
   |
   +-- strip markdown fences and "Sure! Here's..."
   +-- normalize quotes / trailing commas
   +-- parse
   +-- validate SCHEMA (types, fields, bounds)
   |
   +-- if fail -> repair loop
   |      re-prompt with the EXACT error
   |      ("score: expected int, got str")
   |
   +-- if API is dying -> circuit breaker
          stop hammering a corpse

Same idea as QC on a factory floor: make it -> check it -> fix or scrap. No hoping.

I didn't start in software. I spent years around real production. A part goes out against a drawing, gets inspected, and scrap doesn't move down the line. An LLM is a machine too. It just lies with more confidence.

Code (no magic)

from llm_output_guard import Schema, Field, RepairLoop

schema = Schema(
    fields={
        "title": Field(str),
        "score": Field(int, min_value=0, max_value=100),
    }
)

def call_model(err=None):
    # err = exact validation error from last attempt
    return generate(title=..., hint=err)

result = RepairLoop(schema=schema, max_attempts=3).run(call_model)

Repair is not "roll the dice again". It is feed the model its own error. In our runs, attempt 2 fixes most type mistakes.

Separately: circuit breaker — 5 failures -> cooldown -> reset. Otherwise a clever repair loop becomes a hammer that finishes off a dying API.

Numbers from our systems (not from a blog)

49 Instructor injection points in one production pipeline — all through one JSON path
60+ schema contracts
12 behavioral tests just for "dirty" LLM outputs
finish_reason=length counts as failure, not success

When you have dozens of calls, "it'll parse somehow" is not a strategy.

Steal this checklist (practical, Monday-morning)

If you only take one thing from this post — take this:

[ ] Never json.loads(llm_text) raw. Strip fences first.
[ ] Always validate shape after parse (types + required fields).
[ ] Treat finish_reason=length as failure, not success.
[ ] On schema error: re-prompt with the exact error. Don't roll dice.
[ ] Put a circuit breaker around the API. 5 fails -> cooldown.
[ ] Log why validation failed. Future-you will need it.
[ ] Test with ugly outputs: fences, prose, trailing commas, wrong types.
[ ] One shared contract for all call sites. Not 49 special cases.

Quick "before you ship" check

Ask three questions:

What happens if the model returns markdown?
What happens if a field has the wrong type?
What happens if the API dies mid-run?

If the answer is "we crash" or "we ship garbage" — you're not done.

What we do not do

No sending secrets to random "JSON fixer" services
No treating json.loads(strip(text)) as enough
No hiding "almost passed" results

Takeaways

LLM is not a JSON API. It's a text API. JSON is your job.
Silent failures hurt more than crashes. Schema catches what try/except misses.
Repair with the exact error beats regenerate-from-scratch.
Circuit breaker protects you from your own stubbornness.
One contract for every call beats 49 special snowflakes.

We published this as small open-source libraries so nobody rebuilds it again:

structured-llm-output-guard — the safe JSON path
guards-and-laws — quality gates + domain laws with hard veto
checkpointed-task-pipeline — long jobs survive crashes

If you run LLMs in production, check your json.loads. If it has no schema and no repair — you may already be paying for it. Just haven't seen the invoice.

What's the ugliest LLM JSON you've seen in production? I'm collecting war stories.