DEV Community

Maxim
Maxim

Posted on

I treated my LLM like a machine on a factory floor — and it stopped lying

#testing #python #ai #productivity

I didn't start in software. I spent years around real production lines.

A part goes out against a drawing. Someone checks it. Scrap doesn't move to the next station.
Nobody says "hopefully it's fine".

Then I started building systems with LLMs — and watched teams do exactly that.
Prompt. Hope. Print the output. Ship.

The shift that changed everything

Stop treating the model like a colleague who tries their best.
Treat it like a noisy machine:

  1. Draw a contract (what the output must look like)
  2. Inspect every unit (schema validation)
  3. If scrap: rework with the defect note (repair loop)
  4. If the machine is broken: turn it off (circuit breaker)

That's it. Same as QC. Just for tokens instead of steel.

Why LLM reliability is its own engineering

Code fails loudly. An LLM fails politely.

  • Wrong type? Sometimes valid JSON.
  • Missing field? Model says "Sure!" and you're happy until QA.
  • Truncated? Might look complete.

So you need more than try/except. You need gates.

What I actually shipped

Small open-source layers:

Nothing fancy. Boring layers that make models usable in production.

Practical tips (if you want the factory mindset)

  1. Write the drawing first. If you can't say what "good" looks like, the model can't either.
  2. Inspect every unit. One shared validator. Not "we'll see in prod".
  3. Rework beats recreate. Feed the defect note back. Cheaper than a new prompt.
  4. Scrap is allowed. Better to reject output than ship junk and apologize later.
  5. Measure scrap rate. If 20% of outputs fail schema, fix the prompt or the model tier.

Takeaway

If you can feel where a factory line breaks,
you can feel where an LLM pipeline breaks.

Same ear. Different material.

Top comments (0)