DEV Community: Dimitris Kyrkos

"I'm 93% confident" is a string, not a probability

Dimitris Kyrkos — Thu, 22 Oct 2026 06:30:29 +0000

The gap between the demo and the pipeline

In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write if confidence > 0.9: auto_approve() and move on.

In production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk.

A confidence score earns the word "confidence" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration.

Failure mode 1: the threshold that was never tested

Illustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great.

What they never measured is accuracy within the >0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point.

Failure mode 2: one accuracy number hiding two systems

If your system can defer (hand off to a human, ask for clarification, fall back to a rule), then you're not evaluating one number. You're evaluating a trade-off:

Coverage: how often does the system answer instead of deferring?
Selective accuracy: when it answers, how often is it right?

Push the threshold up and coverage drops while selective accuracy (hopefully) rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it:

import numpy as np

def coverage_accuracy_curve(conf, correct, thresholds):
    conf, correct = np.asarray(conf), np.asarray(correct)
    rows = []
    for t in thresholds:
        answered = conf >= t
        coverage = answered.mean()
        sel_acc = correct[answered].mean() if answered.any() else float("nan")
        rows.append((t, round(coverage, 3), round(sel_acc, 3)))
    return rows

def calibration_bins(conf, correct, n_bins=10):
    conf, correct = np.asarray(conf), np.asarray(correct)
    edges = np.linspace(0, 1, n_bins + 1)
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (conf >= lo) & (conf < hi)
        if m.any():
            print(f"[{lo:.1f}, {hi:.1f})  n={m.sum():4d}  "
                  f"stated={conf[m].mean():.2f}  actual={correct[m].mean():.2f}")

If "stated" and "actual" drift apart in the bins you care about, your threshold is lying to you.

Failure mode 3: the global threshold

Even a calibrated score can be calibrated on average and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart.

One threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice:

for slice_name, idx in slices.items():   # e.g. by task type, language, source quality
    print(slice_name, coverage_accuracy_curve(conf[idx], correct[idx], [0.7, 0.8, 0.9]))

If the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all.

Failure mode 4: verbalized vs. measured confidence

Asking the model "how sure are you?" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence (verbal, log-prob, ensemble agreement, a separate verifier) as a hypothesis to test, not a fact to consume.

The reframe

Confidence isn't a model property. It's a system property you have to measure. The real question isn't "does the model feel sure?" It's "is the system's decision to proceed reliable for this task, in this context, on this kind of input?"

Practical checklist

Log every decision with its confidence, the slice metadata, and the eventual ground truth.
Plot coverage vs. selective accuracy, globally and per slice.
Check calibration bins in the range where your threshold actually sits.
Re-run it when the model, prompt, or input distribution changes. Calibration doesn't survive a model swap.

How are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?

Your agent's retry bug isn't an AI problem. It's a distributed-systems problem with an LLM on top.

Dimitris Kyrkos — Mon, 19 Oct 2026 06:42:45 +0000

Most agent demos have a happy path: the model picks a tool, the tool returns 200, the agent moves on. Production is different. In production, the tool call sometimes returns nothing at all.

That is where a correct decision turns into a wrong outcome.

The gap between "decided" and "done"

Here is an illustrative scenario (not a real incident, but a very ordinary one).

An agent handles a support ticket and decides the customer is owed a refund. It calls POST /refunds. The payments service processes the refund, writes it to its ledger, and starts sending the response. The network hiccups. The agent's HTTP client hits its timeout and raises an error.

From the agent's point of view, the call failed. From the payment service's point of view, it succeeded. The agent's loop sees an error, and the most natural thing for it to do is try again.

Now the customer has two refunds.

The model reasoned perfectly. It picked the right action with the right arguments. Nothing about better prompting, a bigger model, or a smarter planner fixes this, because the failure happened after the reasoning was over.

Failure mode 1: treating a timeout as a "no"

A timeout tells you one thing: you didn't get an answer. It does not tell you the action didn't happen. Agent frameworks that wrap tool errors into "the tool failed, decide what to do next" collapse three different situations into one:

the request never reached the service
the request reached it and was rejected
the request reached it, succeeded, and the response was lost

Only the first two are safe to retry blindly.

Failure mode 2: no idempotency key

If the agent had sent an idempotency key, the second refund call would have been a no-op that returned the original result. The key has to be generated by the application before the first attempt and reused on every retry of that same intent:

intent = actions.record_intent(
    kind="refund",
    order_id=order.id,
    amount=amount,
    idempotency_key=uuid4(),   # created once, stored with the intent
)

try:
    result = payments.refund(
        order_id=order.id,
        amount=amount,
        idempotency_key=intent.idempotency_key,
    )
    actions.mark_confirmed(intent.id, result)
except Timeout:
    actions.mark_unknown(intent.id)   # NOT "failed"

The important part is where the key lives. If the model generates it, or if it's regenerated per attempt, it protects nothing.

Failure mode 3: state that only exists in the context window

If the record of "I already tried to refund order 4812" lives only in the conversation history, it disappears on a crash, a context truncation, or a restart of the worker. The intent needs to be written to durable storage before execution, so that whatever process picks the work back up can see it.

Failure mode 4: no "outcome unknown" state

Most action models have two terminal states: success and failure. Reliable ones need at least:

submitted: we sent it, no answer yet
confirmed: the downstream system acknowledged it
failed: the downstream system explicitly rejected it
unknown: we can't tell

unknown is the one that matters. It's the state that should block automatic retries and trigger reconciliation instead.

Failure mode 5: no reconciliation path

When an action lands in unknown, something has to go find out the truth: query the payment service by idempotency key or order ID, compare against the ledger, and then move the intent to confirmed or failed. That can be a background job, a scheduled check, or a human queue for high-value actions. What it can't be is the agent deciding "probably fine" or "probably failed" on its own.

The reframe

Agent tool use is really remote procedure calls with an LLM choosing which procedure to call. Everything we already know about RPCs over unreliable networks still applies: idempotency, durable intent logs, explicit uncertainty, reconciliation.

The clean split is:

The model proposes an action.
The application owns the execution state: the key, the intent record, the status, the retry policy.

"Try again" is a fine default for reads. For anything that changes state, it's only safe once you know the first attempt didn't land, or once the downstream system guarantees a retry can't double-apply.

Build vs. integrate

If you already run payment or order flows with idempotency keys and outbox/ledger patterns, you mostly need to route agent actions through that same machinery instead of letting the agent call APIs directly. If you don't, that's the piece to build (or adopt) first, before giving an agent write access to anything that moves money or data.

How are you handling the "we don't know if it happened" case in your agent workflows? Are you blocking retries, reconciling, or escalating to a human?

Your model router is a classifier, and you probably aren't evaluating it like one

Dimitris Kyrkos — Thu, 15 Oct 2026 07:12:00 +0000

Most LLM cost discussions end with the same advice: don't send everything to the biggest model. Put a small, fast model in front, escalate when you need to, and watch the bill drop.

That advice is fine. The problem is what it leaves out. The moment you add a router, you've added a second model to your system: one whose job is to make a decision, whose mistakes are mostly invisible, and which almost nobody evaluates with the same care as the models it routes between.

The tutorial version vs. the production version

The tutorial version of a cascade looks like this:

def answer(request):
    draft = small_model(request)
    if looks_ok(draft):
        return draft
    return large_model(request)

Three lines of logic, and every one of them hides a decision. What does looks_ok actually check? What happens when it's wrong in either direction? How long did the user wait when the first call was thrown away?

In production, those questions are the system. Here's where it tends to go wrong. (The scenarios below are illustrative, not drawn from a specific deployment.)

Failure mode 1: confident wrong answers never escalate

Escalation usually triggers on things you can detect: malformed JSON, a failed schema check, a refusal, a low self-reported confidence score. What it rarely catches is a well-formed, fluent, wrong answer.

Picture a support assistant that routes "simple" billing questions to the small model. A customer asks about a proration edge case. The small model returns a clean, polite, perfectly structured answer that quotes the wrong number. Every validator passes. The cascade reports a successful cheap resolution. Your dashboard says routing is working.

The failures your router can't see are the ones that cost the most.

Failure mode 2: "difficulty" isn't a feature you can compute

Many routers decide based on a proxy for difficulty: prompt length, keyword lists, a classifier trained on "easy vs. hard" labels someone made up in an afternoon.

But difficulty is relative to the model. A short prompt asking for a regex that handles a nasty edge case is hard. A long prompt asking to summarize a meeting transcript might be easy. If your routing signal is a vague human intuition about difficulty, you're encoding that intuition as a production decision without ever measuring it.

Failure mode 3: the cheap path can be the slow path

A cascade that escalates often pays twice: one call to the small model, then a full call to the large one. If your escalation rate on some task type is high, the "fast first pass" is adding latency, not removing it.

Average latency hides this. Look at end-to-end latency per task type, including retries and validation time, and the picture can flip: for some categories, going straight to the larger model is both faster and simpler.

Failure mode 4: treating all errors as equally expensive

A wrong answer in an internal draft generator costs someone thirty seconds. A wrong answer in a compliance summary or a code change that ships can cost far more. A router optimized purely on token spend will happily save cents on requests where an error costs real money.

What to actually measure

Evaluate the router as its own component, against real traffic, broken down by task type:

per task type:
  - quality of the small-model path vs. the large-model path
  - escalation rate
  - false "ok" rate (validator passed, answer was wrong)
  - p50 / p95 end-to-end latency, including retries
  - cost per request, and estimated cost of an error

The false "ok" rate is the one most teams don't have, because it requires sampling routed answers and grading them. It's also the one that tells you whether your cheap path is actually cheap.

Where to draw the line

A practical starting point:

Route to the small model where the output is verifiable (structured extraction, classification against a fixed label set) and errors are cheap or caught downstream.
Go straight to the large model where errors are expensive, hard to detect automatically, or where the measured escalation rate is high enough that the cascade costs latency.
Re-measure whenever the models, the prompts, or the traffic mix change. The boundary moves.

Model routing isn't a cost optimization with a classifier attached. It's a decision system, and it deserves the same evaluation discipline as any model you'd put in front of users.

How are you deciding where your cheap path ends? Measured boundaries, or gut feel that hasn't been revisited?

Human-in-the-loop isn't a safety net. It's a UX problem with a compliance label.

Dimitris Kyrkos — Mon, 12 Oct 2026 08:53:11 +0000

The diagram vs. the Tuesday afternoon

On the architecture diagram, human-in-the-loop looks great. The model makes a decision, a box labelled "human review" sits between the model and production, and everyone signs off on the design doc feeling safer.

Then the system ships. The reviewer is a real person with a real queue, and on a normal Tuesday afternoon that queue holds 340 items. Most of them are obviously fine. By item 40, they are reading the first line. By item 120, they are clicking "approve" on rhythm.

The box on the diagram is still there. The review isn't.

This is the gap: we treat the human as a component with infinite attention, when attention is the scarcest resource in the whole pipeline. Below are the failure modes I see most often, with illustrative examples (composites, not specific companies).

Failure mode 1: Everything goes to the human

The simplest design routes every model output to review. It feels conservative. In practice it is the least safe option, because it trains the reviewer that approval is the expected answer.

Illustrative example: an invoice-classification model is right 97% of the time. The reviewer sees 300 invoices a day, so roughly 9 are wrong. Those 9 are buried among 291 correct ones that look identical at a glance. After two weeks, the reviewer's override rate drops to near zero. Not because the model improved, but because the human stopped looking.

The fix is routing. Only escalate when the model is uncertain, or when the cost of being wrong is high:

def route(decision, confidence, impact):
    # High-impact actions always get a human, whatever the confidence
    if impact == "high":
        return "human_review"
    # Low confidence gets a human
    if confidence < CONFIDENCE_THRESHOLD:
        return "human_review"
    # Everything else is auto-approved, but sampled for audit
    if random.random() < AUDIT_SAMPLE_RATE:
        return "audit_queue"
    return "auto_approve"

Two details matter here. First, confidence alone is not enough: a model can be confidently wrong, and raw scores from many models are poorly calibrated. Check calibration on held-out data before you trust a threshold, and pair confidence with an impact rule. Second, the audit sample on auto-approved items is how you find out whether your threshold is still right.

Failure mode 2: The reviewer has to reconstruct the reasoning

The second failure is a review screen that shows the model's answer and nothing else. The reviewer now has two jobs: redo the task from scratch, or trust the answer. Under time pressure, they trust the answer.

Illustrative example: a loan-document extraction tool flags "annual income: 84,000". To check it, the reviewer opens a 22-page PDF and searches manually. They do it carefully twice, then stop.

A good review screen shows the exact inputs the model relied on: the highlighted span in the source document, the fields that drove the score, the rule that triggered the escalation. Verification becomes a glance instead of an investigation. If you can't surface what the model used, that is worth fixing before you add more reviewers.

Failure mode 3: Overriding is harder than approving

If "approve" is one click and "reject" opens a form with six required fields, you have designed a system that approves things.

The override path should be as cheap as the approve path: one action to correct, an optional short reason, and the corrected value captured automatically. Every override should be logged with the original output, the correction, who made it and when.

That log is not paperwork. It is the most useful dataset you have, because it tells you exactly where the model fails and whether your threshold is set in the right place.

Failure mode 4: Nobody measures the reviewers

If you don't track review behavior, you won't notice when the loop goes quiet. A few signals are worth watching:

Time spent per review (a sudden drop is a warning sign)
Override rate over time (near zero is suspicious, not reassuring)
Errors found in the audit sample that a reviewer had approved

These tell you whether the human in the loop is still doing anything.

The reframe

Human-in-the-loop is really an attention-allocation problem with a model in front of it. The model's job is to decide what is safe to skip. The interface's job is to make the remaining decisions fast and easy to get right. The log's job is to show you when either of those stops working.

We think about this a lot at Cyclopt, because every AI action that touches a codebase raises the same question: which ones need a person, and how do you make that person's review count?

So here's my question for you: what is the override rate in your human review queue, and do you actually know what it is?

Shipping faster with AI isn't engineering maturity. It's a demo that hasn't met year two yet.

Dimitris Kyrkos — Thu, 08 Oct 2026 06:53:18 +0000

Every AI rollout comes with a dashboard. PRs merged per week, time from ticket to prototype, model accuracy on the eval set. All of those numbers go up and to the right in the first quarter, and all of them are measuring the same thing: how fast you can produce something new.

None of them measure what happens to that something eighteen months later.

That's the gap. The demo version of a system is judged on how quickly it appeared. The production version is judged on whether it's still understandable, still stable, and still changeable after three teams, two reorgs, and a vendor price hike have happened to it. AI made the first part dramatically cheaper. It did nothing for the second part, and in some teams it made it worse.

Here are three tests I use instead of velocity. None of them are new. That's the point.

Test 1: can a new developer read it without an AI explaining it?

Illustrative scenario: a team generates most of a service with an assistant. It works. Six months later, a new hire is asked to change how refunds are calculated. They open the code and find four helpers named processData, handleResult, transformPayload and finalizeResponse, each one 200 lines, each one doing a slightly different slice of refund logic. Their first move is to paste it into a chat window and ask what it does.

That's the smell. If the only way to understand your codebase is to ask a model to summarize it, the code isn't documenting itself, and you've made comprehension a dependency on a tool.

What to check:

Can someone find where a business rule lives by reading file and function names alone?
Are the domain concepts (Refund, Invoice, Tenant) visible as types, or buried in dictionaries passed between generic helpers?
Would a code review catch a wrong rule, or would the reviewer also need to ask the model?

# Hard to read without help
def handle_result(data, cfg, flag=False):
    ...

# Readable on its own
def calculate_partial_refund(order: Order, returned_items: list[LineItem]) -> Money:
    ...

AI-generated code isn't the problem. Unreviewed, unnamed, unstructured code is, and AI lets you produce it at a volume nobody can review.

Test 2: does it run without someone babysitting it?

A mature system is boring in production. A system that needs a person to restart a worker every Tuesday, re-run a stuck job, or manually clear a queue is not stable. It's being held up by a human who hasn't been written into the architecture diagram.

Illustrative scenario: an ingestion pipeline built quickly with lots of generated glue code. No idempotency, no dead-letter queue, retries implemented three different ways in three places. It mostly works. "Mostly" means one engineer spends a few hours a week nudging it, and when that engineer goes on holiday, the on-call rotation discovers the runbook is "ask Sam".

What to check:

How many manual interventions did production need last month? Is anyone counting?
Are failures handled by design (retries with limits, idempotent handlers, clear failure states), or by people?
Could the system survive the departure of the person who knows its quirks?

Test 3: can you swap a core dependency without rewriting the business logic?

This one matters more now than it did three years ago, because the dependency most teams just added is the one most likely to change. Model providers change pricing, deprecate versions, and change behavior between releases. If your provider's SDK is imported in forty files and its response shape has leaked into your domain objects, switching is a rewrite.

// Business logic coupled to one vendor
const res = await openai.chat.completions.create({ ... });
order.summary = res.choices[0].message.content;

// Business logic depends on a capability, not a vendor
interface Summarizer {
  summarize(text: string): Promise<string>;
}
order.summary = await summarizer.summarize(order.notes);

The same applies to your database, your queue, your payment provider, and your auth vendor. The question isn't whether you'll ever swap them. It's whether the decision is yours to make, or already made for you by how deeply they're wired in.

The reframe

Velocity tells you how fast you can create code. Maturity tells you how cheaply you can live with it. AI changed the price of the first one. The second one is still priced in discipline: clear naming, stable failure handling, and clean boundaries around things you don't control.

AI is a tool. A very good one. But the systems still standing in five years will be the ones that would have been well built without it.

What's the one signal you trust more than velocity when you judge whether a codebase is actually healthy?

Your bugs used to burn CPU. Now they burn money.

Dimitris Kyrkos — Mon, 05 Oct 2026 09:03:13 +0000

In traditional software, a badly written loop gives you a CPU spike. Somebody gets paged, the graph goes back down, and the cost is a slow afternoon.

In an AI system, the same mistake is a billing event. Every iteration is a paid API call, and the meter doesn't care whether the call was useful. A loop that runs for ten minutes before anyone notices can cost more than the feature earns in a month.

The tutorial version of an LLM feature has one call, one response, done. The production version has retries, self-correction steps, agents calling tools that call models, and thousands of users doing all of that at once. Every one of those multipliers is a place where cost can run away from you.

Here are the three failure modes I see most often.

Failure mode 1: the self-correcting loop that never converges

Illustrative scenario: an agent generates JSON, validates it, and if validation fails, sends the error back to the model and asks it to fix the output. Reasonable pattern. Then a schema change ships, and one field becomes impossible to satisfy. The model keeps "fixing" the output, the validator keeps rejecting it, and the loop keeps going.

Nothing crashes. No exception reaches your error tracker. The only signal is the invoice.

Fix: every retry and correction loop gets a hard ceiling, in attempts and in tokens, and hitting the ceiling is a logged failure, not a silent fallback.

async function generateValid<T>(prompt: string, validate: (x: unknown) => T) {
  const MAX_ATTEMPTS = 3;
  const MAX_TOKENS_TOTAL = 20_000;
  let tokensUsed = 0;
  let lastError = "";

  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    const res = await callModel(prompt, lastError);
    tokensUsed += res.usage.totalTokens;
    if (tokensUsed > MAX_TOKENS_TOTAL) throw new BudgetExceeded(tokensUsed);

    try {
      return validate(res.output);
    } catch (e) {
      lastError = String(e);
    }
  }
  throw new MaxAttemptsExceeded(MAX_ATTEMPTS, lastError);
}

The same applies to agents: cap the number of tool calls and model calls per task, not just per request.

Failure mode 2: traffic scales linearly, cost doesn't

A traffic spike on a normal endpoint costs you some extra compute. A traffic spike on an endpoint that fans out to several model calls per request, each with a long context window, costs you multiples of that.

If one user action triggers a summarization call, a classification call, and an embedding call, your cost per request is the sum of three metered APIs. Double the users and you've doubled all three, before retries.

Fix: know your cost per request the same way you know your latency per request. Put a budget on the feature, and decide in advance what degrades when the budget is hit: a cheaper model, a shorter context, a cached answer, or no AI output at all.

Failure mode 3: no limits per user

Global rate limits protect your provider quota. They don't protect you from one user (or one script, or one bug in your own frontend that re-fires a request on every render) consuming a disproportionate share of it.

Illustrative scenario: a chat feature with no per-user cap. A single integration test, pointed at production by mistake, sends requests in a tight loop over a weekend. The global limit is never reached, so nothing alerts.

Fix: rate limits and token quotas per user, per API key, and per tenant. Treat them like you treat auth: not optional, and enforced at the edge, not deep in a handler.

Monitoring is now a finance problem

Uptime dashboards tell you whether the system is running. They don't tell you what it cost to keep it running in the last hour. For AI features, you need both:

tokens and spend per feature, per user, per tenant, in near real time
alerts on spend rate, not just error rate
a kill switch that turns a feature off when it crosses a threshold, without a deploy

The reframe: an LLM call isn't a function call. It's a purchase. Your architecture should treat every loop, retry, and fan-out as a spending decision with a limit attached.

If you don't have hard cost controls in production, you're not running a feature. You're running an open tab.

What's the most expensive loop you've seen an AI feature get stuck in, and what limit would have caught it?

Your AI feature isn't a feature. It's a dependency you don't control.

Dimitris Kyrkos — Thu, 01 Oct 2026 06:34:06 +0000

In the tutorial, the AI call always works. You import the SDK, paste in an API key, await a completion, and render the result. Ship it.

In production, that same line of code is a network call to someone else's infrastructure, running someone else's capacity planning, under someone else's incident response. Sometimes it's slow. Sometimes it rate-limits you. Sometimes it's just down.

The question isn't whether your AI provider will have an outage. It's what your users see when it does. If the answer is "a frozen screen" or "Something went wrong," the problem isn't the provider. It's the architecture.

Here are the failure modes I see most often, and what graceful degradation looks like for each.

Failure mode 1: the whole page waits on the model

Illustrative scenario: a project management app adds an AI-generated summary at the top of every ticket. The summary is fetched server-side before the page renders. One afternoon the provider's latency jumps from 2 seconds to 40. Now every ticket page takes 40 seconds to load, including for users who never read the summary.

The AI part was optional. The architecture made it mandatory.

Fix: never put a model call on the critical rendering path. Load the core page first, fetch AI output asynchronously, and give it a hard timeout. If it doesn't arrive, the slot stays empty or shows a quiet "summary unavailable." The ticket still opens.

Failure mode 2: one provider, hardcoded

If your code calls one vendor's SDK directly in twenty places, you have twenty single points of failure and no switch to flip.

Put a thin routing layer between your app and the providers, and make fallback a first-class behavior rather than a try/catch afterthought:

const chain = [
  { name: "primary", call: callPrimaryModel, timeoutMs: 8000 },
  { name: "secondary", call: callSecondaryProvider, timeoutMs: 8000 },
  { name: "local", call: callLocalModel, timeoutMs: 4000 },
];

async function complete(task: Task): Promise<Result | null> {
  for (const step of chain) {
    if (breaker.isOpen(step.name)) continue;
    try {
      const out = await withTimeout(step.call(task), step.timeoutMs);
      breaker.recordSuccess(step.name);
      return { ...out, servedBy: step.name };
    } catch (err) {
      breaker.recordFailure(step.name);
    }
  }
  return null; // caller must handle "no AI available"
}

Two details matter more than the loop itself. The circuit breaker stops you from hammering a provider that's clearly down (and paying the full timeout on every request). And null is a legitimate return value, which forces every caller to decide what happens without AI.

Failure mode 3: the fallback model gets the same job

A small local model (served through something like Ollama or llama.cpp) is a great last line of defense, but it isn't a drop-in replacement for a frontier model. Prompts tuned for one often produce garbage on the other.

Decide in advance which tasks are essential enough to fall back locally: classification, short extraction, simple rewrites. Give those their own prompts and test them. Everything else should degrade to "not available right now" rather than to a confidently worse answer.

Failure mode 4: the core features depend on the AI path

This is the one that turns a provider outage into your outage. Search that only works through embeddings. A form that can't be submitted until the AI validator responds. An onboarding flow that stalls because the welcome message is generated.

Every AI feature should have a non-AI path that still lets the user finish their job: keyword search behind semantic search, rule-based validation behind AI validation, a static template behind the generated one.

Resilience is a day-one decision

Graceful degradation is really just treating your AI provider like any other external dependency: timeouts, fallbacks, circuit breakers, and a clear answer for "what if it's gone?" The teams that struggle are the ones who treated the model as part of their own code instead of as a service they rent.

Retrofitting this later means touching every call site. Designing it in from the start means one routing layer and a few decisions.

What does your app actually do today when your AI provider goes down? Have you tested it, or are you assuming?

Half the AI agents in production are if-statements with a GPU bill

Dimitris Kyrkos — Mon, 28 Sep 2026 07:45:38 +0000

There's a new kind of technical debt, and it doesn't come from cutting corners. It comes from reaching for the most impressive tool in the room.

Call it resume-driven AI engineering: picking an agent framework, a vector database, or a multi-model orchestration layer because it looks great on a CV, not because the problem needs it. The result works in the demo. It's also slower, more expensive, harder to debug, and nondeterministic in places where it didn't need to be.

The demo vs. the pager

In a tutorial, complexity is free. You spin up an agent, wire in a vector store, and watch it do something clever with ten sample documents.

In production, every moving part has a cost: latency, token spend, a new failure mode, a new thing someone has to understand at 3 a.m. The question isn't "can an LLM do this?" (it usually can). It's "is an LLM the simplest thing that does this reliably?"

Here are three places where the answer is often no. The scenarios are illustrative, but if you've been around AI projects for a while, they'll look familiar.

Failure mode 1: a model call where a regex would do

A team needs to pull invoice numbers out of incoming emails. Invoice numbers follow a fixed format: INV- plus eight digits. They send every email to an LLM with a prompt asking it to extract the number.

It works 98% of the time. The other 2%, the model "helpfully" reformats the number, or picks up a purchase order number instead. Each call costs money and adds a few hundred milliseconds.

import re

INVOICE_RE = re.compile(r"\bINV-\d{8}\b")

def extract_invoice_ids(text: str) -> list[str]:
    return INVOICE_RE.findall(text)

Deterministic, testable, effectively free, and it runs in microseconds. Keep the model for the messy cases the pattern can't handle, and route to it only when the regex finds nothing.

Failure mode 2: vector search where SQL would do

"Show me all orders from customer 4417 in the last 30 days that are still unpaid."

That's not a semantic question. It's a filter. Yet it's common to see this kind of query embedded, pushed through a vector store, and answered by an LLM summarizing the top-k chunks, which may or may not include every matching order.

SELECT id, total, created_at
FROM orders
WHERE customer_id = 4417
  AND status = 'unpaid'
  AND created_at >= NOW() - INTERVAL '30 days';

Exact, complete, indexed, auditable. Vector search is great when you're matching meaning ("tickets similar to this complaint"). It's the wrong tool when you're matching facts.

Failure mode 3: an autonomous agent where a decision tree would do

A support workflow: if the customer is on the enterprise plan and the issue is billing, route to account management; if it's a bug, open a ticket; otherwise, send the FAQ link.

That's four branches. Someone builds it as an autonomous agent with tool access, a planning loop, and a memory store. Now the routing is probabilistic, occasionally loops, and nobody can explain why ticket #8812 went to the wrong team.

def route(customer, issue):
    if customer.plan == "enterprise" and issue.type == "billing":
        return "account_management"
    if issue.type == "bug":
        return "open_ticket"
    return "send_faq"

If you can draw the logic on a whiteboard, you probably don't need an agent to rediscover it every request.

The reframe

A lot of "AI systems" are really ordinary software with an LLM bolted onto a step that didn't need one. The model isn't the problem. Using it as the default instead of the exception is.

A practical decision checklist

Before adding a framework, a model call, or an agent, ask:

Is the input structured or the output fixed-format? Start with parsing, regex, or schema validation.
Is the question about facts or about meaning? Facts go to SQL. Meaning can go to embeddings.
Can the logic be enumerated? If yes, write the branches. Agents are for open-ended tasks where you genuinely can't.
What happens when it's wrong? If the answer is "silent bad data," you want determinism.
Who maintains this in a year? Every framework is a dependency someone has to upgrade, understand, and debug.

None of this means "never use AI." Use it where ambiguity actually lives: unstructured text, fuzzy matching, generation. Just make it the tool you reach for on purpose, not by reflex.

The best engineers aren't the ones with the most complex stack. They're the ones whose systems are still simple enough to understand when something breaks.

What's the most over-engineered AI setup you've seen (or built) that could've been replaced with something boring?

Your model doesn't need more training. It needs a better search index.

Dimitris Kyrkos — Thu, 24 Sep 2026 09:40:25 +0000

Intro

Every team that plugs an LLM into its business hits the same moment. Someone asks the model about an internal pricing rule, a product SKU, or a clause in the standard contract, and it answers confidently and wrongly.

The reflex that follows is almost universal: "The model doesn't know our business. Let's fine-tune it."

So the team spends weeks exporting tickets and docs, cleaning them, formatting them into JSONL, and paying for training runs. The new model sounds more like the company. It uses the right jargon. And it still makes things up.

That's not bad luck. It's a category error.

What fine-tuning is actually good at

Fine-tuning adjusts a model's weights so it behaves differently. That's powerful when the thing you want to change is behavior:

A specific tone or writing style (support replies that sound like your brand, not like a chatbot)
A strict output format (always return this JSON schema, always produce this report layout)
A narrow, repetitive task where a smaller tuned model can replace a bigger general one

Notice what all of those have in common. They're about how the model answers, not what facts it knows.

When the goal is "the model should know our refund policy, our product catalog, and last quarter's changes," you're asking weights to act like a database. They're a bad one.

Failure mode 1: it still hallucinates, just in your accent

A fine-tuned model doesn't gain a reliable sense of what it knows and what it doesn't. Training on your documents shifts probabilities toward your vocabulary, but when a question lands in a gap, the model does what it always does: it produces the most plausible-sounding continuation.

Illustrative scenario: a support assistant tuned on two years of tickets is asked about a warranty extension introduced last month. It has never seen it. It answers anyway, in perfect company voice, with the terms of the old warranty. The answer is more convincing than a generic model's would have been, which makes it more dangerous, not less.

Fluency in your domain is not the same as accuracy about your domain.

Failure mode 2: every fact change is a training run

Business knowledge isn't static. Prices change, policies get revised, products get deprecated, a regulation lands and three documents get rewritten.

If that knowledge lives in the weights, every update means rebuilding the dataset, retraining, re-evaluating, and redeploying. In practice, teams don't do that weekly. So the model drifts out of date, quietly, and nobody knows exactly which facts are stale.

Compare that with a retrieval setup: a document changes, you re-index it, and the next query sees the new version. The update cycle is minutes, not a project.

Failure mode 3: you can't show your work

When an answer comes from retrieval, you can point to the exact chunk of the exact document it was grounded in. You can show it to the user. You can log it. When someone disputes an answer, you can check whether the source was wrong or the model misread it.

When an answer comes from fine-tuned weights, there is no source. The knowledge is smeared across billions of parameters. You can't cite it, you can't audit it, and you can't tell a compliance team where a claim came from.

For anything customer-facing, regulated, or contractual, that alone should end the discussion.

What to build first instead

Before training custom weights, invest in the boring part: retrieval.

user question
   -> query rewriting / expansion
   -> hybrid search (keyword + embeddings) over a clean, deduplicated index
   -> reranking
   -> top-k chunks + source metadata into the prompt
   -> answer with citations

Most of the quality comes from things that have nothing to do with the model:

Clean source documents (no three conflicting versions of the same policy)
Sensible chunking that keeps related context together
Metadata (dates, owners, product, region) you can filter on
Hybrid search, because embeddings alone miss exact identifiers like SKUs and error codes
An evaluation set of real questions with known correct answers, so you can measure whether changes help

None of this is glamorous. All of it pays off more than a training run for knowledge-heavy use cases.

When fine-tuning does make sense

This isn't "never fine-tune." It's "fine-tune for the right reason, and usually later":

Retrieval is solid, answers are grounded, but the output format or tone keeps drifting. Tune for behavior.
You need a smaller, cheaper model to handle a narrow, high-volume task a big model already does well. Tune for cost.
The model struggles to use retrieved context well in your domain (for example, dense technical or legal text). Tune on examples of reading and citing context, not on the facts themselves.

In all three cases, facts still come from the index. The weights handle behavior.

The reframe

Fine-tuning teaches a model how to talk. Retrieval tells it what's true right now. Most "the model doesn't understand our business" problems are the second kind wearing the costume of the first.

What pushed your team toward fine-tuning (or away from it), and did it actually fix the knowledge problem you were trying to solve?

Your LLM has no memory. Your application had better have one.

Dimitris Kyrkos — Mon, 21 Sep 2026 10:05:47 +0000

Intro

Every LLM tutorial has the same shape. You send a list of messages, you get a reply, you append the reply, you send the list again. It works, it demos well, and it quietly teaches you the wrong mental model.

That loop is not state management. It is the absence of it, dressed up as a feature.

The APIs behind LLM platforms are stateless. Each request is independent of the last one. The model does not "remember" your conversation, it re-reads whatever you hand it, every single time. Meanwhile the thing you are building, a support flow, a code migration, an approval process, a multi-step agent, is stateful by nature. Somebody started it, it is halfway done, and it has to survive interruptions, retries and crashes.

That gap between a stateless processor and a stateful process is where production systems break. Here are the three places I see it break first.

1. "Just send the history" stops scaling

The tutorial answer to "how does the model know what happened earlier?" is to resend the whole transcript. It is fine for ten turns. At a hundred turns you are paying for the same tokens over and over, latency creeps up, and you start hitting context limits at the worst possible moment.

The usual patch is summarization: compress old turns into a paragraph and carry on. That helps with size but introduces a subtler problem. The summary is now the model's opinion of what mattered, and nothing in your system can verify it.

Imagine a multi-step refund workflow (illustrative scenario, not a real case). At turn 4 the user confirms the order ID. At turn 30 the summary says "user wants a refund for a recent order." The ID got compressed away, and the model now confidently picks the wrong one.

The fix is to stop treating the transcript as the state. Facts that the process depends on belong in a structured object your code owns:

@dataclass
class RefundState:
    order_id: str | None = None
    reason: str | None = None
    amount_confirmed: bool = False
    step: str = "collect_order"   # explicit position in the workflow

def build_prompt(state: RefundState, recent_turns: list[Message]) -> list[Message]:
    # The model gets the current state as input, plus only the turns it needs.
    return [system_prompt(), state_message(state), *recent_turns[-6:]]

Now the context you send is small, deterministic and reviewable. The model reads state, it does not store it.

2. The user interrupts halfway through

Real users do not wait politely for a long-running process to finish. They close the tab, change their mind, or send "actually, cancel that" while three tool calls are still in flight.

If the only record of progress is "whatever the model said last," you cannot answer basic questions. Which steps already ran? Which are safe to abandon? Does the half-finished action need to be rolled back?

A workflow that can be interrupted needs explicit steps with explicit statuses, persisted outside the model:

class Step(Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE = "done"
    CANCELLED = "cancelled"

# Persisted per run, updated by your code, never inferred from model output.
run.steps["reserve_inventory"] = Step.DONE
run.steps["charge_card"] = Step.RUNNING

When the interrupt arrives, your code reads the run record, decides what a cancellation means at this point, and tells the model the outcome. The model is not asked to work out where it was.

3. The API call fails in the middle of a transaction

Retries are where stateless design bites hardest. If a request times out after your tool executed but before you saw the response, resending it can execute the tool twice. Charge the card twice. Send the email twice. Open two tickets.

This is an old distributed systems problem, and the old answers apply: idempotency keys, an append-only event log, and recovery by replay.

def execute_tool(run_id: str, step_id: str, call: ToolCall):
    key = f"{run_id}:{step_id}"
    if (result := store.get_result(key)) is not None:
        return result                      # already ran, do not run again
    result = tools[call.name](**call.args, idempotency_key=key)
    store.save_result(key, result)
    return result

Notice that none of this depends on the model behaving well. That is the point. A model can be asked to "remember not to repeat itself," and it will still eventually repeat itself.

The pattern underneath

All three failures come from the same mistake: letting the conversation double as the system of record. Once you separate the two, the design gets much less mysterious.

State lives in your application: structured, persisted, versioned, recoverable.
The model is a stateless processor. It receives the relevant state, produces a proposed next action, and your code validates and applies it.
The transcript is a log for humans and for debugging, not the source of truth.

An LLM workflow is really a state machine with a language model on top of it.

Build it or lean on the platform?

Some providers now offer conversation or thread objects that manage history for you. They are convenient for prototypes and for simple chat. Before relying on them for a business process, ask a few questions:

Can you inspect and export the state in a form your own code can reason about?
Can you resume, replay or roll back a run after a failure?
Can you move to another model or provider without losing the process?
Who is accountable when the stored history and reality disagree?

If the answers are "not really," keep the state in your own store and treat the platform's memory as a cache at best. Managed history is fine as an optimization. It should not be where correctness lives.

Closing thought

If you rely on the model to remember the state of the conversation, your system will eventually break, and it will break in the least reproducible way available. Keep your application code as the source of truth and let the model do what it is good at: processing what you hand it, one request at a time.

Where does your team keep workflow state for multi-step LLM interactions: your own database, an event log, or the provider's thread objects? And what made you pick it?

Two-second latency isn't an AI problem. It's an architecture problem your stack was never built to hide.

Dimitris Kyrkos — Fri, 18 Sep 2026 06:25:25 +0000

Intro

There's a gap between the demo and the deployment that nobody puts in the tutorial.

In the demo, you call the model, you get a response, you render it. Done. In production, that same call sits in your request path for one, two, sometimes four seconds, and your users are staring at whatever you left on screen while they wait.

We spend weeks shaving milliseconds off database queries and tuning cache layers to get a page from 180ms to 90ms. Then we bolt an LLM call onto the same request/response cycle and it takes two full seconds to resolve. Nobody notices the discrepancy until users start bouncing.

Here's the actual gap: traditional backend work happens in a latency band humans don't perceive. LLM inference happens in a band they very much do. Same request/response pattern, completely different experience on the other end of it. If you don't change how you architect for it, users feel the friction immediately, no matter how good the model's output is.

Failure mode 1: the silent request

Picture a support-ticket triage feature. User submits a ticket, the backend calls an LLM to classify and route it, and the endpoint doesn't return until the model finishes. On a slow model day, that's a three-second white screen with no feedback. The user assumes the button didn't register and clicks again. Now you've got two calls in flight for the same ticket. This is a made-up example, but if you've shipped an LLM feature behind a synchronous POST, you've lived some version of it.

The fix isn't a faster model, it's not treating the call as instant in the first place.

Failure mode 2: streaming as an afterthought

Streaming gets treated as a nice-to-have polish pass instead of the default. Teams ship the blocking version first, get it working end to end, and plan to "add streaming later." Later rarely comes, because by then the blocking version is load-bearing and touching it means retesting the whole flow. Token-by-token streaming isn't a UX nicety here, it's the difference between "the interface feels instant" and "the interface feels broken."

Failure mode 3: heavy work still living in the request path

Some LLM work genuinely doesn't need to block the response at all: summarizing a document after upload, generating suggested tags, running a background analysis pass. These still get built as synchronous calls because that's the default pattern every other endpoint in the codebase already uses. Moving that work to an async job with a webhook or polling endpoint is often a bigger UX win than any amount of prompt optimization, because the user was never meant to wait on it.

Failure mode 4: no optimistic path forward

Even with streaming and async jobs in place, there's often no way for the user to keep moving while the model works. Optimistic UI, showing the next step as available, prefilling a draft state, letting the user act on a provisional result, keeps people in the workflow instead of parked on a spinner watching a progress indicator that means nothing to them.

The reframe

None of this is really an "AI latency" problem. It's a request/response architecture problem that AI just made visible, because AI is the first thing in most stacks slow enough that users notice the assumption baked into synchronous request handling. The fix isn't specific to LLMs: stream what you can, push what doesn't need to block into the background, and give the user something to do while they wait.

When you're deciding which technique to reach for: if the output needs to be seen as it's produced, stream it. If it doesn't need to block the next user action at all, take it out of the request path entirely. If it has to block but you can predict the shape of the result, use optimistic UI to bridge the gap.

We hit this directly while building the analysis pipeline behind Cyclopt: the moment an LLM call entered the loop, the whole interaction model around it had to change, not just the backend call itself.

How is your team handling this? Streaming everything by default, pushing more to background jobs, or still treating the LLM call like any other backend request?

Your LLM Isn't Bad At Math. It Was Never Doing Math In The First Place.

Dimitris Kyrkos — Tue, 15 Sep 2026 06:31:43 +0000

Your LLM Isn't Bad At Math. It Was Never Doing Math In The First Place.

In a tutorial, an LLM call looks like a function: pass in text, get back an answer, move on. It's easy to start treating the model like it's evaluating your business logic the same way a function would, deterministically, the same input always producing the same output.

In production, that assumption breaks in a specific, predictable way, and it's worth naming precisely why.

Traditional software runs on deterministic logic: if input A meets condition B, output C happens, every time, with a proof you could write on a whiteboard. An LLM runs on statistical probability: it predicts the most likely next token given the patterns in its training data. Those are two different kinds of "correct." The friction shows up exactly where a team asks the second kind to do the first kind's job.

Below are the places that friction actually shows up in a live system. These are illustrative scenarios based on the shape this failure takes, not one specific incident.

Asking a model to do arithmetic is asking it to guess what arithmetic usually looks like

Say a pipeline extracts line items from a batch of invoices, then asks the same model call to also compute the subtotal, apply tax, and return a total. On short, simple invoices, it's right almost every time, because short simple math is heavily represented in training data and easy to pattern-match. On a 40-line invoice with mixed tax rates and a rounding rule, it starts being right most of the time, which is a different and much worse thing than right. Nothing throws an error. The number is just plausible instead of correct, and "plausible instead of correct" is invisible until someone reconciles the books.

A statistically-applied business rule is not the same rule as a logically-applied one

Say the business rule is "refunds are allowed within 30 days of purchase." Put that rule in a prompt and ask the model to decide eligibility case by case, and it will get the easy cases right: a purchase from six months ago, obviously no; a purchase from yesterday, obviously yes. Where it gets interesting is the boundary: day 29, day 30, day 31, across time zones, with a purchase timestamp in one format and today's date passed in another. A deterministic date comparison gets this right every single time by construction. A model is producing its best guess at what "a refund decision near the boundary" looks like, based on how those decisions were phrased in its training data, and that's a meaningfully different operation even when it happens to output the correct answer nine times out of ten.

The output still looks like a normal response, which is exactly the problem

This is what makes the failure mode dangerous rather than just annoying: a wrong statistical answer doesn't look different from a right one. It's the same JSON shape, the same confident tone, the same absence of an exception. There's no signal in the response itself that tells you this was a guess rather than a computation. You find out later, from a customer service escalation or a finance reconciliation, not from anything in your logs.

Where the line actually goes

None of this means don't use the model near your business logic. It means be precise about which half of the job you're handing it.

What the model is good at: turning unstructured input, an email, a contract clause, a support ticket, a scanned form, into structured data. That's genuinely ambiguous work: language is fuzzy, humans phrase the same request ten different ways, and a statistical model that's seen millions of phrasings is the right tool for mapping "hey can I send this back, it's been like a month" into structured fields like intent and stated days since purchase.

What the model should never be the last word on: the validation, the calculation, and the rule enforcement that runs on that structured data once you have it. That's deterministic code's job, on purpose, with a strict schema at the boundary so a malformed or out-of-range extraction fails loudly instead of quietly flowing downstream as a confident-looking guess.

We lean on this split in how Cyclopt's automated checks work: the model interprets unstructured signals in a codebase or a pull request, but the actual rule enforcement, the pass or fail line, runs through deterministic logic against a defined schema, not through the model re-deciding the rule each time. The interpretation layer changes. The rule layer doesn't get to.

The reframe

Statistical vs. logical correctness isn't a model-quality problem that gets solved by a better model. It's an architecture decision, and better models make it easier to ignore, not less necessary to make. Let the model handle the ambiguity. Let your code handle the rules.

Where do you actually draw that line in a system you've shipped? What's the one thing you learned the hard way should never have been the model's call to make?