The Vertical AI Bubble: We Keep Forgetting That LLMs Roll Dice

I’ve spent the last year watching an entire generation of vertical AI startups get funded, launched, and celebrated. AI for law firms. AI for insurance claims. AI for radiology. AI for accounting. AI for customer support in a specific niche of healthcare billing. Every pitch deck looks the same: take a foundation model, wrap it in a narrow workflow, add some proprietary data, call it a moat, and raise a Series A at a valuation that assumes this thing behaves like software.

That last part is where I think we’ve collectively lost the plot. It’s the reason I’ve become genuinely worried we’re building a bubble on a foundation that doesn’t hold the weight we’re putting on it.

The Quiet Assumption Everyone Is Making
Every vertical AI product, no matter how it’s marketed, is quietly making a bet: that if you constrain a large language model tightly enough, with prompts, fine-tuning, retrieval, guardrails, evals, you can make it behave deterministically enough to run a business process. Approve the claim. Flag the contract clause. Code the diagnosis. Reconcile the ledger.

The word enough is doing an enormous amount of work in that sentence, and I don’t think most people building these companies have sat with what it actually means.

Traditional software is deterministic. Give it the same input, you get the same output, every time, forever, unless someone changes the code. That’s the property that made SaaS investable in the first place. You could underwrite reliability. You could say a system does X and mean it categorically, not statistically.

LLMs don’t have that property. They are, structurally, probabilistic next-token predictors. Even at temperature zero, even with the same prompt, the same weights, the same everything, you are sampling from a distribution over possible outputs, not executing a fixed procedure. Most of the time the sample looks like the procedure. Some of the time it doesn’t. Critically, you often can’t tell which is which just by looking at the output. A wrong answer can be just as fluent and confident as a right one.

Why This Matters More in Vertical Than in Horizontal
For a general chatbot, this is a minor annoyance. If I ask a general-purpose assistant to summarize an article and it makes a small factual slip, I read it, I notice something’s off, I double check, life goes on. The failure mode is visible and low stakes.

Vertical AI is different by design. The entire pitch is removing the human from the loop in a specific, repeatable business process, because that’s where the margin is. Underwriting insurance policies. Approving or denying medical claims. Drafting legal filings. Auto-coding invoices. Triaging patients. The value proposition is explicitly that you should trust the system to make the judgment call so a human doesn’t have to.

But you can’t have it both ways. You can’t sell the idea that you removed the human because the model is reliable enough to be trusted, while also being honest that the underlying technology is probabilistic and will occasionally produce a confidently wrong answer that looks exactly as polished as a correct one. That tension doesn’t go away because you fine-tuned on fifty thousand examples of your domain. It gets quieter, not smaller. The failure rate might drop from 8% to 1%, but at scale, across millions of insurance claims or legal documents, 1% is a catastrophic number of wrong, confidently stated decisions. Worse, it’s a 1% you often can’t predict in advance, because it isn’t correlated with the kinds of inputs a human would flag as hard.

The Bubble Mechanics
Here’s how I think the bubble actually inflates, step by step.

Step one: the demo works. Vertical AI demos are almost always built on cherry-picked or lightly adversarial-tested examples. Probabilistic systems are extremely good at looking deterministic in a demo, because you’re implicitly sampling from the part of the distribution where the model is confident and correct. Nobody demos the failure mode on purpose.

Step two: the pilot works, mostly. Early customers run it on a limited slice of real data, with a human reviewing outputs for now. The error rate looks acceptable because the review step is quietly absorbing the probabilistic tail. The pitch to investors and to the customer’s leadership is that the review step is temporary, a bridge to full autonomy.

Step three: the humans get removed, or their review gets rubber-stamped. This is where the economics of the vertical AI thesis actually kick in. The human-in-the-loop step is expensive, and it’s the thing standing between the current valuation and the ten-times-cheaper-than-a-human-analyst valuation everyone is underwriting. So it gets thinned out. Review sampling drops from 100% to 10% to spot checks. Fatigue sets in, and rubber-stamping becomes the norm, the same way it does with any repetitive QA task performed on a system that’s right the overwhelming majority of the time.

Step four: the tail event happens at scale. These aren’t independent coin flips reviewed one at a time by a careful human anymore. They’re a probabilistic system operating with a thinned safety net across a huge volume of consequential decisions. Somewhere in there is a wrongly denied medical claim, a hallucinated case citation in a legal filing, a misclassified transaction that cascades through a ledger. It isn’t a bug in the traditional sense. It’s the expected behavior of a probabilistic system finally showing you its distribution.

Step five: the moat turns out to be thin. Once trust cracks, through a lawsuit, a regulatory inquiry, a viral failure story, the vertical AI company discovers that its actual defensibility was never the model, since everyone has access to roughly the same foundation models, and was never really the proprietary data either, which turns out to matter less than advertised for general reasoning tasks. The moat was the appearance of reliability. Appearance is not a moat once the illusion is punctured.

I think we are somewhere between step two and step three across a huge number of vertical AI companies right now, and I think this is exactly the mechanism that will separate the ones that survive from the ones that don’t over the next few years.

But the Models Are Getting Better
They are. I’m not arguing that LLMs are stagnant or that this is a permanent ceiling. I use these systems every day and the trajectory is real. But better in this context usually means the distribution shifts so that correct outputs are more probable and errors are rarer, not that the system becomes deterministic. Those are fundamentally different claims, and vertical AI marketing consistently conflates them.

Even a model that’s right 99.9% of the time is still a probabilistic system, not a deterministic one, and the businesses being built on top of it need to be architected as if failures are a certainty at scale, not as an embarrassing edge case to be quietly patched later. A 99.9%-accurate system processing ten million claims a year is still generating ten thousand wrong decisions a year, delivered with exactly the same confident tone as the correct ones.

There’s also a subtler problem. Improvements on benchmarks don’t necessarily transfer to the narrow, high-stakes, out-of-distribution edge cases that vertical products actually live or die on. A model can get dramatically better at general reasoning while still failing in eerily consistent ways on the specific messy, ambiguous, adversarial inputs that show up in real insurance claims or real contract redlines, which are exactly the inputs a vertical AI company was supposedly built to handle better than a generalist model.

The EU AI Act Is the Bill Coming Due
If everything above is the slow-motion mechanism, the EU AI Act is the thing that pull the timeline forward violently, at least for anyone operating in or selling into Europe. When I read it closely, I keep coming back to the same conclusion: for a lot of vertical AI as currently built, this regulation isn’t a compliance cost. It’s closer to a death warrant.

Here’s why, and it comes back to the exact same probabilistic-versus-deterministic tension.

Most vertical AI use cases fall straight into high-risk. Credit scoring, insurance underwriting and claims, employment decisions, medical triage, law enforcement-adjacent tools, access to essential services. This is almost a checklist of the categories vertical AI startups have been racing into for the last two years, and it’s almost exactly the high-risk AI system list under the Act. That’s not a coincidence. Those are the domains with the biggest, most attractive inefficiencies to automate. They’re also precisely where regulators decided the stakes were too high for trust-us-the-eval-scores-were-good.

High-risk systems don’t get to be black boxes anymore. The Act’s requirements read like a direct rebuttal to how most of these products are actually built. A documented risk-management system covering the entire lifecycle, not a one-time eval before launch. Training data governance, meaning provenance, bias examination, and gaps, that most teams fine-tuning or doing retrieval on scraped or licensed domain data simply cannot fully reconstruct after the fact. Technical documentation detailed enough for a regulator to reconstruct why the system produced a given output. Automatic logging sufficient to trace decisions after the fact. Human oversight that’s real and structurally capable of overriding the system, not a rubber-stamp UI. Accuracy, robustness, and cybersecurity levels that have to be disclosed, not just claimed in a pitch deck.

Every one of those requirements assumes you can explain, reconstruct, and audit a specific decision after the fact. That assumption is trivially true for deterministic software: you look at the code path and the logs and you’re done. It’s not straightforwardly true for a large language model. You can log the prompt and the output. You generally cannot produce a causal account of why the model landed on one token sequence instead of another, because that reasoning is distributed across billions of parameters shaped by training data you often don’t fully control and can’t fully inspect. Explainability research is real and improving, but it’s nowhere near mature enough to satisfy reconstruct-this-specific-high-stakes-decision-for-a-regulator as a routine operational capability.

This is the same probabilistic problem, just wearing a legal costume. My whole argument so far is that vertical AI companies have been selling the appearance of determinism built on top of a genuinely probabilistic substrate. The EU AI Act, in effect, forces that gap into the open and puts a legal price tag on it. You can’t paper over the fact that the model is probabilistic and you don’t fully know why it said what it said with better UX or a confident dashboard, when a regulator or a plaintiff’s lawyer is asking you to produce the audit trail for a specific denied claim or a specific rejected loan. Saying the model thought this was the most likely correct answer, based on patterns in training data you can’t fully enumerate, doesn’t satisfy an auditability requirement. It’s a restatement of the exact problem the requirement exists to catch.

The penalties are sized to be existential, not incidental. Fines for the most serious violations run up to the higher of a fixed cap in the tens of millions of euros or a percentage of global annual turnover, the kind of number that isn’t a line item, it’s a company-ending event for a startup still burning venture money. For a vertical AI company whose entire go-to-market has been trust our model’s judgment instead of a human’s, that liability profile changes the unit economics of the business overnight. Suddenly the cost isn’t just inference and fine-tuning. It’s building, and proving to a regulator, an entire scaffolding of logging, human oversight, and documentation around a system that was pitched, and priced, as if it didn’t need one.

The uncomfortable fork this creates. I think this regulation forces every vertical AI company operating in high-risk categories into one of two positions, and neither is comfortable. Either they genuinely rebuild around the fact that the core decision-making component is irreducibly probabilistic, which means heavy human-in-the-loop by design, not by pretense, real logging and explainability infrastructure built in from day one, and honestly accepting that some categories of decision may never be fully delegable to the model at all. That’s expensive, slows down the we-replaced-the-human pitch, and undercuts the exact margin story that got them funded. Or they keep operating the way they do now and quietly accept regulatory risk as a cost of doing business, a bet that works until the day it very publicly doesn’t, in a jurisdiction with both the will and the fine schedule to make an example of someone.

Either way, raise fast, automate the human away, worry about compliance later, which has basically been the vertical AI playbook, doesn’t survive contact with a regulatory regime built explicitly to demand something a probabilistic system cannot, by its nature, fully supply: a complete, reconstructable account of why it produced one specific output rather than another.

Why I Think This Has to Be Solved in the Open
I don’t want to end on pure doom, because I don’t think the answer is that vertical AI is impossible. I think the answer is that the infrastructure layer everyone skipped, the boring, unglamorous plumbing of actually recording and reconstructing what an agent did and why, is exactly what closes the gap between a probabilistic model and an auditable system. This is the layer I’ve been spending my own time on, and I want to be upfront that what follows is me talking about my own project, not a neutral third-party recommendation.

I’m the founder of ZizkaDB, an operational database built specifically for AI agents, and the EU AI Act is the reason it exists in its current form. Rather than pretending the underlying agent is deterministic, it takes the opposite approach: treat every agent decision, tool call, and outcome as a causally linked event, so instead of a single opaque output you get a reconstructable chain of why the system landed where it did, queryable after the fact, not reconstructed from memory, screenshots, or application logs that were never designed for this.

It maps directly onto the specific articles I’ve described as the hard part in this piece, not as a marketing gloss but as a point-by-point correspondence. Continuous event logging and session reconstruction against Article 12’s automatic logging and traceability duties. Causal lineage and drift detection feeding the kind of ongoing risk assessment and post-market monitoring Articles 72 and 79 expect, rather than a one-time launch eval. Dashboards and point-in-time state retrieval giving deployers the transparency Article 13 requires and the human oversight Article 14 assumes is actually possible. Tamper-evident event checksums speaking to Article 15’s accuracy and robustness requirements rather than just the audit-trail ones. You can see the full mapping here.

But here’s the part I want to be honest about, and it’s the actual reason the project is open source. I don’t think this is a problem any single team or company gets to solve alone. Auditability for probabilistic systems isn’t a feature you bolt onto a product roadmap, it’s infrastructure, in the same sense that logging, observability, and version control are infrastructure. Nobody expects a single company to own the one true observability standard for distributed systems, and I don’t think anyone should expect that for agent auditability either. The moment this becomes proprietary, closed, single-vendor infrastructure, it becomes exactly the kind of black box the EU AI Act is designed to distrust, and it puts every company depending on it at the mercy of one vendor’s roadmap, pricing, and continued existence, for a compliance obligation that isn’t going away.

That’s why ZizkaDB is built as an open-core engine on time-tested, well-understood pieces like Postgres, pgvector, and Qdrant, under an AGPL-3.0 license, with the full source available on GitHub. You can self-host it entirely inside your own infrastructure, which matters enormously for anyone thinking seriously about data residency and not wanting their causal decision logs living in someone else’s cloud. It’s model-agnostic by design, because the point was never to lock you into a particular LLM provider. The point is to give you a durable, inspectable record layer no matter which model happens to be making the decisions underneath.

To be fair to where the real work still is: no single tool, mine included, makes an AI system compliant. There’s still real governance work, risk classification, a quality management system, a genuine conformity assessment, that no database can substitute for, and I’d be doing exactly the thing I criticized earlier in this piece if I pretended otherwise. What I do think is that the auditability problem underneath all of that governance work is too large, too foundational, and too urgent to be solved by any one vendor behind a paywall. It needs to be solved the way the internet’s actual infrastructure got solved, in the open, with the people who have the problem contributing to the tool that fixes it, not waiting for a single startup to get it exactly right on the first try.

That, to me, is the difference between vertical AI companies that survive contact with the EU AI Act and the ones that don’t. Not better models. A better, shared, inspectable paper trail for the imperfect ones we already have.

What I Think Actually Survives This
I don’t think the answer is to stop building vertical AI. I think the answer is being honest about what kind of system you’re actually building, and designing around the probabilistic reality instead of marketing around it.

The companies I find myself more optimistic about are the ones that treat the LLM as a fast, cheap, fallible component, permanently probabilistic, never fully tamed, and design the rest of the system around that fact instead of around the wish that it weren’t true. Hard constraints and verifiable checks on the decision boundaries that matter most. Human review calibrated to actual risk rather than vibes. Honest reporting of confidence and error rates instead of a dashboard that implies certainty. The value they’re selling isn’t that they replaced the human. It’s that they made the human dramatically more leveraged, and they know exactly where the model’s judgment needs a second set of eyes.

The companies I’m worried about are the ones whose entire cap table is quietly betting that probabilistic, if you squint, will keep behaving like deterministic long enough for them to reach the next round. That bet works fine for a while, because most of the time, the model is right. That’s exactly what makes it dangerous. It’s the most-of-the-time businesses that make bubbles, right up until the tail shows up.

I don’t know exactly when this particular bubble pops, or whether it pops all at once or leaks out slowly through a hundred smaller stories that never quite make national news. But I’m fairly confident of the mechanism, because it isn’t really about AI at all. It’s the oldest story in finance: you can price a system as if it’s reliable for a surprisingly long time, and the market only really finds out it wasn’t when the tail event finally arrives.

The technology isn’t the problem. The pretending is.