The label that gets the sequence backwards
"AI-first" has become a branding exercise. Startups put it in their pitch decks. Enterprise t...
For further actions, you may consider blocking this person and/or reporting abuse
Strong piece, and the friction-first framing is right. I would push it one level deeper. Most AI-first failures I get called to clean up are not "should have been a rule engine." They are one monolithic LLM call doing a job that was 90 percent deterministic parsing and 10 percent real ambiguity. The fix is not AI or no AI, it is drawing the line inside the problem so the model only owns the slice that genuinely needs judgment, and everything it returns gets checked against something deterministic.
The 91 versus 95 example lands for a second reason too. A rule engine that is wrong 5 percent of the time knows its boundaries. A model that is confidently wrong 9 percent of the time, with no signal for which 9 percent, is worse, because you cannot even route around it. Calibration, the model knowing when it does not know, is usually worth more than a few points of raw accuracy.
The "draw the line inside the problem" framing is more precise than what I wrote. You're right that the real decision isn't AI vs no AI, it's identifying which 10% of the problem actually needs judgment and isolating the model to just that slice. The other 90% should be deterministic because it can be. That architecture is also way easier to debug because when something goes wrong you know immediately whether the failure was in the deterministic layer (your bug) or the model layer (ambiguity the model mishandled). The calibration point is the sharper version of the same argument. A rule engine with known failure boundaries lets you build routing around those boundaries. A model that's confidently wrong with no uncertainty signal gives you no information about when to trust it, which means you either trust it always or verify it always, and both of those defeat the purpose.
On the trust-always-or-verify-always fork, there is a third door, and it is the one that made calibration practical for me. The model rarely volunteers a usable uncertainty signal, self-reported confidence is just more sampled text from the distribution that produced the answer. So stop asking it, and build the signal outside the model from properties you can check without trusting it: does the output satisfy the schema, is every claim traceable to the source you handed it, does it survive a few resamples. None of those ask the model to grade itself.
That also softens "verify always defeats the purpose," since the checks are not one cost. Schema and grounding are cheap and deterministic, so run them on every call; resampling is the expensive one, so spend it only on the slice the cheap checks flagged. And set the threshold on the cost of a false accept, not raw accuracy, since a wrong value that reads clean and ships is a different kind of expensive from one you caught.
The "stop asking the model to grade itself" point is important and I should have been clearer about that in my response. Self-reported confidence is just more generated text, it has no privileged relationship to actual correctness. Building the uncertainty signal externally from schema validation, source traceability, and resampling is the right architecture because those checks are independent of the thing being checked.
The tiered cost model is practical and fixes the objection I raised about verification defeating the purpose. Schema and grounding checks on every call is essentially free compared to inference cost, so it's not "verify everything" in the expensive sense, it's "run the cheap deterministic filter on everything and spend the expensive resampling budget only where the cheap filter can't give you a clear answer." That's the same pattern as the 90/10 split from your first comment applied to the verification layer instead of the generation layer.
Setting the threshold on cost of false accept rather than raw accuracy is the framing that ties it all together. A wrong answer that looks right and ships undetected is a completely different cost from a wrong answer that got caught at the schema check. Same error rate, completely different business impact. Most teams set accuracy targets without weighting for how expensive each type of failure actually is.
Exactly, and one step further: once you price the false accept instead of accuracy, the threshold stops being hand-tuned and falls out of the numbers. Put a cost on a wrong answer that ships and a cost on a human review, and the optimal cut is where the marginal item's expected false-accept cost crosses the review cost. The routing coordinate becomes arithmetic.
The reason teams still default to accuracy is not that they missed this. Accuracy needs no such number; this needs the price of a wrong answer nobody caught, and that number lives with the business, not engineering, so it never gets set. The math was never the hard half.
"The math was never the hard half" is the perfect closing line for this entire thread. The engineering for tiered verification, external calibration, cost-weighted thresholds, all of that is buildable the moment someone provides the input number. The input number is "what does it cost us when a wrong answer ships undetected" and that's a business question that requires a product manager or a domain owner to sit down and actually quantify it. Most never do because it forces them to put a dollar amount on a failure scenario they'd rather not think about concretely. So engineering defaults to accuracy because accuracy is self-contained, no business input required, and the system stays unoptimized because the optimization function was never fully specified. This thread basically mapped the entire architecture from "AI vs no AI" down to "the reason production AI systems underperform is that nobody asked the business to price their own failure modes." That's a good place to land.
Very true, and AI is just the latest instantiation of this recurring behavior pattern. Every new technology wave comes with it. Part of the problem is that we, as engineers, are forced to stay up to date, and our current project is often the only chance to gain real-world exposure to a new technology. The same is true for managers. If you want to climb the corporate ladder, you better make sure your team uses the latest tech. This problem will remain as long as we have careers, but maybe AI will "solve" that one day 😆
The career incentive angle is the part nobody wants to say out loud but it drives more technical decisions than anyone admits. "We used AI" looks better on a resume than "we solved it with a cron job and three SQL queries" even if the second one was the better engineering decision. Same goes for managers, "led AI transformation initiative" gets you promoted, "prevented team from over-engineering with AI" doesn't even get mentioned. Until "chose the right tool for the problem" carries the same career weight as "used the exciting tool," the pattern will keep repeating with every new technology wave. And yeah, maybe AI will eventually solve the problem of humans adopting AI for the wrong reasons. That would be appropriately recursive.
the 'prompt that worked in March produces different results in September' line is the maintenance cost nobody puts in the estimate. we've seen this exact pattern — got budget for the build, none for the drift.
the friction first framing works for individual features, tbh. the part that gets harder at org scale is that 'what is slow or error prone right now' often has 6 different answers across 6 teams, and AI projects need a sponsor who can pick one and defend it. without that, friction first becomes friction wherever the loudest stakeholder points.
what's your heuristic for distinguishing friction that AI actually resolves vs friction that needs a process fix instead?
Really great read! I think the core issue is a disconnect between strategy planners and the teams who actually build things. Also, there’s a natural pattern for tech evolution: quantity always comes before quality—you can’t nail refined, well-fit AI solutions without running through lots of messy attempts first.
The strategy-builder disconnect is real and it's usually the root cause of the over-engineering pattern. The people deciding "we should use AI" aren't the ones who have to maintain the pipeline when it drifts. Your point about quantity before quality is interesting and probably true at the industry level. We're still in the phase where teams need to build a lot of bad AI implementations before the patterns for good ones become obvious. The problem is that each of those messy attempts costs real money and engineering time, so the faster teams can learn which problems actually need AI and which don't, the less they waste on the learning curve. Thanks for reading.