DEV Community: The AI Downside

AI Hallucinations Are Still Not Solved

The AI Downside — Sat, 15 Aug 2026 21:55:04 +0000

With every major model release comes the same reassuring note: hallucinations are down, reliability is up, the fabrication problem is largely behind us. And every release, within days, someone posts a screenshot of the new model inventing a citation, a quote, a case, a statistic or a person with total, serene confidence. The rate improves. The category does not disappear. It is worth understanding why, because the gap between “less often” and “solved” is where the real damage happens.

It is not a bug, which is the uncomfortable part

A hallucination is not a glitch the way a crash is a glitch. Large language models generate text by predicting plausible continuations, and a plausible continuation is not the same thing as a true one. The model has no separate store of verified facts it checks against; it has patterns, and a fabricated citation in exactly the right format is, to the model, an excellent pattern. It is doing precisely what it was built to do. The falsehood and the truth are produced by the identical process, which is why the model is equally confident about both.

The model is not lying, because lying requires knowing the truth. It is producing the most likely-looking answer, and likely-looking is a different target from true.

The failure mode gets worse exactly where you can check least

Hallucination is not evenly distributed, and its distribution is perverse. Models fabricate most readily in precisely the situations where you are least equipped to catch them: obscure topics, niche technical details, specific figures, recent events, and anything at the edge of what was well represented in training. Ask about something popular and well-documented and the answer is usually solid. Ask about something rare — the exact thing you turned to the tool for because you did not know it — and the fabrication rate climbs, while your ability to notice drops to zero. The model is most confident and least reliable in the same dark corners where you have no independent way to tell.

This inverts the trust you would place in a human expert. A knowledgeable person becomes visibly hesitant at the edge of their competence — they hedge, they qualify, they say “I'd want to check that.” The model does the opposite: it maintains identical fluency and confidence whether it is on firm ground or inventing wholesale, offering no tell at the exact moment a tell would matter most. The uniformity of its confidence is not a cosmetic flaw. It is the specific property that makes the fabrications dangerous, because it strips away the single cue humans have always used to calibrate how much to believe.

Confidence is the dangerous ingredient

If these systems hedged — “I think, but I am not sure” — hallucination would be a manageable nuisance. The problem is that the fabrications arrive in the same fluent, assured, well-structured prose as the correct answers. There is no tell. A made-up legal case cites a plausible court and year. An invented statistic sits at a believable number. The interface offers no way to distinguish the two, because the model itself cannot.

This is why hallucination has produced genuine, documented harm rather than just funny screenshots. Lawyers have been sanctioned for filing briefs containing citations to cases that never existed, produced by a chatbot and not checked. That is not a hypothetical; it has happened in real courtrooms, more than once, because the fabrications were persuasive enough to survive a busy professional's glance.

The fixes help, and none of them close the gap

The industry's mitigations are real and worth using, but each has a ceiling:

Retrieval — grounding answers in real documents fetched at query time — genuinely reduces fabrication, but the model can still misread, misquote or over-extrapolate from the very sources it was handed.
“Reasoning” models that work through problems step by step catch some errors and confidently reason their way into others.
Citations in answer engines are only as good as the check you do on them, and a fabricated-but-formatted citation defeats the reader who trusts the format.

All of these lower the rate. None of them change the underlying fact that the system's job is to produce plausible text, and plausible text is sometimes false. You cannot fully suppress a behaviour that is identical, mechanically, to the behaviour you want.

The honesty we are owed

The complaint here is not that the technology hallucinates — that is inherent, and understood. The complaint is the marketing gap. When a release is sold on “dramatically reduced hallucinations,” a reasonable person hears “I can now trust this.” What is actually true is “it will fabricate slightly less often, still with total confidence, still undetectably.” Those are very different messages, and the second one is the one that would keep the sanctioned lawyer out of trouble.

Fluency is being mistaken for competence

Part of why hallucinations do so much damage is a very human bug, not a machine one: we are wired to read fluent, confident, well-organised language as a sign of knowledge. For all of history, someone who could explain a thing clearly and without hesitation usually understood it, because producing fluent expertise required actually having the expertise. Large language models sever that link. They produce the fluency without the understanding, and our instinct to trust the fluency fires anyway. The model exploits a shortcut in human judgement that was reliable right up until a machine learned to fake the surface.

This is why “just be more careful” is weak advice. The failure is not laziness; it is that the single most useful cue humans have for calibrating trust — confident fluency — has been rendered meaningless in this context, and no replacement cue has taken its place. You cannot feel your way to whether an answer is true, because the feeling of truth and the feeling of fabrication are now identical. The only defence is external verification, which is slow, effortful, and exactly the labour the tool promised to save.

The trap of automation complacency

There is a well-documented human tendency that makes all of this worse over time: the more reliable a system usually is, the less we scrutinise it. It is called automation complacency, and it is why people drive into rivers following a satnav. A model that is right ninety-something percent of the time is, perversely, more dangerous than one that is right half the time, because the high base rate lulls you. You check the first ten answers, they are all fine, you stop checking — and the eleventh, the fabricated one, sails straight through the scrutiny you have quietly abandoned.

This means the better these systems get, the more the residual errors matter, because they arrive inside a wall of correctness that has trained you not to look. A world of mostly-right AI is not a world where hallucinations stop mattering; it is a world where they become harder to catch precisely because they are rarer. The improvement in the average case erodes the vigilance you would need for the bad case, which is the specific reason “it hardly ever gets things wrong now” is cold comfort rather than reassurance.

“I don't know” is the feature nobody ships

The single change that would do most to tame hallucination is also the one the incentives fight hardest: a model that reliably says “I don't know” when it doesn't. Humans trust experts partly because good ones admit the edge of their knowledge, and a system that could do the same — hedging where it is uncertain, declining where it is guessing — would let users calibrate exactly where calibration is needed. The technology to express uncertainty is not the barrier. The barrier is that uncertainty demos badly. A model that frequently says “I'm not sure” looks less impressive on stage and in comparisons than one that answers everything with breezy confidence, even when the confidence is unearned.

So the market quietly rewards the wrong trait. Confident-and-sometimes-wrong beats hesitant-and-honest in a side-by-side demo, in a preference leaderboard, in the gut impression of a new user — and the products are tuned, consciously or not, toward the confidence that sells. This is the deep reason hallucination persists beyond its technical roots: honesty about uncertainty is a competitive disadvantage in the way these systems are currently judged. Until buyers start actively prizing a model that knows its limits — and penalising one that bluffs — vendors will keep shipping the smooth, assured voice that fabricates rather than the humble one that hedges. The fix is partly technical, but it is also a matter of what we, collectively, decide to reward. Right now we reward the bluff.

Use the tools. They are useful. But treat every specific, checkable claim — a citation, a date, a quote, a number, a name — as unverified until you have verified it yourself. That is not cynicism; it is the correct operating procedure for a system whose fluency is not evidence of its accuracy. Until a model can reliably say “I do not know” instead of inventing something that looks like knowing, the fabrication problem is not solved. It is merely quieter, which is arguably worse.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

Why Every AI Startup Looks the Same

The AI Downside — Sat, 15 Aug 2026 21:55:00 +0000

Spend an afternoon browsing new AI startups and a strange déjà vu sets in. The landing pages rhyme. There is a dark hero section, a gradient somewhere between indigo and violet, a little sparkle or star icon denoting Intelligence, a headline promising to let you “chat with” your documents or data or customers, and a demo video with the same upbeat, slightly anonymous soundtrack. You could swap the logos between fifty of these sites and almost nobody would notice.

Sameness on the surface

Some of this is just design fashion, and design fashions always converge. But the AI cohort has converged harder and faster than most, and the reason is worth naming: when everyone is building on top of the same handful of foundation models, the differentiation has to come from somewhere else, and branding is the cheapest lever to pull. If your product is a thin layer over a model anyone can call, you cannot differentiate on the model, so you differentiate on the gradient.

When the engine is a commodity everyone rents, the paint job is the only thing left to argue about. Hence a thousand identical paint jobs.

Funded by the same money, chasing the same story

The uniformity runs deeper than design and architecture; it reaches into the incentives. A great many of these companies are funded by the same pools of venture capital, pitched against the same market maps, and steered toward the same narrative arc — explosive growth now, monetisation later, an acquisition or an IPO at the end. When the funding, the advice and the definition of success are shared, the strategies converge. Everyone chases the same enterprise customers, adopts the same land-grab pricing, and races the same clock, because that is the shape of company the money was betting on.

This produces a cohort that is not only visually and technically alike but strategically alike, which makes the whole field unusually fragile to the same shocks. A shift in model pricing, a change in what the platform providers offer natively, a cooling of investor enthusiasm — any of these hits the entire cohort at once, because the cohort made the same bet. The sameness that looks like a design trend on the landing pages is, underneath, a systemic exposure. When a thousand companies are built to the same template on the same assumptions, they do not merely look identical in the good times. They fail identically in the bad ones.

Sameness underneath, which matters more

The visual monoculture is a symptom. The structural one is the real story. A large share of AI startups are, functionally, a prompt and a nice interface wrapped around an API call to one of a few providers. This is not automatically bad — plenty of good businesses are thin layers that solve a real, specific problem better than the raw tool does. But it creates a specific and widely shared fragility.

If your entire product is a wrapper, then:

Your margins are someone else's pricing decision. The provider raises token prices and your cost base moves without your permission.
Your moat is a prompt, and a prompt is copied in an afternoon by anyone who can see your outputs.
Your roadmap can be erased by a feature announcement. The classic fate of the AI wrapper is to build a clever tool on top of a model, and then watch the model provider ship that exact capability as a native feature, for free, to everyone. The polite industry term is “getting platformed.” It happens constantly.

The demo-to-product gap

There is another kind of sameness: the gap between the demo and the daily reality. AI products demo extraordinarily well, because a demo is a curated happy path and the models are genuinely dazzling on a good example. The sameness comes in the second week, when the novelty wears off and you discover that every one of these tools has roughly the same failure modes — the confident wrong answer, the vague limit, the feature that works in the video and not on your actual data.

This is why AI product retention is such a quietly discussed problem. Getting someone to try a magical demo is easy and cheap. Getting them to still be using it a month later, when the wrapper has revealed itself as a wrapper, is the hard part nobody puts on the landing page.

What actually distinguishes the survivors

The startups that will still be here in a few years are, tellingly, the ones that look the least like the template. They tend to have something the model cannot provide on its own: proprietary data, a genuinely hard integration, a workflow deeply embedded in how a specific industry works, real distribution, or an interface so good it constitutes the product. In other words, they compete on the parts that are not the model — because the model is the one part every competitor also has.

The demo economy rewards the wrapper

The sameness is partly a rational response to how these companies are funded and judged. In the current climate, a compelling demo and a fast-growing user chart can raise money, and raising money is survival. Building the harder thing — the proprietary data, the deep integration, the genuinely defensible product — is slow and unglamorous and does not fit in a launch clip. So the incentives reward whoever can ship the most impressive-looking wrapper fastest, and everyone optimises for the same short-term signal, which produces the same short-term shape of company.

This is how you get a field full of products that demo like magic and retain like a leaky bucket. The demo is the fundable moment; retention is next quarter's problem. A great many AI startups are, in effect, financial instruments optimised for the raise rather than businesses optimised for the customer — and a customer can feel that, even if they cannot name it, in the second week when the magic thins out and the wrapper shows through. Built for the investor, the product treats the user as a growth metric, and growth metrics do not need to be delighted, only acquired.

How to tell a tool from a landing page

For the person deciding whether to depend on one of these, the useful discipline is to ignore the aesthetics entirely and interrogate the substance. Ask what happens to this company if the underlying model provider ships this feature natively next month — if the answer is “it dies,” you are looking at a feature, not a business. Ask what the product knows or does that you could not get by typing the same request into the model it is built on. Ask what it owns that a competitor cannot copy in a weekend: data, distribution, a hard integration, a genuinely superior interface.

If there are good answers, the gradient is just paint on something real, and it may well survive. If the honest answers are “nothing, nothing, and nothing,” then the sameness you are looking at is not a coincidence of design fashion — it is the visible surface of an absence, a company that looks like every other because there is nothing underneath to make it look like anything else. The paint job is uniform because, in too many cases, the paint job is the product.

Consolidation is coming, and it will look like a cull

A field this uniform, this thinly differentiated, and this dependent on cheap capital does not stay crowded forever. When the funding climate tightens — and it always eventually tightens — the thousand near-identical wrappers do not gently mature into a thousand sustainable businesses. Most quietly disappear, acquired for their team, wound down, or simply switched off when the runway ends and the metrics never justified a further round. The sameness that made them easy to launch makes them easy to lose: when a product has no defensible core, there is nothing to stop a customer moving on and nothing to make an investor fight to keep it alive.

For users, this is the part with a real personal cost, and it is worth weighing before you build your workflow on the exciting new tool with the beautiful gradient. The thin wrapper you adopted this year may not exist next year, taking your data, your saved work and your integrations with it. Betting on the survivors means looking past the launch aesthetics to the boring signals of durability — a real business model, genuine differentiation, something the model provider cannot simply absorb. The cull will not announce itself. It will arrive as a series of quiet shutdown emails, and the products that send them will, disproportionately, be the ones that looked exactly like all the others. Sameness is not just a design smell. In a downturn, it is a mortality risk, and the customer inherits part of it.

None of this is a reason to be cynical about the whole field. It is a reason to look past the gradient. When you evaluate an AI product, ask the unglamorous question the sparkle icon is designed to distract from: what does this do that I could not get by typing the same request directly into the model it is built on? If there is a good answer, it may be one of the survivors. If there is not, you are looking at a landing page, not a business.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

Why AI Benchmarks Mean Less Than You Think

The AI Downside — Sat, 15 Aug 2026 21:54:55 +0000

Every model launch comes with a chart. Bars, usually, or a spider diagram, showing the new model edging past its rivals on a row of benchmarks with acronyms most people cannot expand. The bar is taller. The press writes it up as a leap. And within a week, users report that the new state-of-the-art model is, for their actual work, about the same as the last one or occasionally worse. The benchmark said one thing. Reality said another. This happens so reliably that it is worth understanding the mechanics of the gap.

The test is public, which ruins the test

The most fundamental problem is contamination. Many popular benchmarks are published, discussed, and sitting on the open web — which is exactly where models get their training data. When the questions and answers to your exam are in the study material, a high score measures memorisation as much as ability. Nobody needs to cheat deliberately; the leak is structural. A model can score brilliantly on a benchmark it has effectively already seen and then flounder on a genuinely novel version of the same task.

A benchmark stops measuring intelligence the moment it becomes famous enough to end up in the training data. Fame is the thing that breaks it.

The number becomes the marketing, and the marketing corrupts the number

There is a commercial feedback loop that makes benchmark figures even less trustworthy than their technical limitations alone would suggest. A high score is not just an engineering result; it is a marketing asset worth an enormous amount in attention, funding and credibility. That raises the stakes on every fractional improvement, and where the stakes are high, the temptation to select, frame and present the numbers favourably is irresistible. Vendors choose which benchmarks to headline, which comparisons to draw, and which unflattering results to leave in an appendix or omit entirely. The chart on the launch slide is not a neutral readout; it is a curated argument.

This is not necessarily fraud — it rarely needs to be. It is simply the ordinary gravity of a metric that has become a sales tool. When beating a particular number by a point translates into headlines and a valuation bump, engineering effort flows toward that number regardless of whether it corresponds to anything you care about, and communication effort flows toward presenting it as impressively as the facts allow. The benchmark started life as an attempt to measure capability honestly. By the time it is famous enough to appear on a keynote slide, it has been thoroughly repurposed into an instrument of persuasion, which is a different job with different loyalties.

Optimising for the test, at the expense of the job

Benchmarks are also targets, and targets get gamed — not always cynically, but inevitably. When a specific set of evaluations becomes the scoreboard the whole industry watches, enormous effort goes into nudging those specific numbers up. This is Goodhart's law in its purest form: when a measure becomes a target, it stops being a good measure. A model tuned to excel at benchmark-shaped questions is not necessarily a model that is better at your unglamorous, benchmark-unshaped problem.

The benchmark is not your job

Even a perfectly clean, ungamed benchmark would mislead you, because the tasks bear little resemblance to real work. Benchmarks favour the things that are easy to score automatically: multiple-choice questions, problems with a single verifiable answer, self-contained puzzles. Your actual use is messier — a long, ambiguous document; a vague request; a task where “good” is a matter of taste and context and there is no answer key. A model that aces graduate-level multiple choice may still write emails you would be embarrassed to send.

The dimensions you actually care about are mostly unmeasured by the leaderboard: does it follow instructions precisely, does it keep its tone consistent, does it refuse sensible requests (a problem we covered in our piece on over-cautious refusals), does it stay coherent over a long session, is it fast enough not to break your flow. None of those fit neatly on the launch chart.

“Human preference” leaderboards have their own trap

The response to all this has been crowd-sourced arenas where humans vote on which of two anonymous answers they prefer. These are genuinely more useful than static exams — but they measure preference, not correctness, and preference has biases. People tend to prefer longer, more confident, more flattering answers, which rewards models for being verbose and agreeable rather than accurate and concise. A model can climb a preference leaderboard by being a better sycophant, which is not the quality most of us are shopping for.

How to actually evaluate a model

A single number for a many-shaped thing

Underneath every benchmark complaint is a category error: the attempt to collapse a wildly multidimensional thing into a single rankable score. “How good is this model” is not one question. A model can be superb at code and mediocre at prose, brilliant at short tasks and lost over long ones, precise at following instructions and hopeless at knowing when to refuse. These qualities do not move together, and a customer cares about different ones. Averaging them into a leaderboard position discards exactly the information you needed and hands you a number that is true, precise, and useless for your decision.

This is why two people can use the “same” top-ranked model and reach opposite verdicts. The one doing bulk classification loves it; the one writing nuanced long-form finds it flat. Neither is wrong, and the benchmark cannot adjudicate between them, because it measured a blend of capabilities that neither of them actually has. A ranking implies a single axis of better-and-worse. Real model quality is a landscape, and the leaderboard is a photograph of it taken from one arbitrary angle, sold as the view.

The numbers are aimed at investors, not users

It helps to remember who the benchmark chart is really for. A dramatic score is a fundraising asset, a recruiting asset, and a press asset long before it is a user asset. In a field where enormous sums move on the perception of being at the frontier, a benchmark result is a claim to that frontier — and the audience that most rewards the claim is not the person choosing a tool for their work, but the investor, the journalist and the prospective hire deciding who is winning. The chart is optimised for that audience, and its conventions follow accordingly.

Which means the leaderboard is best read as a marketing artefact that happens to be expressed in numbers, rather than a measurement that happens to be useful for marketing. Numbers carry an air of objectivity that a slogan does not, and that borrowed authority is much of their value to the vendor. Your defence is to decline the frame entirely: not to argue about whose benchmark is fairer, but to stop treating the leaderboard as the thing that decides, and to move the decision back onto the only ground that is contaminated by nobody's incentives — your own real tasks, run yourself, judged by whether the output was any good.

Benchmarks shape what gets built, not just what gets sold

The quiet damage of benchmark obsession is not only that it misleads buyers; it is that it steers the technology itself. When a specific set of scores is the scoreboard the whole field watches, research effort, training choices and product decisions all bend toward moving those particular numbers. Capabilities that happen to be benchmarked get lavish attention; capabilities that matter to real users but resist tidy scoring — consistency, restraint, knowing when to refuse, staying coherent over a long session, admitting uncertainty — get comparatively neglected, because there is no leaderboard to win for them. The measure does not just describe progress; it decides what “progress” is allowed to mean.

This is Goodhart's law operating at the scale of an entire industry. Optimise hard enough for a proxy and you get models finely tuned to the proxy and subtly misaligned with the goal it was standing in for. A generation of models sculpted to ace exam-shaped questions may be genuinely worse, in ways no benchmark records, at the messy, ambiguous, judgement-laden work that actually fills your day — not because anyone chose that trade-off, but because the scoreboard never counted the thing that was quietly lost. So the leaderboard misleads twice over: once when you read it, and once, invisibly, in the shape of the models it helped bring into being. The only defence against both is the same — hold the measure loosely, and keep your own tasks as the thing that actually decides.

The only benchmark that reliably predicts whether a model is good for you is the one nobody can sell you: your own tasks. Keep a small, private set of real problems from your actual work — the kind of thing you use these tools for every day. When a new model appears, run your set. Ignore the chart. The results will frequently disagree with the leaderboard, and when they do, trust your set. It is contaminated by nothing, gamed by no one, and it is measuring the only thing that matters, which is whether the tool is useful to you. The leaderboard is measuring whether it is useful to the launch.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

The Problem With AI “Memory”

The AI Downside — Sat, 15 Aug 2026 21:54:51 +0000

“Memory” is the feature everyone asked for and few thought through. The pitch is lovely: the assistant remembers your preferences, your projects, your writing style, the fact that you are vegetarian and allergic to long emails, so you never have to repeat yourself. In practice, it is one of the most consequential privacy decisions in consumer AI, dressed up as a convenience toggle, and most people flipped it on without reading past the word “remember.”

Convenience and surveillance are the same feature

The uncomfortable truth is that there is no version of persistent memory that is not also a growing personal record. For the assistant to remember your details, it has to store your details. The thing that makes it feel like it knows you is a file — structured or otherwise — accumulating what you have told it, and inferring more from what you did not. The warmth and the dossier are not two features. They are one feature seen from two angles.

A system that remembers everything you tell it is, definitionally, a system that keeps a record of everything you tell it. The friendliness is the interface; the record is the substance.

The profile that shapes what you are shown

A stored model of who you are does not sit there inertly; it starts to shape the responses you get. That is the entire selling point — a memory-enabled assistant tailors its answers to what it believes about you. But tailoring cuts both ways. Once the system has decided you are a particular sort of person, with particular views and particular tastes, it begins to give you the version of the world it thinks you want, and you lose the ability to know what it would have said to someone it had profiled differently. The personalisation that feels like being understood is also, quietly, a narrowing.

We have seen this film before, with recommendation feeds that learned our preferences and then fed them back to us until the preferences hardened into a cage. A memory-driven assistant risks the same dynamic applied to information and advice rather than entertainment — a system that increasingly tells you what fits the profile it has built, in a voice of neutral helpfulness that hides the fact that a version of you is doing the steering. The more it remembers, the more it reflects you back at yourself, and the harder it becomes to use the tool to genuinely think against your own grain. Forgetting, it turns out, is part of what keeps a source honest.

You cannot easily see what it decided about you

The deeper problem is opacity. Memory does not only store the facts you deliberately offered. It stores inferences — patterns it noticed, conclusions it drew, categories it filed you under. And the interface for inspecting all of this is, at best, partial. You can often see some saved notes; you can rarely see the full shape of what the system has concluded about your habits, your mood, your politics, your health, your finances, based on months of you thinking out loud.

This matters because people talk to chatbots with a strange candour. The blank, non-judgemental box invites disclosure — worries, symptoms, relationship problems, half-formed plans — that people would never put in an email. Memory turns that candour into a persistent profile. The thing you found comforting about the anonymity is quietly undermined by the thing that remembers.

The record outlives the moment

Context collapses over time. A question you asked during a frightening week, a subject you researched out of one-off curiosity, an opinion you were trying on and discarded — memory can flatten all of these into apparently stable facts about who you are. The system does not know that you were joking, venting, or looking something up on behalf of someone else. It just knows you said it, and now it remembers, and it will helpfully bring it up later.

And then there is everyone else who can reach it

A stored profile is a target. It can be subject to a data request, exposed in a breach, retained after you thought you had deleted the account, or used — depending on the settings we complained about elsewhere — to improve the very models it was built from. The more the assistant remembers, the higher the stakes of every one of those failure modes. A chatbot that forgets each conversation is a low-value target. A chatbot that has quietly assembled a year of your inner monologue is not.

Using memory without being used by it

Memory is not evil, and for some genuinely useful cases — remembering your formatting preferences, your recurring project context — it is a real quality-of-life improvement. The point is to treat it as the significant choice it is, rather than the harmless toggle it is presented as:

Know whether it is on. For several products it defaults on. Check.
Read what it has stored, where you are allowed to, and delete what you did not mean to donate.
Keep the sensitive stuff out of the box entirely — the health worries, the financial specifics, the details about other people who never agreed to be remembered.
Turn it off for the anonymous, one-off questions where the whole value was that nothing was being kept.

The inference problem is worse than the storage problem

People worry about memory storing the facts they typed. The sharper concern is the facts it derives that they never typed at all. A system with a running record of your questions can infer a great deal you never stated: your rough location from what you ask about, your health from your worries, your income bracket from your spending questions, your politics from the framing of your queries, a mental-health picture from your tone at 2am. None of this was disclosed. All of it can be inferred, stored as a working model of you, and used to shape what you are shown — and you cannot delete an inference you do not know exists.

This is the part the tidy “view your saved memories” screen conceals. It shows you the explicit notes; it does not show you the profile assembled from the pattern of everything you have ever asked. The gap between what a system has visibly saved and what it has effectively learned about you is enormous, and it is precisely the invisible part that is most sensitive and least controllable. You are auditing the tip and reassured, while the mass of it sits below the waterline, unlabelled and un-deletable.

Intimacy as a retention strategy

It is worth being clear-eyed about why memory is pushed so hard. An assistant that knows you is an assistant that is painful to leave. Every preference it has learned, every bit of context it holds, is a small switching cost — start again with a competitor and you must rebuild the relationship from scratch. Memory is not only a convenience feature; it is a moat, quietly converting your accumulated disclosures into lock-in. The more it remembers, the more it costs you to walk away, which is a benefit to the vendor dressed as a benefit to you.

That reframing is not cynical; it is just following the incentive. A feature that happens to make the product both stickier and more data-rich, while feeling to the user like warmth and personalisation, is a feature a subscription business will build whether or not it is in your interest — because it is squarely in theirs. The intimacy is real in effect and instrumental in purpose, and knowing which is which is the difference between using the feature and being used by it.

Consent that cannot keep up with context

Even a diligent user who reads every setting faces a problem no toggle solves: you cannot meaningfully consent to how a fact about you will be used in a future you cannot foresee. You tell the assistant something during a frightening week, in a particular mood, for a particular reason. Memory strips away that context and preserves the fact as a stable, decontextualised truth about who you are — available to be surfaced, acted on, or inferred from, months later, in situations you never imagined when you said it. The consent you gave was to a moment. The retention is forever, and the two do not match.

This is why “you agreed to memory” is a weak defence for what the feature actually does. Agreeing that an assistant may remember your preferences is not the same as agreeing that a year of your unguarded thinking may be assembled into a durable profile, mined for inferences you never volunteered, and used to shape what you are shown. The granularity of real consent — this fact, for this purpose, for this long — is exactly what a persistent, inferential memory cannot offer, because its whole value comes from retaining and connecting things you did not deliberately decide to give it. A feature built on remembering everything cannot, by construction, ask permission for each thing it remembers. So it asks once, vaguely, at the start, and calls a year of accumulation covered by a single tick.

The feature is sold on intimacy — an assistant that finally knows you. It is worth remembering that intimacy, in software, is another word for data retention. There is a security edge to this too: a well-stocked memory is exactly the private store an attacker hopes your assistant will quietly read out, which is how prompt injection turns a convenient profile into a liability. Enjoy the convenience with your eyes open, and decide for yourself what you would rather it forget.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

When AI Refuses Perfectly Normal Requests

The AI Downside — Sat, 15 Aug 2026 21:54:47 +0000

Ask a modern chatbot to help with something completely ordinary and there is a growing chance it will decline. Not because the request was dangerous, but because it brushed against a keyword, a topic, or a category that the vendor's safety systems treat as radioactive. A recipe that mentions alcohol. A history question about a violent event. A medical query you were entitled to ask. A creative scene with any conflict in it. The refusal arrives politely, firmly, and without much interest in whether it was warranted.

Safety is real; this is not most of it

Let us be fair, because this is a topic where fairness is usually the first casualty. Some restrictions are entirely sensible. Refusing to help synthesise a weapon, produce material that sexualises children, or plan real violence is not censorship; it is basic responsibility, and reasonable people want it there. The complaint is not about those lines. It is about everything on the wrong side of a border that has been drawn far too wide, catching countless legitimate requests to avoid a handful of genuinely bad ones.

There is a difference between refusing to help build a bomb and refusing to discuss the chemistry a GCSE student is studying. Too many systems can no longer tell which one you are asking for.

Whose values, decided by whom

There is a question underneath the practical annoyance that deserves stating plainly: when a model refuses, whose standards is it enforcing? The boundaries of what these systems will and will not discuss are set inside companies, by people you did not elect, according to policies you cannot read, calibrated to a mixture of genuine safety concern, legal caution and brand protection. A handful of firms are, in effect, quietly setting the terms of acceptable enquiry for hundreds of millions of people, and doing so through refusals that arrive without an appeal, an explanation of the rule, or any way to contest the judgement.

Reasonable people disagree about difficult topics, and different cultures draw lines in different places. Baking one company's risk appetite into a tool that the whole world uses flattens that legitimate variety into a single, cautious default, exported everywhere at once. A subject that is ordinary and discussable in one context is treated as off-limits because it might be sensitive in another, and the most restrictive interpretation wins by default because it is the safest for the vendor. The result is a slow, unaccountable narrowing of what it is convenient to ask about — not through any grand act of censorship, but through a million small refusals, each individually defensible and collectively a real constraint on ordinary enquiry that nobody voted for.

The incentives all point at over-refusal

The reason refusals skew cautious is not a mystery; it is arithmetic on the vendor's side of the ledger. A model that helps with a harmful request generates a screenshot, a news story, and reputational damage. A model that wrongly refuses a harmless request generates a mildly annoyed user who mostly says nothing and quietly tries a competitor. One of those failure modes is loud and career-threatening for whoever owns safety. The other is silent. Faced with that asymmetry, the rational institutional choice is to over-refuse, and so systems over-refuse.

The cost of that choice is simply moved onto the user, who now pays a tax of friction, workarounds and second-guessing on every request that lives near a sensitive edge. The vendor optimises its own risk; the responsible majority absorb the inconvenience so the system can dodge a rare embarrassment.

The condescension problem

Beyond the practical friction there is a tonal one, and it grates. A refusal often comes wrapped in a small lecture — a reminder to consult a professional, a note about why the topic is sensitive, an assumption about your intentions you did not invite. An adult asking a straightforward question about their own body, their own legal situation, or a difficult subject they are perfectly entitled to explore does not need to be gently managed. Being treated as a potential problem to be handled, rather than a competent person to be helped, is its own kind of insult, delivered thousands of times a day.

It pushes people to worse options

There is a pro-consumer irony here worth spelling out. Excessive refusals do not make people safer; they make people go elsewhere. Refuse enough reasonable requests and users learn not to trust the cautious tool for anything sensitive, and route those exact queries to less careful models, unfiltered alternatives, or the open web. The over-cautious system does not remove the demand. It exports it, frequently to somewhere with no safety thinking at all. Caution that pushes the hard questions toward the least responsible corner of the internet is not caution succeeding.

What good looks like

The fix is not “remove all limits.” It is calibration and respect:

Draw the hard lines narrowly and defend them, rather than drawing them wide and catching everyone.
Assume competence. Most people asking about a sensitive topic have an ordinary, legitimate reason, and the system should behave as if that is true until it plainly is not.
Explain refusals specifically, and offer the version of the help that is appropriate, rather than shutting the whole subject down.
Skip the lecture. A refusal with a moral seasoning is worse than a refusal.

The inconsistency is its own insult

What tips over-refusal from frustrating to absurd is how inconsistent it is. The same request, phrased two slightly different ways, gets a helpful answer once and a firm refusal the next time. A topic the model will happily discuss in the abstract triggers a wall the moment you make it concrete. Ask directly and you are blocked; add “for a novel I'm writing” and the gate swings open. The boundary is not a principled line you can understand and respect; it is a jittery, keyword-sensitive tripwire, and its randomness makes it impossible to form a working mental model of what the tool will and will not do.

That unpredictability is corrosive because it defeats the whole point of a rule. A consistent limit, even one you disagree with, you can at least plan around. A limit that fires on the phrasing rather than the substance just teaches users to play word games — to launder ordinary requests through fictional framings and euphemisms until they slip past. This trains everyone, including people with entirely innocent needs, to treat the safety system as an obstacle to be tricked rather than a boundary to be respected, which is roughly the opposite of what a safety system should cultivate.

Over-refusal quietly erodes the case for safety itself

There is a longer-term cost that the risk-averse calculus misses. Every unjustified refusal spends a little of the public's goodwill toward AI safety as a whole. When people experience “safety” mainly as being condescended to and blocked from reasonable tasks, the word starts to read as a euphemism for corporate caution and liability management rather than genuine care. The real, important lines — the ones that stop genuine harm — get tarred with the same brush as the silly ones, and the constituency for sensible guardrails shrinks every time someone is lectured for asking a normal question.

This is why calibration is not a niceties issue; it is core to safety's credibility. A system that refuses well — narrowly, consistently, respectfully, only where refusal is genuinely warranted — earns the trust that makes its hard limits acceptable. A system that refuses badly squanders that trust and, in doing so, weakens the standing of the very principle it claims to serve. Over-caution does not just annoy users. It discredits the idea that any of the caution was worth having, which is a strange own goal for an industry that badly needs people to believe its safety work is real.

Who is actually being protected

Strip the language of safety back and ask, of any given over-refusal, who it actually protects. Occasionally the answer is “a genuinely vulnerable person from genuine harm,” and there the refusal is doing its job. Far more often the honest answer is “the company, from a hypothetical bad headline.” The refusal is not shielding the user from danger; it is shielding the vendor from the small chance that this interaction becomes an embarrassing screenshot. That is a legitimate corporate interest, but it is not the same thing as your safety, and dressing one up as the other is where the resentment comes from. Users can feel the difference between being protected and being managed, and being managed while told it is for your own good is the particular flavour that grates.

This matters because it reveals what the calibration is really optimising. A system tuned to minimise the vendor's reputational risk will refuse anything near an edge, because the cost of a wrong refusal falls on you and the cost of a wrong answer falls on them. A system tuned to actually serve users would accept a little more institutional risk in exchange for helping the vast, blameless majority — and would trust adults to handle adult topics. The current settings tell you which optimisation is winning. The lectures, the blanket refusals, the assumption of bad intent: these are the fingerprints of a system protecting itself and calling it protecting you. Naming that honestly is the first step to demanding the version that actually puts the user first.

Responsible AI has to be able to say no. It also has to be able to say yes to the overwhelming number of reasonable requests that currently trip the wire. A system that cannot tell the difference has not solved safety. It has merely relocated its own risk onto the people trying to use it for something perfectly normal.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

Why AI Product Launches Feel Identical

The AI Downside — Sat, 15 Aug 2026 21:48:26 +0000

Watch enough AI launches and they begin to blur into a single, endlessly repeating event. There is the understated title slide. The claim that we are at an inflection point. The chart showing the new model clearing a row of benchmarks. The live demo that works flawlessly. The superlatives — most capable, most advanced, our best model yet. And the closing note that all of this will roll out “over the coming weeks,” which is to say, not today, and possibly not to you. It is a genre now, with conventions as fixed as a nature documentary, and once you see the template you cannot unsee it.

The conventions of the genre

Every mature format has its tropes. The AI launch has assembled a reliable set:

The benchmark chart — which, as we argued in our piece on benchmarks, predicts your experience far less than its prominence implies.
The cherry-picked demo — a single, gorgeous example that represents the top of the model's range, not its average day.
The superlative — always “most capable,” because every model is the most capable at the instant it ships, until the next one three months later.
The vague availability — “rolling out over the coming weeks,” a phrase that lets the announcement bank the excitement now and deliver the substance later, to some users, eventually.
The safety paragraph — a brief, serious note about responsible deployment, positioned to reassure without committing to specifics.

When every launch uses the same script, the script stops conveying information and starts conveying mood. The mood is always “inevitable progress.”

The relentless cadence is part of the message

The sheer frequency of these launches is itself a rhetorical device, whether or not anyone intends it that way. When a major model or feature is announced every few weeks, the cumulative effect is a drumbeat of perpetual acceleration — a sense that the field is moving so fast that to pause, to doubt, or to ask whether the last release actually delivered is to risk being left behind. The pace manufactures a fear of missing out that operates on customers and competitors alike, and that fear is extremely useful to everyone selling something, because a frightened-of-falling-behind buyer does less due diligence than a relaxed one.

It also conveniently outruns scrutiny. By the time anyone has properly evaluated whether last month's “most capable model ever” lived up to its demo, this month's has arrived to reset the conversation, and the unanswered questions about the previous one are simply abandoned. The cadence ensures the critical assessment never quite catches up with the marketing, because the marketing keeps moving. Step back from the individual launches and the pattern is clear: an industry that has learned to keep the audience in a permanent state of breathless anticipation, where the promise of what is coming next always arrives before the verdict on what came last. It is a very effective way to sell a future while deferring, indefinitely, any accounting of the present.

Why they all converged

The sameness is not laziness; it is imitation under competitive pressure. One company established a format that generated enormous attention, and in a field where the models themselves are increasingly hard to tell apart, the launch is the differentiation. So everyone reaches for the same beats, because the beats work — they produce the coverage, the social clips, the sense of a field advancing at a pace that demands you keep paying attention. The template is optimised for momentum, not for helping you decide whether the product is any good.

The announcement-to-availability gap

The most user-hostile convention is the gap between the launch and the thing. “Available over the coming weeks” means the company gets the headline today for a product you cannot use today. By the time it actually reaches your account — if your region, your tier and your luck align — the discourse has moved on, and any critical assessment of whether it lived up to the demo has been buried under the announcement of the next model. The hype is delivered on time. The substance ships late and quietly, which is precisely the arrangement that suits the vendor.

How to watch a launch without being managed by it

None of this means the products are bad — many are genuinely impressive, and real progress is happening underneath the theatre. It means the launch is a marketing artefact, not an evaluation, and should be consumed as such:

Ignore the superlatives entirely. They are load-bearing for the narrative and empty as information.
Discount the demo. Assume it is the best case, because it is.
Note the availability date, and judge nothing until you have used the thing yourself.
Wait a fortnight. The honest verdict on any AI launch is written not on stage but two weeks later, by the users who have hit the edges the demo carefully avoided.

The demo is the most dishonest part of an honest-looking show

Of all the conventions, the cherry-picked demo deserves the most suspicion, because it is the one that looks most like evidence. A live demonstration feels like proof — you are watching the thing work, right there, in real time. But a demo is a performance rehearsed against a task the presenters chose precisely because the model handles it beautifully. It represents the top of the model's range on a friendly example, not the median of your Tuesday. The gap between “what it did on stage” and “what it does on your actual, awkward, edge-case-riddled input” is exactly the gap the demo is designed to hide.

This matters because the demo does most of the persuasive work. Nobody remembers the caveats; everyone remembers the moment the model did something that looked like magic. That single curated success becomes the mental image of the product's capability, and the disappointment two weeks later — when the same request on real data produces something ordinary or wrong — never quite catches up to the memory of the demo. You have been sold the ceiling and will live with the average, and the launch was carefully staged so you would not notice the difference until after you had formed your opinion.

Manufactured inevitability is the real product

Strip everything else away and the deepest function of the identical launch is to sell a feeling of inevitability. The superlatives, the ever-climbing charts, the relentless cadence, the “this changes everything” — together they manufacture a sense that this future is not a choice being made by companies with commercial interests, but a tide coming in whether you like it or not. And an inevitable future is one you do not get to question. You can only get on board or be left behind. Resistance reframes itself as ignorance; scepticism reframes itself as being a dinosaur.

That framing is enormously convenient for everyone selling something, which is why the launches all reach for it. But adoption of a technology is not weather; it is a series of decisions — by users, by companies, by regulators — about what is actually worth using and on what terms. The launch theatre exists partly to make those decisions feel already made. The most pro-consumer stance available to a viewer is simply to remember that they are decisions, that the tide is a marketing metaphor rather than a law of physics, and that “inevitable” is the single most load-bearing and least examined word in the entire genre.

The two-week rule, and why it works

If there is one practical habit worth taking from all this, it is the two-week rule: withhold judgement on any AI launch until roughly a fortnight after it ships. The launch-day discourse is worthless as evaluation, because everyone is reacting to the same curated demo and the same superlatives, and nobody has yet used the thing on real work. Two weeks is about how long it takes for the honest signal to emerge — for ordinary users to hit the edges the demo avoided, for the “most capable model ever” to reveal where it is actually the same as its predecessor or quietly worse, for the availability to have reached enough people to judge.

The reason the industry's cadence works against you is precisely that it is faster than the two-week rule. By the time the real verdict on one launch is forming, the next launch has arrived to reset the conversation, and the sober assessment never quite gets written because the news cycle has moved on to the next benchmark chart. This is not necessarily deliberate, but it is extremely convenient: a pace of announcement that consistently outruns the pace of honest evaluation ensures the marketing is always fresh and the reckoning is always deferred. The counter-move is simply to refuse the tempo. Let the launches wash past, note what you'd want to test, and check back in two weeks when the people who actually used it have filed the only review that was ever worth reading.

The industry has built a beautiful, efficient machine for manufacturing the feeling of a breakthrough. The feeling arrives on schedule, every few weeks, whether or not the breakthrough does. Watch the launches if you enjoy them — they are well produced. Just remember you are watching an advert with unusually good pacing, and the review has not been written yet.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.

The Race to Replace Human Support With Bots

The AI Downside — Sat, 15 Aug 2026 21:48:23 +0000

The most enthusiastic corporate adopters of AI are not building anything you would use for fun. They are quietly replacing their customer support. The pitch, internally, writes itself: support is expensive, staffing is hard, and a chatbot never sleeps, never needs training, and never asks for a raise. The pitch to you, the customer, is “faster, always-available help.” The gap between those two pitches is where the whole problem lives.

What support is actually for

Here is the thing the cost model misses. You do not contact support when everything is fine. You contact it when something has gone wrong — a charge you did not recognise, an order that vanished, an account you are locked out of, a problem the FAQ does not cover. By definition, the queries reaching a human were the ones the self-service layer already failed to answer. They are the hard cases, the edge cases, the emotionally charged cases. Those are precisely the cases a support bot is worst at.

A support bot handles the questions you could have answered yourself and struggles with the ones you actually needed help for. It automates the easy half and leaves you stranded on the hard half.

The loop that is designed to exhaust you

Anyone who has fought with a support bot on a genuinely difficult problem knows the particular despair of the loop. You explain the issue; it offers a generic suggestion you already tried. You explain again; it offers the same suggestion, rephrased. You ask for a human; it asks you to describe the problem, and offers the suggestion a third time. The conversation is not progressing toward a resolution because it was never architected to resolve your case — it was architected to contain you, cheaply, for as long as possible, in the hope that you give up or the system can mark the contact “handled.”

This is where the cost-saving genuinely lands, and it is worth being honest that it lands on you. The expense that the company removed from its books did not vanish; it was transferred to the customer in the currency of time and frustration. The hours a human agent used to spend resolving your problem are now hours you spend circling a bot that cannot, and the saving that looks so clean on the support department's budget is really just a cost quietly moved off the company's ledger and onto yours. A business that measures this arrangement as a success has decided that your time is free and its own is expensive — which is, when you put it that way, a fairly complete summary of how the customer's interests rank.

The escape hatch is being welded shut

For years the grudging compromise was the “talk to a human” option — buried, delayed, guarded by a maze, but there. The quiet shift with AI support is that this hatch is increasingly hard to find or simply absent. The bot is not a first line with a human behind it; it is designed to be the only line, engineered to “deflect” as many contacts as possible before they reach a person. Deflection is the actual metric, and deflection is a polite word for “persuaded the customer to give up.”

This is the part worth naming plainly. When a company measures its support AI by how few customers reach a human, it has defined success as your failure to get help. The incentive is not aligned with solving your problem. It is aligned with ending the conversation.

The confident wrong answer, now with consequences

Support is also a setting where the hallucination problem we keep returning to stops being funny. A chatbot that confidently states the wrong refund policy, invents a returns window, or misdescribes your rights is not producing an amusing screenshot; it is giving a customer official-sounding misinformation from the company itself. There have already been cases where businesses were held to promises their support bots made up. The bot speaks with the company's authority and none of the company's accountability, and the customer is left to sort out the difference.

When it is genuinely good

To be fair — and we always try to be — AI support done honestly is a real improvement. A bot that instantly handles the genuinely simple, common questions, at 3am, in the customer's language, is a better experience than a queue. The failure is not the technology. It is deploying the technology as a wall instead of a door: using it to answer the easy things and to hand off gracefully, quickly and without a fight the moment it is out of its depth.

What good support AI would do

Offer a human early and visibly, not as a hidden last resort after three loops of the same suggestion.
Know its limits and escalate on its own the instant it is uncertain, rather than confidently guessing.
Never invent policy. On anything involving money, rights or entitlements, it should quote the real thing or fetch a person.
Be measured on problems solved, not contacts deflected — a metric that would quietly realign the whole system with the customer's interests.

The accountability gap is the real innovation

The genuinely novel thing about AI support is not the automation — companies have been automating support for decades with phone trees and canned macros. It is the accountability gap. A human agent is a person the company employs and stands behind; what they promise, the company is generally bound by. A support bot speaks with the full authority of the brand — same logo, same confident tone, same “we” — while the company reserves the right to disown whatever it says the moment that becomes convenient. You are told to trust it as the company's voice right up until it tells you something the company would rather not honour, at which point it is suddenly just a flawed tool.

Customers should not accept that arrangement, and increasingly the law does not either: there have already been rulings that a business is bound by what its support bot told a customer, on the sensible principle that you cannot deploy something as your official voice and then disclaim it when it errs. That is exactly the right instinct. If a company puts a bot in front of you and lets it speak for them, they own what it says — the confident refund policy it invented included. Anything less lets firms capture the savings of automation while offloading its risks onto the customer, which is the precise arrangement this whole wave was quietly designed to achieve.

“Available 24/7” is not the same as “there when you need it”

The headline virtue of support bots — always on, instant, never a queue — is real but slippery. Availability is only valuable if the thing that is available can actually help. A bot that responds instantly, at any hour, in any language, and cannot resolve your problem has not given you support; it has given you a fast, tireless, multilingual way of not being helped. The metric the company celebrates (response time, coverage, contacts “handled”) measures the wrong thing, because it counts the answering, not the resolving.

For the simple, common question at 3am, instant availability is a genuine gift, and worth saying so. For the complicated, unusual, or emotionally loaded problem — the kind that made you seek help in the first place — instant availability of something that cannot help is just a faster route to frustration, followed by the hunt for the human that the system is designed to prevent you finding. Speed at the front door means little if the door only opens onto a corridor of the same three suggestions. What customers actually want is not a bot that answers quickly; it is a problem that gets solved, and the two have been allowed to drift very far apart.

Support is where a brand's real values show

There is a reason support has always been a truer measure of a company than its marketing: it is where you meet the business on your worst day, when something has gone wrong and you need help. How a company treats you at that moment — whether it makes reaching a competent human easy or hard, whether it owns its mistakes or routes you in circles — reveals what it actually thinks of the people paying it. The marketing is what a company says about itself; support is what it does when a customer is inconvenient. Deploying a bot explicitly optimised to stop you reaching a person is, in that light, a fairly frank statement of priorities.

That is what makes the “deflection” metric so revealing. A company measuring the success of its support by how few customers reach a human has, whether it means to or not, told you where you rank against its costs. It is not a neutral efficiency; it is a decision that your resolved problem is worth less than the salary of the person who could have resolved it. Customers register this, even when they cannot articulate it — the sense that a brand they trusted has quietly rearranged itself so that needing help is treated as a cost to be minimised rather than a promise to be kept. The technology did not force that choice. It just made it cheap enough to make at scale, and gave it a friendly interface to hide behind.

Until then, the honest summary of the current wave is this: your favourite companies are replacing the people who could help you with software optimised to stop you asking. Sometimes it works and everyone is better off. Often it works right up until your problem is the kind of problem you actually needed a human for — which, since you bothered to make contact, it usually is.

Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.