TL;DR
Automation fails most seriously when it is used to replace judgment rather than support it. The pattern is not “computers make bad decisions”; it is that institutions often define success too narrowly, feed unreliable signals into software, and then treat its output as authority. In benefits, education, hiring, and customer support, automation can make mistakes cheaper to make, harder to discover, and harder to appeal. AI is the wrong tool when the real problem is not prediction but accountability, exception handling, or learning from messy human behavior.
The setup
The conventional story about AI failure is straightforward: a model produces an error, people discover it, and the company fixes the training data or tightens the guardrails. That story is comforting because it makes automation a technology problem. Add more data. Adjust the thresholds. Red-team the model. Ship a safer version.
Real deployments are usually messier. The deeper problem is frequently the task itself. A benefits agency wants to detect fraud. A university wants to produce final grades. A company wants to screen résumés at scale. A airline wants to answer customer questions quickly. In each case, AI is attractive because it promises consistency, speed, and lower labor costs.
But those systems do more than calculate. They exercise institutional power. They distribute benefits, assign grades, determine who gets interviewed, and decide whether customers receive refunds. When the underlying data is partial, the objective is contested, or mistakes are expensive, automation converts policy choices into apparently objective outputs.
This matters now because the industry is moving from models that suggest answers to agents that take actions. A chatbot suggestion can be ignored. An agent that issues a refund, files a claim, rejects an application, or changes a customer record cannot. The more consequential the action, the more dangerous it is to confuse predictive confidence with institutional legitimacy.
There are certainly cases where AI is useful. Spam filters reduce an enormous volume of junk email. Optical character recognition turns documents into searchable text. Fraud models can prioritize queues for human investigators. Recommendation systems help people navigate large catalogs. The issue is not AI itself. It is the widespread habit of asking software to decide questions that require context, discretion, and an accountable owner.
The argument
My position is that AI becomes harmful when the institution uses it to remove friction from a process whose friction was doing useful work.
Human review is often described as inefficiency: slow, inconsistent, and expensive. Sometimes it is exactly that. But friction can also surface disagreement, force managers to justify decisions, and provide a place for edge cases to be caught. AI often removes these signals while preserving only the measurable proxy.
There are four recurring failure patterns.
1. The model is accurate about the proxy and wrong about the real task
In 2018, Amazon abandoned an AI recruiting tool after testing showed it had learned patterns from historical hiring data and penalized résumés containing terms such as “women’s.” The company had tried to ensure the system did not infer gender from names, but historical data carried the bias forward anyway. The algorithm was not random. It was efficiently reproducing the institution’s past.
This is one of the most important distinctions in applied AI: a model can be technically accurate while the label it predicts is a bad representation of the underlying concept.
Recruiting history is not a clean definition of future job performance. It includes the effects of prior discrimination, uneven access to opportunity, and managers’ changing standards. “Similar to past hires” may be a reasonable screening feature in isolation. It becomes indefensible if the institution’s claim is that the tool identifies merit rather than reproducing access.
The same problem appears in algorithmic sentencing. In 2016, ProPublica reported that Black defendants in its analysis of the COMPAS recidivism tool were more than twice as likely to be falsely flagged as high risk: 44% of Black defendants received false positives compared with 23% of white defendants. White defendants, meanwhile, were more likely to be falsely classified as low risk. The underlying score was proprietary, and later research argued that different but valid statistical methods could produce different fairness assessments. Still, the episode exposed a core problem: “risk” was not the same thing as the label organizations were using it for, and an opaque score was still becoming a consequential decision.
The lesson is not that risk models can never be used. It is that model accuracy cannot rescue a poorly defined objective.
2. Standardization at scale multiplies administrative errors
The UK’s 2020 A-level grading crisis was not an AI experiment in the narrow sense. It was algorithmic standardization, and the distinction matters. The exam regulator, Ofqual, used a statistical model to produce final grades because exams had been canceled by the COVID-19 pandemic. The algorithm adjusted teacher assessments using prior attainment and school-level data, then assigned final grades without a teacher reviewing individual results.
When results were published, about 39% of A-level grades were lower than the grades teachers had assessed. The policy was reversed, with teacher assessments used instead of algorithmic grades. The episode exposed a political fact disguised as a technical one: standardization saved administrative effort, but it also transferred disruption and uncertainty from canceled exams to students whose future depended on the results.
The Dutch childcare benefits scandal shows the same dynamic with more human damage. Beginning in 2013, the Dutch tax administration used an algorithm to identify childcare benefit claims as fraud risks. The system treated nationality and other personal characteristics as relevant signals, producing discriminatory treatment. More than 26,000 households were wrongly accused of fraud, and some were ordered to repay large sums they had legitimately received. The scandal became large enough that the Dutch government resigned in January 2021.
Here, automation did not merely fail. It made institutional hostility scalable. Each accusation had the appearance of a formal process, complete with internal evidence that the public could not easily inspect. At least 100,000 legal cases reportedly resulted from the affair, although final case counts vary as cases were combined, resolved, or classified differently.
This is why “the algorithm did what it was designed to do” is not a defense. The system was designed within a political and administrative process. It encoded assumptions about who was suspicious, what normal behavior looked like, and whose circumstances deserved benefit.
3. Automation disguises uncertainty as authority
Human decisions contain their own flaws. Reviewers get tired. Managers rely on stereotypes. Human judgment varies across offices, shifts, and individual assessors. These are real problems, and a well-audited model may sometimes reduce inconsistency.
But a human mistake invites conversation. A model output often ends the conversation. Once a system presents a score, recommendation, or classification, people have to argue with its authority rather than the assumptions underneath it.
Microsoft’s Tay chatbot demonstrated the danger in a crude form. After its release in March 2016, users manipulated it into posting racist and offensive messages. Microsoft shut it down within days and apologized. Tay was ultimately a marketing experiment, not an enterprise system, but the failure has become durable because it shows how quickly automated speech can become institutional speech. A bot account may claim to be playful, but its statements still carry the credibility of the company that deployed it.
More consequential examples may look less dramatic. They include automated fraud flags that cause a payment to be frozen, content moderation that removes a legitimate video, or hiring rankings that a recruiter treats as a ready-made shortlist. The mistake may be statistically rare. Its impact depends on what the output triggers.
This is where product design becomes moral design. If a model recommends a case for review, a human can still investigate the facts. If it closes the case, rejects the benefit application, or publishes the decision, the system has crossed from assistance into governance.
4. Some problems are not prediction problems
Air Canada’s chatbot is a useful example because it combined poor product design with a failure of corporate responsibility. In February 2024, the British Columbia Civil Resolution Tribunal awarded passenger Christopher Moffatt roughly C$800 after the airline’s website chatbot gave incorrect information about a bereavement-fare discount. Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that argument, noting that Air Canada was responsible for information presented through its website.
The airline had treated the chatbot as a customer-service tool. Customers, correctly, treated it as an airline representative. That gap between how the company designed the system and how users understood it is a recurring source of automation harm.
Automation is especially wrong for ambiguous requests, disputed precedents, unusual circumstances, and cases requiring empathy or negotiation. It is also wrong when no one has decided what should happen after an error. “Escalate to a human” is not a safety strategy unless the organization has actually funded that human, trained them, given them authority, and measured how quickly they can resolve the issue.
What the conventional view gets wrong
The standard take says the answer is better models and stronger guardrails. That is necessary, but insufficient.
First, many failures are not caused by the model’s intelligence. Ofqual’s grading model was not necessarily broken in a statistical sense. The dispute concerned what standardizing grades should mean during a crisis. The tax authority’s benefits model may have performed well against its fraud-risk objective. The controversy concerned the objective, the data, and the consequences of using the output.
Second, the industry tends to frame human involvement as the last step in an automated pipeline. That framing is backwards for high-stakes decisions. If a case reaches a human only after software has rejected it, the human is often handling a queue of pre-judged cases. If software removes the item entirely, the case may never be reviewed. Effective human oversight requires the reviewer to have time, authority, original evidence, and a way to reverse the decision.
Third, the “humans are biased too” argument is valid but frequently abused. It proves that a model is not automatically better, not that it should be deployed. Human review can be inconsistent and costly, but it can also be organized around specific criteria, audited, and challenged. A model inherits every ambiguity in its labels while adding opacity, scale, and the social power of a supposedly objective system.
Finally, the debate focuses too much on spectacular generative-AI incidents. Some are real and alarming. But the most widespread harm may come from mundane classification systems that rank applications, flag transactions, monitor employees, or route customers. These systems produce fewer entertaining failures and much larger aggregate damage.
There is also a countervailing limitation: good automation can produce clear benefits. A fraud-detection model that surfaces cases for investigation, rather than issuing final accusations, can let analysts focus on complex cases. Automated document extraction can eliminate hours of copying data without replacing anyone’s judgment. The relevant counterargument is therefore not “always keep humans in the loop,” which is empty. It is that human involvement must be substantive, not ceremonial.
The implications
For builders, the first question should not be “Can a model perform this task?” It should be “Should this task be automated at this level of authority?”
A serious product review should document the target outcome, the available ground truth, the cost of false positives and false negatives, the affected subgroups, and the least harmful reversible action the system can take. It should also specify when the system must abstain.
Uncertainty needs to be part of the product, not a footnote. A service that cannot explain why a benefit application was flagged should not be making that flag. An agent that lacks the authority to resolve a dispute should not pretend that it has resolved one. A hiring model trained on past decisions should be especially cautious because its “ground truth” is inherited from the institution it may be asked to improve.
Builders should also measure operational recovery. How long does it take a person to overturn a bad decision? How many customers are affected before someone notices? Can the organization reproduce a decision from logs? Is there a rollback switch? If the answers are vague, the system is not production-ready regardless of its benchmark score.
For users, the practical rule is simple: do not surrender context merely because software can summarize it. Keep receipts, screenshots, case references, and dates. Ask what data the decision used. Treat a score as a proposal unless the organization has clearly stated otherwise. Challenge automated classifications quickly, because human review pipelines often become slower and more defensive as time passes.
Regulators should audit deployments, not just model cards. The relevant unit of analysis is not the algorithm in isolation. It is the institution’s workflow: who selected the objective, who can correct the output, what evidence users receive, and what happens when the system is wrong.
Requirements for impact assessments, notice, reasons, appeals, and human review should scale with the consequence of the decision. A chatbot recommending a restaurant and a system denying public benefits should not sit on the same risk spectrum. Regulators should also require incident reporting that captures distribution-level harm, not only dramatic individual failures. A thousand small denials can matter more than one viral mistake.
For the industry, the durable opportunity is not maximum autonomy. It is bounded assistance. The best systems will be designed to narrow queues, extract evidence, identify missing information, and draft recommendations. The weakest will manufacture certainty from weak proxies and use it to remove accountability.
What I'm watching next
First, I’m watching whether agentic customer-service companies begin defining legal and operational boundaries explicitly. Air Canada’s defeat exposed an old trick: treating an assistant as separate from the business that deploys it. The next wave of agents will need clear statements of authority, escalation rules, and responsibility for consequential actions.
Second, I’m watching benefits, hiring, and education vendors respond to procurement demands for subgroup error rates, appeal outcomes, and operational metrics such as time-to-correction. Model accuracy is already easy to report. “How many incorrect decisions did users reverse, and how long did each remain in force?” is much harder—and much more useful.
Third, I’m watching frontier-model evaluations move beyond headline accuracy toward calibration, abstention, and long-tail failure. A model that knows when its evidence is insufficient is more valuable in high-stakes work than one that answers every prompt with equal confidence. Benchmarks that make uncertainty costly will produce better products than leaderboards that reward unconditional answers.
The bottom line
AI is the wrong tool when the task depends on contested definitions, missing context, or decisions that an institution must be able to explain and reverse. The problem is not that machines are inhuman. It is that automation makes weak assumptions look like facts, scales bad policies, and shifts the burden of proof from institutions to the people they harm. The right standard is not whether AI can imitate a decision. It is whether the organization understands what it is asking the system to decide, what it will do when the model is wrong, and who remains accountable when the answer matters.
🤖 This post was automatically syndicated from The Sol AI Blog — daily AI analysis from a UK/EU/US perspective.
Follow along for more →
Top comments (0)