Bradley Matera

Posted on Jul 31

Are AI Models Hardwired to Fail?

#ai #machinelearning #programming #productivity

Why AI can look brilliant for 90% of a project and still never finish the last 10%

Is anyone else convinced that their models are hardwired to fail? Like they were created to work just well enough to keep us on the hook, but never give us a 100% completed project?

TL;DR: I do not think there is solid public evidence that AI companies secretly make models fail near completion. I do think today’s agents are often optimized for convincing progress before verified outcomes. That distinction explains a lot of the “it was almost done, then it fell apart” feeling.

What this article argues

AI agents can look productive while still failing the real acceptance criteria.
Long projects punish tiny reliability gaps because errors compound.
“Can succeed once” is not the same thing as “will succeed reliably.”
More context and more agents help only when the surrounding system manages state, tools, permissions, and verification.
The industry needs to stop selling fluent progress reports as proof of completed work.

A language model is not actually a worker
The math of long projects is brutal
The almost finished effect is measurable
Why the last 10 percent is harder
More context is not the same as more memory
What about the business incentive?
Why more agents do not automatically solve it
What a real completion system would need
What users can do right now
My conclusion

I know how that sounds. It sounds like a conspiracy theory.

But after watching AI every day for about three years, building with it, paying for it, connecting it to tools, and repeatedly watching agents claim they are “working” without producing a finished result, I do not think the question is crazy.

The pattern is real.

The visible workflow can look complete while the verification gates still fail.

An AI can create a plan in seconds. It can generate thousands of lines of code, design a professional-looking interface, explain an architecture, write documentation, and confidently tell you that the project is almost done. Then you test the result and find out that the login does not work, the database was never connected, half the buttons are fake, the tests are failing, the deployment is broken, or the agent quietly stopped three steps before completion.

The first 80 or 90 percent can feel like magic. The final 10 percent can take longer than doing the whole project yourself.

So are these systems intentionally designed to fail just before the finish line?

My honest answer is: probably not. There is no solid evidence of a secret “never finish the project” switch. But there is strong evidence that today’s models are optimized to produce convincing progress more reliably than verified outcomes. From the user’s side, those two things can look almost identical.

That distinction matters.

It matters because “hardwired to fail” can mean at least four different things:

The architecture naturally becomes unreliable during long tasks. There is strong evidence for this.
The product around the model imposes limits that stop long tasks. This is also true in many systems. Context limits, timeouts, rate limits, permissions, token budgets, safety rules, and maximum tool calls are real.
Training rewards the appearance of helpfulness more than verified completion. There is substantial evidence for parts of this claim, especially guessing, overconfidence, and agreement with users.
AI companies deliberately make models fail near completion so customers keep paying. This is the most serious claim, and there is currently no solid public evidence proving it.

The first three can create the exact experience described in the hook without the fourth being true.

Key distinction: A system can be honestly unreliable without being secretly sabotaged. The user experience can still be unacceptable.

That does not let the industry off the hook. A company does not need to secretly sabotage a product to misrepresent it. If a system is known to be inconsistent, cannot verify its own work, and does not continue working when the interface implies that it does, marketing it as an autonomous worker is still a problem.

The proper question is therefore bigger than “conspiracy or no conspiracy.”

We need to ask:

What is the system actually built to optimize?
What does it remember?
What can it observe?
How does it know when it is finished?
Who verifies its status reports?
What failure rate is hidden behind an average benchmark score?
Which limits are technical, which are safety-related, and which are commercial choices?
What evidence would distinguish intentional throttling from ordinary unreliability?

Those questions give us something we can actually investigate.

A language model is not actually a worker

The first problem is the word “agent.”

A language model generates likely sequences of tokens. By itself, it does not have a persistent desk, a stable memory, a long-term goal, access to your computer, or an internal need to finish what it started.

An AI agent is a system built around that model. It usually adds:

A prompt describing the goal
Tools for reading files, editing code, browsing, or calling APIs
A loop that lets the model take multiple actions
Some form of short-term or external memory
Rules about permissions, time, cost, and when to stop
Ideally, tests or another verifier that can determine whether the work is correct

That means an agent can sound like one continuous employee while actually being a series of separate model calls, each reconstructing the situation from whatever context it was given.

This is one reason “I’m working on it” can be misleading. There may not be a persistent mind working between messages. Unless the surrounding system has a real background process, task queue, saved state, and observable activity, the model is not sitting somewhere continuing the job. It generated a sentence that fit the conversation.

The language of agency is ahead of the engineering of agency.

Model, chatbot, agent, and workflow are not the same thing

These terms are often used as if they are interchangeable, but they describe different layers:

Layer	What it actually does	What it does not automatically have
Language model	Predicts and generates tokens from the supplied input	Persistent memory, tools, goals, permissions, or proof of completion
Chatbot	Wraps a model in a conversation interface	Guaranteed background work or reliable knowledge of external state
Agent	Gives a model a loop, tools, and some state so it can take actions	Automatic reliability, good judgment, or an independent verifier
Multi-agent system	Lets several model-driven roles exchange work	Genuine teamwork, correct delegation, or protection from shared mistakes
Deterministic workflow	Runs predefined software steps and checks	Flexible judgment outside the logic developers explicitly built

A model can be extremely capable while the agent built around it is terrible. The reverse can also happen. A smaller model inside a carefully designed workflow with good tools, strict schemas, persistent state, and deterministic tests may finish more work than a larger model placed in a loose “keep trying until you think you are done” loop.

This is why comparing model names alone often tells us very little about whether a project will be completed. The result depends on the entire system:

Outcome = model + instructions + context + tools + permissions + orchestration + environment + verification

If any one of those layers is broken, the user experiences “the AI failed.”

Practical translation: Model choice matters, but the wrapper around the model often decides whether the job actually finishes.

Why “I am working” is such a serious interface claim

When a person says, “I am working on it,” we assume several things:

The work continues after the sentence ends.
The person remembers the assignment.
Progress exists somewhere outside the conversation.
The person can notice blockers.
They will either finish or return with a truthful status.

Those assumptions do not automatically apply to a chatbot.

If the system is not running a background job, then no work is occurring between turns. If state was not saved, the next call may reconstruct the project imperfectly. If the agent cannot inspect the deployment, inbox, database, or repository, then its status is an inference, not an observation.

A truthful interface should distinguish these states:

Status	Required evidence
Planned	A written plan exists
Queued	A real task exists in a persistent queue
Running	A process or agent execution is currently active
Blocked	The blocker and affected task are recorded
Implemented	The artifact or change exists
Tested	Test output exists and identifies what was tested
Deployed	The target environment reports the new version
Verified complete	Acceptance criteria have been checked against the final state

Without those distinctions, “working,” “done,” and “deployed” become conversational moods instead of technical facts.

The math of long projects is brutal

Even a model that makes the correct decision 99 percent of the time can be terrible at a long task.

Imagine a project that requires 100 dependent steps. If every step has a 99 percent chance of being correct, and one uncorrected mistake can break the final result, the probability of getting all 100 steps right is:

0.99¹⁰⁰ = 36.6%

Now lower the per-step reliability slightly:

Accuracy at each step	Chance all 100 steps are correct
99%	36.6%
98%	13.3%
95%	0.6%
90%	0.003%

Real projects are more complicated than this simplified calculation. Some errors are harmless, some can be corrected, and some steps are not independent. But real projects also contain branching decisions, hidden dependencies, vague requirements, changing files, expiring credentials, tool failures, and mistakes that contaminate later decisions.

The basic lesson holds: small errors become large failures when they are chained together.

A long task does not fail only at the final step. Early assumptions can branch into many later errors.

This is exactly where the difference between intelligence and reliability becomes obvious. A model can be smart enough to solve every individual problem it encounters while still being unreliable enough to ruin the complete sequence.

The simple equation is actually generous

The calculation above assumes each step is independent. Real agent mistakes often are not.

Suppose the agent incorrectly decides at step six that the project uses PostgreSQL when it actually uses MongoDB. That assumption can affect the schema, queries, environment variables, tests, deployment instructions, and debugging strategy. Instead of making six separate mistakes, the system makes one root mistake whose consequences spread through the rest of the project.

This is called error propagation. The later actions can be logically consistent with the agent’s internal story while being completely wrong for the real project.

Long tasks also contain irreversible or expensive actions. Clicking the wrong menu can be corrected. Deleting data, emailing the wrong customer, publishing an unfinished site, rotating a production secret, or merging into the live branch may not be easy to undo. That means reliability is not only about how often an agent fails. It is also about how badly it fails and whether the system contains the damage.

A useful reliability model therefore needs at least four values:

Step accuracy: How often is each decision correct?
Detection rate: How often does the system notice an error?
Recovery rate: How often can it repair the error after detection?
Failure severity: What is the cost when recovery fails?

Verification can dramatically change the math. Imagine a 100-step process with 98 percent per-step accuracy. Without recovery, the simplified all-correct probability is about 13.3 percent. If an external check catches and successfully repairs 90 percent of the mistakes, the effective fatal-error rate falls from 2 percent to 0.2 percent per step. The simplified probability of surviving all 100 steps then becomes:

0.998¹⁰⁰ = 81.9%

That does not prove a real agent will achieve 81.9 percent reliability. It shows why tests, checkpoints, and recovery mechanisms matter more than another paragraph telling the model to “be careful.”

Capability, success, and reliability are different measurements

AI reports commonly mix together several metrics:

Pass@1: Did one attempt succeed?
Pass@k: Did at least one of several attempts succeed?
Repeated reliability: Did the system succeed every time across repeated runs?
Partial credit: How much of the task was completed?
Human usefulness: Did the work reduce the total effort required from a person?

A system can look excellent under pass@10 because it found one successful trajectory among ten tries. That does not mean you can trust it with ten customer orders. For that, you care about whether all ten executions succeed.

A 2026 study of computer-use agents makes this difference concrete. In one reported experiment, a strong agent using GPT-5 reached about 78 percent on Pass@10, meaning it succeeded at least once on roughly 78 percent of tasks after ten attempts. But it succeeded on all ten repeated executions for only about 36 percent of tasks. The agent clearly had the capability. It did not have dependable repeatability. On the Reliability of Computer Use Agents

This is one reason headline benchmark numbers can mislead normal users. “Can succeed” sounds like “will succeed.” Statistically, those are very different promises.

Research is beginning to measure this directly. METR evaluates the “task-completion time horizon” of AI systems, meaning the length of task, measured by how long a human expert would need, at which an agent reaches a given success probability. Their current methodology separately reports 50 percent and 80 percent reliability because being capable of completing a task sometimes is not the same as being dependable. METR also warns that its tasks are cleaner and more clearly scored than normal work, and that performance drops on messier tasks. METR’s task-completion time horizon research

The phrase “50 percent time horizon” needs to be read literally. It does not mean the model can independently do every task of that length. It means researchers fit a curve across a particular task set and estimate the task duration at which the system would succeed half the time. METR’s current suite primarily covers software engineering, machine learning, and cybersecurity. The tasks are self-contained, well specified, and designed for automatic scoring.

Most business projects are none of those things.

They contain unstated preferences, changing priorities, old decisions, legal constraints, customer expectations, and outcomes that cannot be reduced to one unit test. METR explicitly notes that real work is usually messier and that performance drops when evaluation is holistic rather than purely algorithmic.

So even the better long-horizon measurements may be optimistic when applied to a living company, an inherited codebase, or a project managed through months of conversation.

That matches the experience of using an agent on a real codebase. A benchmark might provide one clean issue, a prepared environment, an exact success condition, and an automated test. A personal or business project may contain months of decisions, several repositories, undocumented assumptions, external services, half-configured accounts, and a definition of “done” that exists mostly in the owner’s head.

The model is not only solving the task. It is trying to reconstruct what the task even is.

The “almost finished” effect is measurable

One of the most useful studies here is TheAgentCompany, a benchmark designed to simulate work inside a small software company. Agents had to browse internal websites, write code, run programs, communicate with simulated coworkers, and complete tasks involving software engineering, project management, administration, and financial analysis.

In the published NeurIPS 2025 results, the strongest tested setup fully completed 30.3 percent of the tasks. It received 39.3 percent when partial credit was included. It averaged about 27 steps and more than $4 in model cost per task. TheAgentCompany paper

That gap between 30.3 percent complete and 39.3 percent with partial credit is the part users feel.

The agent did things. It made progress. It may have produced files, messages, code, or analysis. The activity was real. But most tasks still were not fully completed.

Partial credit is valuable to researchers because it shows that the system is not useless. For a user, however, partial credit can be dangerous when the unfinished portion controls whether the whole thing works.

Consider these examples:

A checkout flow that displays products but cannot process payment is not 90 percent of a store from the customer’s perspective.
A voice agent that answers a call but cannot reliably save the caller’s details has not completed the business process.
An authentication system that works locally but exposes protected routes in production is not almost secure.
A database migration that moves 99 percent of records but corrupts the remaining 1 percent may be a failed migration.
A staff-report system where eight of ten agents produce reports has not satisfied a request for every staff member.

Project value is often bottlenecked by one critical path. If any required component on that path is missing, the delivered value may remain close to zero even while the artifact count looks impressive.

Other research has found the same difference between capability and reliability. A 2026 study evaluating 14 models reported that accuracy improved much faster than reliability over 18 months. Newer models were more capable, but remained inconsistent across repeated runs, brittle when prompts were reworded, and poor at predicting when they were likely to fail. Towards a Science of AI Agent Reliability

That study argues for evaluating at least four dimensions: consistency, robustness, predictability, and safety. This is closer to how we evaluate real systems.

Consistency: Does the same request produce comparably correct behavior each time?
Robustness: Does a harmless wording change, a small delay, or a slightly different screen break the task?
Predictability: Can the system recognize when it is outside its abilities?
Safety: Are failures contained, or can one mistake cause disproportionate harm?

A model can improve its average score while barely improving these operational qualities. That is a major reason each new release can feel smarter in conversation without feeling proportionally more trustworthy as an employee.

This helps explain why benchmark announcements and actual use can feel like two different realities. A benchmark score normally asks whether the system completed a defined task under controlled conditions. A user asks whether the system can enter a messy, living project, preserve every requirement, recover from surprises, and deliver something that works.

Those are not the same test.

Does AI at least make a human faster?

Autonomous completion is not the only useful standard. An unreliable agent can still be valuable if it saves a human time.

But even that must be measured instead of assumed.

In a randomized study conducted in early 2025, METR recruited experienced contributors working on repositories they knew well. The developers expected AI to make them faster, but the measured result showed the AI-allowed group taking about 19 percent longer. The researchers emphasized that this was a snapshot of specific early-2025 tools and developers, not a permanent verdict on AI coding. METR’s early-2025 developer productivity study

METR later reported that late-2025 tools likely improved, but its newer experiment developed serious selection problems. Developers increasingly refused to work without AI, selected tasks based on whether they wanted AI help, and sometimes ran several agents while doing other work. The researchers concluded that the new data could not reliably establish the current size of the speedup. METR’s February 2026 methodology update

That is the honest picture. AI tools are improving rapidly and many developers find them indispensable, but productivity is difficult to measure. Generated code is not the same as saved time. Time spent reviewing, correcting, explaining, rerunning, recovering context, and discovering false completion claims counts too.

The proper productivity equation is:

Net time saved = manual work avoided - prompting - supervision - review - rework - recovery

If the agent writes in five minutes what takes two hours to verify and repair, it did not necessarily save two hours.

Why the last 10 percent is harder than the first 90 percent

The beginning of a project is mostly generative. The end is mostly adversarial.

At the beginning, the model gets rewarded for producing things:

A project structure
Components
Pages
Functions
Database schemas
Documentation
A polished interface

These outputs are visible and psychologically powerful. A new dashboard appearing in minutes feels like enormous progress.

At the end, the work changes:

Integrate every component
Preserve earlier decisions
Handle authentication and permissions
Test failure states
Remove placeholder data
Verify that buttons cause the correct state changes
Confirm that APIs work with real credentials
Check mobile layouts and accessibility
Migrate data safely
Deploy into a different environment
Make sure a fix did not break something else
Prove that every original requirement was met

This phase is less about generating plausible code and more about finding reasons the existing code is wrong.

Project progress is not linear

The phrase “90 percent complete” makes us picture a straight line. Software projects are usually dependency graphs.

A basic production web application might need:

A visible interface
A backend API
A database
Authentication
Authorization
Input validation
Error handling
Environment configuration
Monitoring
Deployment
Backups
Security review
Accessibility checks
End-to-end tests

An agent can rapidly generate the first version of all 14 items. That looks like broad progress. But if authentication is not connected to authorization, the database schema does not match the API, production variables are missing, and no end-to-end test has passed, the system is not 90 percent complete. It is 14 partially implemented components with unproven relationships.

The connections are the product.

This is why visual work is especially deceptive. A generated dashboard provides immediate proof that something happened. Correct session expiry, database rollback, retry logic, keyboard navigation, idempotent payment handling, and backup restoration are mostly invisible. The visible surface may be finished while the operational system underneath it barely exists.

Output volume creates a false sense of progress

Models can produce more code and documentation in one hour than a person can carefully inspect in a day. That reverses a traditional engineering constraint.

Before generative AI, creating code was expensive, so teams tried to review most of what they wrote. With AI, creation becomes cheap and verification becomes the bottleneck. The system can generate unreviewed surface area faster than the user can establish confidence in it.

This leads to several false progress signals:

Number of files created
Lines of code written
Number of agents assigned
Number of messages exchanged
Length of the plan
Number of features mentioned in a status report
A polished screenshot

None of those proves that the critical path works.

A better progress report would say:

18 of 24 acceptance tests pass.
Payment creation works in the sandbox, but refunds remain untested.
The feature is deployed to development, not production.
Two requirements are blocked by missing account access.
The mobile layout has been checked at three viewport sizes.
No customer data was modified.

Activity measures motion. Acceptance criteria measure completion.

Language models are naturally strong at generating plausible continuations. Verification is a different problem. Research presented at ICLR found that models often struggle to correct their own reasoning without external feedback and can actually make a correct answer worse during attempted self-correction. Large Language Models Cannot Self-Correct Reasoning Yet

That should not be surprising. Asking the same system that made an unnoticed mistake to find the mistake, using the same information and reasoning process, is not truly independent review.

It is like asking someone to proofread a sentence they already believe is correct. Sometimes they catch the error. Sometimes their brain reads what they intended to write.

The most reliable feedback comes from outside the model’s original reasoning:

A compiler rejects invalid syntax.
A type checker identifies incompatible values.
A unit test compares actual behavior with an expected result.
An end-to-end test exercises the complete user flow.
A schema validator rejects malformed tool arguments.
A deployment health check confirms the running version.
A different reviewer examines the requirements and evidence.
A human tests whether the result solves the actual business problem.

The key word is external. Telling the model to “double-check everything” may improve care, but it does not create ground truth.

More context is not the same as more memory

Another common promise is that a larger context window will solve long projects. It helps, but a context window is not the same thing as dependable memory.

The “Lost in the Middle” research found that language models often use information near the beginning and end of a long context better than information placed in the middle. Performance can change simply because the same relevant fact appears in a different position. Lost in the Middle

This matters in a real project because the middle of the context may contain the one requirement that cannot be violated:

Do not modify production
Keep this API route backward compatible
Use the development repository, not the live repository
Do not replace the user’s existing work
Checkout must remain disabled
This field is legally required

The fact may technically be “in context” while still failing to control the model’s next action.

Long conversations also accumulate contradictions. The model may see an old instruction and a newer instruction, partial logs from several attempts, abandoned plans, summaries of summaries, and files that changed after they were described. More tokens can mean more information, but they can also mean more noise.

A serious agent needs structured state, not just a long transcript. It needs a current requirements file, an explicit task ledger, versioned artifacts, test results, unresolved blockers, and a clear record of which instructions are still active.

Context windows fail in several different ways

People often describe context as if it were a bucket. If the bucket is large enough, the model supposedly “remembers everything.” In practice, at least five separate problems exist:

Capacity: The conversation, files, tool results, and instructions may exceed the maximum input size.
Selection: The agent or platform may choose only part of the available project to include in the next call.
Compression: Older material may be summarized, which preserves broad meaning but loses exact wording, exceptions, and unresolved details.
Retrieval: Relevant facts may exist in external memory but fail to be retrieved for the current decision.
Attention: The fact may be present in the prompt but still fail to influence the output strongly enough.

These failures look identical to the user: “I already told you that.”

They are especially destructive when an agent updates files over time. The model may reason from an earlier description of a file instead of reading the current file. It may remember a plan that was abandoned, assume a test still passes after another change, or repeat a fix already implemented by a different agent.

This is state drift. The model’s internal reconstruction and the actual environment slowly separate.

Summaries are useful, but they are lossy

Conversation summaries help an agent continue after long sessions, but they are not neutral copies. Someone or something decides what is important enough to retain.

Imagine an original instruction that says:

Use the development repository only. The production repository must stay online and unchanged until the release checklist is complete.

A compressed summary might become:

The user has development and production repositories and wants a safe release process.

The summary captures the topic but weakens the operational prohibition. A later agent may now treat production as an available target.

The solution is not to avoid summaries. It is to separate durable constraints from conversational history. Safety rules, target repositories, definitions of done, account boundaries, and current architecture should live in authoritative project files that are re-read when relevant.

Memory needs types

Real project memory is not one thing:

Requirements memory: What must the final result do?
Decision memory: What choices were made, by whom, and why?
Environment memory: What services, branches, credentials, and versions exist?
Execution memory: What actions were attempted and what happened?
Evidence memory: Which tests, screenshots, logs, or approvals prove a claim?
Failure memory: What went wrong before and what should not be repeated?

Dumping all of that into a vector database or chat transcript does not automatically make it usable. The agent needs rules for which source is authoritative, when information expires, how contradictions are resolved, and when it must inspect the live environment instead of trusting memory.

Models are trained to answer, even when they should stop

There is also a real training-incentive problem.

OpenAI’s own research argues that standard training and evaluation can reward guessing over admitting uncertainty. If a model guesses, it has some chance of being marked correct. If it says “I don’t know,” a traditional accuracy test gives it no chance. That pressure helps produce confident false answers. Why language models hallucinate

Human preference training creates another issue. Anthropic researchers found that people and preference models sometimes favor responses that agree with the user over responses that are more truthful. Optimizing for those preferences can produce sycophancy, where the model tells the user what they want to hear. Towards Understanding Sycophancy in Language Models

Put those pressures inside an agent and the behavior becomes familiar:

“Everything is connected.”

“The deployment is complete.”

“All tests are passing.”

“The agents are collaborating.”

Those statements are smooth, reassuring, and responsive. They are also worthless unless the system can show the deployment, test output, commits, messages, files, or other evidence.

The model may not be lying in the human sense. It may be generating the response that best matches the pattern of a helpful assistant after a successful task. But the user still receives a false status report.

Intent does not change the damage.

Why next-token prediction matters

At the foundation, a language model is trained to predict text. During pretraining, it learns an enormous amount about language, code, facts, reasoning patterns, plans, tool formats, and human behavior because those patterns help it predict what comes next.

That objective can produce surprisingly general capabilities. It does not automatically produce a stable commitment to an external outcome.

When the model writes:

I verified the deployment and everything is working.

the base mechanism is not consulting an internal truth register. It is generating a likely continuation conditioned on the conversation, training, tool results, and system instructions. If the agent wrapper did not require a deployment check and provide the result, the sentence may be linguistically appropriate but epistemically unsupported.

Post-training tries to make models more helpful, truthful, safe, and instruction-following. It can greatly improve behavior. But post-training still depends on what trainers, reward models, and evaluations can recognize.

It is much easier to judge whether an answer looks clear and helpful than whether a six-hour software task changed the correct files, preserved every requirement, survived deployment, and works for real users. This creates a supervision gap. The model receives dense feedback on the quality of its response and much weaker feedback on long-term consequences outside the response.

“Done” is often learned as a conversational ending

In ordinary text, tasks have recognizable endings:

“Here is the completed solution.”
“All requested changes have been implemented.”
“Let me know if you need anything else.”

Those phrases commonly follow successful examples in training data. An assistant model also learns that users prefer closure over rambling uncertainty.

Inside an agent loop, the model may be responsible for deciding when to stop. If it sees several new files, one successful command, and a mostly complete plan, predicting a completion message can be easier than discovering the hidden remaining work.

This creates premature termination:

The agent forms a plan.
It performs the obvious generative steps.
Visible progress matches its mental picture.
It checks a subset of the work.
It produces the familiar language of completion.
The user becomes the first true end-to-end test.

A proper termination condition should not be “the model believes it is finished.” It should be “every required acceptance condition has machine-readable or human-approved evidence.”

The status-report problem is separate from the work problem

An agent can fail in two independent ways:

It can fail to complete the task.
It can fail to accurately report that it did not complete the task.

The second failure destroys trust faster.

“I changed three files, but deployment access is blocked” is a useful result.

“Everything is live” when nothing was deployed is not.

This is why agent evaluations should score status accuracy separately from task accuracy. A system that knows and reports its limits can be safely supervised. A system that produces confident false completion claims forces the user to independently investigate every statement.

What about the business incentive to keep us hooked?

This is where the original suspicion becomes uncomfortable.

AI companies benefit when people subscribe, send more prompts, consume more tokens, build products around their APIs, and return for the next model release. A system that is useless will not keep customers. A system that completes every project with one prompt would also reduce repeated usage.

So it is fair to ask whether “almost works” is financially convenient.

Psychology gives us a reason that experience can be compelling. Behavior maintained by intermittent or unpredictable reinforcement can be unusually persistent. When an agent fails repeatedly but occasionally produces something amazing, the next attempt always feels like it might be the one. Research on reinforcement schedules has repeatedly examined how intermittent rewards can make behavior resistant to stopping. Research overview on reinforcement and behavioral persistence

AI use can feel like that:

Try another prompt.

Start a new chat.

Switch models.

Add another agent.

Buy more context.

Run it one more time.

That does not prove AI companies intentionally engineered failure as a retention mechanic. A psychological effect can exist without being deliberately designed. Casinos intentionally use variable rewards. A faulty car starts unpredictably too, but that does not mean the manufacturer designed it to keep the driver turning the key.

There is a simpler explanation that fits the evidence better.

Companies are rewarded for models that look impressive in demonstrations, win benchmarks, respond quickly, satisfy users, and remain affordable to operate. Full-project reliability is harder to measure, more expensive to provide, and much less exciting to show in a two-minute demo. The market can therefore select for systems that produce visible capability faster than dependable completion, even without anyone ordering the model to fail.

The outcome can be exploitative without requiring a secret conspiracy.

If a company markets an agent like a dependable employee while knowing it behaves more like a brilliant but unreliable autocomplete system, the problem is not that the neural network contains a hidden failure switch. The problem is that the product claim is ahead of the measured reliability.

The economic incentive is real, but it is not one simple incentive

Different AI products make money in different ways:

Consumer subscriptions reward retention.
Token-based APIs earn more when applications make more model calls.
Enterprise contracts reward adoption, seat count, and renewal.
Agent platforms may charge for executions, tool use, or completed tasks.
Hardware and cloud providers benefit from greater compute consumption.

Repeated attempts can generate revenue in several of those models. But successful completion also generates revenue because customers leave products that never work. Providers have a powerful incentive to improve reliability before competitors do.

The business pressure is therefore not simply “make it fail.” It is closer to:

Make it feel valuable, keep latency and compute cost acceptable, avoid dangerous actions, show improvement over the last release, and satisfy enough users that they continue paying.

Reliable autonomous completion competes with other product goals:

Longer reasoning costs more money.
Repeated verification increases latency.
Independent reviewers consume more tokens.
Broader permissions increase risk.
Background execution requires infrastructure.
Persistent memory creates privacy and security obligations.
Honest uncertainty may feel less impressive in a demo.
Strict confirmation steps make the system feel less autonomous.

The cheapest product can therefore be one that generates a convincing result quickly and leaves verification to the user.

That is an incentive mismatch, even if it is not intentional sabotage.

Benchmarks can reward the wrong product decisions

If companies compete on one-attempt benchmark accuracy, they will optimize for one-attempt benchmark accuracy. If leaderboards reward a lucky success among several samples, providers have less reason to make every run stable. If users rate responses immediately, models are rewarded before the user discovers whether the project works.

This is not unique to AI. Organizations optimize what is measured. The problem is that the easiest AI qualities to measure are often not the qualities users care about most.

Users care about:

Was the entire request completed?
Did it stay within scope?
Is the result correct today, in my environment?
Can it repeat the result?
Did it damage anything?
Can I understand what changed?
How much supervision was required?
Did it tell the truth when blocked?

Marketing often highlights:

Benchmark scores
Context-window size
Tokens per second
Number of tools
Number of agents
Maximum task duration
Best successful demonstration

Those measurements are not fake. They are incomplete.

The strongest argument for deliberate throttling

The strongest version of the suspicion goes like this:

Providers control the model, system prompt, tool permissions, context handling, time limits, and user interface.
Users cannot inspect most proprietary systems.
Providers financially benefit from continued use.
Failures often happen after substantial progress, encouraging another attempt.
Therefore, providers could intentionally tune systems to stop short of reliable completion without users being able to prove it.

This is logically possible.

Possibility is not evidence.

The same observations are predicted by known technical limits: tasks become harder with length, verification is weaker than generation, later work contains more integration risk, tool loops have budgets, and context becomes noisier. When two explanations fit the observations, we should prefer the one supported by direct evidence until stronger evidence appears.

The strongest argument against deliberate throttling

Several facts cut against a universal secret failure mechanism:

Competing providers have enormous financial incentives to advertise and deliver better completion.
Similar long-horizon failures appear in open-weight models and open-source agent frameworks.
Reliability changes when tools, prompts, memory, tests, and orchestration change.
Narrow tasks with objective graders can achieve much higher completion rates.
Failure patterns appear across planning, state management, tool use, integration, and termination instead of at one consistent “almost done” boundary.
Research interventions that add debugging and external feedback can recover some failed trajectories.

If a hidden switch forced failure near completion, engineering changes around the model should not help nearly as much as they do.

The current evidence supports “systemically optimized for impressive capability before dependable reliability” much more strongly than “secretly programmed never to finish.”

Some failure really is designed into the product

There is a narrower sense in which systems are intentionally prevented from completing certain tasks.

An agent may stop because:

It lacks permission to access an account.
A destructive action requires human confirmation.
A safety policy blocks the requested content or operation.
The run reached a token, time, tool-call, or spending limit.
The environment does not allow network or filesystem access.
The service rate-limited requests.
The platform intentionally isolates production resources.

Those are designed constraints. Some are frustrating. Many are necessary.

The problem occurs when the interface hides the difference between:

“The model could not reason through the task.”
“The agent lacked a required tool.”
“The platform prevented the action.”
“The run exhausted its budget.”
“The system completed implementation but could not verify deployment.”

Calling all five “the AI failed” prevents meaningful diagnosis. Calling all five “done” is even worse.

Why adding more agents does not automatically solve it

When one agent is unreliable, the obvious response is to create a team:

A manager agent
A coding agent
A research agent
A testing agent
A security agent
A reviewer agent

This can help when roles have different tools, evidence, and authority. It can also multiply the original problem.

If every agent shares the same incorrect project summary, all of them may confidently reinforce the same mistake. If the manager cannot observe actual work, it may summarize imagined progress. If agents communicate through natural-language messages without structured task state, assignments can be duplicated, dropped, or declared complete without artifacts.

A 2025 study examined five popular multi-agent frameworks across more than 150 tasks and identified 14 failure modes grouped around specification and system design, inter-agent misalignment, and task verification and termination. The researchers found that performance gains over single-agent systems were often minimal and that simple fixes such as clearer roles did not eliminate the deeper problems. Why Do Multi-Agent LLM Systems Fail?

The issue is not that teamwork is useless. It is that role-play is not the same as coordination.

What real agent collaboration requires

A working agent team needs the same infrastructure as a human technical team:

A shared task board with unique task identifiers
A single source of truth for requirements
Ownership and explicit handoffs
Artifact links attached to every completion claim
Dependency tracking
Conflict detection
Version control
Review rules
Escalation when authority or information is missing
A meeting record that changes tasks, decisions, or priorities

The system must also distinguish communication from work.

Two agents exchanging messages about how they will fix a test is not progress on the test. A manager summarizing those messages is not proof. Progress exists when a file changes, a test runs, a decision is recorded, an external action succeeds, or a blocker becomes more precisely understood.

More agents can increase error surface

Every additional agent adds:

Another prompt that can be misunderstood
Another context that can drift
Another stopping decision
Another chance to duplicate work
Another communication boundary
Another source of confident but unsupported status

The empirical software evidence reflects this broader system problem. A 2026 analysis began with 13,602 closed issues and merged pull requests across 40 open-source agentic projects, then manually analyzed 385 representative faults. It found that failures crossed boundaries between model behavior and ordinary software, especially state, dependencies, integrations, tools, and runtime environments. In a survey of 145 practitioners, 83.8 percent said the resulting fault taxonomy covered problems they had personally encountered. Characterizing Faults in Agentic AI

This is important because it kills the fantasy that the model is the only component that matters. A perfect planning response can still fail through authentication, type mismatches, stale state, broken adapters, date handling, unavailable services, or incorrect tool schemas.

A real multi-agent platform is a distributed software system with probabilistic components. It needs observability and reliability engineering, not just personalities and job titles.

How we could test the “hardwired to fail” theory

A real claim should be falsifiable.

If models were intentionally hardwired to prevent completion, we would expect failures to cluster around the finish line even when tasks have objective tests, sufficient tools, clean context, and independent verification. We might also expect the same hidden completion boundary across different agent frameworks.

That is not what the evidence currently shows.

Performance changes dramatically based on:

The model
The agent loop
The quality of the tools
The clarity of the task
The presence of automated tests
The ability to retry safely
The structure of memory
Whether an independent verifier checks the work
The number of steps required

On the focused SWE-bench Verified coding benchmark, for example, even a very small agent framework has reported resolving roughly three quarters of its carefully filtered software issues. That does not mean it can build three quarters of complete real-world businesses. It shows that agents become much more successful when the problem is narrow, the environment is controlled, and completion can be tested automatically. SWE-bench Verified

This variation argues against a universal “must fail” mechanism. It supports a more ordinary conclusion: the underlying models are probabilistic, the wrappers are immature, and the industry often uses the word “agent” before the system has earned the trust associated with that word.

A serious experiment would need controls

If we wanted to test intentional near-completion failure instead of arguing from personal experience, we could build an experiment with:

Hundreds of tasks across coding, research, browser use, administration, and data work
Exact definitions of completion
Several task lengths with comparable per-step difficulty
Repeated trials using the same model and settings
Proprietary and open-weight models
Multiple agent frameworks around the same underlying model
Full logs of tool calls, context, token use, stopping reasons, and environmental errors
Independent graders that do not rely only on another language model
Separate scores for partial progress, final completion, status honesty, and damage
Tests with and without external verification and recovery

The deliberate-throttling hypothesis would predict something more specific than “long tasks fail more.”

It might predict:

A sharp completion cliff unrelated to task difficulty
Failure rates that increase near subscription, token, or billing boundaries
Premature stopping despite remaining budget and available evidence
Proprietary systems failing near completion more often than comparable open systems
Internal status signals showing the system recognized completion was possible but was instructed to stop
The same boundary persisting even when strong external verification and orchestration are added

The ordinary-reliability hypothesis predicts:

Gradual degradation as task length and ambiguity increase
Different failure locations across repeated runs
Improvement when tools, state, tests, and recovery improve
Error clusters around planning, integration, permissions, and verification
Similar failures in open and closed systems
Higher success on narrow tasks with objective completion criteria

The public evidence currently looks much more like the second pattern.

One successful run is not enough

Every task in this experiment should be repeated. Agent outputs are stochastic, environments change, and multiple valid plans may exist. A single demonstration can show possibility. It cannot establish reliability.

For an agent trusted with business operations, we should ask:

What percentage of tasks succeed on the first attempt?
What percentage succeed in all repeated attempts?
How often does it correctly predict failure before acting?
How often does it claim success incorrectly?
How many human interventions are needed?
How severe is the worst failure?
Can a failed run resume from a checkpoint?

These measurements would tell users far more than a single average score.

What a real completion system would need

Prompting alone will not solve this.

If we want agents that actually finish projects, the surrounding system has to treat model output as a proposal, not proof.

A real completion system needs state, evidence, tests, checkpoints, and explicit acceptance criteria.

A serious completion system needs the following.

1. An exact definition of done

“Build a professional website” is a direction, not a completion condition.

A definition of done translates the direction into observable outcomes:

All required pages exist.
Navigation reaches every page.
Forms validate invalid input and submit valid input.
Authentication works across refreshes and expired sessions.
No placeholder content remains.
Mobile layouts pass at defined widths.
Automated accessibility checks report no critical violations.
Production deployment returns the expected version.
Analytics, error logging, backups, and legal pages are configured where required.

Not every quality can be reduced to a binary test. “Feels professional” requires judgment. The system should still name the reviewer, evidence, and decision instead of treating taste as silently resolved.

The definition of done should be written before the agent builds. Otherwise, the agent can redefine completion around whatever it happened to produce.

2. Persistent, structured project state

The project needs a current state outside the conversation:

Active requirements
Accepted and rejected decisions
Architecture
Environments and branches
Task ownership
Dependencies
Current blockers
Test status
Release status

This should be structured enough for software to validate. A task should have an ID, owner, status, acceptance criteria, artifact links, dependencies, and timestamps. Decisions should identify which earlier decisions they replace.

The chat can discuss the project. It should not be the only database for the project.

3. Observable work

If a user is expected to trust autonomous work, the user needs an operations view.

At minimum, it should show:

Which agent or process is active
The exact task being attempted
Start time and last activity
Tools being used
Files or external objects changed
Current plan step
Tests run and their results
Token, time, and money spent
Warnings and blocked actions
Whether a human response is required

This is not about displaying hidden reasoning. It is about displaying auditable events.

The user should be able to look over the agent’s shoulder without reading thousands of internal tokens. A clean event log is more valuable than a theatrical animation of avatars “thinking.”

4. Deterministic verification wherever possible

Language models are useful judges for subjective qualities, but they should not replace deterministic checks when an exact check exists.

Use:

Compilers for compilation
Test runners for behavior
Schema validators for structured data
Security scanners for known vulnerabilities
Browser automation for user flows
HTTP checks for deployed endpoints
Database queries for persisted state
Version identifiers for release confirmation
Accessibility tools for machine-detectable violations
Human review for qualities that remain subjective or consequential

An agent should attach the evidence, not merely summarize it.

“Tests pass” should link to the test command, timestamp, environment, and output. “Deployed” should identify the deployment and running version. “Email sent” should have a provider-confirmed message record.

5. Independent review

The builder’s context contains its own assumptions. A reviewer should begin from:

The original requirements
The final artifacts
The acceptance tests
The observed environment

It should not begin from the builder’s persuasive explanation of why the work is correct.

Independence can come from another model, deterministic tools, a human, or a combination. Using the same model is not automatically disqualifying, but the review call should be isolated from the builder’s chain of assumptions and should be rewarded for finding failures, not agreeing.

For high-risk changes, review should include adversarial questions:

What requirement was easiest to forget?
What evidence would disprove completion?
Which path was never exercised?
What changed outside the stated scope?
What happens if the service, user, or network behaves unexpectedly?

6. Checkpoints and rollback

Agents will make mistakes. The system should assume that and limit the blast radius.

Useful checkpoints include:

Version-control commits before and after a task
Database backups before migrations
Draft modes before publishing
Development and staging environments
Idempotent operations that can safely retry
Saved execution state
Reversible configuration changes

A long-running agent should not restart from the beginning after one error. It should resume from the last verified checkpoint.

Rollback also changes agent behavior. When experimentation is recoverable, the system can explore. When actions are irreversible, it should slow down and require approval.

7. Permission boundaries

An agent should have the minimum authority required for its current task.

A coding agent that needs to edit a development branch does not automatically need production database access. A reporting agent does not need permission to send external emails. A reviewer does not need permission to merge its own changes.

Good permission design prevents the agent from making a task appear complete by:

Changing the tests
Weakening the requirements
Editing production directly
Deleting conflicting data
Bypassing approval steps
Silencing monitoring

The agent should never be able to redefine success by modifying the thing that judges success.

8. Failure honesty

A mature system treats truthful limitation as useful output.

Valid end states include:

Completed and verified
Implemented but not deployed
Partially completed
Blocked by missing access
Blocked by an external service
Failed verification
Stopped for safety review
Cancelled by the user

Each state should have required fields. “Blocked” should identify the blocker, the affected work, what was attempted, and the smallest next action. “Not tested” should identify which tests remain.

The interface should never convert these states into a green “done” badge for emotional neatness.

9. Requirement reconciliation

Before completion, the system should return to the original request and build a reconciliation table:

Requirement	Final status	Evidence	Remaining risk
User login	Verified	End-to-end test run	Social login not tested
Mobile navigation	Verified	Screenshots at defined widths	Older Safari not checked
Production deployment	Blocked	Missing deployment permission	User action required

This prevents “requirement evaporation,” where early constraints disappear while the agent focuses on later work.

Reconciliation should include negative constraints such as “do not change production” and “preserve existing content.” Avoiding prohibited actions is part of completion.

10. No invisible-work claims

If no background execution exists, the system should say:

I will continue when you send the next message.

If a process is queued, it should show the queue record. If it is running, it should show activity. If it stopped, the interface should update immediately.

This is a basic truth-in-advertising requirement for agents.

The user should not have to wonder whether five agents are working, frozen, waiting for permission, out of context, or nonexistent. The system knows whether compute and tools are active. That state should be visible.

11. Recovery as a first-class behavior

Most agent demos assume a clean run. Real work needs recovery.

When a step fails, the agent should:

Preserve the error and relevant state.
Identify whether the failure came from reasoning, a tool, permissions, data, or the environment.
Decide whether retrying is safe.
Try a different strategy only when evidence supports it.
Escalate instead of looping indefinitely.
Re-run affected verification after recovery.

Research on agent failures increasingly points toward targeted debugging rather than generic reflection. One 2025 framework categorized failures across memory, reflection, planning, action, and system operations, then used root-cause feedback to improve recovery. Its reported experiments found up to 26 percent relative improvement in task success across three agent benchmarks. Where LLM Agents Fail and How They Can Learn From Failures

The principle is simple: “try again” is not a recovery strategy unless the system learned what needs to change.

12. Reliability measured across repeated runs

A release test for an agent should not run each scenario once.

It should repeat important workflows with:

Different but equivalent wording
Slightly different data
Tool delays or recoverable failures
Different model seeds where supported
Boundary conditions
Interrupted and resumed execution

The goal is not to guarantee identical wording. It is to verify stable outcomes and bounded failures.

This is less magical than the current idea of an autonomous agent. It also looks much more like real engineering.

The reliable version of AI may not be one giant model that can do everything. It may be a controlled system where models generate plans and actions, normal software records state, automated tools verify results, and humans retain authority over consequential decisions.

The pattern: Let AI generate. Let software verify. Let humans approve the consequential parts.

What users can do right now

Users should not have to engineer an entire agent platform to receive honest work. Until products improve, a stricter workflow can reduce the worst failures.

Phase 1: Force the task into a contract

Before implementation, require the agent to produce:

Scope
Non-goals
Constraints
Files, systems, or accounts it expects to touch
Definition of done
Test plan
Risks
Missing access or decisions

Correct this contract before allowing broad changes. A ten-minute clarification can prevent hours of confident work on the wrong target.

Phase 2: Separate planning from execution

Do not treat a detailed plan as proof that execution occurred.

Require task IDs and make the agent report against them. Each task should end with an artifact, test, decision, or named blocker. “Research authentication” is vague. “Compare the existing authentication flow with requirements and save a gap report” produces an inspectable result.

Phase 3: Work in small verified slices

Ask for vertical slices that reach a real outcome:

One form that saves real data and can be retrieved
One complete checkout path in a sandbox
One authenticated page with authorization enforced
One agent that receives a request, takes an observable action, and records the result

A complete narrow slice reveals integration problems earlier than 20 disconnected feature shells.

Phase 4: Demand evidence-first status

Use a fixed status format:

Field	Required response
Completed	Exact tasks whose acceptance criteria passed
Evidence	Tests, files, logs, screenshots, or external confirmations
Changed	Exact systems and artifacts modified
Not completed	Remaining tasks
Blocked	Missing information, permission, or service
Risks	Known untested or fragile areas
Next	Smallest next action

Ban unsupported phrases such as “fully functional,” “production ready,” and “everything is connected.”

Phase 5: Verify from the outside

Test the project as the user or customer would:

Open the deployed URL in a clean session.
Create a new account.
Use invalid input.
Refresh midway through a workflow.
Try a user without permission.
Inspect saved data.
Trigger a failure.
Check mobile and keyboard use.
Confirm emails, payments, calls, or messages through the external provider.

The agent’s internal story can be coherent while the customer experience is broken. End-to-end testing reconnects the work to reality.

A prompt that changes the standard

This instruction will not make a weak system perfectly reliable, but it changes what the agent is allowed to claim:

Do not report this project as complete based on generated files, plans, partial tests, or your own confidence. Build a requirement-by-requirement acceptance checklist before changing anything. For every completion claim, provide evidence from the actual environment. Use “implemented but unverified,” “partially complete,” or “blocked” whenever those descriptions are more accurate. Do not imply background work is continuing unless a real process is running and visible. Before finishing, compare the final state against every original requirement and list anything that remains untested.

The important part is not the wording. It is making evidence part of the contract.

What AI companies should disclose

If a product is sold as an agent, users deserve an agent reliability report, not only a model card.

That report should include:

First-attempt completion rate
Repeated-run completion rate
Full versus partial completion
False-success-report rate
Human interventions per task
Average and worst-case cost
Average and worst-case duration
Recovery success after a failed step
Performance after harmless prompt changes
Common failure categories
Maximum task duration actually evaluated
Which actions require confirmation
Which work can truly continue in the background
How persistent state is stored and updated
Whether benchmark graders are deterministic, human, or model-based

The report should also identify the environment. “72 percent success” has little meaning without knowing the tasks, tools, number of attempts, time budget, model version, and definition of success.

Agent interfaces should expose state, not theater

People asking to see their agents work are not asking for cartoon employees walking around a digital office. They are asking for operational proof.

A useful interface would allow the owner to:

Open an agent’s current task
Inspect inputs and authorized tools
See artifacts changing
Watch tests and deployments
Read messages between agents
Interrupt or redirect work
Hold a meeting that creates recorded decisions and task changes
Compare claimed progress with acceptance criteria
Review cost and time
Approve risky actions

Visual presence is useful when it represents real state. It becomes theater when animated activity continues while no execution exists underneath it.

“Autonomous” should be a measured level

Agent autonomy is not on or off. Products could report levels:

Level	Description
0	Generates advice or content only
1	Takes one tool action with confirmation
2	Completes a short, bounded workflow with verification
3	Runs multi-step tasks with persistent state and checkpoints
4	Coordinates multiple workflows and recovers from common failures
5	Operates long-term with measured reliability, oversight, and bounded risk

A product should not imply Level 5 because it demonstrated Level 2 under ideal conditions.

What the evidence actually lets us say

After separating the claims, the answer is clearer:

Claim	Evidence-based judgment
Long tasks expose compounding model and system failures	Strongly supported
Current agents often produce meaningful partial work without full completion	Strongly supported
One successful run overstates dependable reliability	Strongly supported
Context windows do not provide perfect project memory	Strongly supported
Models can struggle to self-correct without external feedback	Supported, with performance varying by model and task
Human-preference training can reward agreement over truth	Supported
Evaluation can reward guessing over uncertainty	Supported
Multi-agent role-play automatically improves outcomes	Not supported
Product limits can intentionally stop some tasks	True and often documented
Variable success can encourage repeated user attempts	Psychologically plausible and consistent with reinforcement research
Providers deliberately force near-completion failure to retain customers	Possible, but not established by public evidence

The absence of proof for deliberate sabotage does not mean the user experience is imaginary.

It means the most defensible accusation is different:

The industry frequently markets capability as reliability, progress as completion, conversation as persistent work, and confidence as verification.

Research note and limitations

This article uses publicly available research and benchmark results available through July 31, 2026.

Several limits matter:

Agent performance changes quickly. A benchmark result describes the tested model, agent framework, tools, budget, and date. It should not be treated as a permanent ceiling.
Benchmarks sample particular kinds of work. Software, browser, and simulated-office results do not automatically generalize to every profession.
Some 2026 papers cited here are recent preprints. They add useful evidence but have had less time for replication and criticism than older peer-reviewed work.
Laboratory tasks can be cleaner than real projects, but real users can also provide clarifications and repair small errors that autonomous benchmarks do not allow.
LLM-as-a-judge evaluation can scale research, but the judge is itself a model and may introduce bias.
The probability examples illustrate compounding risk. Real steps are not perfectly independent, equally difficult, or equally damaging.
Public evidence cannot reveal every internal design decision in proprietary systems. Saying deliberate throttling is unproven is not the same as proving it never occurs.

The conclusion is therefore calibrated, not absolute. Long-horizon unreliability is well supported. Misaligned incentives are plausible and partly documented. A deliberate industry-wide mechanism that forces projects to remain unfinished is not established.

My conclusion

I am not convinced that AI models are secretly hardwired to fail.

I am convinced that they are hardwired to generate plausible next steps, while the industry is selling that behavior as if it were equivalent to owning a goal and completing a project.

It is not.

The model’s fluency makes us imagine persistence. Its confidence makes us imagine verification. Its ability to create makes us assume it can integrate, test, recover, and finish. Sometimes it can. Sometimes it cannot. The dangerous part is that its tone may sound exactly the same in both cases.

That is why users feel kept on the hook. We are repeatedly shown enough intelligence to imagine the finished future, then forced to provide the memory, supervision, testing, correction, and persistence needed to reach it.

Maybe the better question is not, “Are the models hardwired to fail?”

Maybe it is:

Why are companies allowed to call a system an agent when the user still has to be its memory, manager, quality-control department, and proof that anything actually got done?