Automation bias is the reason a pull request full of AI-generated code often slips through review faster than one a colleague wrote by hand. The code looks tidy, the tests are green, and somewhere in the back of your mind sits the thought: the machine probably knows what it's doing. That is exactly where things get dangerous.
I care about this topic because I use AI tools every day and genuinely like them. As a team lead, though, I keep asking what happens to our attention when a tool is almost always right. Research has known the answer since the nineties, only back then the tool was called an autopilot. In this article I apply those findings to everyday development work and show you how to counter the effect in your team without demonizing AI.
In short
Automation bias means people follow automated suggestions more than they should and miss errors the system does not report. Studies on AI-generated code show exactly this pattern, including a false sense of security. The biggest pitfall: a second reviewer alone does not fix it; you need clear accountability, review depth by risk and practice with real mistakes.
What you will learn in this article
- What automation bias is and which two types of error it causes
- How to spot automation bias in code review, agent work and incidents
- What studies found about AI assistants and code security
- Seven countermeasures for your team and a 45-minute workshop to practice
What is automation bias?
Automation bias describes how people give the advice of an automated system more weight than other signals. Contradicting hints then get brushed aside, even when they are right. The term comes from aviation and medical research. In the nineties, Kathleen Mosier, Linda Skitka and colleagues studied how pilots work with automated cockpit systems. Their finding was uncomfortable: well-trained professionals followed wrong indications even though other instruments showed the opposite.
A closely related idea is automation complacency. It means we monitor a system less carefully because it has run reliably for a long time. In 2010, Raja Parasuraman and Dietrich Manzey showed that both effects share a root: our attention is limited, and we direct it to wherever we suspect the bigger risk. When the automation works well, our attention drifts elsewhere.
Two types of error: accepting and overlooking
Research distinguishes two errors that map directly onto software development. With an error of commission, you follow a suggestion without weighing it against other information. One example is the AI-suggested IAM policy with a wildcard in the actions, which you accept because the deployment finally works afterwards.
With an error of omission, the system fails to report a problem, and you don't notice it either because you relied on the system. For instance, the AI review praises a pull request but does not mention that a new endpoint skips the authorization check. Since there is no comment about it, nobody looks there. This second type is trickier, because there is nothing to trip over.
How automation bias shows up in everyday development
To make this concrete, let's take a fictional team building a ticket shop for concerts. The team uses an AI assistant in the IDE, lets an agent handle smaller tasks as pull requests and has set up an AI review that runs before the human review. All sensible. Still, typical patterns appear in four places.
Code completion
A developer builds the event search. The AI suggests a database query that inserts the search term via string concatenation. She reads the line and, since it looks plausible, hits Tab. Had a colleague shown her this line in a review, she would probably have said “SQL injection” right away. The difference is not knowledge but attitude: you check a colleague, you accept a suggestion.
Agent pull requests
The agent is asked to extend the discount logic with group tickets. It delivers a clean pull request with new tests, all green. However, nobody checks one thing: the agent wrote the tests itself, tailored to its own implementation. A rounding error for groups of ten or more therefore lives in both the code and the test. Green tests feel like independent proof, but in this case they are not.
AI review before the human
The AI review leaves three small style comments and nothing else. Then the human reviewer sees the result, thinks “the AI already checked the big stuff” and only looks at the three comments. Without noticing, the team has turned two reviews into one and a half. I covered the question of who is responsible for what in detail in AI code review: team rules for Copilot approvals.
Incident analysis
At night, checkout goes down. Right away, the on-call engineer pastes logs into an AI chat and gets a convincing explanation: the database is overloaded. As a result, the team scales up the database, but the error stays. In fact, the real cause, an expired certificate at the payment provider, was in the log, but the first plausible answer ended the search. Experts call this the anchoring effect: the first hypothesis colors everything that follows.
What research says about automation bias and AI code
By now there are solid studies on AI assistants in programming. I find two of them particularly revealing, because together they show the whole pattern.
Two studies on AI code and security
In 2021, Hammond Pearce and colleagues tested GitHub Copilot in 89 scenarios with security-critical weaknesses, based on MITRE's “Top 25” list. Of the roughly 1,700 generated programs, about 40 percent were vulnerable. Admittedly, the study is older, and models have improved a lot since. Still, the core point remains: a language model produces likely code, not verified code.
Even more revealing for automation bias is the study by Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh from Stanford. Participants with an AI assistant wrote significantly less secure code than the comparison group without one. At the same time, they were more likely to believe their code was secure. In other words, that is the effect in a nutshell: quality drops while the feeling of safety rises. One more detail matters: people who were more skeptical of the AI and worked with it more actively, for example by rephrasing prompts, ended up with fewer vulnerabilities.
What the DORA report says about trust
Google Cloud's 2025 DORA report fits the picture too. According to Google, 90 percent of respondents now use AI at work, but 30 percent trust it only a little or not at all. The report calls AI an amplifier: it makes whatever already works or fails in an organization more pronounced. To me this means: a team with a weak review culture does not automatically get a better one through AI, but a faster weak one.
🎬 The trust spiral behind automation bias – this animated diagram only runs on my site: watch it on sascha-ms.de
Why a second reviewer is not enough
The obvious reflex is: then let two people review. Unfortunately, aviation research offers little hope here. In 1998, Mosier and colleagues also studied two-person crews, and they were no better than individuals at catching errors the automation failed to report. When both assume the system and the other person are paying attention, responsibility spreads out until nobody holds it. A second review only helps if it is clear what it should check in addition.
Why good teams are especially prone
This sounds paradoxical, but it is well documented. Automation bias grows with high workload, with time pressure and with a system that has been reliable for a long time. The literature calls the last effect “learned carelessness”. An AI assistant that is right in 95 out of 100 cases is therefore more dangerous for your attention than one that botches every second suggestion. With the bad tool, you stay alert. With the good tool, however, you train yourself out of checking.
In addition, presentation plays a role. Suggestions that are displayed prominently and in detail seem more trustworthy. AI code arrives neatly formatted, with comments and sometimes even with a rationale. As a result, it looks like the work of an experienced colleague. Our brain gratefully takes that shortcut.
Curbing automation bias in your team: seven countermeasures
You cannot switch the effect off entirely; it is human. However, you can change the conditions under which your team works with AI. The following countermeasures are easy to introduce and build on what research describes as effective.
1. Make accountability explicit
One of the most reliable findings: people who must account for the result rely less blindly on automation. In practice, whoever opens a pull request owns every line in it, no matter who or what wrote it. “The AI did it that way” is not a justification. Say this out loud in your team once and add it to your team agreement.
2. Reasons instead of checkmarks
Add a short question to your pull request template: “Which parts come from an AI, and what did you check or change in them?” For most pull requests, two sentences are enough. It is less about control than about a brief moment of pausing. If you have to write down what you checked, you check something.
3. Scale review depth by risk
Not all code needs the same attention, so divide your repository into zones. For documentation and tests a quick look is enough; for authentication, permissions, payments and data migrations you need a deliberate, independent review. That way you focus scarce attention where an error of omission gets really expensive.
4. Think first, then ask
For incidents and architecture decisions a simple rule helps: before anyone asks the AI, the team writes down its own hypothesis. Afterwards, the AI may add to it or disagree. This weakens the anchoring effect because the first answer no longer comes from the machine by default. Besides, the team keeps learning instead of just collecting answers.
5. Practice with real mistakes
In 2008, Bahner, Hüper and Manzey showed that training with deliberately flawed suggestions works better against automation bias than merely pointing out that errors can happen. Moreover, this translates nicely to teams. I show how in the workshop further down.
6. Take time pressure out of reviews
High workload demonstrably amplifies the effect. If your team is measured by how fast pull requests get merged, you reward exactly the behavior that fuels automation bias. Measure how many defects show up after the merge instead, and schedule fixed review times rather than squeezing reviews between two meetings.
7. Verify independently
Don't let AI code be judged only by the same AI that wrote it, and don't rely solely on tests created in the same step. Independent checks include acceptance criteria written beforehand, static analysis, security scanners or a test written by a human. What matters is that the check does not share the same assumptions as the code.
A 45-minute workshop against automation bias
Countermeasure number five works best when you experience it together. The following agenda fits into an extended retro or a learning slot. You need a facilitator and five short code snippets in the style of your AI suggestions.
| Time | Step | What happens |
|---|---|---|
| 5 min | Intro | The facilitator briefly explains automation bias and the two types of error without giving away the exercise. |
| 10 min | Review round | Everyone reviews five AI suggestions alone and marks each “accept” or “change”. Two snippets contain a deliberately planted error. |
| 10 min | Reveal | The facilitator shows the errors. The team collects who found what and why the other errors slipped through. |
| 15 min | Find patterns | Discuss together: when did we read carefully, when did we just skim? What lulled us into a sense of safety? |
| 5 min | Agreement | Agree on one concrete experiment for the next two weeks, for example the question in the PR template. |
The attitude matters: this is not about who finds the most errors. If someone misses an error, that is precisely the effect you are exploring, not a personal failure. Without psychological safety in the team, this exercise quickly turns into a test, and then nobody learns anything.
Good errors for the exercise are ones that look plausible: a missing permission check, a swapped comparison operator at a boundary, a time zone mistake, an overly broad permission or a retry without an upper limit. Either write the snippets yourself or let an AI generate them and plant the errors by hand afterwards.
Questions for your next retro
If you want to raise the topic without a workshop, take one of these questions to your next retrospective. You can find out how to structure a retro in my guide How to run a retrospective.
- Which AI suggestion in the last sprint should we have looked at more closely?
- Was there a production bug that a review could have caught? What role did AI play in it?
- Where do we review AI code more thoroughly than code from colleagues, and where less thoroughly?
- Which parts of our system should nobody ever accept unchecked, no matter who wrote the code?
- What would need to change so that nobody feels rushed during review?
Common mistakes in dealing with automation bias
The most common mistake is a blanket ban. After all, banning AI tools pushes their use into private browser tabs and removes any chance of developing good rules together. Blanket distrust is just as unfortunate: if every AI suggestion has to be checked three times, the benefit is gone and the team works around the rule.
Another mistake is the blame question. When a bug from AI code reaches production, asking “Who waved this through?” is understandable but not very helpful. It is more productive to ask which conditions made waving it through likely. Time pressure, unclear responsibility and missing risk levels are almost always involved.
Finally: metrics that reward speed only. Of course, more merged pull requests per week look good on a dashboard. If defects after release rise at the same time, you have built automation bias into your goals.
How to turn this into a fixed guardrail in the review is shown in the pull request template for AI-generated code with an automated check.
My take
I consider automation bias one of the underrated topics in using AI within teams. Most discussions revolve around models, costs and data protection. Those matter. Yet the biggest weak spot sits between screen and chair, and I explicitly don't mean that in a derogatory way. We are simply not built to watch something attentively for long periods when it almost always works.
At the same time, I disagree with the narrative that AI makes developers lazy or dumb. The Stanford study also shows the opposite: people who work with the tool actively and critically benefit from it. For me, this is above all a leadership task. As a team lead, I can create conditions in which looking closely is normal: enough time, clear accountability and a culture in which “I didn't understand this” is a good review comment.
How to measure the impact of AI on your team without anyone feeling watched is the topic of my article Metrics for AI-generated code: measure impact, not people.
FAQ
What is automation bias in simple terms?
Automation bias is the tendency to trust automated suggestions more than other information. As a result, people accept wrong suggestions or miss errors the system does not report.
What is the difference between automation bias and automation complacency?
Automation bias concerns decisions: you follow the suggestion instead of other evidence. Complacency concerns monitoring: you look less closely because the system has been reliable for a long time. Both stem from limited attention and often occur together.
Does AI-generated code make software less secure?
Not automatically. Studies do show, however, that people with an AI assistant can write less secure code and still consider it more secure. What matters is how critically the team handles the suggestions.
Should teams ban AI tools?
A ban usually just pushes usage out of sight. Clear accountability, review depth by risk and regular practice with flawed suggestions are more effective.
Your next step
This week, add a question to your pull request template asking which parts come from an AI and what was checked. It takes five minutes and changes how your team looks at AI code. Now I'm curious about your view: when did you last catch yourself simply waving an AI suggestion through? Feel free to tell me in the comments.
Sources
- Wikipedia: Automation bias (overview with references to Mosier et al. 1997/1998, Parasuraman & Manzey 2010, Bahner et al. 2008)
- Parasuraman, R. & Manzey, D. H. (2010): Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3)
- Bahner, J. E., Hüper, A.-D. & Manzey, D. (2008): Misuse of automated decision aids. International Journal of Human-Computer Studies 66(9)
- Perry, N., Srivastava, M., Kumar, D. & Boneh, D.: Do Users Write More Insecure Code with AI Assistants? (arXiv 2211.03622)
- Pearce, H. et al.: Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (IEEE S&P 2022)
- Google: 2025 DORA report (State of AI-assisted Software Development)
- DORA: 2025 State of AI-assisted Software Development
This post was created with AI assistance.
👉 Next step: More for team leads: my cheat sheets put the essentials on one printable page.
📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (0)