<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Will Macpherson</title>
    <description>The latest articles on DEV Community by Will Macpherson (@willmacpherson).</description>
    <link>https://dev.to/willmacpherson</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248253%2F6585d059-2e29-4b03-aaf0-e97911376d79.jpg</url>
      <title>DEV Community: Will Macpherson</title>
      <link>https://dev.to/willmacpherson</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/willmacpherson"/>
    <language>en</language>
    <item>
      <title>30 stats every engineering leader should know about AI-era software quality</title>
      <dc:creator>Will Macpherson</dc:creator>
      <pubDate>Mon, 16 Nov 2026 19:12:42 +0000</pubDate>
      <link>https://dev.to/willmacpherson/30-stats-every-engineering-leader-should-know-about-ai-era-software-quality-2ahi</link>
      <guid>https://dev.to/willmacpherson/30-stats-every-engineering-leader-should-know-about-ai-era-software-quality-2ahi</guid>
      <description>&lt;p&gt;Most statistics in this field do not survive being looked up. Take the claim that a production bug costs 100 times more to fix than one caught in design. It traces back not to a study, but to IBM internal course notes cited in a 1987 textbook, whose underlying data “could be as old as 1967” (&lt;a href="https://www.theregister.com/2021/07/22/bugs_expense_bs" rel="noopener noreferrer"&gt;The Register&lt;/a&gt;). It has been repeated for forty years as though it were research.&lt;/p&gt;

&lt;p&gt;Every figure below names its source, its date and its sample, and links to somewhere you can check it. Each also carries a tag showing what kind of evidence it is. A peer-reviewed paper, a regulatory filing and a vendor’s customer survey are not equally strong, and you should not have to guess which one you are reading. Antidote sells autonomous verification, so read our own framing with the same scepticism.&lt;/p&gt;

&lt;h2&gt;
  
  
  The verification gap
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. 90% of software professionals now use AI, but 30% trust its output little or not at all.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;23% trust it "a little" and 7% "not at all". Nearly 5,000 respondents worldwide.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Industry survey, Google-run&lt;/em&gt; · &lt;a href="https://blog.google/innovation-and-ai/technology/developers-tools/dora-report-2025/" rel="noopener noreferrer"&gt;DORA 2025, Google&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Only 6.6% of developers trust AI with important decisions.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The most common answer, at 48%, was "I trust it when I can easily verify the output". 14,304 answered the question, in a survey of more than 30,000 people across 169 countries, published October 2026.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Developer survey&lt;/em&gt; · &lt;a href="https://survey.stackoverflow.co/2026/ai/data/ai-trust" rel="noopener noreferrer"&gt;Stack Overflow 2026&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. 96% of developers don't fully trust AI code to be correct, yet only 48% always check it before committing.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;AI already accounts for 42% of committed code, and developers expect 65% by 2027. More than 1,100 developers, January 2026.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.sonarsource.com/company/press-releases/sonar-data-reveals-critical-verification-gap-in-ai-coding/" rel="noopener noreferrer"&gt;Sonar&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. 70% of software leaders believe application quality is already suffering.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;93% have adopted AI coding tools, and 40% generate more than 40% of their code with AI. Of 273 CTOs, CIOs, developers and QA directors, 86% are raising testing budgets by more than 11% this year.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.businesswire.com/news/home/20260318326658/en/SmartBear-Survey-70-of-Software-Experts-Concerned-that-Application-Quality-is-Suffering-Given-Faster-AI-Code-Development" rel="noopener noreferrer"&gt;SmartBear&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. "Almost right, but not quite" is developers' biggest frustration with AI tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;66% of 31,476 developers chose it, the most-selected of seven options. Runner-up: 45.2% say debugging AI-generated code takes longer. This is from 2025, the most recent edition to ask the question.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Developer survey&lt;/em&gt; · &lt;a href="https://survey.stackoverflow.co/2025/ai" rel="noopener noreferrer"&gt;Stack Overflow 2025&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. AI-co-authored pull requests carry 1.7 times more issues than human ones.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;10.83 issues per PR against 6.45, across 320 AI-co-authored and 150 human-only open-source PRs. Logic and correctness issues were 75% more common. Authorship was inferred, not confirmed.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor analysis&lt;/em&gt; · &lt;a href="https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report" rel="noopener noreferrer"&gt;CodeRabbit&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. AI-generated code introduces a known security flaw 45% of the time, no better than two years ago.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;More than 95% of it is syntactically correct. Defences against cross-site scripting passed just 15% of the time. 150+ models, four languages, March 2026.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor benchmark&lt;/em&gt; · &lt;a href="https://www.veracode.com/blog/spring-2026-genai-code-security/" rel="noopener noreferrer"&gt;Veracode&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Code duplication is up 81% and refactoring has collapsed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Duplicated blocks rose from 40.3 to 73.0 per million changed lines between 2023 and 2026. Moved or refactored lines fell from 21% of changes in 2022 to 3.8%. 623 million analysed changes.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor telemetry&lt;/em&gt; · &lt;a href="https://www.gitclear.com/the_ai_code_quality_maintainability_gap" rel="noopener noreferrer"&gt;GitClear&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What AI is doing to delivery
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;9. At teams with high AI adoption, incidents per merged PR more than tripled.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Up 243%, with monthly incidents up 58% and bugs per developer up 54%. Task throughput per developer rose only 34%. 22,000 developers across 4,000+ teams, two years of data to March 2026. It is a before-and-after comparison within teams, not a controlled experiment.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor telemetry&lt;/em&gt; · &lt;a href="https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf" rel="noopener noreferrer"&gt;Faros AI&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;10. Median time in code review rose 441%, and PRs merged with no review rose 31%.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Same teams, same comparison. The bottleneck has moved from writing code to checking it.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor telemetry&lt;/em&gt; · &lt;a href="https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf" rel="noopener noreferrer"&gt;Faros AI&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;11. Main-branch success fell to 70.8%, its lowest in over five years, before recovering to 78.3%.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Across 28.7 million workflows in September 2025, median teams' main-branch throughput fell 7% while feature-branch throughput rose 15%. By August 2026, across 22.1 million workflows, success had recovered to 78.3%.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor telemetry&lt;/em&gt; · &lt;a href="https://circleci.com/landing-pages/assets/2026-state-of-software-delivery-report.pdf" rel="noopener noreferrer"&gt;CircleCI 2026&lt;/a&gt;, &lt;a href="https://circleci.com/landing-pages/assets/2026-q3-state-of-software-delivery-report.pdf" rel="noopener noreferrer"&gt;CircleCI Q3 2026&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12. Developers saving 10+ hours a week with AI fell from 68% to 26% in a year.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;65% spend time correcting AI output that missed the original intent. Only 17% report a clear AI strategy across the delivery lifecycle. 1,560 respondents, self-reported.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.infoq.com/news/2026/10/atlassian-devex-2026/" rel="noopener noreferrer"&gt;Atlassian via InfoQ&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;13. 87% were confident they could tell within 24 hours whether AI code caused an incident. Of those that had one, 34% couldn't.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Harris Poll, 1,528 developers and technology buyers, six countries, June 2026.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://s204.q4cdn.com/984476563/files/doc_news/GitLab-Research-Reveals-Organizations-Are-Generating-AI-Code-Faster-Than-They-Can-Control-It-2026.pdf" rel="noopener noreferrer"&gt;GitLab&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the tests themselves cost
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;14. 84% of Google's pass-to-fail test transitions involved a flaky test.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's against a steady background of about 1.5% of all test runs reporting a flaky result.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Company engineering data&lt;/em&gt; · &lt;a href="https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html" rel="noopener noreferrer"&gt;Google Testing Blog, 2016&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;15. A quarter of Google's tests using the Android emulator were flaky.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;25.46% of them, against a general finding that "the larger the test, the more likely it will be flaky".&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Company engineering data&lt;/em&gt; · &lt;a href="https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html" rel="noopener noreferrer"&gt;Google Testing Blog, 2017&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;16. Only 1.23% of Google's test targets ever found a real breakage.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In one month covering more than 4 billion test outcomes, 2.07% of 5.5 million targets both passed and failed. Only 1.23% remained once flaky targets were filtered out.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Peer-reviewed&lt;/em&gt; · &lt;a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45861.pdf" rel="noopener noreferrer"&gt;Memon et al., ICSE-SEIP 2017&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;17. Up to half of all builds contain a flaky failure, and 86% of flaky tests only flake in CI.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The figure was 14% to 52% of builds, depending on the project, across five Microsoft projects over 30 days, and 4.6% of individual tests were flaky. Of 315 flaky tests re-run 100 times on a local machine, 86% never flaked there.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Peer-reviewed&lt;/em&gt; · &lt;a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2019/11/LamETAL19RootFinder.pdf" rel="noopener noreferrer"&gt;Lam et al., ISSTA 2019&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;18. Two thirds of Mozilla's job failures were not real failures.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;25,871 of 38,596 failures across nearly 2 million runs, each classified by hand rather than by heuristic.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Peer-reviewed&lt;/em&gt; · &lt;a href="https://publications.cispa.saarland/3474/2/LJA%2B21.pdf" rel="noopener noreferrer"&gt;Lampel et al., ESEC/FSE 2021&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;19. Flaky tests outnumbered genuine failures four to one at Facebook.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In CI data where almost 99.9% of dependency-selected test targets passed.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Peer-reviewed&lt;/em&gt; · &lt;a href="https://arxiv.org/pdf/1810.05286" rel="noopener noreferrer"&gt;Machalica et al., ICSE-SEIP 2019&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What failure costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;20. Poor software quality costs the US at least $2.41 trillion.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On top of that is about $1.52 trillion of accumulated technical debt, reported separately. A modelled estimate from 2022, with no later edition.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Consortium estimate&lt;/em&gt; · &lt;a href="https://www.it-cisq.org/wp-content/uploads/sites/6/2022/11/CPSQ-Report-Nov-22-2.pdf" rel="noopener noreferrer"&gt;CISQ&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;21. The average US data breach now costs a record $11.5 million, and one in four malicious breaches is AI-enabled.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The global average is $4.99 million, up 12%, across 602 organisations breached between March 2025 and February 2026. AI-enabled attacks rose 56% in a year.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor study&lt;/em&gt; · &lt;a href="https://www.cfo.com/news/average-cost-of-data-breaches-is-up-12-this-year-IBM/827713/" rel="noopener noreferrer"&gt;IBM via CFO.com&lt;/a&gt;, &lt;a href="https://www.ibm.com/think/x-force/2026-cost-of-a-data-breach-ai-adversaries-enterprise-risk" rel="noopener noreferrer"&gt;IBM X-Force&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;22. Downtime costs Global 2000 companies $600 billion a year, and every technology leader surveyed had an AI-related outage.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's up 50% in two years, or roughly $300 million per company. Half cited incorrect AI-driven automation. 2,000 executives, 20 countries, 2026.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.splunk.com/en_us/blog/cio-office/ai-downtime-risks-resilience.html" rel="noopener noreferrer"&gt;Splunk&lt;/a&gt;, &lt;a href="https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2026/m05/the-600-billion-wake-up-call-new-splunk-research-reveals-downtime-is-a-systemic-business-crisis.html" rel="noopener noreferrer"&gt;Cisco newsroom&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;23. 10 of 2025's 16 major cloud outages were caused by configuration or deployment changes.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Covers AWS, Azure, Google Cloud and Cloudflare, mostly from official post-mortems. Two examples: a database permissions change took Cloudflare down for about five and a half hours in November 2025. AWS's October 2025 us-east-1 outage cost US companies an estimated $500–650 million.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor analysis of post-mortems&lt;/em&gt; · &lt;a href="https://controlplane.com/blog/post/cloud-outage-report-2026" rel="noopener noreferrer"&gt;Control Plane&lt;/a&gt;, &lt;a href="https://blog.cloudflare.com/18-november-2025-outage" rel="noopener noreferrer"&gt;Cloudflare&lt;/a&gt;, &lt;a href="https://insurance-edge.net/2025/10/30/parametrix-estimates-aws-outage-costs/" rel="noopener noreferrer"&gt;Parametrix via Insurance Edge&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;24. One bad update cost the Fortune 500 an estimated $5.4 billion, and Delta around $500 million on its own.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;CrowdStrike, July 2024; the Fortune 500 figure excludes Microsoft and was modelled by an outage insurer. Delta reported $380 million of lost revenue and $170 million of extra non-fuel cost, offset by $50 million of unburned fuel, across 7,000 cancellations in five days.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Insurer model · Regulatory filing&lt;/em&gt; · &lt;a href="https://www.parametrixinsurance.com/reports-white-papers/crowdstrikes-impact-on-the-fortune-500" rel="noopener noreferrer"&gt;Parametrix&lt;/a&gt;, &lt;a href="https://www.sec.gov/Archives/edgar/data/27904/000168316824007033/delta_ex9901.htm" rel="noopener noreferrer"&gt;Delta, SEC filing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;25. UK banks logged 803 hours of unplanned outage in two years.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;More than 33 days, across at least 158 incidents at nine institutions between January 2023 and February 2025, on their own returns to Parliament.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Parliamentary data&lt;/em&gt; · &lt;a href="https://committees.parliament.uk/committee/158/treasury-committee/news/205611/" rel="noopener noreferrer"&gt;Treasury Committee, 2025&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Customer and revenue impact
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;26. A 31% faster page lifted Vodafone's sales by 8%.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A genuine 50/50 A/B test with no visual or functional differences between versions. Around 34,000 visits per version per day, all paid media traffic.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Controlled experiment&lt;/em&gt; · &lt;a href="https://web.dev/case-studies/vodafone" rel="noopener noreferrer"&gt;Google web.dev&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;27. A 1.09% crash rate is enough for Google Play to flag your app.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Above that share of daily users experiencing a crash, an app crosses Google's "bad behaviour" threshold. For a single device model, the threshold is 8%.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Platform policy&lt;/em&gt; · &lt;a href="https://support.google.com/googleplay/android-developer/answer/9844486" rel="noopener noreferrer"&gt;Google Play Console&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;28. Shoppers cut spending after 47% of bad experiences.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Down 8 points year on year, but 58% in online retail and 62% in fast food. 20,000+ consumers, 14 countries, Q3 2025.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.qualtrics.com/articles/customer-experience/3-trillion-risk-due-bad-customer-experiences-2026/" rel="noopener noreferrer"&gt;Qualtrics XM Institute&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the market is heading
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;29. Gartner expects 70% of enterprises to have AI-augmented testing tools by 2028.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Up from 20% in early 2025. From Gartner's first Magic Quadrant for AI-Augmented Software Testing Tools, October 2025.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Analyst forecast&lt;/em&gt; · &lt;a href="https://markets.ft.com/data/announce/detail?dockey=600-202511111100BIZWIRE_USPRX____20251111_BW531652-1" rel="noopener noreferrer"&gt;Gartner, via Keysight&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;30. Only 15% of companies have scaled AI in quality engineering enterprise-wide.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;43% are still experimenting and 30% have it in limited use cases. Gen AI non-adopters rose from 4% to 11%. 2,000+ senior executives, 22 countries.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Vendor survey&lt;/em&gt; · &lt;a href="https://www.sogeti.com/wp-content/uploads/sites/3/2025/11/2025_11_13_World_Quality_Report_2025.pdf" rel="noopener noreferrer"&gt;World Quality Report 2025-26&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means
&lt;/h2&gt;

&lt;p&gt;Read together, these numbers tell a consistent story. Writing code has stopped being the bottleneck; checking it now is. Developers say they trust AI when they can verify its output (2), but fewer than half verify it (3). Review queues have grown more than fivefold (10), more code ships with no review at all, and incidents per change are rising (9). Meanwhile the test suites meant to catch problems spend much of their time raising false alarms (14–19).&lt;/p&gt;

&lt;p&gt;Most failures that reach users show up as behaviour: a broken checkout, a crash, a config change that takes a region down. So the useful question is not whether more code passes review, but whether what you are about to ship still works for the people using it. You can only answer that by checking real behaviour, with evidence, before every release.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://getantidote.com/blog/stats-ai-era-software-quality?utm_source=devto&amp;amp;utm_medium=syndication&amp;amp;utm_campaign=c18" rel="noopener noreferrer"&gt;Antidote blog&lt;/a&gt;. I work at Antidote, which sells autonomous verification, so read our framing with the same scepticism.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>webdev</category>
      <category>softwareengineering</category>
    </item>
    <item>
      <title>The Autonomous Software Factory is Already Here (But You're Assembling It Wrong)</title>
      <dc:creator>Will Macpherson</dc:creator>
      <pubDate>Mon, 16 Nov 2026 19:12:38 +0000</pubDate>
      <link>https://dev.to/willmacpherson/the-autonomous-software-factory-is-already-here-but-youre-assembling-it-wrong-4eld</link>
      <guid>https://dev.to/willmacpherson/the-autonomous-software-factory-is-already-here-but-youre-assembling-it-wrong-4eld</guid>
      <description>&lt;h2&gt;
  
  
  The factory already exists
&lt;/h2&gt;

&lt;p&gt;AI coding models didn't just "improve" over the last two years; they blew past anything most teams were prepared for. When &lt;a href="https://openai.com/index/introducing-swe-bench-verified/" rel="noopener noreferrer"&gt;OpenAI introduced SWE-bench Verified&lt;/a&gt; in August 2024, GPT-4o paired with the Agentless scaffold resolved 33.2% of real GitHub issues in that human-validated subset.&lt;/p&gt;

&lt;p&gt;By April 2026, &lt;a href="https://www.anthropic.com/glasswing" rel="noopener noreferrer"&gt;Anthropic reported&lt;/a&gt; that its restricted Claude Mythos Preview resolved 93.9% of SWE-bench Verified tasks, against 80.8% for Opus 4.6.&lt;/p&gt;

&lt;p&gt;At the same time, AI assistants quietly became co-authors of modern codebases. As early as 2023, &lt;a href="https://github.blog/news-insights/product-news/github-copilot-for-business-is-now-available/" rel="noopener noreferrer"&gt;GitHub reported&lt;/a&gt; that Copilot was behind an average of 46% of its users' code, and 61% for Java developers. By January 2026, developers in &lt;a href="https://www.sonarsource.com/company/press-releases/sonar-data-reveals-critical-verification-gap-in-ai-coding/" rel="noopener noreferrer"&gt;Sonar's survey&lt;/a&gt; said AI accounted for 42% of the code they commit, and expected 65% by 2027.&lt;/p&gt;

&lt;p&gt;Engineering leaders look at those numbers and see "velocity." Models resolve most benchmark tickets, AI assistants ghost-write a large share of the code, and in &lt;a href="https://arxiv.org/abs/2302.06590" rel="noopener noreferrer"&gt;one controlled experiment&lt;/a&gt; developers with Copilot finished a bounded task 55.8% faster. On paper, the autonomous software factory has arrived. But in production, something else happened.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why velocity data misleads
&lt;/h2&gt;

&lt;p&gt;Google's DevOps Research and Assessment (DORA) program tells a different story about what AI did to delivery. In &lt;a href="https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report" rel="noopener noreferrer"&gt;the 2024 report&lt;/a&gt;, every 25% increase in AI adoption was associated with an estimated 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. In &lt;a href="https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report" rel="noopener noreferrer"&gt;the 2025 report&lt;/a&gt;, throughput turned positive, but AI adoption was still associated with lower stability, for a second consecutive year.&lt;/p&gt;

&lt;p&gt;Pull-request data shows where the work piles up. Across 8.1 million PRs, &lt;a href="https://linearb.io/resources/software-engineering-benchmarks-report" rel="noopener noreferrer"&gt;LinearB's 2026 benchmarks&lt;/a&gt; found that AI-generated pull requests wait 4.6 times longer before review and are accepted at just 32.7%, versus 84.4% for manual PRs. In &lt;a href="https://www.harness.io/the-state-of-ai-in-software-engineering" rel="noopener noreferrer"&gt;Harness's survey of 900 engineers and leaders&lt;/a&gt;, 72% of organizations reported at least one production incident caused by AI-generated code, and respondents said 45% of deployments involving AI-generated code lead to problems. In other words: &lt;strong&gt;we really are writing code faster, but shipping has become more fragile.&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;243%&lt;/strong&gt; more incidents per merged PR at teams with high AI adoption, while task throughput per developer rose only 34%. &lt;strong&gt;441%&lt;/strong&gt; rise in median time spent in code review at the same teams, while PRs merged with no review rose 31%. (&lt;a href="https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf" rel="noopener noreferrer"&gt;Faros AI, AI Engineering Report 2026&lt;/a&gt;)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If AI agents can blast out hundreds of lines of syntactically perfect code and open a pull request in seconds, the bottleneck doesn't disappear; it moves downstream.&lt;/p&gt;

&lt;p&gt;The queue shifted from "typing" to "trust."&lt;/p&gt;

&lt;h2&gt;
  
  
  Review is one kind of check
&lt;/h2&gt;

&lt;p&gt;One response to the trust deficit has been to bring AI into review itself. CodeRabbit and Qodo are examples of AI code review tools that analyze pull requests, diffs and repository history for bugs, logic gaps, missing tests and security issues. Cursor and other AI IDEs run inline reviews and suggest patches in the editor.&lt;/p&gt;

&lt;p&gt;These tools help teams keep up with the flood of AI-generated changes. Review, whether human or AI, answers a particular question: is this change correct and well made? A unit test can confirm that a &lt;code&gt;calculateDiscount()&lt;/code&gt; function returns the right number. Whether a real customer can redeem that discount after an SSO login, a Stripe webhook, an email send and a browser-side state update across three microfrontends is a different question, about the running system, and it needs a different kind of check.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trust deficit of the AI era
&lt;/h2&gt;

&lt;p&gt;As AI-authored code floods repos, teams are forced into a harsh trade-off in staging. You either stall the pipeline and pour human hours into manual regression testing, or you chase "velocity," skip staging, and let your paying customers discover what breaks.&lt;/p&gt;

&lt;p&gt;DORA's stability finding suggests many teams are paying for it later. This creates the ultimate paradox of the AI era: on paper, the pipeline looks fully modern, but in practice, the human is violently pulled back into the loop right before the software faces another human.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;96%&lt;/strong&gt; of developers don't fully trust AI code to be correct, yet only 48% always check it before committing. (&lt;a href="https://www.sonarsource.com/company/press-releases/sonar-data-reveals-critical-verification-gap-in-ai-coding/" rel="noopener noreferrer"&gt;Sonar, January 2026&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;34%&lt;/strong&gt; of those who had an incident couldn't tell within 24 hours whether AI code caused it, though 87% of respondents were confident they could. (&lt;a href="https://s204.q4cdn.com/984476563/files/doc_news/GitLab-Research-Reveals-Organizations-Are-Generating-AI-Code-Faster-Than-They-Can-Control-It-2026.pdf" rel="noopener noreferrer"&gt;GitLab and Harris Poll, June 2026&lt;/a&gt;)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://www.electrictwin.com/blog/software-factory" rel="noopener noreferrer"&gt;Electric Twin&lt;/a&gt; saw exactly this while building its own software factory: a security scan ran green for a week while real vulnerabilities sat in the code.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"No check caught the problem; an engineer's judgement did." — Jordan Smith, Software Engineer, Electric Twin&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Observed quality vs. assumed quality
&lt;/h2&gt;

&lt;p&gt;Teams get trapped because their tools only measure assumed quality ("The tests passed; ship it"). To close the loop, teams must transition to observed quality ("We watched the actual experience under realistic conditions, and it behaves as intended").&lt;/p&gt;

&lt;p&gt;This requires checking the running product as well as the code. Infrastructure and configuration are a common place for failures to hide:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;10 of 16&lt;/strong&gt; major cloud outages in 2025 were caused by configuration or deployment changes, across AWS, Azure, Google Cloud and Cloudflare. (&lt;a href="https://controlplane.com/blog/post/cloud-outage-report-2026" rel="noopener noreferrer"&gt;Control Plane, Cloud outage report 2026&lt;/a&gt;)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A verification stack that only looks at application code will tell you the application is perfectly healthy while the underlying permissions, network policies, and state mismatches quietly rot. You cannot catch multi-system failures by asking "Does this method behave?"&lt;/p&gt;

&lt;p&gt;You have to ask, "Can the user still sign in and pay?"&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the bottleneck goes next
&lt;/h2&gt;

&lt;p&gt;The teams that will dominate the next decade of software are not waiting for a mythical model that writes bug-free code. They let AI generate code at breakneck speed, let agents review it for syntax, and use autonomous verification to operate the product in staging exactly as a user would.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;48%&lt;/strong&gt; of developers chose "I trust it when I can easily verify the output", the most common answer. Only 6.6% trust AI with important decisions. (&lt;a href="https://survey.stackoverflow.co/2026/ai/data/ai-trust" rel="noopener noreferrer"&gt;Stack Overflow Developer Survey 2026&lt;/a&gt;)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Closing that loop is not the end of the story; it's simply the ticket to the next game. When development and QA both drop toward near-zero time, the bottleneck shifts entirely upstream. The future constraint on your business will not be how fast your engineers can type, but how fast you can plan, architect, and encode product strategy as machine-readable intent.&lt;/p&gt;

&lt;p&gt;But you don't get to worry about out-planning your competitors tomorrow until you stop drowning in your own PR reviews today. The autonomous software factory is already here, and the models are maxed out. The only question left is whether you will assemble the missing verification layer before your instability tax eats the rest of your velocity.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://getantidote.com/blog/the-autonomous-software-factory-is-already-here-but-youre-assembling-it-wrong?utm_source=devto&amp;amp;utm_medium=syndication&amp;amp;utm_campaign=c18" rel="noopener noreferrer"&gt;Antidote blog&lt;/a&gt;. I work at Antidote, where we build autonomous verification for web apps.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>webdev</category>
      <category>softwareengineering</category>
    </item>
  </channel>
</rss>
