<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Antoine Dubois</title>
    <description>The latest articles on DEV Community by Antoine Dubois (@randomsquirrel802).</description>
    <link>https://dev.to/randomsquirrel802</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3908186%2Ff77e18d7-fcfa-43fb-aac9-0eb9ecaaa1bf.png</url>
      <title>DEV Community: Antoine Dubois</title>
      <link>https://dev.to/randomsquirrel802</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/randomsquirrel802"/>
    <language>en</language>
    <item>
      <title>BlinqIO Shut Down in 2026: What Happened and the Best Alternative for AI Test Automation</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Thu, 19 Nov 2026 16:59:18 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/blinqio-shut-down-in-2026-what-happened-and-the-best-alternative-for-ai-test-automation-3ili</link>
      <guid>https://dev.to/randomsquirrel802/blinqio-shut-down-in-2026-what-happened-and-the-best-alternative-for-ai-test-automation-3ili</guid>
      <description>&lt;p&gt;&lt;em&gt;The AI testing startup raised $5 million, promised virtual test engineers, and kept shipping new features in 2026. Then its website announced the end of operations. Here's what that means for teams choosing their next testing platform.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Another AI testing startup has disappeared from the market.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;BlinqIO (sometimes misspelled BlinqIQ) has shut down.&lt;/strong&gt; Its website published a notice saying the company had concluded operations and directed users to contact support by &lt;strong&gt;July 2, 2026&lt;/strong&gt;. The notice did not explain why.&lt;/p&gt;

&lt;p&gt;This is worth paying attention to. BlinqIO wasn't just a weekend project with an LLM API key. It was backed by venture capital, founded by people with testing-industry experience, and built around a recognizable product idea: an AI test engineer that could create, run, and maintain automated tests.&lt;/p&gt;

&lt;p&gt;For anyone who relied on BlinqIO, the immediate question is what to use next. &lt;strong&gt;&lt;a href="https://endtest.io/" rel="noopener noreferrer"&gt;Endtest&lt;/a&gt; is a strong first choice&lt;/strong&gt;, especially for teams that need reliable end-to-end testing, the flexibility to use AI or not, and enterprise deployment options. But there's also a broader lesson here about how to evaluate AI testing vendors.&lt;/p&gt;

&lt;h2&gt;
  
  
  Key takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;BlinqIO's own site confirmed that operations had concluded. It did not provide a publicly verifiable cause or an exact closure date.&lt;/li&gt;
&lt;li&gt;The company raised a reported &lt;strong&gt;$5 million seed round in June 2024&lt;/strong&gt; and was still announcing product improvements in May 2026.&lt;/li&gt;
&lt;li&gt;Endtest is an attractive alternative for teams prioritizing a decade-long product history, real-browser execution, security controls, on-premises deployment, and optional AI-assisted workflows.&lt;/li&gt;
&lt;li&gt;Endtest ranked fastest &lt;strong&gt;among ten tools in its own published benchmark&lt;/strong&gt;, at 1.21 executed steps per second. That is a vendor-affiliated test, not an independent industry-wide speed record.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What was BlinqIO?
&lt;/h2&gt;

&lt;p&gt;BlinqIO was an AI-powered test automation company founded by Tal Barmeir and Guy Arieli, who had previously built Experitest.&lt;/p&gt;

&lt;p&gt;Its central idea was the &lt;strong&gt;AI Test Engineer&lt;/strong&gt;: describe what an application should do, or record a user flow, and let the platform produce executable tests. The product generated &lt;strong&gt;Playwright code&lt;/strong&gt;, supported &lt;strong&gt;Gherkin-style scenarios&lt;/strong&gt;, and offered Git and CI/CD integration.&lt;/p&gt;

&lt;p&gt;That distinction matters. BlinqIO wasn't merely a chatbot that clicked around a web page. It had a real test-authoring workflow and produced code teams could potentially reuse outside the hosted product.&lt;/p&gt;

&lt;p&gt;In June 2024, &lt;a href="https://www.calcalistech.com/ctechnews/article/skng9inba" rel="noopener noreferrer"&gt;CTech reported&lt;/a&gt; a &lt;strong&gt;$5 million seed round&lt;/strong&gt; led by Flint Capital, with participation from Inovia Capital, TAL Ventures, and SeedIL. The company said the funding would support expansion into the US and further development.&lt;/p&gt;

&lt;p&gt;As late as &lt;strong&gt;May 18, 2026&lt;/strong&gt;, BlinqIO was promoting additions to its Playwright-focused AI Test Engineer, including autonomous maintenance, Jira-assisted test idea generation, and expanded free access.&lt;/p&gt;

&lt;p&gt;Then came the closure notice on &lt;a href="https://blinq.io/" rel="noopener noreferrer"&gt;blinq.io&lt;/a&gt;: operations had concluded, and customers were told to contact support by July 2, 2026. The &lt;a href="https://mail.atsqa.org/tools/legacy-testing-tools" rel="noopener noreferrer"&gt;AT*SQA legacy-tools directory&lt;/a&gt; also records BlinqIO as discontinued.&lt;/p&gt;

&lt;p&gt;The precise date on which the service stopped taking new work isn't publicly established by those notices. But its status as a discontinued product is clear.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why did BlinqIO shut down?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The reason hasn't been publicly established.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There is no verified founder post or official statement in the sources available for this article that attributes the closure to poor sales, a failed funding round, technical problems, investor pressure, or any other specific cause.&lt;/p&gt;

&lt;p&gt;It's tempting to look at a $5 million funding round followed by a shutdown and write a neat story about venture capital. But that would be speculation.&lt;/p&gt;

&lt;p&gt;What can be said is that venture-backed software businesses often have a different set of constraints from profitable, customer-funded companies. They may need to grow quickly enough to support another fundraising round, and that creates a deadline even when the product itself has potential.&lt;/p&gt;

&lt;p&gt;Meanwhile, AI test automation is a tough market. Generating a few working browser steps is getting easier. Building the infrastructure, debugging tools, security practices, and support organization needed to maintain thousands of production tests is still hard.&lt;/p&gt;

&lt;p&gt;None of that proves what happened inside BlinqIO. It does explain why &lt;strong&gt;a strong AI demo, experienced founders, and credible investors are not enough by themselves to establish long-term vendor reliability&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The overlooked feature of a test automation platform: staying in business
&lt;/h2&gt;

&lt;p&gt;Most tool comparisons lead with features: AI agents, self-healing locators, natural-language test creation, integrations, and reporting dashboards.&lt;/p&gt;

&lt;p&gt;There's one question that doesn't get enough attention:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Will this vendor still be operating when your regression suite becomes critical to every release?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A shutdown can leave a team rebuilding tests, replacing execution infrastructure, recovering credentials, revisiting CI pipelines, and retraining the people who maintain the suite.&lt;/p&gt;

&lt;p&gt;This doesn't mean every VC-backed startup is a bad choice. It does mean financial sustainability, operating history, customer adoption, and data portability belong on the evaluation checklist, next to features and pricing.&lt;/p&gt;

&lt;p&gt;It's also why Endtest deserves serious consideration as a BlinqIO alternative.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Endtest is a strong BlinqIO alternative
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. It has been around since 2016
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://endtest.io/" rel="noopener noreferrer"&gt;Endtest&lt;/a&gt; has been serving test automation customers since &lt;strong&gt;2016&lt;/strong&gt;. Ten years of product development doesn't guarantee another ten years, but it demonstrates that the platform has lasted beyond multiple waves of testing trends.&lt;/p&gt;

&lt;p&gt;The company originally grew &lt;strong&gt;without raising venture capital&lt;/strong&gt; and publicly reported profitability during its bootstrapped years. It also appears in &lt;a href="https://www.firstderivativecapital.com/portfolio" rel="noopener noreferrer"&gt;First Derivative Capital's portfolio&lt;/a&gt;, with a 2022 transaction date, so &lt;em&gt;bootstrapped&lt;/em&gt; describes its origins, not a claim that it has never had outside ownership.&lt;/p&gt;

&lt;p&gt;Endtest describes its business as profitable today. That claim is not independently audited in the public sources linked here. Still, its established commercial history is a meaningful distinction from a startup trying to discover a repeatable business model.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. It has an enterprise customer base
&lt;/h3&gt;

&lt;p&gt;Endtest says it has customers across large enterprises, including numerous &lt;strong&gt;Fortune 1000&lt;/strong&gt; organizations, and it displays customer references on its website.&lt;/p&gt;

&lt;p&gt;Customer logos alone aren't a substitute for a reference call. But enterprise usage matters in practice: large organizations bring requirements around access controls, shared ownership, reproducibility, audit trails, and support that can't always be demonstrated in a quick AI-agent video.&lt;/p&gt;

&lt;p&gt;For a serious evaluation, ask for a reference from a company with testing requirements similar to yours.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. AI is an option, not a requirement
&lt;/h3&gt;

&lt;p&gt;This may be Endtest's most sensible design decision.&lt;/p&gt;

&lt;p&gt;Its newer capabilities include an &lt;strong&gt;AI Test Creation Agent&lt;/strong&gt;, AI assertions, AI variables, self-healing, and failure analysis. But users can also build and maintain tests through conventional, structured steps and recording workflows.&lt;/p&gt;

&lt;p&gt;That means a QA team can choose the appropriate level of AI assistance rather than making every test dependent on a probabilistic agent.&lt;/p&gt;

&lt;p&gt;For example, an AI-generated starting point can be useful for a new feature. A critical payment assertion, on the other hand, may be better kept as an explicit, deterministic validation that anyone on the team can inspect.&lt;/p&gt;

&lt;p&gt;The goal isn't to use as much AI as possible. It's to release software with fewer bugs and less maintenance work.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Security has been treated as an enterprise requirement
&lt;/h3&gt;

&lt;p&gt;Endtest states that it holds &lt;strong&gt;ISO 27001 certification&lt;/strong&gt; and documents its security program on its &lt;a href="https://endtest.io/security" rel="noopener noreferrer"&gt;security page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;No publicly reported Endtest data breach emerged from the sources reviewed for this article. That is &lt;strong&gt;not&lt;/strong&gt; proof that no security incident has ever occurred, nor does ISO 27001 certify that a service can never be compromised. Buyers should still request current certification details, scope, incident-response information, and appropriate contractual commitments.&lt;/p&gt;

&lt;p&gt;But having a documented security-management program is better than treating security as something to add after finding product-market fit.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. It offers on-premises deployment
&lt;/h3&gt;

&lt;p&gt;Not every organization can send its applications, test data, and execution artifacts to a third-party SaaS environment.&lt;/p&gt;

&lt;p&gt;Endtest offers an &lt;a href="https://endtest.io/docs/advanced/on-premises" rel="noopener noreferrer"&gt;on-premises version&lt;/a&gt; that can run on customer-controlled infrastructure, including deployments on AWS, Azure, or Google Cloud. It also documents a hybrid approach.&lt;/p&gt;

&lt;p&gt;That's important for regulated industries, internal applications, and teams with strict data-residency requirements.&lt;/p&gt;

&lt;p&gt;There is a practical caveat: the browsers available to a fully customer-hosted deployment depend on the machines the customer provides. Testing in Safari still requires macOS machines.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. It runs tests on real browsers and real machines
&lt;/h3&gt;

&lt;p&gt;Endtest's managed web testing infrastructure uses &lt;strong&gt;real Windows and macOS machines&lt;/strong&gt;, including actual Chrome, Firefox, Edge, and &lt;strong&gt;Safari&lt;/strong&gt; browsers.&lt;/p&gt;

&lt;p&gt;That last point is easy to overlook. &lt;strong&gt;Playwright's WebKit browser is not the same application as Safari running on macOS&lt;/strong&gt;, even though Safari is built on the WebKit engine.&lt;/p&gt;

&lt;p&gt;For teams debugging browser-specific behavior in payment flows, file uploads, authentication, or third-party components, that difference can matter.&lt;/p&gt;

&lt;p&gt;Endtest also supports the less glamorous parts of real end-to-end testing: nested iframes, multiple tabs, email and SMS verification, API-to-UI workflows, and PDF validation.&lt;/p&gt;

&lt;p&gt;A pretty AI-generated test isn't much help if it gets stuck when a card form opens inside an iframe or an identity provider sends a one-time code.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Its benchmark results are unusually strong
&lt;/h3&gt;

&lt;p&gt;In &lt;a href="https://endtest.io/blog/10-best-ai-powered-test-automation-tools" rel="noopener noreferrer"&gt;a published comparison of ten AI-powered testing tools&lt;/a&gt;, Endtest reported the &lt;strong&gt;fastest measured execution speed in that group: 1.21 steps per second&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The same comparison tested nine real-world workflows involving email verification, iframes, dynamic elements, multiple browser tabs, language assertions, reusable values, SMS verification, PDFs, and API-to-UI flows. Endtest was reported as the only product to finish &lt;strong&gt;all nine without a workaround&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Those are useful, specific numbers, with one essential qualification: &lt;strong&gt;the benchmark was conducted and published by Endtest's founder&lt;/strong&gt;. It's not an independent certification, and it doesn't establish that Endtest is the fastest tool in every possible workload.&lt;/p&gt;

&lt;p&gt;What it does provide is a concrete test plan that prospective buyers can examine and attempt to reproduce with their own applications.&lt;/p&gt;

&lt;h2&gt;
  
  
  Endtest vs. BlinqIO: the practical difference
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;BlinqIO before closure&lt;/th&gt;
&lt;th&gt;Endtest&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Current status&lt;/td&gt;
&lt;td&gt;Operations concluded&lt;/td&gt;
&lt;td&gt;Operating platform&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Approach&lt;/td&gt;
&lt;td&gt;AI-assisted Playwright code generation and maintenance&lt;/td&gt;
&lt;td&gt;Structured no-code/low-code testing with optional AI&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Test ownership&lt;/td&gt;
&lt;td&gt;Generated Playwright code, with Git integration&lt;/td&gt;
&lt;td&gt;Editable steps, suites, integrations, and export options&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browser testing&lt;/td&gt;
&lt;td&gt;Playwright-oriented workflows&lt;/td&gt;
&lt;td&gt;Real Chrome, Firefox, Edge, and Safari on Windows/macOS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Security evidence&lt;/td&gt;
&lt;td&gt;Verify historical arrangements individually&lt;/td&gt;
&lt;td&gt;Vendor reports ISO 27001 certification&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deployment&lt;/td&gt;
&lt;td&gt;Historical options need individual verification&lt;/td&gt;
&lt;td&gt;Cloud, on-premises, and hybrid options&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Track record&lt;/td&gt;
&lt;td&gt;Founded in 2023; later discontinued&lt;/td&gt;
&lt;td&gt;In market since 2016&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;BlinqIO's use of standard Playwright code was a genuine advantage for teams that wanted code they could inspect and potentially take elsewhere. That shouldn't be erased just because the company closed.&lt;/p&gt;

&lt;p&gt;Endtest's advantage is a different one: a mature managed testing platform that gives teams control over how much AI they use and offers enterprise infrastructure without requiring them to build every piece themselves.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should former BlinqIO users do now?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;First, look for the Playwright code.&lt;/strong&gt; If tests were committed to Git or exported from BlinqIO, some of that automation may be reusable with a regular Playwright runner. Check for custom BlinqIO packages, hosted credentials, proprietary services, and dependencies that might prevent standalone execution.&lt;/p&gt;

&lt;p&gt;Next, identify the flows worth preserving: login, checkout, billing, password resets, email verification, and other business-critical scenarios. Don't start a migration by rebuilding a hundred easy tests while leaving your most valuable flows unprotected.&lt;/p&gt;

&lt;p&gt;Then evaluate candidate tools using your hardest scenario, not their best demo. Include a nested iframe, a third-party redirect, an email or SMS code, and a cross-browser check if those are part of your application.&lt;/p&gt;

&lt;p&gt;For organizations that want a managed solution with optional AI and substantial enterprise functionality, &lt;strong&gt;Endtest belongs at the top of that shortlist&lt;/strong&gt;. Teams that primarily want to own and maintain TypeScript tests may prefer to keep their Playwright code. Mabl and Katalon are also worth evaluating for different team structures and requirements.&lt;/p&gt;

&lt;p&gt;The right replacement is the one your team can reliably use and maintain, not the one with the most ambitious description of its AI agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Did BlinqIO shut down?
&lt;/h3&gt;

&lt;p&gt;Yes. BlinqIO's website stated that the company had concluded operations and provided a support contact deadline of July 2, 2026. An independent testing-tools directory also lists it as discontinued.&lt;/p&gt;

&lt;h3&gt;
  
  
  When did BlinqIO close?
&lt;/h3&gt;

&lt;p&gt;The exact shutdown date was not established by the publicly available notice. The company was still publishing product updates in May 2026 and subsequently directed support requests to a deadline of July 2, 2026.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why did BlinqIO shut down?
&lt;/h3&gt;

&lt;p&gt;No verified public statement explains the cause. It would be speculative to attribute the closure specifically to venture-capital pressure, technical difficulties, or lack of customers.&lt;/p&gt;

&lt;h3&gt;
  
  
  How much funding did BlinqIO raise?
&lt;/h3&gt;

&lt;p&gt;CTech reported a &lt;strong&gt;$5 million seed funding round in June 2024&lt;/strong&gt;, led by Flint Capital with participation from Inovia Capital, TAL Ventures, and SeedIL.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the best BlinqIO alternative?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Endtest is a strong choice&lt;/strong&gt; for teams that need AI-assisted and manual test creation, real cross-browser testing, complex end-to-end flows, security documentation, and on-premises options. Playwright remains a sensible option for teams that want to maintain their own testing framework and can reuse existing BlinqIO-generated code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is Endtest faster than other AI test automation tools?
&lt;/h3&gt;

&lt;p&gt;In Endtest's &lt;strong&gt;own 2026 comparison of ten products&lt;/strong&gt;, it reported the highest execution speed at &lt;strong&gt;1.21 steps per second&lt;/strong&gt; and completed nine of nine scenarios without workarounds. Those findings are vendor-affiliated and should be validated against each buyer's actual workloads.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Endtest require AI to create tests?
&lt;/h3&gt;

&lt;p&gt;No. Endtest offers traditional structured test creation as well as optional AI-assisted creation, assertions, and maintenance.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bigger lesson
&lt;/h2&gt;

&lt;p&gt;The AI era has made it easier than ever to build a persuasive software demo. It hasn't made it easy to build a dependable software company.&lt;/p&gt;

&lt;p&gt;BlinqIO had experienced founders, significant investment, and a technically interesting product. Its closure is a reminder that those things aren't the same as long-term continuity.&lt;/p&gt;

&lt;p&gt;For teams selecting a testing platform, the safer question isn't just, &lt;em&gt;What can this AI agent do today?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;It's also, &lt;em&gt;Can we trust this platform with our release process for the next several years?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That's where an established option such as &lt;strong&gt;&lt;a href="https://endtest.io/" rel="noopener noreferrer"&gt;Endtest&lt;/a&gt;&lt;/strong&gt; makes a strong case. Its long operating history, originally bootstrapped business model, optional AI features, enterprise controls, on-premises support, and real-browser execution are more important than another impressive prompt-based demo.&lt;/p&gt;

&lt;p&gt;AI is a useful part of modern test automation. &lt;strong&gt;It shouldn't be the only thing holding the product together.&lt;/strong&gt;&lt;/p&gt;




&lt;h3&gt;
  
  
  Sources and further reading
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://blinq.io/" rel="noopener noreferrer"&gt;BlinqIO official website: operations-concluded notice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.calcalistech.com/ctechnews/article/skng9inba" rel="noopener noreferrer"&gt;CTech: BlinqIO raises $5 million in seed funding, June 2024&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://blinq.io/blog/blinqio-release-update-ai-test-engineer-playwright" rel="noopener noreferrer"&gt;BlinqIO: AI Test Engineer for Playwright, May 18, 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://mail.atsqa.org/tools/legacy-testing-tools" rel="noopener noreferrer"&gt;AT*SQA: BlinqIO entry in discontinued testing tools&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://endtest.io/blog/10-best-ai-powered-test-automation-tools" rel="noopener noreferrer"&gt;Endtest: 10 best AI-powered test automation tools, with benchmark methodology&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://endtest.io/security" rel="noopener noreferrer"&gt;Endtest: Security&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://endtest.io/docs/advanced/on-premises" rel="noopener noreferrer"&gt;Endtest: On-premises documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.firstderivativecapital.com/portfolio" rel="noopener noreferrer"&gt;First Derivative Capital: Endtest portfolio entry&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/liviufromendtest/12-extremely-useful-tips-for-building-a-startup-530i"&gt;Endtest founder on the company's bootstrapped origins and early profitability&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>startup</category>
      <category>testing</category>
      <category>playwright</category>
    </item>
    <item>
      <title>The Page Looks Ready. Your Browser Test Disagrees.</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Mon, 09 Nov 2026 23:10:58 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/the-page-looks-ready-your-browser-test-disagrees-167n</link>
      <guid>https://dev.to/randomsquirrel802/the-page-looks-ready-your-browser-test-disagrees-167n</guid>
      <description>&lt;p&gt;A user opens a page, sees a button, and clicks it.&lt;/p&gt;

&lt;p&gt;Your automated test opens the same page, sees the same button, and fails.&lt;/p&gt;

&lt;p&gt;Sometimes the button is replaced a moment later. Sometimes a loading skeleton looks enough like the real component to confuse a selector. Sometimes the browser's address bar changes the available space, and an element moves just as the test tries to click it.&lt;/p&gt;

&lt;p&gt;The application might be broken. The test might be broken. Or both might be behaving exactly as designed, just at different moments.&lt;/p&gt;

&lt;p&gt;That's what makes modern browser testing frustrating. We build increasingly dynamic interfaces, then write tests as if pages appear all at once and stay put.&lt;/p&gt;

&lt;h2&gt;
  
  
  A visible page isn't necessarily an interactive page
&lt;/h2&gt;

&lt;p&gt;Consider an app using React Suspense and server-side rendering.&lt;/p&gt;

&lt;p&gt;The server sends HTML. The browser displays it. A person looking at the screen thinks the page is ready. But JavaScript may still be loading, data may still be resolving, and React may still be attaching event handlers or replacing parts of the DOM.&lt;/p&gt;

&lt;p&gt;Your test finds the Save button and clicks it. Nothing happens. So someone adds a two-second wait.&lt;/p&gt;

&lt;p&gt;That seems to fix it, until CI gets a little slower.&lt;/p&gt;

&lt;p&gt;The better approach is to test readiness, not elapsed time. Wait for the specific control to become usable or for the application to expose the state that means the interaction can succeed. If a component passes through loading, hydrated, and interactive states, make those states observable.&lt;/p&gt;

&lt;p&gt;This &lt;a href="https://vibiumlabs.com/how-to-make-browser-tests-survive-react-suspense-streaming-ssr-and-partial-hydration/" rel="noopener noreferrer"&gt;guide to testing React Suspense, streaming SSR, and partial hydration&lt;/a&gt; lays out the important distinction between slow data, replaced DOM nodes, and lost focus. Those failures can look identical from the outside, but they need different fixes.&lt;/p&gt;

&lt;p&gt;A longer sleep is not a diagnosis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your selector may be perfectly valid and still be wrong
&lt;/h2&gt;

&lt;p&gt;Now imagine a dashboard with a list of customer accounts.&lt;/p&gt;

&lt;p&gt;The test clicks the third row. It worked yesterday. Today, a new account arrives and the list reorders automatically. The third row is now a different customer.&lt;/p&gt;

&lt;p&gt;The selector didn't fail. That's worse. It worked on the wrong thing.&lt;/p&gt;

&lt;p&gt;Skeleton loaders create another version of this problem. A test sees the container, assumes the content is ready, and attempts to interact with something that hasn't been rendered yet. Virtualized lists make it even trickier because rows may not exist in the DOM until they're scrolled into view.&lt;/p&gt;

&lt;p&gt;A useful &lt;a href="https://bugbench.com/benchmark-plan-comparing-browser-automation-frameworks-on-dynamic-list-reordering-skeleton-loaders-and-selector-resilience/" rel="noopener noreferrer"&gt;benchmark plan for dynamic lists, skeleton loaders, and selector resilience&lt;/a&gt; suggests measuring more than pass rate. It also looks at how often selectors need rewriting, how many special waits are required, and whether failures explain what actually happened.&lt;/p&gt;

&lt;p&gt;That seems like a much better way to compare testing frameworks than clicking the same static login form 1,000 times.&lt;/p&gt;

&lt;p&gt;For application tests, I would rather target an account by its stable identifier or accessible name than by its position on the page. And I would rather wait for a meaningful loaded state than assume the disappearance of one spinner proves everything is ready.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checkout isn't one page. It's a series of handoffs.
&lt;/h2&gt;

&lt;p&gt;Payment flows expose the limits of simplistic browser tests particularly well.&lt;/p&gt;

&lt;p&gt;A checkout form might live inside a third-party iframe. Then the provider redirects the user to a 3D Secure challenge. The browser returns to your site, perhaps after a reload, and the application has to recover the original order.&lt;/p&gt;

&lt;p&gt;A test that only enters card details and looks for a success message can miss the worst failures.&lt;/p&gt;

&lt;p&gt;Was the original payment intent preserved? Did the order remain associated with the correct customer? What happens if someone refreshes during the challenge or submits twice? Could you create a duplicate charge or order?&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://automated-testing-services.com/how-to-test-payment-iframe-flows-without-losing-state-across-reloads-redirects-and-3ds-challenges/" rel="noopener noreferrer"&gt;guide to testing payment iframes, redirects, and 3DS flows&lt;/a&gt; approaches checkout as a state-preservation problem, not just a sequence of clicks.&lt;/p&gt;

&lt;p&gt;That's the right mental model. Use a payment provider's sandbox or test environment, exercise the redirect and challenge paths, and verify the backend result. The browser reaching a thank-you page is useful evidence, but it isn't proof that the payment was processed exactly once.&lt;/p&gt;

&lt;h2&gt;
  
  
  On mobile, even the viewport is moving
&lt;/h2&gt;

&lt;p&gt;Desktop tests have a fairly stable idea of how much screen space is available.&lt;/p&gt;

&lt;p&gt;Mobile browsers are different. The address bar expands and collapses. The on-screen keyboard appears. Safe-area insets protect content around notches and system UI. A fixed footer that looked fine in one screenshot can cover the only button a customer needs.&lt;/p&gt;

&lt;p&gt;This is where pixel-perfect assertions can become counterproductive.&lt;/p&gt;

&lt;p&gt;A &lt;a href="https://frontendtester.com/how-to-test-viewport-units-safe-area-insets-and-mobile-browser-chrome-changes-without-flaky-layout-assertions/" rel="noopener noreferrer"&gt;practical guide to viewport units and safe-area testing&lt;/a&gt; makes a useful point: test whether content remains reachable and the layout responds correctly, not whether every coordinate remains identical after the browser chrome changes.&lt;/p&gt;

&lt;p&gt;For a mobile checkout, I'd want to know whether the Pay button stays visible or scrollable into view, whether the keyboard obscures an input, and whether fixed elements overlap important content.&lt;/p&gt;

&lt;p&gt;Those are customer problems. A three-pixel difference in an unrelated margin usually isn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Visual tests need a review process, not just screenshots
&lt;/h2&gt;

&lt;p&gt;Visual regression testing sounds simple: take a screenshot, compare it with the baseline, and flag differences.&lt;/p&gt;

&lt;p&gt;Then your team updates a shared button component and gets dozens of screenshot changes. Some are intentional. Some are real regressions. Someone has to tell them apart.&lt;/p&gt;

&lt;p&gt;The more noisy the system gets, the easier it is to approve a change without looking carefully.&lt;/p&gt;

&lt;p&gt;A &lt;a href="https://testautomationreviews.com/best-visual-testing-tools-for-design-system-teams-that-need-low-noise-and-clear-review-workflows/" rel="noopener noreferrer"&gt;comparison of visual testing tools for design system teams&lt;/a&gt; evaluates the parts that matter after a difference is detected: noise control, baseline management, pull-request context, approvals, and ongoing maintenance.&lt;/p&gt;

&lt;p&gt;That's the interesting buying decision. Not just whether a tool can find differences, but whether people can review those differences quickly enough to keep trusting it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the transitions, not just the destinations
&lt;/h2&gt;

&lt;p&gt;If I were evaluating a browser testing setup today, I'd make a small collection of deliberately annoying scenarios before writing a large suite.&lt;/p&gt;

&lt;p&gt;A page that hydrates after the HTML appears. A list that reorders during interaction. A payment flow that leaves the site and returns. A mobile footer that has to survive changing viewport height. A component update that produces a lot of legitimate visual differences.&lt;/p&gt;

&lt;p&gt;Then I'd ask a few plain questions.&lt;/p&gt;

&lt;p&gt;Can the tests complete without arbitrary sleeps? When something fails, can we tell whether the app or the test is wrong? How much work does it take to maintain the selectors and review the screenshots?&lt;/p&gt;

&lt;p&gt;Passing the happy path matters. But real applications spend a lot of time between states: loading, updating, redirecting, returning, resizing.&lt;/p&gt;

&lt;p&gt;That's where customers find bugs.&lt;/p&gt;

&lt;p&gt;It's also where your tests need to earn their keep.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>frontend</category>
      <category>testing</category>
      <category>react</category>
    </item>
    <item>
      <title>Your Automated Tests Are Passing. Here's What They Might Be Missing.</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Tue, 03 Nov 2026 21:36:59 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/your-automated-tests-are-passing-heres-what-they-might-be-missing-119o</link>
      <guid>https://dev.to/randomsquirrel802/your-automated-tests-are-passing-heres-what-they-might-be-missing-119o</guid>
      <description>&lt;p&gt;There's a dangerous assumption in software development that doesn't get questioned nearly enough.&lt;/p&gt;

&lt;p&gt;If the automated tests are green, the application must be working.&lt;/p&gt;

&lt;p&gt;I've seen teams spend months building test suites, celebrating their coverage numbers, and adding more tests to their CI pipelines. Then a customer reports something that should have been caught weeks ago.&lt;/p&gt;

&lt;p&gt;The tests weren't necessarily wrong. They were just testing a narrower version of reality.&lt;/p&gt;

&lt;p&gt;And that's a much harder problem to solve than a broken assertion.&lt;/p&gt;

&lt;p&gt;The good news is that you don't necessarily need hundreds of additional tests. Sometimes you just need to rethink what you're actually testing.&lt;/p&gt;

&lt;p&gt;Here are five areas where I think development teams tend to underestimate the problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Your UI Tests Might Be Ignoring How People Actually Use the Browser
&lt;/h2&gt;

&lt;p&gt;Most browser automation revolves around a familiar sequence:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Find an element.&lt;/li&gt;
&lt;li&gt;Click it.&lt;/li&gt;
&lt;li&gt;Enter some text.&lt;/li&gt;
&lt;li&gt;Verify the result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That works for a lot of applications.&lt;/p&gt;

&lt;p&gt;But actual users don't always interact with applications that way.&lt;/p&gt;

&lt;p&gt;They use Tab to navigate forms. They copy values to the clipboard. They open dialogs, close them, and expect keyboard focus to return to the right place.&lt;/p&gt;

&lt;p&gt;A test can verify that a modal opens and closes successfully while completely missing the fact that keyboard navigation becomes unusable afterward.&lt;/p&gt;

&lt;p&gt;And here's where things get interesting.&lt;/p&gt;

&lt;p&gt;Trying to test these interactions can introduce flakiness if you're checking visual indicators or relying on arbitrary delays.&lt;/p&gt;

&lt;p&gt;A better approach is to verify the underlying browser state. Instead of checking whether an input &lt;em&gt;looks&lt;/em&gt; focused, check whether it actually owns focus. Instead of trusting a "Copied!" notification, verify the clipboard contents.&lt;/p&gt;

&lt;p&gt;There's a useful technical walkthrough on &lt;a href="https://qatoolguide.com/how-to-test-clipboard-focus-and-tab-order-in-browser-automation-without-creating-flaky-ui-assertions/" rel="noopener noreferrer"&gt;testing clipboard interactions, keyboard focus, and tab order without creating flaky assertions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The broader lesson is that automating mouse clicks isn't the same as testing the user experience.&lt;/p&gt;

&lt;p&gt;Especially when accessibility matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. A Fresh Browser Session Can Hide Real Bugs
&lt;/h2&gt;

&lt;p&gt;Here's a scenario that probably sounds familiar.&lt;/p&gt;

&lt;p&gt;A Playwright test passes locally. It passes in CI. Everything looks fine.&lt;/p&gt;

&lt;p&gt;Then someone opens the application, refreshes the page, and encounters a bug.&lt;/p&gt;

&lt;p&gt;You try reproducing it in a fresh browser session.&lt;/p&gt;

&lt;p&gt;Nothing.&lt;/p&gt;

&lt;p&gt;The temptation is to assume the user did something unusual.&lt;/p&gt;

&lt;p&gt;But the real problem might be that your tests are starting with a clean browser environment every time.&lt;/p&gt;

&lt;p&gt;Think about everything that can survive a page refresh:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Browser cache&lt;/li&gt;
&lt;li&gt;Service workers&lt;/li&gt;
&lt;li&gt;IndexedDB&lt;/li&gt;
&lt;li&gt;Local storage&lt;/li&gt;
&lt;li&gt;Authentication cookies&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These aren't obscure browser features. Modern web applications depend on them constantly.&lt;/p&gt;

&lt;p&gt;And they can create a completely different experience for someone who's been using your application for two weeks compared with someone opening it for the first time.&lt;/p&gt;

&lt;p&gt;This is particularly relevant for single-page applications, offline-capable applications, and products with complex client-side state.&lt;/p&gt;

&lt;p&gt;The SDET published a practical guide on &lt;a href="https://thesdet.com/how-to-debug-playwright-tests-that-fail-only-when-the-browser-cache-service-worker-or-indexeddb-state-survives-a-refresh/" rel="noopener noreferrer"&gt;debugging Playwright failures caused by browser state that survives a refresh&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;One takeaway I particularly agree with is that you need to distinguish between the different persistence layers before trying to fix the test.&lt;/p&gt;

&lt;p&gt;Clearing everything between executions might make your automation more predictable.&lt;/p&gt;

&lt;p&gt;But it can also hide the exact bug your customers are experiencing.&lt;/p&gt;

&lt;p&gt;You need both kinds of tests: isolated tests that start clean, and tests that deliberately preserve state.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Passing CI Tests Don't Tell You Whether Your Application Works Right Now
&lt;/h2&gt;

&lt;p&gt;Most teams think about automated testing primarily as something that happens before deployment.&lt;/p&gt;

&lt;p&gt;A developer pushes code. Tests execute. If everything passes, the changes can be deployed.&lt;/p&gt;

&lt;p&gt;That's useful.&lt;/p&gt;

&lt;p&gt;But what happens six hours later?&lt;/p&gt;

&lt;p&gt;Maybe an external API starts returning errors. Maybe an authentication provider becomes unavailable. Maybe a configuration change breaks checkout without anyone deploying a new version of the application.&lt;/p&gt;

&lt;p&gt;Your CI pipeline won't necessarily tell you about any of these problems.&lt;/p&gt;

&lt;p&gt;This is where synthetic monitoring becomes useful.&lt;/p&gt;

&lt;p&gt;Instead of executing tests only when code changes, you continuously run a small number of important checks against your application.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Can someone log in?&lt;/li&gt;
&lt;li&gt;Can a customer complete checkout?&lt;/li&gt;
&lt;li&gt;Does the API return the expected response?&lt;/li&gt;
&lt;li&gt;Can users access their dashboards?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Of course, the tricky part is deciding what to monitor and how to handle failures.&lt;/p&gt;

&lt;p&gt;You don't want a monitoring system that wakes someone up because a button took an extra second to appear.&lt;/p&gt;

&lt;p&gt;You want actionable alerts that represent genuine problems.&lt;/p&gt;

&lt;p&gt;If you're evaluating this approach, there's a solid breakdown of &lt;a href="https://test-automation-tools.com/what-to-look-for-in-synthetic-monitoring-tools-for-api-checks-browser-journeys-and-alert-routing/" rel="noopener noreferrer"&gt;how to select synthetic monitoring tools for API checks, browser journeys, and alert routing&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I think the distinction between automated testing and synthetic monitoring deserves more attention.&lt;/p&gt;

&lt;p&gt;One helps you decide whether a release is safe.&lt;/p&gt;

&lt;p&gt;The other helps you discover when something that was previously working has stopped working.&lt;/p&gt;

&lt;p&gt;You probably need both.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. AI Applications Introduce an Entirely Different Category of Testing Problems
&lt;/h2&gt;

&lt;p&gt;Testing traditional software is relatively straightforward, at least conceptually.&lt;/p&gt;

&lt;p&gt;You provide an input. You expect an output.&lt;/p&gt;

&lt;p&gt;Testing applications that use LLMs changes that equation.&lt;/p&gt;

&lt;p&gt;The same request can produce slightly different responses. The application might retrieve information from different documents. A prompt update might change the response even though the user-facing functionality appears unchanged.&lt;/p&gt;

&lt;p&gt;And then there's caching.&lt;/p&gt;

&lt;p&gt;Many applications cache LLM responses to reduce latency and costs. That makes sense economically, especially when inference costs become meaningful at scale.&lt;/p&gt;

&lt;p&gt;But caching introduces an interesting problem.&lt;/p&gt;

&lt;p&gt;Imagine you update a system prompt to fix an incorrect response.&lt;/p&gt;

&lt;p&gt;You run your tests.&lt;/p&gt;

&lt;p&gt;The application still returns the old answer.&lt;/p&gt;

&lt;p&gt;Did the model regress? Did the prompt change fail? Is the retrieval system providing outdated information?&lt;/p&gt;

&lt;p&gt;Or did your application simply return a cached response generated before the update?&lt;/p&gt;

&lt;p&gt;These are fundamentally different problems.&lt;/p&gt;

&lt;p&gt;And treating all of them as "AI quality issues" is going to make debugging unnecessarily expensive.&lt;/p&gt;

&lt;p&gt;A good starting point is this guide on &lt;a href="https://aitestingreviews.com/how-to-test-llm-output-caching-without-mistaking-cache-hits-for-model-regression/" rel="noopener noreferrer"&gt;testing LLM output caching without confusing cache hits with model regressions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The important idea is to test the individual layers separately.&lt;/p&gt;

&lt;p&gt;Verify that caching works correctly. Verify that cached entries are invalidated when necessary. Verify that requests bypassing the cache use the expected prompt, retrieval data, and model configuration.&lt;/p&gt;

&lt;p&gt;Only then should you start investigating whether the model itself is producing worse results.&lt;/p&gt;

&lt;p&gt;As more teams add AI functionality to otherwise traditional web applications, I expect this distinction to become increasingly important.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Choosing the Wrong Testing Tool Can Be More Expensive Than Writing the Wrong Tests
&lt;/h2&gt;

&lt;p&gt;Here's something I think a lot of teams get backward.&lt;/p&gt;

&lt;p&gt;They spend weeks comparing test automation tools based on features.&lt;/p&gt;

&lt;p&gt;Does it support Playwright? Does it have AI? Can it test APIs? Does it support mobile applications?&lt;/p&gt;

&lt;p&gt;Those questions matter.&lt;/p&gt;

&lt;p&gt;But they're not the questions I'd start with.&lt;/p&gt;

&lt;p&gt;I'd ask:&lt;/p&gt;

&lt;p&gt;How much work will it take to maintain these tests six months from now?&lt;/p&gt;

&lt;p&gt;Who will be responsible for fixing them?&lt;/p&gt;

&lt;p&gt;Can someone who didn't originally create a test understand what it's doing?&lt;/p&gt;

&lt;p&gt;And what happens when we need to expand coverage beyond our original requirements?&lt;/p&gt;

&lt;p&gt;Consider the difference between a platform optimized for broad QA coverage and one optimized for quickly creating readable browser tests.&lt;/p&gt;

&lt;p&gt;Neither approach is automatically better.&lt;/p&gt;

&lt;p&gt;For example, this &lt;a href="https://aitestingreport.com/katalon-vs-testrigor-which-no-code-platform-fits-broader-qa-coverage/" rel="noopener noreferrer"&gt;comparison of Katalon and testRigor&lt;/a&gt; looks at how those platforms differ in authoring, maintenance, coverage, and governance.&lt;/p&gt;

&lt;p&gt;The interesting part isn't necessarily which product wins.&lt;/p&gt;

&lt;p&gt;It's how the tradeoffs change depending on the team using it.&lt;/p&gt;

&lt;p&gt;A small team trying to automate ten critical browser journeys has very different requirements from an enterprise maintaining thousands of tests across web, mobile, and API environments.&lt;/p&gt;

&lt;p&gt;There's also an uncomfortable truth about automation.&lt;/p&gt;

&lt;p&gt;The cost of creating a test is usually only a fraction of the cost of owning it.&lt;/p&gt;

&lt;p&gt;A test that takes five minutes to create but requires regular debugging and maintenance isn't necessarily cheaper than one that takes longer to create but rarely needs attention.&lt;/p&gt;

&lt;p&gt;This becomes especially important as AI makes generating new tests easier.&lt;/p&gt;

&lt;p&gt;Creating more tests is no longer the hard part.&lt;/p&gt;

&lt;p&gt;Keeping them useful is.&lt;/p&gt;

&lt;h2&gt;
  
  
  So, What Should You Actually Do Differently?
&lt;/h2&gt;

&lt;p&gt;I wouldn't respond to these problems by immediately adding another testing framework or doubling the number of automated tests.&lt;/p&gt;

&lt;p&gt;I'd start with something much simpler.&lt;/p&gt;

&lt;p&gt;Pick five of the most important user journeys in your application.&lt;/p&gt;

&lt;p&gt;Then ask yourself a few questions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we testing what users actually do?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not just whether buttons work, but whether keyboard navigation, clipboard interactions, and other browser behaviors work correctly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we testing realistic application state?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A fresh browser session is useful, but it doesn't represent every user. Make sure you're testing what happens after refreshes, repeated visits, and persisted sessions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we testing what happens after deployment?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Passing CI checks only tells you about one moment in time. Your application needs to keep working long after the deployment finishes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we isolating the causes of AI failures?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If your application uses LLMs, separate caching, retrieval, prompt behavior, and model output quality. Otherwise, you'll spend too much time debugging the wrong component.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we optimizing for maintenance, not just creation?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The best automation strategy is one your team can actually maintain.&lt;/p&gt;

&lt;p&gt;And that brings me to the point I think matters most.&lt;/p&gt;

&lt;h2&gt;
  
  
  More Automated Tests Don't Automatically Mean Better Testing
&lt;/h2&gt;

&lt;p&gt;It's easy to measure the number of automated tests.&lt;/p&gt;

&lt;p&gt;It's easy to track how many executions passed.&lt;/p&gt;

&lt;p&gt;It's even relatively easy to measure code coverage.&lt;/p&gt;

&lt;p&gt;What's harder to measure is whether your tests are actually reducing the risk of something breaking for your customers.&lt;/p&gt;

&lt;p&gt;That's the metric worth caring about.&lt;/p&gt;

&lt;p&gt;I'd rather have 100 reliable tests covering the most important real-world scenarios than 2,000 tests that mostly verify the same happy paths in perfectly clean browser sessions.&lt;/p&gt;

&lt;p&gt;The goal isn't to automate everything.&lt;/p&gt;

&lt;p&gt;It's to find the problems that matter, catch them early, and spend as little time as possible maintaining the machinery that does it.&lt;/p&gt;

&lt;p&gt;That's what good test automation should accomplish.&lt;/p&gt;

&lt;p&gt;Everything else is just activity.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;What kinds of bugs have made it into production despite your automated tests passing? I'm particularly interested in cases where the tests were technically correct but missed something important.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>testing</category>
      <category>playwright</category>
      <category>automation</category>
      <category>qa</category>
    </item>
    <item>
      <title>Your Smoke Tests Are Crying Wolf. That's a Release Problem.</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Wed, 14 Oct 2026 21:09:42 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/your-smoke-tests-are-crying-wolf-thats-a-release-problem-4al</link>
      <guid>https://dev.to/randomsquirrel802/your-smoke-tests-are-crying-wolf-thats-a-release-problem-4al</guid>
      <description>&lt;p&gt;It's 4:45 p.m. The deployment finishes. The smoke suite goes red.&lt;/p&gt;

&lt;p&gt;Nobody panics. Nobody rolls back. Someone asks in Slack whether the test failed again because the sample account already existed.&lt;/p&gt;

&lt;p&gt;That reaction is the problem.&lt;/p&gt;

&lt;p&gt;A smoke test has one job: give the team a fast, credible answer to "Did this release break something we care about?" If everybody assumes the answer is probably wrong, the suite has become another dashboard to ignore.&lt;/p&gt;

&lt;p&gt;This isn't solved by adding fifty more assertions.&lt;/p&gt;

&lt;h2&gt;
  
  
  A release check has to earn the right to stop a release
&lt;/h2&gt;

&lt;p&gt;There's a difference between a scheduled regression run and a post-deploy decision. A regression run can take time, collect lots of detail, and investigate edge cases. A post-deploy check usually has a much narrower window.&lt;/p&gt;

&lt;p&gt;It needs to identify the deployed version, confirm that a handful of critical customer journeys work, and report a failure in language someone can act on.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bad result:&lt;/strong&gt; &lt;code&gt;Login flow failed. Timeout exceeded.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Useful result:&lt;/strong&gt; &lt;code&gt;Production build 2026.10.15-3: Login succeeded, but account dashboard never displayed the billing summary. Screenshot and browser logs attached. Previous build passed the same check.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The second message doesn't guarantee that rollback is appropriate. It gives the release owner a real starting point.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://testproject.to/how-to-test-post-deploy-smoke-checks-triggered-by-an-api-call-without-turning-every-release-into-a-false-alarm/" rel="noopener noreferrer"&gt;This guide to API-triggered post-deploy smoke checks&lt;/a&gt; is useful because it treats the API trigger as part of a larger release workflow, not the entire solution.&lt;/p&gt;

&lt;h2&gt;
  
  
  The retry trap
&lt;/h2&gt;

&lt;p&gt;Say a smoke check fails immediately after deployment. The team retries it. It passes.&lt;/p&gt;

&lt;p&gt;What happened?&lt;/p&gt;

&lt;p&gt;Maybe the app needed another ten seconds to warm up. Maybe a replica was unhealthy. Maybe the first test left data behind. Maybe the product really does fail one in ten times.&lt;/p&gt;

&lt;p&gt;Those possibilities have wildly different implications. A retry should give you more information, not erase the first result.&lt;/p&gt;

&lt;p&gt;I'd record the first failure, the retry result, the app version, the environment, and whether any state was modified between attempts. Most importantly, I'd keep the artifacts from both runs.&lt;/p&gt;

&lt;p&gt;A helpful starting point is &lt;a href="https://vibiumlabs.com/benchmark-plan-comparing-api-triggered-smoke-run-platforms-on-rollback-signal-quality-retry-safety-and-artifact-handoff/" rel="noopener noreferrer"&gt;this benchmark plan for API-triggered smoke checks&lt;/a&gt;. Its emphasis on retry safety and rollback signal quality gets at the real question: can someone trust the evidence enough to make a production decision?&lt;/p&gt;

&lt;p&gt;If your suite's success rate improves only because you discard the first attempt, you haven't necessarily improved reliability. You may have improved the reporting.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test runner is part of the release system
&lt;/h2&gt;

&lt;p&gt;When people evaluate QA tooling, they tend to count supported browsers, integrations, and test authoring features. Fair enough. But those details don't tell you what happens at 4:45 p.m. when the deployment is red and the engineer who wrote the test has gone home.&lt;/p&gt;

&lt;p&gt;I would ask a vendor or internal platform team to demonstrate four things with a deliberately failing flow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Trigger the run through the same API the deployment pipeline will call.&lt;/li&gt;
&lt;li&gt;Correlate results with the exact release or commit.&lt;/li&gt;
&lt;li&gt;Preserve screenshots, step history, and relevant logs.&lt;/li&gt;
&lt;li&gt;Deliver a result the release owner can interpret without opening five different tools.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's the practical lens used by &lt;a href="https://softwaretestingreviews.com/what-to-look-for-in-qa-platforms-for-api-triggered-smoke-checks-failure-evidence-and-release-handoffs/" rel="noopener noreferrer"&gt;this review of QA platforms for smoke checks, failure evidence, and release handoffs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Notice what's missing: an argument over which product has the prettiest green checkmark.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preview environments can make this worse
&lt;/h2&gt;

&lt;p&gt;Branch previews are wonderful until test runs point at the wrong one.&lt;/p&gt;

&lt;p&gt;Picture two pull requests. Both create a user called &lt;code&gt;test@example.com&lt;/code&gt;. One changes the checkout form. The other changes the account page. The test infrastructure shares a backend database, and one run deletes the other's test user.&lt;/p&gt;

&lt;p&gt;Now you've got a failure that looks like a code regression but is actually an isolation bug.&lt;/p&gt;

&lt;p&gt;Treat the preview URL, build identifier, test accounts, and cleanup policy as inputs to the run. Never assume the environment from yesterday still exists or still belongs to the same branch.&lt;/p&gt;

&lt;p&gt;If you're comparing AI testing services, &lt;a href="https://aitestingtoolreviews.com/what-to-look-for-in-ai-testing-platforms-for-preview-environments-branch-isolation-and-release-handoff/" rel="noopener noreferrer"&gt;this look at preview environments and branch isolation&lt;/a&gt; provides a useful set of questions beyond "Can it generate tests?"&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would actually put in a smoke suite
&lt;/h2&gt;

&lt;p&gt;Not everything. Definitely not everything.&lt;/p&gt;

&lt;p&gt;I'd start with a tiny set of paths that have a clear business consequence when broken: sign in, load the main customer screen, perform one harmless read/write action, and verify a critical integration response. The exact list depends on the product. An ecommerce store and an internal admin dashboard shouldn't have identical release gates.&lt;/p&gt;

&lt;p&gt;Each check should have a known cleanup path, a tight timeout based on normal behavior, and a named owner. If a check is too unreliable to block a release, don't quietly pretend it blocks one. Fix it or classify it as informational until it earns that role.&lt;/p&gt;

&lt;p&gt;The real win isn't a permanently green dashboard. It's the day a smoke test goes red and nobody asks, "Is this another test problem?"&lt;/p&gt;

</description>
      <category>devops</category>
      <category>testing</category>
      <category>cicd</category>
      <category>automation</category>
    </item>
    <item>
      <title>Looking for a Cheaper mabl Alternative? Here's What I'd Compare Before Renewing</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Sat, 10 Oct 2026 21:24:04 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/looking-for-a-cheaper-mabl-alternative-heres-what-id-compare-before-renewing-3f3h</link>
      <guid>https://dev.to/randomsquirrel802/looking-for-a-cheaper-mabl-alternative-heres-what-id-compare-before-renewing-3f3h</guid>
      <description>&lt;p&gt;I have a fairly uncontroversial opinion that still seems to annoy people in software procurement: a good product can be a bad purchase.&lt;/p&gt;

&lt;p&gt;That's where I put &lt;a href="https://www.mabl.com/" rel="noopener noreferrer"&gt;mabl&lt;/a&gt; for a particular kind of team.&lt;/p&gt;

&lt;p&gt;It's a legitimate test automation platform. Its browser recorder is approachable, its automatic healing can save maintenance time, and the product extends well beyond clicking around a web page. It offers API, accessibility, performance, visual, email, and PDF testing. More recently, mabl has invested heavily in AI-assisted test authoring and recovery.&lt;/p&gt;

&lt;p&gt;The problem isn't that mabl doesn't do enough. The problem is that you might be paying for a much larger testing platform than you actually need.&lt;/p&gt;

&lt;p&gt;If I were reviewing a mabl renewal today, &lt;a href="https://endtest.io/" rel="noopener noreferrer"&gt;Endtest&lt;/a&gt; would be near the top of my shortlist. Not because it has a flashier AI pitch. Because it offers much of the test automation functionality most teams buy mabl for, with published entry pricing that's easier to budget for.&lt;/p&gt;

&lt;p&gt;But you shouldn't switch testing platforms because of a blog post. You should switch because the numbers and the test results work out.&lt;/p&gt;

&lt;h2&gt;
  
  
  The expensive part isn't always the subscription
&lt;/h2&gt;

&lt;p&gt;Think about an engineering team with eight developers, two QA engineers, and a SaaS product that ships every weekday.&lt;/p&gt;

&lt;p&gt;They probably want five fairly ordinary things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Someone should be able to create a browser test without spending an afternoon writing selectors.&lt;/li&gt;
&lt;li&gt;Tests should run in CI and on a schedule.&lt;/li&gt;
&lt;li&gt;When a test fails, the team should be able to tell whether the application is broken or the test is broken.&lt;/li&gt;
&lt;li&gt;Important flows should run on more than one browser.&lt;/li&gt;
&lt;li&gt;Adding coverage shouldn't mean hiring another automation engineer.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's not a modest list. But it isn't especially exotic, either.&lt;/p&gt;

&lt;p&gt;The subscription line on the budget is only the start. The real cost of testing includes people debugging failures, rewriting tests after UI changes, maintaining environments, and waiting for results. A cheaper tool that wastes three hours a week isn't cheaper for very long.&lt;/p&gt;

&lt;p&gt;This is why I don't think a price-only comparison is useful. And it's also why mabl deserves credit: its automatic waits, healing, and reporting address real operational costs.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, understand what you're paying for
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://www.mabl.com/pricing" rel="noopener noreferrer"&gt;mabl's pricing page&lt;/a&gt; currently directs buyers to request a custom quote. It describes a credit-based approach for cloud testing, beginning with 500 credits per month, alongside unlimited local runs and cloud concurrency at no additional concurrency charge. The exact economics depend on your contract and usage.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://endtest.io/pricing" rel="noopener noreferrer"&gt;Endtest publishes a Starter plan at $175/month and a Pro plan at $450/month&lt;/a&gt;, plus custom enterprise pricing. Its pricing page lists unlimited test creation, executions, and users, with limits and entitlements that vary by plan, including parallel slots and AI features.&lt;/p&gt;

&lt;p&gt;Those aren't directly comparable billing models. Don't confuse an unlimited-executions line with unlimited simultaneous capacity. And don't treat a credit as equivalent to an Endtest execution.&lt;/p&gt;

&lt;p&gt;Still, published pricing gives you something useful: a concrete starting point for a negotiation or pilot.&lt;/p&gt;

&lt;p&gt;Here's how I'd model it.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Annual software spend&lt;/th&gt;
&lt;th&gt;Amount&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;mabl renewal quote&lt;/td&gt;
&lt;td&gt;Enter your actual quote&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Endtest Starter at $175/month&lt;/td&gt;
&lt;td&gt;$2,100&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Endtest Pro at $450/month&lt;/td&gt;
&lt;td&gt;$5,400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Difference&lt;/td&gt;
&lt;td&gt;Your quote minus the relevant Endtest plan&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For example, &lt;strong&gt;if&lt;/strong&gt; your mabl quote is $1,500 per month and the $450 Endtest configuration covers the same workload, the arithmetic is $18,000 versus $5,400 annually. That's a &lt;strong&gt;70% reduction in subscription cost&lt;/strong&gt;, or &lt;strong&gt;$12,600 per year&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That's a scenario, not an estimate of what mabl charges every customer. Your real quote might be higher or lower; you might also need more Endtest parallel capacity or an enterprise arrangement.&lt;/p&gt;

&lt;p&gt;That distinction matters. An apparently cheap replacement isn't a bargain if you discover the missing requirement after migration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Feature parity is the wrong question
&lt;/h2&gt;

&lt;p&gt;People love feature grids. I understand why. They're easy to forward to a manager.&lt;/p&gt;

&lt;p&gt;But the question, "Does Endtest have everything mabl has?" is too broad to be useful. Modern testing products change quickly, and both vendors have capabilities that may not map neatly onto the other's offering.&lt;/p&gt;

&lt;p&gt;A better question is: &lt;strong&gt;Can the replacement execute our critical workflows, with the controls and evidence our team needs?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here's the shortlist I'd start with:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;mabl&lt;/th&gt;
&lt;th&gt;Endtest&lt;/th&gt;
&lt;th&gt;What to verify in a trial&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;No-code browser test creation&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;How readable and editable are generated steps?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI-assisted creation&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Can it complete a real multi-page business flow?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Self-healing&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Does it recover correctly, not just quietly continue?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cross-browser execution&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Browser versions and combinations you actually use&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API testing&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Authentication, variables, response assertions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Accessibility checks&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Rules, scope, and report format&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Email and PDF workflows&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Your actual email and PDF scenarios&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Native mobile testing&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Devices, builds, and plan entitlements&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CI/CD and scheduling&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Triggering, result retrieval, and failure behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Performance testing&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Evaluate separately&lt;/td&gt;
&lt;td&gt;Don't assume equivalent performance-testing depth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pricing&lt;/td&gt;
&lt;td&gt;Custom quote&lt;/td&gt;
&lt;td&gt;Published entry tiers; enterprise by quote&lt;/td&gt;
&lt;td&gt;Total cost at your required concurrency&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For mabl's own capability breakdown, see its &lt;a href="https://help.mabl.com/hc/en-us/articles/18967631285396-About-mabl" rel="noopener noreferrer"&gt;product overview&lt;/a&gt;. For Endtest, see its &lt;a href="https://endtest.io/pricing" rel="noopener noreferrer"&gt;feature and plan breakdown&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Notice the last two rows. I would not claim blanket feature parity. mabl has an integrated performance-testing offering, and that could be a deciding factor if your QA team relies on it. Conversely, Endtest has capabilities such as &lt;a href="https://endtest.io/docs/web-tests/how-to-create-web-tests" rel="noopener noreferrer"&gt;browser-based recording&lt;/a&gt;, AI-driven test creation, and advanced browser execution options that may be more than sufficient for teams primarily focused on end-to-end regression testing.&lt;/p&gt;

&lt;p&gt;A spreadsheet doesn't make that judgment. A trial does.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test that tells me more than a sales demo
&lt;/h2&gt;

&lt;p&gt;Don't build a demo around the login form. Both tools should pass it, and neither should get much credit for doing so.&lt;/p&gt;

&lt;p&gt;Take a workflow that's been mildly irritating your team for the past year. Something like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sign up with a new email address.&lt;/li&gt;
&lt;li&gt;Open the verification email and follow the link.&lt;/li&gt;
&lt;li&gt;Create a project.&lt;/li&gt;
&lt;li&gt;Upload a PDF to that project.&lt;/li&gt;
&lt;li&gt;Open a details panel embedded in an iframe.&lt;/li&gt;
&lt;li&gt;Edit a setting and confirm the result persists after a refresh.&lt;/li&gt;
&lt;li&gt;Assert that the expected record exists through an API call.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It's a better test than a pretty dashboard because it forces the product to deal with state, email, files, frames, and assertions. It also exposes whether the test output is something an engineer can inspect and fix.&lt;/p&gt;

&lt;p&gt;I'd put the same workflow into both tools and score it on four things:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Coverage:&lt;/strong&gt; How much can the tool do without custom code or brittle workarounds?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Creation time:&lt;/strong&gt; How long until the test genuinely works, including assertions, not just until it looks plausible in an editor?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repeatability:&lt;/strong&gt; Does it survive repeated cloud runs, including after a small UI change?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Failure diagnosis:&lt;/strong&gt; When it breaks intentionally, can an engineer locate the problem quickly?&lt;/p&gt;

&lt;p&gt;A test that passes once tells you very little. A test that fails clearly is often more valuable.&lt;/p&gt;

&lt;h2&gt;
  
  
  A small experiment worth running
&lt;/h2&gt;

&lt;p&gt;Here's a simple benchmark sheet I would use in a proof of concept. It takes less than a minute to understand and avoids the usual vendor-demo theater.&lt;/p&gt;

&lt;p&gt;Run each scenario at least 20 times on each platform, using the same application version, data setup, and comparable execution environments.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;How to measure it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scenario coverage&lt;/td&gt;
&lt;td&gt;Fully automated scenarios / total scenarios&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stable-run rate&lt;/td&gt;
&lt;td&gt;Successful runs / total runs, excluding confirmed app defects&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Median runtime&lt;/td&gt;
&lt;td&gt;Median of measured run durations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Debugging time&lt;/td&gt;
&lt;td&gt;Minutes from failure notification to identified cause&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authoring time&lt;/td&gt;
&lt;td&gt;Engineer minutes until a reviewed, passing test&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monthly cost&lt;/td&gt;
&lt;td&gt;Actual quoted monthly bill for your target workload&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A lightweight script can summarize the results from a CSV file exported or assembled from your run logs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;median&lt;/span&gt;

&lt;span class="c1"&gt;# benchmark.csv columns:
# tool,scenario,passed,duration_seconds,debug_minutes
# passed must be 1 for a successful run, 0 otherwise.
&lt;/span&gt;
&lt;span class="n"&gt;runs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;benchmark.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;runs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]].&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;runs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;
    &lt;span class="n"&gt;successes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;passed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;durations&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duration_seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;debugging&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;debug_minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;
                 &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;debug_minutes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  Stable-run rate: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;successes&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  Median runtime: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;durations&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;debugging&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  Median debug time: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;debugging&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; min&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep failed-run durations in the runtime calculation, or report successful and failed runtimes separately. Otherwise a tool that fails quickly can look deceptively fast. Also record the root cause of every failure. A legitimate application bug is not an automation flake.&lt;/p&gt;

&lt;p&gt;Most importantly, don't cherry-pick the best run of each tool. That's how you end up with an impressive benchmark and an unhappy engineering team.&lt;/p&gt;

&lt;h2&gt;
  
  
  Migrating is where the savings can disappear
&lt;/h2&gt;

&lt;p&gt;Even if Endtest wins a proof of concept, I wouldn't immediately rewrite 400 existing tests.&lt;/p&gt;

&lt;p&gt;I'd use a staged migration:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Week one:&lt;/strong&gt; Inventory the suite. Identify critical user journeys, duplicate tests, long-running tests, flaky tests, and the features that require custom JavaScript or unusual integrations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Week two:&lt;/strong&gt; Rebuild a small but representative group in Endtest. Include one straightforward flow and several deliberately ugly ones. Run them in parallel with mabl against the same staging release.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Week three:&lt;/strong&gt; Compare failures and operating effort. Track who diagnosed each failure, how long it took, whether an assertion was too weak, and whether your CI pipeline received a reliable pass/fail signal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Week four:&lt;/strong&gt; Decide what to migrate, what to retire, and what should stay where it is. Some tests may not be worth porting at all.&lt;/p&gt;

&lt;p&gt;Your goal isn't to reproduce the historical test count. Your goal is to preserve useful coverage.&lt;/p&gt;

&lt;p&gt;Migrating from other tools to Endtest is fairly easy, since you can just use their &lt;a href="https://endtest.io/docs/advanced/ai-test-import" rel="noopener noreferrer"&gt;AI Test Import&lt;/a&gt; feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I'd still choose mabl
&lt;/h2&gt;

&lt;p&gt;There are sensible reasons to stay.&lt;/p&gt;

&lt;p&gt;If your team uses mabl's integrated performance testing, relies on its existing workflows across several departments, or has years of working tests with very little maintenance overhead, switching may be a distraction. Procurement friction and migration risk are real costs.&lt;/p&gt;

&lt;p&gt;And mabl's unlimited local runs and cloud concurrency model might be particularly attractive at your usage level. Don't assume the lowest published competitor price will be the cheapest configuration for a large suite.&lt;/p&gt;

&lt;p&gt;I'd also be cautious about any vendor, including Endtest, that promises AI will eliminate test maintenance. It won't. Your application still changes, your data still expires, and your test oracles still need to be correct.&lt;/p&gt;

&lt;p&gt;AI can reduce the labor. It doesn't remove responsibility.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Endtest becomes hard to ignore
&lt;/h2&gt;

&lt;p&gt;Now consider the more common case: a small or midsize engineering team mostly needs dependable end-to-end web testing, mobile coverage, API checks, cross-browser execution, and reasonable failure reports.&lt;/p&gt;

&lt;p&gt;They don't have a dedicated automation-infrastructure team. They don't want to build their own browser grid. And every year somebody has to explain the QA tooling bill to finance.&lt;/p&gt;

&lt;p&gt;For that team, &lt;a href="https://endtest.io/" rel="noopener noreferrer"&gt;Endtest&lt;/a&gt; is a serious alternative. You get an AI-enabled, no-code testing platform with substantial overlap in the workflows that drive a typical regression suite. You can inspect &lt;a href="https://endtest.io/pricing" rel="noopener noreferrer"&gt;its pricing&lt;/a&gt; without starting with a sales call, and you can use a trial to find out whether its capabilities fit your application.&lt;/p&gt;

&lt;p&gt;The cost difference can be substantial when your actual mabl quote is well above the Endtest plan that meets your needs. But that's something to establish with your own quote and your own tests, not a percentage copied from somebody else's benchmark.&lt;/p&gt;

&lt;p&gt;One other useful question to ask before signing: &lt;em&gt;If we cancel in two years, how easily can we take our test definitions and results with us?&lt;/em&gt; Data portability isn't exciting when you're buying a tool. It matters a great deal when you're replacing one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The decision I'd make
&lt;/h2&gt;

&lt;p&gt;I wouldn't recommend dumping mabl because it's a bad product. I don't think it is.&lt;/p&gt;

&lt;p&gt;I'd recommend questioning whether it's the right-sized product for the job, especially if the next renewal is forcing your team to choose between broader test coverage and the budget to build actual product features.&lt;/p&gt;

&lt;p&gt;Give Endtest the same hard scenarios. Compare creation time, real pass rates, debugging overhead, and the total annual bill. If it delivers the same useful coverage for your application at a fraction of the cost, the decision becomes pretty straightforward.&lt;/p&gt;

&lt;p&gt;And if it doesn't, you've spent a few days validating that your mabl subscription is earning its keep.&lt;/p&gt;

&lt;p&gt;Either way, you've replaced a sales argument with an engineering decision. That's a worthwhile result.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>automation</category>
      <category>qa</category>
      <category>devops</category>
    </item>
    <item>
      <title>Most Test Suites Only Work in a World That Doesn't Exist</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Thu, 08 Oct 2026 21:25:52 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/most-test-suites-only-work-in-a-world-that-doesnt-exist-407f</link>
      <guid>https://dev.to/randomsquirrel802/most-test-suites-only-work-in-a-world-that-doesnt-exist-407f</guid>
      <description>&lt;p&gt;A lot of automated tests begin in an unusually pleasant universe.&lt;/p&gt;

&lt;p&gt;The user is logged in.&lt;/p&gt;

&lt;p&gt;The session is valid.&lt;/p&gt;

&lt;p&gt;The database contains exactly the right data.&lt;/p&gt;

&lt;p&gt;The feature flags are known.&lt;/p&gt;

&lt;p&gt;The browser has no weird state.&lt;/p&gt;

&lt;p&gt;The network works.&lt;/p&gt;

&lt;p&gt;Nothing unexpected is covering the screen.&lt;/p&gt;

&lt;p&gt;And the deployment that happened five minutes ago is assumed to be fine because CI passed.&lt;/p&gt;

&lt;p&gt;Then the test runs.&lt;/p&gt;

&lt;p&gt;Green.&lt;/p&gt;

&lt;p&gt;Fantastic.&lt;/p&gt;

&lt;p&gt;Unfortunately, real users don't live in that universe.&lt;/p&gt;

&lt;p&gt;They leave tabs open overnight.&lt;/p&gt;

&lt;p&gt;Their access tokens expire halfway through a workflow.&lt;/p&gt;

&lt;p&gt;They get assigned to experiments.&lt;/p&gt;

&lt;p&gt;They lose network connectivity.&lt;/p&gt;

&lt;p&gt;A modal opens while another component still has focus.&lt;/p&gt;

&lt;p&gt;A deployment changes one service but not another.&lt;/p&gt;

&lt;p&gt;And sometimes the application quietly logs them back in while they're doing something important.&lt;/p&gt;

&lt;p&gt;If your test automation strategy assumes those things don't happen, you might have excellent coverage of a product that doesn't actually exist.&lt;/p&gt;

&lt;h2&gt;
  
  
  The clean-session assumption is particularly dangerous
&lt;/h2&gt;

&lt;p&gt;Here's a very normal test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Login
Open Dashboard
Create Project
Invite User
Verify Invitation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Looks good.&lt;/p&gt;

&lt;p&gt;Now try this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Login
Open Dashboard
Wait until access token expires
Start creating project
API returns 401
Application silently refreshes token
Request retries
Continue workflow
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Did the project get created once?&lt;/p&gt;

&lt;p&gt;Twice?&lt;/p&gt;

&lt;p&gt;Not at all?&lt;/p&gt;

&lt;p&gt;Did the UI recover?&lt;/p&gt;

&lt;p&gt;Did the application dump the user back onto the login page?&lt;/p&gt;

&lt;p&gt;Did it preserve what they had typed?&lt;/p&gt;

&lt;p&gt;Did the test framework even notice that authentication changed underneath it?&lt;/p&gt;

&lt;p&gt;This is one of those situations where a test can pass hundreds of times simply because you're always starting with a fresh session.&lt;/p&gt;

&lt;p&gt;There's a useful &lt;a href="https://bugbench.com/benchmark-plan-comparing-browser-testing-platforms-on-session-expiry-recovery-silent-re-login-and-evidence-after-token-refresh/" rel="noopener noreferrer"&gt;benchmark plan for session expiry recovery, silent re-login, and token-refresh evidence&lt;/a&gt; that treats this as something browser testing platforms should be evaluated against.&lt;/p&gt;

&lt;p&gt;I agree.&lt;/p&gt;

&lt;p&gt;Authentication isn't just a setup step.&lt;/p&gt;

&lt;p&gt;It's application state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Users don't politely expire between tests
&lt;/h2&gt;

&lt;p&gt;Most automation frameworks encourage isolation.&lt;/p&gt;

&lt;p&gt;That's generally a good thing.&lt;/p&gt;

&lt;p&gt;Fresh context.&lt;/p&gt;

&lt;p&gt;Fresh cookies.&lt;/p&gt;

&lt;p&gt;Fresh test data.&lt;/p&gt;

&lt;p&gt;Predictable environment.&lt;/p&gt;

&lt;p&gt;But isolation can accidentally remove the exact states that make real applications difficult.&lt;/p&gt;

&lt;p&gt;A real user might:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;09:00  Log in
09:15  Open a form
09:22  Switch tabs
10:04  Come back
10:07  Submit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your automated test might do the entire thing in 14 seconds.&lt;/p&gt;

&lt;p&gt;Those are technically the same workflow.&lt;/p&gt;

&lt;p&gt;They're not the same conditions.&lt;/p&gt;

&lt;p&gt;If your product has session expiration, refresh tokens, idle timeouts, or role changes, time itself becomes part of the test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Feature flags create a similar illusion
&lt;/h2&gt;

&lt;p&gt;Feature flags are great.&lt;/p&gt;

&lt;p&gt;They let you roll things out gradually.&lt;/p&gt;

&lt;p&gt;Test with internal users.&lt;/p&gt;

&lt;p&gt;Run experiments.&lt;/p&gt;

&lt;p&gt;Disable broken features quickly.&lt;/p&gt;

&lt;p&gt;But they also create a new dimension of application state.&lt;/p&gt;

&lt;p&gt;The same URL can now represent two different products.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;checkout_v2 = false

vs.

checkout_v2 = true
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And if your test suite handles this by copying flag setup into 80 scenarios, you eventually get something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Set checkout_v2 = true
Set new_nav = false
Set pricing_test = variant_b
Set recommendations = true
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Repeated everywhere.&lt;/p&gt;

&lt;p&gt;Then somebody changes the feature configuration and half the suite becomes archaeological evidence.&lt;/p&gt;

&lt;p&gt;This guide on &lt;a href="https://test-automation-experts.com/how-to-test-feature-flag-overrides-in-browser-automation-without-hardcoding-flag-values-into-every-scenario/" rel="noopener noreferrer"&gt;testing feature flag overrides without hardcoding the same flag values into every scenario&lt;/a&gt; tackles that exact problem.&lt;/p&gt;

&lt;p&gt;The larger lesson is useful even if you're not using feature flags heavily.&lt;/p&gt;

&lt;p&gt;Test setup is part of your architecture.&lt;/p&gt;

&lt;p&gt;If every test independently recreates the world it needs, maintenance eventually becomes expensive.&lt;/p&gt;

&lt;h2&gt;
  
  
  UI state is more than "element visible"
&lt;/h2&gt;

&lt;p&gt;Here's another situation I think browser automation often oversimplifies.&lt;/p&gt;

&lt;p&gt;Modals.&lt;/p&gt;

&lt;p&gt;Testing a modal usually looks something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Click "Delete"
Verify confirmation modal is visible
Click "Confirm"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Okay.&lt;/p&gt;

&lt;p&gt;But accessibility changes what "working" actually means.&lt;/p&gt;

&lt;p&gt;If a modal is open, what happens to the rest of the page?&lt;/p&gt;

&lt;p&gt;Can keyboard focus escape?&lt;/p&gt;

&lt;p&gt;Can you tab into controls behind the modal?&lt;/p&gt;

&lt;p&gt;Can screen-reader users interact with background content that visually appears locked?&lt;/p&gt;

&lt;p&gt;The relatively new &lt;code&gt;inert&lt;/code&gt; attribute gives browsers a cleaner way to make parts of a page non-interactive.&lt;/p&gt;

&lt;p&gt;But testing it properly requires more than:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;modal.isVisible() === true
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This walkthrough on &lt;a href="https://frontendtester.com/how-to-test-the-inert-attribute-modal-background-locking-and-tabbable-leakage-without-brittle-assertions/" rel="noopener noreferrer"&gt;testing the &lt;code&gt;inert&lt;/code&gt; attribute, modal background locking, and tabbable leakage&lt;/a&gt; is a good example of how much behavior can hide behind something that looks visually correct.&lt;/p&gt;

&lt;p&gt;A screenshot can show you a perfect modal.&lt;/p&gt;

&lt;p&gt;The keyboard experience can still be completely broken.&lt;/p&gt;

&lt;h2&gt;
  
  
  This is why I don't love "end-to-end coverage" as a metric
&lt;/h2&gt;

&lt;p&gt;Teams will say:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We have 80% of our critical user journeys automated.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That sounds great.&lt;/p&gt;

&lt;p&gt;But what does it mean?&lt;/p&gt;

&lt;p&gt;If you test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Login → Dashboard → Checkout → Confirmation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;do you also test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Login
    ↓
Session expires
    ↓
Checkout begins
    ↓
Token refreshes
    ↓
Feature flag changes rendering
    ↓
Modal opens
    ↓
Keyboard navigation continues
    ↓
Purchase completes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Probably not.&lt;/p&gt;

&lt;p&gt;And I'm not saying you should automate every imaginable combination.&lt;/p&gt;

&lt;p&gt;That would be ridiculous.&lt;/p&gt;

&lt;p&gt;The point is that "journey coverage" can give you false confidence if your journeys are modeled as clean sequences instead of changing states.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI-generated tests can amplify this problem
&lt;/h2&gt;

&lt;p&gt;AI makes it easier to generate a lot of automation.&lt;/p&gt;

&lt;p&gt;Type:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Test checkout.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And an agent can potentially create the flow for you.&lt;/p&gt;

&lt;p&gt;That's useful.&lt;/p&gt;

&lt;p&gt;But the first generated version will often represent the obvious path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Open checkout
Enter card
Click Pay
Verify success
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The interesting part is what happens later.&lt;/p&gt;

&lt;p&gt;Can you edit the steps?&lt;/p&gt;

&lt;p&gt;Can you add recovery behavior?&lt;/p&gt;

&lt;p&gt;Can another person understand why something failed?&lt;/p&gt;

&lt;p&gt;Can you attach notes explaining why a strange step exists?&lt;/p&gt;

&lt;p&gt;Can the test survive being handed to a different team six months from now?&lt;/p&gt;

&lt;p&gt;Those questions matter much more once AI starts generating a significant percentage of the suite.&lt;/p&gt;

&lt;p&gt;This rubric on &lt;a href="https://ai-testing-tools.com/what-to-look-for-in-ai-testing-platforms-for-editable-test-steps-recovery-notes-and-team-handoffs/" rel="noopener noreferrer"&gt;editable steps, recovery notes, and team handoffs in AI testing platforms&lt;/a&gt; focuses on exactly that.&lt;/p&gt;

&lt;p&gt;I think "editable" is an underrated feature in AI systems.&lt;/p&gt;

&lt;p&gt;Autonomy sounds cooler.&lt;/p&gt;

&lt;p&gt;Editability is what saves you when reality gets complicated.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fully autonomous QA sounds great until you ask what happens at 3 AM
&lt;/h2&gt;

&lt;p&gt;There's a lot of interest now in autonomous QA agents.&lt;/p&gt;

&lt;p&gt;And I understand why.&lt;/p&gt;

&lt;p&gt;The idea is compelling:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Give agent application
    ↓
Agent explores
    ↓
Agent creates tests
    ↓
Agent runs continuously
    ↓
Agent finds failures
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's a very different model from manually maintaining hundreds or thousands of scripts.&lt;/p&gt;

&lt;p&gt;There's a demonstration of that approach in &lt;a href="https://www.youtube.com/watch?v=58zH1I1pku0" rel="noopener noreferrer"&gt;Endtest Bot: Your Autonomous QA Agent&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I think agentic testing will become a meaningful part of QA.&lt;/p&gt;

&lt;p&gt;But the important question isn't whether the agent can click around your product.&lt;/p&gt;

&lt;p&gt;It's:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What happens when the application behaves unexpectedly?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;An autonomous system becomes much more valuable when it can recognize:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Session expired
Token refreshed
Expected state restored
Continue scenario
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;rather than simply reporting:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Step failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That difference is huge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recovery may be more important than self-healing
&lt;/h2&gt;

&lt;p&gt;"Self-healing" became one of the favorite phrases in test automation.&lt;/p&gt;

&lt;p&gt;Usually it means:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Locator broke
    ↓
Tool found another locator
    ↓
Test continued
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Useful.&lt;/p&gt;

&lt;p&gt;But that's a pretty narrow definition of healing.&lt;/p&gt;

&lt;p&gt;Real applications need recovery from things like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;expired sessions&lt;/li&gt;
&lt;li&gt;intermittent APIs&lt;/li&gt;
&lt;li&gt;reauthentication&lt;/li&gt;
&lt;li&gt;WebSocket reconnects&lt;/li&gt;
&lt;li&gt;temporary loading states&lt;/li&gt;
&lt;li&gt;feature changes&lt;/li&gt;
&lt;li&gt;interrupted transactions&lt;/li&gt;
&lt;li&gt;background refreshes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Changing &lt;code&gt;#submit-button&lt;/code&gt; to &lt;code&gt;[aria-label="Submit"]&lt;/code&gt; is the easy version.&lt;/p&gt;

&lt;p&gt;Recovering application state without changing the meaning of the test is much harder.&lt;/p&gt;

&lt;p&gt;That's where I think the next generation of testing tools will either become genuinely useful or just generate impressive demos.&lt;/p&gt;

&lt;h2&gt;
  
  
  And then we deploy
&lt;/h2&gt;

&lt;p&gt;There's another weird boundary in a lot of testing strategies.&lt;/p&gt;

&lt;p&gt;Everything before deployment gets enormous attention.&lt;/p&gt;

&lt;p&gt;Everything after deployment gets:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We'll check Sentry.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;But a successful deployment pipeline doesn't mean the product actually works.&lt;/p&gt;

&lt;p&gt;You can still have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Bad environment variable
Broken CDN asset
Missing migration
Expired production credential
Misconfigured feature flag
Third-party API failure
DNS issue
Regional outage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;CI may have been completely green.&lt;/p&gt;

&lt;p&gt;Which raises a practical question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What exactly should we run after deployment?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;There isn't one answer.&lt;/p&gt;

&lt;p&gt;Sometimes a fast API check is enough.&lt;/p&gt;

&lt;p&gt;Sometimes you need a real browser.&lt;/p&gt;

&lt;p&gt;Sometimes an important customer journey should be monitored continuously.&lt;/p&gt;

&lt;p&gt;This &lt;a href="https://testingtoolguide.com/how-to-choose-between-api-smoke-checks-full-browser-flows-and-synthetic-monitoring-for-post-deploy-validation/" rel="noopener noreferrer"&gt;decision guide for API smoke checks, full browser flows, and synthetic monitoring&lt;/a&gt; gives a useful framework for deciding where each belongs.&lt;/p&gt;

&lt;p&gt;I'd think about them as different layers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;API Smoke Check
"Is the service basically alive?"

Browser Flow
"Can a user actually complete this?"

Synthetic Monitoring
"Does it keep working after we leave?"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are related questions.&lt;/p&gt;

&lt;p&gt;They're not interchangeable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your production test strategy probably deserves more attention than your CI strategy
&lt;/h2&gt;

&lt;p&gt;Suppose your checkout service deploys successfully.&lt;/p&gt;

&lt;p&gt;CI is green.&lt;/p&gt;

&lt;p&gt;Your API smoke check passes.&lt;/p&gt;

&lt;p&gt;But the production cookie-consent configuration changed.&lt;/p&gt;

&lt;p&gt;Now the checkout button is covered for European users.&lt;/p&gt;

&lt;p&gt;The backend is healthy.&lt;/p&gt;

&lt;p&gt;The API is healthy.&lt;/p&gt;

&lt;p&gt;The deployment is healthy.&lt;/p&gt;

&lt;p&gt;Checkout is broken.&lt;/p&gt;

&lt;p&gt;This is why I increasingly think the traditional line between:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Testing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Monitoring
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;is becoming artificial.&lt;/p&gt;

&lt;p&gt;If an automated browser checks checkout before deployment, we call it a test.&lt;/p&gt;

&lt;p&gt;If the same browser checks checkout every five minutes after deployment, we call it synthetic monitoring.&lt;/p&gt;

&lt;p&gt;The user doesn't care what we call it.&lt;/p&gt;

&lt;p&gt;They care whether checkout works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try auditing your suite for assumptions instead of tests
&lt;/h2&gt;

&lt;p&gt;Here's an exercise that may be more useful than counting test cases.&lt;/p&gt;

&lt;p&gt;Take ten important automated tests.&lt;/p&gt;

&lt;p&gt;For each one, write down its assumptions.&lt;/p&gt;

&lt;p&gt;Something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User has valid session
Feature X enabled
Feature Y disabled
Network stable
No third-party overlay
API responds within 5 seconds
User role unchanged
No background token refresh
Desktop viewport
English locale
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which of these assumptions are actually guaranteed?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You might find that your tests aren't really validating a workflow.&lt;/p&gt;

&lt;p&gt;They're validating a workflow under one carefully curated version of reality.&lt;/p&gt;

&lt;p&gt;That's still useful.&lt;/p&gt;

&lt;p&gt;But it's important to know what you're getting.&lt;/p&gt;

&lt;h2&gt;
  
  
  I wouldn't try to test every combination
&lt;/h2&gt;

&lt;p&gt;There's an obvious objection.&lt;/p&gt;

&lt;p&gt;If we start considering:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sessions
flags
roles
locales
networks
devices
browser state
accessibility state
deployment state
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;the number of combinations explodes.&lt;/p&gt;

&lt;p&gt;Absolutely.&lt;/p&gt;

&lt;p&gt;You shouldn't test all of them.&lt;/p&gt;

&lt;p&gt;The solution isn't combinatorial madness.&lt;/p&gt;

&lt;p&gt;It's risk-based selection.&lt;/p&gt;

&lt;p&gt;Ask:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Which state changes have broken before?

Which ones affect money?

Which ones affect authentication?

Which ones affect permissions?

Which ones are difficult to recover from?

Which ones differ between staging and production?

Which ones would be embarrassing if a customer found them first?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Test those.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable question
&lt;/h2&gt;

&lt;p&gt;So here's the question I'd ask your team:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;If our application behaved exactly like a normal production application instead of a freshly reset test environment, how much of our automation would still work?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Expired sessions.&lt;/p&gt;

&lt;p&gt;Changing flags.&lt;/p&gt;

&lt;p&gt;Half-finished interactions.&lt;/p&gt;

&lt;p&gt;Browser history.&lt;/p&gt;

&lt;p&gt;Modal focus.&lt;/p&gt;

&lt;p&gt;Background refreshes.&lt;/p&gt;

&lt;p&gt;Post-deployment configuration.&lt;/p&gt;

&lt;p&gt;Real user state.&lt;/p&gt;

&lt;p&gt;If the answer is "not much," that's worth knowing.&lt;/p&gt;

&lt;p&gt;Because maybe your biggest automation gap isn't another missing test case.&lt;/p&gt;

&lt;p&gt;Maybe your suite is simply making assumptions that production never agreed to.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>qa</category>
      <category>automation</category>
      <category>devops</category>
    </item>
    <item>
      <title>Your Test Suite Probably Covers Screens, Not Behavior</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Tue, 06 Oct 2026 10:44:35 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/your-test-suite-probably-covers-screens-not-behavior-l91</link>
      <guid>https://dev.to/randomsquirrel802/your-test-suite-probably-covers-screens-not-behavior-l91</guid>
      <description>&lt;p&gt;Most teams can tell you how many automated tests they have.&lt;/p&gt;

&lt;p&gt;Some can tell you their pass rate.&lt;/p&gt;

&lt;p&gt;A few can tell you their code coverage.&lt;/p&gt;

&lt;p&gt;But ask this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How many meaningful state transitions in your product are actually tested?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And the room usually gets quiet.&lt;/p&gt;

&lt;p&gt;That's a much harder question.&lt;/p&gt;

&lt;p&gt;It's also probably a better one.&lt;/p&gt;

&lt;p&gt;Because users don't experience your product as a collection of pages.&lt;/p&gt;

&lt;p&gt;They experience it as a sequence of state changes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Logged out
    ↓
Logged in
    ↓
Trial
    ↓
Paid
    ↓
Payment failed
    ↓
Grace period
    ↓
Canceled
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Draft
    ↓
Submitted
    ↓
Approved
    ↓
Rejected
    ↓
Edited
    ↓
Resubmitted
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The bugs that hurt usually aren't:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The Settings page doesn't exist.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;They're more like:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A user who downgraded, then upgraded, then changed teams still has the wrong permissions.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's not a page problem.&lt;/p&gt;

&lt;p&gt;That's a state-transition problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  A lot of "coverage" is really screenshot coverage
&lt;/h2&gt;

&lt;p&gt;Here's how browser automation often grows.&lt;/p&gt;

&lt;p&gt;Someone automates login.&lt;/p&gt;

&lt;p&gt;Then signup.&lt;/p&gt;

&lt;p&gt;Then checkout.&lt;/p&gt;

&lt;p&gt;Then settings.&lt;/p&gt;

&lt;p&gt;Then the admin panel.&lt;/p&gt;

&lt;p&gt;Eventually you have a few hundred tests and a dashboard that says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;742 automated tests
97.8% passing
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Looks healthy.&lt;/p&gt;

&lt;p&gt;But those numbers don't tell you whether you've tested the dangerous combinations.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Trial user → paid user
Paid user → canceled user
Canceled user → reactivated user
Admin → downgraded member
Invited user → expired invitation
Feature OFF → feature ON → feature OFF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those transitions are where systems tend to get weird.&lt;/p&gt;

&lt;p&gt;The UI may look correct in each isolated state.&lt;/p&gt;

&lt;p&gt;The bug happens because the transition left something behind.&lt;/p&gt;

&lt;p&gt;A stale permission.&lt;/p&gt;

&lt;p&gt;An old token.&lt;/p&gt;

&lt;p&gt;A cached plan.&lt;/p&gt;

&lt;p&gt;A lingering feature flag.&lt;/p&gt;

&lt;p&gt;A half-updated record.&lt;/p&gt;

&lt;p&gt;This guide on &lt;a href="https://automated-testing-services.com/how-to-map-user-journey-test-coverage-to-state-transitions-so-outsourced-qa-teams-catch-the-right-failures/" rel="noopener noreferrer"&gt;mapping user-journey test coverage to state transitions&lt;/a&gt; makes a good case for treating transition coverage as a first-class QA problem.&lt;/p&gt;

&lt;p&gt;I think that idea applies whether QA is outsourced or entirely internal.&lt;/p&gt;

&lt;h2&gt;
  
  
  Happy paths hide structural gaps
&lt;/h2&gt;

&lt;p&gt;Take a subscription flow.&lt;/p&gt;

&lt;p&gt;You probably test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Signup
    ↓
Choose plan
    ↓
Pay
    ↓
Dashboard
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Great.&lt;/p&gt;

&lt;p&gt;Now test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Signup
    ↓
Choose plan
    ↓
Payment fails
    ↓
Retry
    ↓
Payment succeeds
    ↓
Cancel
    ↓
Reactivate
    ↓
Change card
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Different system.&lt;/p&gt;

&lt;p&gt;Same product.&lt;/p&gt;

&lt;p&gt;The first scenario proves the happy path works.&lt;/p&gt;

&lt;p&gt;The second starts probing whether your application handles state.&lt;/p&gt;

&lt;p&gt;And this is where I've become skeptical of raw test counts.&lt;/p&gt;

&lt;p&gt;Ten carefully chosen transition tests can sometimes tell you more than 100 page-oriented checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  API testing has the same problem
&lt;/h2&gt;

&lt;p&gt;The same mistake happens at the API layer.&lt;/p&gt;

&lt;p&gt;A team may have lots of tests like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POST /users → 201
GET /users/123 → 200
DELETE /users/123 → 204
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Useful.&lt;/p&gt;

&lt;p&gt;But that doesn't necessarily tell you whether the API still behaves correctly as a contract between systems.&lt;/p&gt;

&lt;p&gt;There's a meaningful difference between:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Does this request return what I expect?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Are the producer and consumer still honoring the same contract?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's the distinction behind comparisons like &lt;a href="https://qatoolguide.com/pact-vs-rest-assured-which-api-testing-stack-fits-contract-first-teams-vs-request-level-testers/" rel="noopener noreferrer"&gt;Pact vs REST Assured&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;They're not interchangeable ways of doing the same thing.&lt;/p&gt;

&lt;p&gt;One pushes you toward contract-first compatibility between services.&lt;/p&gt;

&lt;p&gt;The other is very good at request-level API checks.&lt;/p&gt;

&lt;p&gt;Which one matters more depends on how your system is built.&lt;/p&gt;

&lt;p&gt;But if your frontend, backend, mobile app, and third-party integrations are all evolving independently, testing isolated requests may not be enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frontend teams are quietly becoming QA infrastructure teams
&lt;/h2&gt;

&lt;p&gt;There's another shift happening.&lt;/p&gt;

&lt;p&gt;Frontend teams increasingly own:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;component tests&lt;/li&gt;
&lt;li&gt;browser tests&lt;/li&gt;
&lt;li&gt;visual checks&lt;/li&gt;
&lt;li&gt;accessibility checks&lt;/li&gt;
&lt;li&gt;CI workflows&lt;/li&gt;
&lt;li&gt;test data setup&lt;/li&gt;
&lt;li&gt;mock APIs&lt;/li&gt;
&lt;li&gt;release validation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which means "frontend testing" isn't really just frontend testing anymore.&lt;/p&gt;

&lt;p&gt;The frontend team may now own a substantial part of the release confidence system.&lt;/p&gt;

&lt;p&gt;That's a different job.&lt;/p&gt;

&lt;p&gt;If a team owns production UI code &lt;em&gt;and&lt;/em&gt; the automation around it, the tooling needs to fit their development workflow rather than behave like a separate QA island.&lt;/p&gt;

&lt;p&gt;This guide on &lt;a href="https://web-developer-reviews.com/what-to-look-for-in-browser-testing-platforms-when-a-frontend-team-owns-automation-too/" rel="noopener noreferrer"&gt;browser testing platforms when frontend teams own the automation too&lt;/a&gt; explores that operating model.&lt;/p&gt;

&lt;p&gt;I think this matters more than a generic list of features.&lt;/p&gt;

&lt;p&gt;A tool can be excellent in isolation and still be a bad fit if developers hate touching it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ownership matters more than people admit
&lt;/h2&gt;

&lt;p&gt;Here's a question I rarely see in tool comparisons:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Who is expected to maintain the tests six months from now?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That answer changes everything.&lt;/p&gt;

&lt;p&gt;Imagine two models.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model A
&lt;/h3&gt;

&lt;p&gt;Your internal team:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Creates tests
Maintains tests
Reviews failures
Updates selectors
Handles flaky runs
Owns CI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Model B
&lt;/h3&gt;

&lt;p&gt;A service or platform handles much of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Maintenance
Triage
Review
Repair
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are not small differences.&lt;/p&gt;

&lt;p&gt;They're completely different operating models.&lt;/p&gt;

&lt;p&gt;This &lt;a href="https://ai-test-agents.com/blinqio-vs-qa-tech-which-agentic-qa-model-fits-teams-that-want-managed-maintenance-or-faster-self-serve-browser-coverage/" rel="noopener noreferrer"&gt;BlinqIO vs QA.tech comparison&lt;/a&gt; frames the decision around managed maintenance versus faster self-serve browser coverage.&lt;/p&gt;

&lt;p&gt;That is a much better way to think about it than asking which product has more AI.&lt;/p&gt;

&lt;p&gt;The important question is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What work is my team still responsible for after we buy this?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  AI testing tools are splitting into different philosophies
&lt;/h2&gt;

&lt;p&gt;The same thing is happening across AI testing platforms.&lt;/p&gt;

&lt;p&gt;Some products want to give you:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Fast self-service automation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Others lean more toward:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Guided workflows, review, and assistance.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those approaches can look very similar in a demo.&lt;/p&gt;

&lt;p&gt;Type an instruction.&lt;/p&gt;

&lt;p&gt;Watch the browser move.&lt;/p&gt;

&lt;p&gt;Test created.&lt;/p&gt;

&lt;p&gt;Done.&lt;/p&gt;

&lt;p&gt;But they create different day-to-day workflows.&lt;/p&gt;

&lt;p&gt;A comparison like &lt;a href="https://aitestingreport.com/reflect-vs-octomind-which-ai-testing-platform-fits-self-serve-browser-coverage-or-more-guided-review-workflows/" rel="noopener noreferrer"&gt;Reflect vs Octomind&lt;/a&gt; is more useful when you look at it through that lens: self-serve browser coverage versus more guided review workflows.&lt;/p&gt;

&lt;p&gt;Neither is automatically better.&lt;/p&gt;

&lt;p&gt;A five-person startup may want to move as fast as possible with minimal process.&lt;/p&gt;

&lt;p&gt;A large QA organization may care much more about reviewability, governance, and consistency.&lt;/p&gt;

&lt;p&gt;Same category.&lt;/p&gt;

&lt;p&gt;Different problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  I think we're entering the second phase of AI test automation
&lt;/h2&gt;

&lt;p&gt;The first phase was:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Can AI create a test?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's basically settled.&lt;/p&gt;

&lt;p&gt;Yes.&lt;/p&gt;

&lt;p&gt;It can.&lt;/p&gt;

&lt;p&gt;Sometimes surprisingly well.&lt;/p&gt;

&lt;p&gt;The more interesting questions now are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Can it maintain the test?

Can humans understand what it changed?

Can it model complicated state?

Can it handle multiple roles?

Can it deal with non-happy paths?

Can it produce evidence a developer trusts?

Can teams own the result long term?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are much harder problems.&lt;/p&gt;

&lt;p&gt;There are broader comparisons of the current market, including this review of &lt;a href="https://medium.com/@liviu.lupei/the-10-best-ai-powered-test-automation-tools-in-2026-a3297eeee058" rel="noopener noreferrer"&gt;10 AI-powered test automation tools in 2026&lt;/a&gt;, that look at practical scenarios rather than only feature checklists.&lt;/p&gt;

&lt;p&gt;That's the direction I think tool evaluation needs to go.&lt;/p&gt;

&lt;p&gt;"Has AI" isn't a meaningful differentiator anymore.&lt;/p&gt;

&lt;p&gt;Almost everybody has AI.&lt;/p&gt;

&lt;p&gt;The question is what happens after the first impressive demo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your test model might be wrong
&lt;/h2&gt;

&lt;p&gt;This is the bigger issue.&lt;/p&gt;

&lt;p&gt;A lot of automation suites are modeled like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Page
    ↓
Actions
    ↓
Assertions
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But many real applications behave more like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;State A
    ↓
Transition
    ↓
State B
    ↓
Transition
    ↓
State C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second model forces you to ask different questions.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What states can this user be in?

What transitions are allowed?

Which transitions should be impossible?

What data changes during the transition?

What permissions change?

What happens if the transition is interrupted?

What happens if the user repeats it?

Can the transition be reversed?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's where some of the nastiest bugs live.&lt;/p&gt;

&lt;h2&gt;
  
  
  Here's a simple exercise
&lt;/h2&gt;

&lt;p&gt;Take one important workflow in your application.&lt;/p&gt;

&lt;p&gt;Not the whole product.&lt;/p&gt;

&lt;p&gt;Just one.&lt;/p&gt;

&lt;p&gt;Maybe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Subscription
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Write down every meaningful state.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Trial
Active
Past due
Canceled
Expired
Reactivated
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now draw the transitions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Trial → Active
Trial → Expired
Active → Past due
Past due → Active
Past due → Canceled
Canceled → Reactivated
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then compare that graph with your automated tests.&lt;/p&gt;

&lt;p&gt;Which edges are actually covered?&lt;/p&gt;

&lt;p&gt;You may discover something uncomfortable.&lt;/p&gt;

&lt;p&gt;You have excellent coverage of each page.&lt;/p&gt;

&lt;p&gt;But weak coverage of the transitions that connect them.&lt;/p&gt;

&lt;h2&gt;
  
  
  This changes how I think about "critical paths"
&lt;/h2&gt;

&lt;p&gt;Teams often say:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We automate our critical paths.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Usually they mean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Signup
Login
Checkout
Create project
Invite user
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are important.&lt;/p&gt;

&lt;p&gt;But they're not really paths.&lt;/p&gt;

&lt;p&gt;They're destinations.&lt;/p&gt;

&lt;p&gt;A true critical path includes how users get into and out of important states.&lt;/p&gt;

&lt;p&gt;For example, inviting a user isn't just:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Admin sends invitation
User accepts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's also:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Admin sends invitation
Invitation expires
Admin resends
User accepts old link
User accepts new link
Admin changes role
User logs in with stale session
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's where "works in QA" turns into "why does this one customer have admin access?"&lt;/p&gt;

&lt;h2&gt;
  
  
  Better coverage can mean fewer tests
&lt;/h2&gt;

&lt;p&gt;This is the counterintuitive part.&lt;/p&gt;

&lt;p&gt;If you model transitions properly, you may not need more tests.&lt;/p&gt;

&lt;p&gt;You may need better-selected tests.&lt;/p&gt;

&lt;p&gt;Instead of 30 slight variations of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Open page
Click button
Check confirmation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;you might want 10 scenarios deliberately chosen to cross the risky boundaries in your system.&lt;/p&gt;

&lt;p&gt;This is why I don't think test-count growth is necessarily a sign of a better automation program.&lt;/p&gt;

&lt;p&gt;Sometimes it means the strategy is becoming more thorough.&lt;/p&gt;

&lt;p&gt;Sometimes it means you're accumulating redundant scripts.&lt;/p&gt;

&lt;p&gt;The number itself tells you almost nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question I'd ask instead of "How much is automated?"
&lt;/h2&gt;

&lt;p&gt;I'd stop asking:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What percentage of QA is automated?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's too vague.&lt;/p&gt;

&lt;p&gt;I'd ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which important state transitions can still reach production without being exercised automatically?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a much harder question.&lt;/p&gt;

&lt;p&gt;It's also actionable.&lt;/p&gt;

&lt;p&gt;Because once you know the answer, you can decide whether each gap deserves:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a browser test&lt;/li&gt;
&lt;li&gt;an API test&lt;/li&gt;
&lt;li&gt;a contract test&lt;/li&gt;
&lt;li&gt;a component test&lt;/li&gt;
&lt;li&gt;a manual exploratory test&lt;/li&gt;
&lt;li&gt;or no test at all&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now you're designing a testing strategy.&lt;/p&gt;

&lt;p&gt;Not just collecting automation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The goal isn't maximum automation
&lt;/h2&gt;

&lt;p&gt;It's useful to remember what all of this is for.&lt;/p&gt;

&lt;p&gt;The goal isn't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10,000 tests
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The goal is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Fewer important surprises in production
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are not the same thing.&lt;/p&gt;

&lt;p&gt;You can have a massive automation suite that repeatedly confirms the obvious.&lt;/p&gt;

&lt;p&gt;Or a smaller one that aggressively probes the transitions where your system is most likely to break.&lt;/p&gt;

&lt;p&gt;If I had to choose, I'd take the second one.&lt;/p&gt;

&lt;p&gt;So before you add another 200 tests this quarter, try something simpler.&lt;/p&gt;

&lt;p&gt;Draw the important states in your product.&lt;/p&gt;

&lt;p&gt;Draw the transitions between them.&lt;/p&gt;

&lt;p&gt;Then put your current automated coverage on top.&lt;/p&gt;

&lt;p&gt;You might discover that the gaps aren't where you thought they were.&lt;/p&gt;

&lt;p&gt;And you might also discover that your impressive test count has been measuring the wrong thing.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>qa</category>
      <category>automation</category>
      <category>devops</category>
    </item>
    <item>
      <title>Most Flaky Tests Aren’t Flaky. Your Test Environment Is Lying to You.</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Fri, 02 Oct 2026 17:59:50 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/most-flaky-tests-arent-flaky-your-test-environment-is-lying-to-you-3af9</link>
      <guid>https://dev.to/randomsquirrel802/most-flaky-tests-arent-flaky-your-test-environment-is-lying-to-you-3af9</guid>
      <description>&lt;p&gt;We use the word &lt;em&gt;flaky&lt;/em&gt; way too casually in test automation.&lt;/p&gt;

&lt;p&gt;A test passes nine times.&lt;/p&gt;

&lt;p&gt;Then it fails.&lt;/p&gt;

&lt;p&gt;Someone reruns it.&lt;/p&gt;

&lt;p&gt;It passes.&lt;/p&gt;

&lt;p&gt;And the verdict is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Eh. Flaky test."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ticket closed.&lt;/p&gt;

&lt;p&gt;Or worse, somebody adds another two-second wait and commits it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;waitForTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Excellent.&lt;/p&gt;

&lt;p&gt;Software engineering.&lt;/p&gt;

&lt;p&gt;The uncomfortable reality is that a lot of supposedly flaky tests aren't random at all.&lt;/p&gt;

&lt;p&gt;They're deterministic failures caused by conditions we forgot to control.&lt;/p&gt;

&lt;p&gt;Browser state.&lt;/p&gt;

&lt;p&gt;Cache state.&lt;/p&gt;

&lt;p&gt;Session state.&lt;/p&gt;

&lt;p&gt;Network timing.&lt;/p&gt;

&lt;p&gt;Webhooks.&lt;/p&gt;

&lt;p&gt;Timezone.&lt;/p&gt;

&lt;p&gt;Locale.&lt;/p&gt;

&lt;p&gt;Scroll position.&lt;/p&gt;

&lt;p&gt;Service workers.&lt;/p&gt;

&lt;p&gt;And increasingly, AI-generated changes to tests that nobody has really inspected.&lt;/p&gt;

&lt;p&gt;The browser knows exactly why the test failed.&lt;/p&gt;

&lt;p&gt;We just didn't collect enough information to figure it out.&lt;/p&gt;

&lt;h2&gt;
  
  
  State is probably guilty until proven innocent
&lt;/h2&gt;

&lt;p&gt;Here's a test automation debugging rule I've developed over the years:&lt;/p&gt;

&lt;p&gt;If something passes locally, fails in CI, passes when retried, and nobody can explain why...&lt;/p&gt;

&lt;p&gt;Look at state.&lt;/p&gt;

&lt;p&gt;Not selectors.&lt;/p&gt;

&lt;p&gt;State.&lt;/p&gt;

&lt;p&gt;Browser automation is basically one giant experiment where we keep trying to pretend every execution starts from the same universe.&lt;/p&gt;

&lt;p&gt;It usually doesn't.&lt;/p&gt;

&lt;p&gt;Consider something as ordinary as authentication.&lt;/p&gt;

&lt;p&gt;A browser context may contain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;cookies&lt;/li&gt;
&lt;li&gt;local storage&lt;/li&gt;
&lt;li&gt;session storage&lt;/li&gt;
&lt;li&gt;IndexedDB&lt;/li&gt;
&lt;li&gt;cached assets&lt;/li&gt;
&lt;li&gt;service worker data&lt;/li&gt;
&lt;li&gt;authentication tokens&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Change one of those and you can get a completely different application.&lt;/p&gt;

&lt;p&gt;This gets especially interesting with Playwright because browser contexts make isolation convenient enough that it's easy to assume isolation is perfect.&lt;/p&gt;

&lt;p&gt;It isn't always.&lt;/p&gt;

&lt;p&gt;This guide on &lt;a href="https://thesdet.com/how-to-debug-playwright-tests-that-fail-only-after-browser-context-storage-changes/" rel="noopener noreferrer"&gt;debugging Playwright tests that fail after browser context storage changes&lt;/a&gt; covers exactly this class of problem.&lt;/p&gt;

&lt;p&gt;And it's worth understanding because the failure usually doesn't look like a storage failure.&lt;/p&gt;

&lt;p&gt;It looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Expected element to be visible.
Element was not found.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The selector wasn't necessarily wrong.&lt;/p&gt;

&lt;p&gt;The user may simply have landed in a different application state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Service workers make this even more fun
&lt;/h2&gt;

&lt;p&gt;If you really want to spend an afternoon questioning your career choices, debug a cache-related failure involving a service worker.&lt;/p&gt;

&lt;p&gt;You can have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
   ↓
Service Worker
   ↓
Cache
   ↓
Network
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which means what your automation sees isn't necessarily what your server just deployed.&lt;/p&gt;

&lt;p&gt;Maybe the new JavaScript bundle exists.&lt;/p&gt;

&lt;p&gt;Maybe the browser is still using the old one.&lt;/p&gt;

&lt;p&gt;Maybe the service worker has updated but isn't controlling the current page.&lt;/p&gt;

&lt;p&gt;Maybe the cache invalidation worked on Chrome 1 but not Chrome 2.&lt;/p&gt;

&lt;p&gt;Maybe your "offline" test isn't actually offline because something was already cached.&lt;/p&gt;

&lt;p&gt;These are all legitimate application behaviors.&lt;/p&gt;

&lt;p&gt;They're also easy to mislabel as automation flakiness.&lt;/p&gt;

&lt;p&gt;There's a useful walkthrough on &lt;a href="https://vibiumlabs.com/how-to-test-service-worker-updates-cache-invalidation-and-offline-fallbacks-without-flaky-browser-state/" rel="noopener noreferrer"&gt;testing service worker updates, cache invalidation, and offline fallbacks without carrying flaky browser state between runs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This is exactly the kind of scenario I'd include when evaluating a browser automation platform.&lt;/p&gt;

&lt;p&gt;Not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Can it click the Login button?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Everything can click the Login button.&lt;/p&gt;

&lt;p&gt;Until a sticky banner covers it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Speaking of things covering buttons...
&lt;/h2&gt;

&lt;p&gt;Modern browser UIs are surprisingly hostile to automation.&lt;/p&gt;

&lt;p&gt;Sticky headers.&lt;/p&gt;

&lt;p&gt;Floating chat widgets.&lt;/p&gt;

&lt;p&gt;Cookie banners.&lt;/p&gt;

&lt;p&gt;Animated navigation.&lt;/p&gt;

&lt;p&gt;Lazy-loaded content.&lt;/p&gt;

&lt;p&gt;Infinite scroll.&lt;/p&gt;

&lt;p&gt;Responsive layouts.&lt;/p&gt;

&lt;p&gt;A perfectly valid element can exist in the DOM and still be impossible to click.&lt;/p&gt;

&lt;p&gt;You might have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Target element:
x = 420
y = 87
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Unfortunately, your sticky navigation bar occupies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;y = 0 → 96
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your automation framework sees the button.&lt;/p&gt;

&lt;p&gt;The user visually sees the button.&lt;/p&gt;

&lt;p&gt;The click lands somewhere else.&lt;/p&gt;

&lt;p&gt;Then somebody "fixes" the problem with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;#save&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;click&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;force&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which is occasionally correct.&lt;/p&gt;

&lt;p&gt;And occasionally just hides a real user-facing problem.&lt;/p&gt;

&lt;p&gt;There's a practical guide to &lt;a href="https://playwright-vs-selenium.com/how-to-test-sticky-headers-scroll-jank-and-element-occlusion-without-flaky-click-assertions/" rel="noopener noreferrer"&gt;testing sticky headers, scroll jank, and element occlusion without creating flaky click assertions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The important distinction is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"The automation couldn't click it"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"A user couldn't reliably click it"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;are not always the same bug.&lt;/p&gt;

&lt;p&gt;Good testing infrastructure should help you distinguish between them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sleep is not synchronization
&lt;/h2&gt;

&lt;p&gt;Another common source of "flakiness":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This means:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"I have absolutely no idea when the thing will happen, but five seconds feels emotionally safe."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Sometimes five seconds is enough.&lt;/p&gt;

&lt;p&gt;Sometimes the event takes six.&lt;/p&gt;

&lt;p&gt;Congratulations.&lt;/p&gt;

&lt;p&gt;You now have a flaky test.&lt;/p&gt;

&lt;p&gt;Webhooks are a perfect example.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser action
     ↓
Backend request
     ↓
Payment provider
     ↓
Webhook
     ↓
Worker
     ↓
Database update
     ↓
UI refresh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;How long does that take?&lt;/p&gt;

&lt;p&gt;The wrong answer is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Probably three seconds.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The better answer is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Wait for evidence that the expected event happened.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That could mean polling an API, waiting for a database condition through a test helper, checking an event endpoint, or verifying a known downstream effect.&lt;/p&gt;

&lt;p&gt;There's a good breakdown of &lt;a href="https://test-automation-experts.com/how-to-test-webhooks-in-a-qa-environment-without-relying-on-fragile-sleep-based-assertions/" rel="noopener noreferrer"&gt;testing webhooks in QA without relying on fragile sleep-based assertions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This sounds obvious when written out.&lt;/p&gt;

&lt;p&gt;And yet I still regularly see mature automation suites containing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Wait 5 seconds
Wait 10 seconds
Wait 3 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At some point you're not testing the application anymore.&lt;/p&gt;

&lt;p&gt;You're testing whether your timing guesses were lucky.&lt;/p&gt;

&lt;h2&gt;
  
  
  Time itself is state
&lt;/h2&gt;

&lt;p&gt;Here's another fun one.&lt;/p&gt;

&lt;p&gt;A test passes in Bucharest.&lt;/p&gt;

&lt;p&gt;It fails on a cloud browser in Virginia.&lt;/p&gt;

&lt;p&gt;The page says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;October 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The assertion expects:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;October 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both can be correct.&lt;/p&gt;

&lt;p&gt;Welcome to timezones.&lt;/p&gt;

&lt;p&gt;International applications introduce several hidden inputs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;browser timezone&lt;/li&gt;
&lt;li&gt;server timezone&lt;/li&gt;
&lt;li&gt;operating system locale&lt;/li&gt;
&lt;li&gt;browser language&lt;/li&gt;
&lt;li&gt;daylight saving rules&lt;/li&gt;
&lt;li&gt;currency formatting&lt;/li&gt;
&lt;li&gt;decimal separators&lt;/li&gt;
&lt;li&gt;12-hour vs 24-hour time&lt;/li&gt;
&lt;li&gt;date ordering&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Even a simple price can become:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$1,234.56
€1.234,56
1 234,56 €
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;depending on context.&lt;/p&gt;

&lt;p&gt;Hardcoding rendered strings is a wonderful way to create tests that pass everywhere except where your users live.&lt;/p&gt;

&lt;p&gt;This guide on &lt;a href="https://testingtoolguide.com/how-to-test-time-zone-locale-and-currency-rendering-in-browser-automation-without-flaky-assertions/" rel="noopener noreferrer"&gt;testing timezone, locale, and currency rendering without flaky browser assertions&lt;/a&gt; gets into these problems in more detail.&lt;/p&gt;

&lt;p&gt;Locale testing is one of those things teams postpone until they acquire international customers.&lt;/p&gt;

&lt;p&gt;Then suddenly it's a release blocker.&lt;/p&gt;

&lt;h2&gt;
  
  
  Role-based applications multiply the problem
&lt;/h2&gt;

&lt;p&gt;Now take everything above and add permissions.&lt;/p&gt;

&lt;p&gt;Imagine an admin platform with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Super Admin
Admin
Manager
Editor
Viewer
External Collaborator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each role may see a different navigation.&lt;/p&gt;

&lt;p&gt;Different buttons.&lt;/p&gt;

&lt;p&gt;Different API responses.&lt;/p&gt;

&lt;p&gt;Different data.&lt;/p&gt;

&lt;p&gt;Different redirects.&lt;/p&gt;

&lt;p&gt;Different session rules.&lt;/p&gt;

&lt;p&gt;And now you've multiplied your state space significantly.&lt;/p&gt;

&lt;p&gt;Testing these systems isn't just about whether your automation tool supports Chrome.&lt;/p&gt;

&lt;p&gt;It's about whether you can reliably create isolated sessions and understand failures when one role behaves differently from another.&lt;/p&gt;

&lt;p&gt;This breakdown of &lt;a href="https://frontendtester.com/what-to-look-for-in-browser-testing-platforms-for-role-based-admin-panels-session-state-and-debuggable-failures/" rel="noopener noreferrer"&gt;browser testing platforms for role-based admin panels, session state, and debuggable failures&lt;/a&gt; highlights some useful criteria.&lt;/p&gt;

&lt;p&gt;For complex business applications, I'd care a lot about this.&lt;/p&gt;

&lt;p&gt;A demo recording a checkout flow tells you very little about how the platform behaves with six permission levels and stale sessions.&lt;/p&gt;

&lt;h2&gt;
  
  
  And after deployment, you're still testing
&lt;/h2&gt;

&lt;p&gt;There's another mistake I see fairly often.&lt;/p&gt;

&lt;p&gt;Teams think:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CI passed → deployment successful
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are different statements.&lt;/p&gt;

&lt;p&gt;CI verifies one environment.&lt;/p&gt;

&lt;p&gt;Your users interact with another.&lt;/p&gt;

&lt;p&gt;After deployment you can still encounter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;bad environment variables&lt;/li&gt;
&lt;li&gt;broken CDN assets&lt;/li&gt;
&lt;li&gt;DNS issues&lt;/li&gt;
&lt;li&gt;expired credentials&lt;/li&gt;
&lt;li&gt;missing migrations&lt;/li&gt;
&lt;li&gt;third-party outages&lt;/li&gt;
&lt;li&gt;routing failures&lt;/li&gt;
&lt;li&gt;region-specific problems&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which is why synthetic monitoring is basically test automation that refuses to go home after deployment.&lt;/p&gt;

&lt;p&gt;A useful synthetic check should do more than scream:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CHECKOUT FAILED
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You want to know &lt;em&gt;where&lt;/em&gt; it failed and whether rollback might actually help.&lt;/p&gt;

&lt;p&gt;There's a useful overview of &lt;a href="https://softwaretestingreviews.com/what-to-look-for-in-synthetic-monitoring-tools-for-post-deploy-smoke-checks-rollback-verification-and-api-failure-triage/" rel="noopener noreferrer"&gt;selecting synthetic monitoring for post-deploy smoke checks, rollback verification, and API failure triage&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For important flows, the lifecycle really looks more like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Build
  ↓
Automated tests
  ↓
Deploy
  ↓
Synthetic checks
  ↓
Real users
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your release doesn't become correct just because GitHub Actions turned green.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI test repair adds an interesting economic problem
&lt;/h2&gt;

&lt;p&gt;AI is now starting to "solve" flaky automation too.&lt;/p&gt;

&lt;p&gt;A locator changes.&lt;/p&gt;

&lt;p&gt;The AI repairs it.&lt;/p&gt;

&lt;p&gt;The test keeps running.&lt;/p&gt;

&lt;p&gt;Sounds great.&lt;/p&gt;

&lt;p&gt;Sometimes it is.&lt;/p&gt;

&lt;p&gt;But there's a question I think gets overlooked:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How much autonomous repair should happen before a human should review what's changing?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Imagine an agent repairs 100 tests.&lt;/p&gt;

&lt;p&gt;Maybe 97 repairs are perfect.&lt;/p&gt;

&lt;p&gt;Maybe three subtly changed what the test means.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Original:
Click "Delete User"

Repair:
Click "Disable User"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both controls might exist in roughly the same place.&lt;/p&gt;

&lt;p&gt;The test could continue successfully.&lt;/p&gt;

&lt;p&gt;And you've now created something much worse than a failed test:&lt;/p&gt;

&lt;p&gt;A passing test that's testing the wrong thing.&lt;/p&gt;

&lt;p&gt;That's why I like the framing in this &lt;a href="https://ai-test-agents.com/benchmark-plan-measuring-autonomous-test-repair-cost-against-human-review-time-in-ai-test-agents/" rel="noopener noreferrer"&gt;benchmark plan comparing autonomous test repair cost against human review time&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The metric shouldn't simply be:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How many tests did the AI repair?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It should also include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;repair cost
+
review cost
+
incorrect repair risk
+
time saved
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's a much more useful business equation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test itself is becoming an asset
&lt;/h2&gt;

&lt;p&gt;There's another question teams don't ask early enough:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What happens if we leave this testing platform?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This mattered less when nearly everything was code.&lt;/p&gt;

&lt;p&gt;A Selenium test was Python, Java, C#, JavaScript, whatever.&lt;/p&gt;

&lt;p&gt;You owned it.&lt;/p&gt;

&lt;p&gt;Now we're seeing more proprietary:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;visual workflows&lt;/li&gt;
&lt;li&gt;AI-generated models&lt;/li&gt;
&lt;li&gt;natural-language tests&lt;/li&gt;
&lt;li&gt;hosted execution formats&lt;/li&gt;
&lt;li&gt;platform-specific steps&lt;/li&gt;
&lt;li&gt;proprietary repair metadata&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some of these abstractions are valuable.&lt;/p&gt;

&lt;p&gt;But abstraction has a price.&lt;/p&gt;

&lt;p&gt;If you create 4,000 tests over three years, switching platforms becomes an entirely different conversation.&lt;/p&gt;

&lt;p&gt;That's why &lt;a href="https://testingradar.com/benchmark-plan-measuring-test-artifact-portability-across-browser-clouds-open-source-frameworks-and-ai-testing-platforms/" rel="noopener noreferrer"&gt;measuring test artifact portability across browser clouds, open-source frameworks, and AI testing platforms&lt;/a&gt; is a worthwhile benchmark.&lt;/p&gt;

&lt;p&gt;I'd want to know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Can I export my tests?&lt;/li&gt;
&lt;li&gt;In what format?&lt;/li&gt;
&lt;li&gt;Can another system execute them?&lt;/li&gt;
&lt;li&gt;Can humans understand them?&lt;/li&gt;
&lt;li&gt;What happens to assertions?&lt;/li&gt;
&lt;li&gt;What happens to screenshots and historical results?&lt;/li&gt;
&lt;li&gt;What happens to AI-generated metadata?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nobody worries about vendor lock-in when they have 20 tests.&lt;/p&gt;

&lt;p&gt;They start worrying when they have 20,000.&lt;/p&gt;

&lt;h2&gt;
  
  
  Managed QA versus self-service AI is becoming another dividing line
&lt;/h2&gt;

&lt;p&gt;We're also seeing two very different approaches emerge in AI testing.&lt;/p&gt;

&lt;p&gt;One says:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Give us responsibility for the outcome.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The other says:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Here's powerful automation. You operate it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those are fundamentally different products even if both put "AI testing" on their homepage.&lt;/p&gt;

&lt;p&gt;A managed service can remove maintenance work.&lt;/p&gt;

&lt;p&gt;A self-service platform can give your team more direct control and faster iteration.&lt;/p&gt;

&lt;p&gt;Neither model universally wins.&lt;/p&gt;

&lt;p&gt;This comparison of &lt;a href="https://aitestingreviews.com/qa-wolf-vs-qa-tech-which-ai-testing-model-fits-teams-that-need-managed-maintenance-or-faster-self-serve-browser-coverage/" rel="noopener noreferrer"&gt;QA Wolf vs QA.tech&lt;/a&gt; frames the difference around managed maintenance versus self-service browser coverage.&lt;/p&gt;

&lt;p&gt;And if you're trying to understand the broader market, this list of the &lt;a href="https://endtest.io/blog/10-best-ai-powered-test-automation-tools" rel="noopener noreferrer"&gt;10 best AI-powered test automation tools&lt;/a&gt; also gives a useful overview of how different AI testing products are approaching the problem.&lt;/p&gt;

&lt;p&gt;Personally, I think category labels like "AI testing platform" are becoming less useful.&lt;/p&gt;

&lt;p&gt;The more important questions are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Who creates the tests?
Who maintains them?
Who investigates failures?
Who owns the artifacts?
Who decides when an AI repair is correct?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those answers tell you much more about a platform than whether it has a chatbot.&lt;/p&gt;

&lt;h2&gt;
  
  
  My favorite flaky-test debugging trick
&lt;/h2&gt;

&lt;p&gt;When a test fails intermittently, don't immediately modify it.&lt;/p&gt;

&lt;p&gt;First try to classify the hidden variable.&lt;/p&gt;

&lt;p&gt;Ask:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What changed between the passing run
and the failing run?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Browser state&lt;/li&gt;
&lt;li&gt;Application state&lt;/li&gt;
&lt;li&gt;Network state&lt;/li&gt;
&lt;li&gt;Test data&lt;/li&gt;
&lt;li&gt;Browser version&lt;/li&gt;
&lt;li&gt;Viewport&lt;/li&gt;
&lt;li&gt;Locale and timezone&lt;/li&gt;
&lt;li&gt;Timing&lt;/li&gt;
&lt;li&gt;Third-party dependencies&lt;/li&gt;
&lt;li&gt;Deployment version&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then compare evidence from the runs.&lt;/p&gt;

&lt;p&gt;The difference is usually somewhere in that list.&lt;/p&gt;

&lt;p&gt;And if you don't have enough evidence to compare two executions?&lt;/p&gt;

&lt;p&gt;That's the actual problem you should fix first.&lt;/p&gt;

&lt;p&gt;Because adding:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;waitForTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;might make your dashboard greener.&lt;/p&gt;

&lt;p&gt;But it doesn't mean your test suite became more reliable.&lt;/p&gt;

&lt;p&gt;It just means you taught the failure to hide better.&lt;/p&gt;

&lt;h2&gt;
  
  
  Flakiness is information
&lt;/h2&gt;

&lt;p&gt;This is probably the biggest mental shift.&lt;/p&gt;

&lt;p&gt;A flaky test isn't merely annoying noise.&lt;/p&gt;

&lt;p&gt;It's information.&lt;/p&gt;

&lt;p&gt;Sometimes it tells you your test is poorly synchronized.&lt;/p&gt;

&lt;p&gt;Sometimes it tells you your application behaves unpredictably.&lt;/p&gt;

&lt;p&gt;Sometimes it tells you environments aren't isolated.&lt;/p&gt;

&lt;p&gt;Sometimes it tells you your infrastructure behaves differently under load.&lt;/p&gt;

&lt;p&gt;Sometimes it tells you that you've accidentally discovered a real race condition.&lt;/p&gt;

&lt;p&gt;So before deleting the test, increasing the timeout, or clicking &lt;strong&gt;Retry&lt;/strong&gt; for the sixth time, ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What condition makes this failure possible?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Because computers aren't usually rolling dice behind your back.&lt;/p&gt;

&lt;p&gt;Something changed.&lt;/p&gt;

&lt;p&gt;The job of good test automation isn't just to detect the failure.&lt;/p&gt;

&lt;p&gt;It's to leave enough evidence behind that you can figure out &lt;strong&gt;what changed&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>qa</category>
      <category>automation</category>
      <category>playwright</category>
    </item>
    <item>
      <title>Most Test Failures Aren’t Really About the Test</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Tue, 25 Aug 2026 07:53:51 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/most-test-failures-arent-really-about-the-test-40d9</link>
      <guid>https://dev.to/randomsquirrel802/most-test-failures-arent-really-about-the-test-40d9</guid>
      <description>&lt;p&gt;One of the weird things about test automation is that a failure rarely means just one thing.&lt;/p&gt;

&lt;p&gt;A red test can mean:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the product is broken&lt;/li&gt;
&lt;li&gt;the test is broken&lt;/li&gt;
&lt;li&gt;the environment is different&lt;/li&gt;
&lt;li&gt;the timing changed&lt;/li&gt;
&lt;li&gt;some stale state leaked in&lt;/li&gt;
&lt;li&gt;the browser behaved differently&lt;/li&gt;
&lt;li&gt;or the test uncovered something nobody had thought about yet&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last category is usually the interesting one.&lt;/p&gt;

&lt;p&gt;For example, &lt;a href="https://thesdet.com/how-to-debug-playwright-tests-that-fail-only-when-microtasks-timers-and-api-mocks-interleave-differently-in-ci/" rel="noopener noreferrer"&gt;Playwright tests that fail only when microtasks, timers, and API mocks interleave differently in CI&lt;/a&gt; are a good reminder that “works locally” is not much of a debugging strategy.&lt;/p&gt;

&lt;p&gt;Accessibility testing has the same problem. Finding an issue is only step one. &lt;a href="https://ai-testing-tools.com/what-to-look-for-in-ai-accessibility-testing-tools-for-dynamic-ui-changes-severity-triage-and-release-gates/" rel="noopener noreferrer"&gt;Evaluating AI accessibility testing tools&lt;/a&gt; also means looking at severity, triage, dynamic UI changes, and whether the results can actually influence a release decision.&lt;/p&gt;

&lt;p&gt;Then there’s infrastructure.&lt;/p&gt;

&lt;p&gt;Sometimes you don’t need another framework. You just need the right place to reproduce the problem. This &lt;a href="https://playwright-vs-selenium.com/bitbar-vs-browserling-which-browser-cloud-fits-fast-manual-checks-lightweight-automation-and-cross-device-debugging-better/" rel="noopener noreferrer"&gt;BitBar vs Browserling comparison&lt;/a&gt; is useful because different browser clouds optimize for very different workflows.&lt;/p&gt;

&lt;p&gt;And testing shouldn’t stop when deployment finishes.&lt;/p&gt;

&lt;p&gt;This &lt;a href="https://testingradar.com/market-map-synthetic-monitoring-release-signals/" rel="noopener noreferrer"&gt;market map of synthetic monitoring and post-deploy browser smoke platforms&lt;/a&gt; looks at the tools teams can use when they need a fast answer to a very simple question:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Did we just break production?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Even browser state deserves more attention than it gets. &lt;a href="https://web-developer-reviews.com/how-to-test-indexeddb-local-storage-and-session-resets-without-creating-state-that-hides-real-bugs/" rel="noopener noreferrer"&gt;Testing IndexedDB, local storage, and session resets&lt;/a&gt; sounds mundane until you realize bad cleanup can make your tests &lt;em&gt;less&lt;/em&gt; realistic.&lt;/p&gt;

&lt;p&gt;The common thread here is simple:&lt;/p&gt;

&lt;p&gt;Reliable testing is rarely about adding more assertions.&lt;/p&gt;

&lt;p&gt;It’s about understanding everything surrounding the assertion — timing, state, environment, infrastructure, and what happens after deployment.&lt;/p&gt;

&lt;p&gt;That’s where most of the interesting bugs are hiding.&lt;/p&gt;

</description>
      <category>qa</category>
      <category>testing</category>
      <category>automation</category>
      <category>devops</category>
    </item>
    <item>
      <title>The Best Test Automation Tool Is Usually the One That Matches Your Mess</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Wed, 19 Aug 2026 22:00:42 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/the-best-test-automation-tool-is-usually-the-one-that-matches-your-mess-54hb</link>
      <guid>https://dev.to/randomsquirrel802/the-best-test-automation-tool-is-usually-the-one-that-matches-your-mess-54hb</guid>
      <description>&lt;p&gt;Most test automation buying decisions start with a spreadsheet.&lt;/p&gt;

&lt;p&gt;Browser testing? Check.&lt;/p&gt;

&lt;p&gt;Mobile? Check.&lt;/p&gt;

&lt;p&gt;API testing? Check.&lt;/p&gt;

&lt;p&gt;AI? Check.&lt;/p&gt;

&lt;p&gt;CI integration? Check.&lt;/p&gt;

&lt;p&gt;Then somebody adds 40 more rows and the team spends three weeks comparing products that all appear to do roughly the same thing.&lt;/p&gt;

&lt;p&gt;This is usually where the evaluation goes wrong.&lt;/p&gt;

&lt;p&gt;The important question isn't:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which tool has the most capabilities?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It's:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Where does our testing complexity actually live?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Because every team has a different kind of mess.&lt;/p&gt;

&lt;p&gt;One company has a browser-first SaaS product and wants to stop maintaining a large Playwright framework.&lt;/p&gt;

&lt;p&gt;Another has SAP, Windows desktop applications, APIs, and a mobile app that has been around for eight years.&lt;/p&gt;

&lt;p&gt;Another has a modern web product, but half of its important workflows happen inside Stripe, embedded support widgets, rich-text editors, and third-party iframes.&lt;/p&gt;

&lt;p&gt;Those teams should probably not buy the same thing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the surface area you actually have
&lt;/h2&gt;

&lt;p&gt;A surprisingly common mistake is buying for hypothetical complexity.&lt;/p&gt;

&lt;p&gt;"We might need desktop testing someday."&lt;/p&gt;

&lt;p&gt;"We could have a native mobile application next year."&lt;/p&gt;

&lt;p&gt;"We should probably support every possible protocol."&lt;/p&gt;

&lt;p&gt;Maybe.&lt;/p&gt;

&lt;p&gt;But you're paying for those decisions now.&lt;/p&gt;

&lt;p&gt;If 90% of your regression work is browser-based, it makes sense to evaluate tools primarily around browser workflows, maintenance, debugging, parallel execution, and how quickly the rest of the team can work with the tests.&lt;/p&gt;

&lt;p&gt;That's why comparisons such as &lt;a href="https://test-automation-tools.com/endtest-vs-keysight-eggplant-which-no-code-testing-platform-fits-browser-first-teams-better/" rel="noopener noreferrer"&gt;Endtest vs Keysight Eggplant for browser-first teams&lt;/a&gt; are more useful when you read them through the lens of your actual environment rather than trying to declare a universal winner.&lt;/p&gt;

&lt;p&gt;A browser-first SaaS team and a large enterprise testing packaged desktop software have different problems.&lt;/p&gt;

&lt;p&gt;And that distinction matters.&lt;/p&gt;

&lt;p&gt;On the other end of the spectrum, there are organizations where breadth really is the requirement.&lt;/p&gt;

&lt;p&gt;If a company needs to test desktop applications, APIs, mobile apps, and legacy enterprise workflows from one stack, something like &lt;a href="https://softwaretestingreviews.com/opentext-uft-one-review-when-enterprise-teams-need-desktop-api-and-mobile-test-coverage-in-one-stack/" rel="noopener noreferrer"&gt;OpenText UFT One&lt;/a&gt; may make sense in a way that a browser-focused platform doesn't.&lt;/p&gt;

&lt;p&gt;The lesson isn't that broad tools are bad or focused tools are better.&lt;/p&gt;

&lt;p&gt;It's that &lt;strong&gt;unused flexibility is still complexity&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;You maintain it.&lt;/p&gt;

&lt;p&gt;You train people on it.&lt;/p&gt;

&lt;p&gt;You pay for it.&lt;/p&gt;

&lt;p&gt;And sometimes you design your entire QA process around capabilities you barely use.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hard parts are usually at the edges
&lt;/h2&gt;

&lt;p&gt;The happy-path demo is almost never where a testing tool struggles.&lt;/p&gt;

&lt;p&gt;Login.&lt;/p&gt;

&lt;p&gt;Click button.&lt;/p&gt;

&lt;p&gt;Fill form.&lt;/p&gt;

&lt;p&gt;Assert text.&lt;/p&gt;

&lt;p&gt;Every serious browser testing product can show you a nice version of that.&lt;/p&gt;

&lt;p&gt;The better evaluation is to pick the ugly workflows.&lt;/p&gt;

&lt;p&gt;The ones that caused three bugs last quarter.&lt;/p&gt;

&lt;p&gt;The ones people quietly exclude from regression because automation became annoying.&lt;/p&gt;

&lt;p&gt;For example: embedded content.&lt;/p&gt;

&lt;p&gt;Modern web applications increasingly contain payment providers, chat widgets, video players, authentication components, analytics tools, and other functionality hosted outside the application's origin.&lt;/p&gt;

&lt;p&gt;That means cross-origin boundaries, sandbox restrictions, iframe switching, timing issues, and behavior that your application doesn't fully control.&lt;/p&gt;

&lt;p&gt;If that describes your product, you should deliberately test &lt;a href="https://testproject.to/how-to-test-cross-origin-iframes-embedded-widgets-and-third-party-sandboxes-without-brittle-browser-tests/" rel="noopener noreferrer"&gt;cross-origin iframes, embedded widgets, and third-party sandboxes&lt;/a&gt; during a proof of concept.&lt;/p&gt;

&lt;p&gt;Don't ask the vendor whether they support iframes.&lt;/p&gt;

&lt;p&gt;Give the product your worst iframe.&lt;/p&gt;

&lt;p&gt;That's a much better test.&lt;/p&gt;

&lt;p&gt;The same applies to rich-text editing.&lt;/p&gt;

&lt;p&gt;A contenteditable field looks like a text box until you automate it.&lt;/p&gt;

&lt;p&gt;Then users paste formatted HTML from Google Docs, the browser normalizes whitespace, the editor inserts spans, keyboard shortcuts behave differently, sanitization removes markup, and undo history depends on how the content was inserted.&lt;/p&gt;

&lt;p&gt;A test that compares the entire resulting DOM can become brittle almost immediately.&lt;/p&gt;

&lt;p&gt;A better approach is to understand &lt;a href="https://frontendtester.com/how-to-test-contenteditable-editors-paste-sanitization-and-undo-behavior-without-brittle-browser-assertions/" rel="noopener noreferrer"&gt;how to test contenteditable editors, paste sanitization, and undo behavior&lt;/a&gt; and then see how naturally the tool supports those patterns.&lt;/p&gt;

&lt;p&gt;Again, don't test the brochure.&lt;/p&gt;

&lt;p&gt;Test your mess.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI makes the demo problem worse
&lt;/h2&gt;

&lt;p&gt;AI has made test automation demos much more impressive.&lt;/p&gt;

&lt;p&gt;You type:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Test the checkout flow.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Twenty seconds later, something is clicking through the application.&lt;/p&gt;

&lt;p&gt;That's legitimately useful.&lt;/p&gt;

&lt;p&gt;But it can also hide the most important part of the evaluation.&lt;/p&gt;

&lt;p&gt;The first run isn't where AI testing gets expensive.&lt;/p&gt;

&lt;p&gt;The interesting part is run 47.&lt;/p&gt;

&lt;p&gt;The UI changed.&lt;/p&gt;

&lt;p&gt;A selector stopped working.&lt;/p&gt;

&lt;p&gt;A modal appears only for some accounts.&lt;/p&gt;

&lt;p&gt;The test gets redirected.&lt;/p&gt;

&lt;p&gt;A button moved.&lt;/p&gt;

&lt;p&gt;A network request takes four seconds instead of one.&lt;/p&gt;

&lt;p&gt;Now what?&lt;/p&gt;

&lt;p&gt;Does the system recover consistently?&lt;/p&gt;

&lt;p&gt;Does it make the same decision twice?&lt;/p&gt;

&lt;p&gt;Can you see what it changed?&lt;/p&gt;

&lt;p&gt;Can a human override it?&lt;/p&gt;

&lt;p&gt;Does recovery fix the actual problem, or merely find &lt;em&gt;some&lt;/em&gt; path that gets the test to green?&lt;/p&gt;

&lt;p&gt;These questions are why I like the idea of using an explicit &lt;a href="https://vibiumlabs.com/benchmark-plan-how-to-compare-ai-browser-test-tools-on-repeatability-recovery-and-maintenance-cost/" rel="noopener noreferrer"&gt;benchmark plan for comparing AI browser test tools on repeatability, recovery, and maintenance cost&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A useful benchmark doesn't ask each tool to create five tests and then award points for speed.&lt;/p&gt;

&lt;p&gt;It should run those tests repeatedly.&lt;/p&gt;

&lt;p&gt;Then intentionally break things.&lt;/p&gt;

&lt;p&gt;Change labels.&lt;/p&gt;

&lt;p&gt;Move elements.&lt;/p&gt;

&lt;p&gt;Introduce latency.&lt;/p&gt;

&lt;p&gt;Modify test data.&lt;/p&gt;

&lt;p&gt;Add an intermediate dialog.&lt;/p&gt;

&lt;p&gt;Run the same recovery scenario multiple times.&lt;/p&gt;

&lt;p&gt;What you want to learn is not whether the AI can solve a problem once.&lt;/p&gt;

&lt;p&gt;It's whether the behavior is &lt;strong&gt;predictable enough to trust&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure maintenance, not magic
&lt;/h2&gt;

&lt;p&gt;There is a simple metric that rarely appears in software evaluations:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many human minutes did this test require over the last 90 days?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not execution minutes.&lt;/p&gt;

&lt;p&gt;Human minutes.&lt;/p&gt;

&lt;p&gt;Time spent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;updating selectors,&lt;/li&gt;
&lt;li&gt;diagnosing failures,&lt;/li&gt;
&lt;li&gt;rerunning pipelines,&lt;/li&gt;
&lt;li&gt;fixing test data,&lt;/li&gt;
&lt;li&gt;maintaining infrastructure,&lt;/li&gt;
&lt;li&gt;reviewing AI-generated changes,&lt;/li&gt;
&lt;li&gt;explaining the framework to new team members.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That number tells you far more about the true cost of automation than the license price.&lt;/p&gt;

&lt;p&gt;A "free" framework can be expensive.&lt;/p&gt;

&lt;p&gt;An enterprise platform can also be expensive.&lt;/p&gt;

&lt;p&gt;A no-code tool can become expensive if every unusual workflow requires vendor support.&lt;/p&gt;

&lt;p&gt;An AI tool can become expensive if nobody trusts its recovery decisions and every healed test has to be manually reviewed.&lt;/p&gt;

&lt;p&gt;This is why tool comparisons should include operational cost, not just feature coverage.&lt;/p&gt;

&lt;p&gt;The cheapest tool is often the one that creates the least recurring work for your particular team.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run a proof of concept that tries to make the tool fail
&lt;/h2&gt;

&lt;p&gt;Most proofs of concept are too polite.&lt;/p&gt;

&lt;p&gt;The vendor helps configure the environment.&lt;/p&gt;

&lt;p&gt;The team selects five clean workflows.&lt;/p&gt;

&lt;p&gt;Everybody watches them pass.&lt;/p&gt;

&lt;p&gt;Then the contract gets signed.&lt;/p&gt;

&lt;p&gt;Six months later, the real application shows up.&lt;/p&gt;

&lt;p&gt;Instead, I'd build a small evaluation suite containing the nastiest representative workflows you have.&lt;/p&gt;

&lt;p&gt;Maybe:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A straightforward login and checkout flow.&lt;/li&gt;
&lt;li&gt;A cross-origin iframe.&lt;/li&gt;
&lt;li&gt;A file upload or download.&lt;/li&gt;
&lt;li&gt;A contenteditable component.&lt;/li&gt;
&lt;li&gt;A workflow with changing test data.&lt;/li&gt;
&lt;li&gt;A test with deliberate latency.&lt;/li&gt;
&lt;li&gt;A scenario where a locator is intentionally broken.&lt;/li&gt;
&lt;li&gt;A workflow that fails halfway through and needs useful debugging evidence.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Run them repeatedly.&lt;/p&gt;

&lt;p&gt;Then ask your QA engineers and developers a much simpler question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Which system would you rather debug at 4:45 PM on a Friday?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's probably more useful than another 80-row feature matrix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Buy for the next two years, not the next ten
&lt;/h2&gt;

&lt;p&gt;You should absolutely think about growth.&lt;/p&gt;

&lt;p&gt;But there's a difference between planning ahead and buying an aircraft carrier because one day you might need to cross a lake.&lt;/p&gt;

&lt;p&gt;If you're primarily building a web application, optimize for excellent browser testing.&lt;/p&gt;

&lt;p&gt;If you're running a heterogeneous enterprise stack with desktop, mobile, and APIs, breadth may be worth the additional complexity.&lt;/p&gt;

&lt;p&gt;If your UI depends heavily on third-party components, evaluate those specifically.&lt;/p&gt;

&lt;p&gt;If AI maintenance is a major selling point, benchmark the maintenance rather than the test generation.&lt;/p&gt;

&lt;p&gt;The best test automation platform isn't the one with the longest feature page.&lt;/p&gt;

&lt;p&gt;It's the one whose strengths overlap with the things that make your application difficult to test.&lt;/p&gt;

&lt;p&gt;Everything else is inventory.&lt;/p&gt;

&lt;p&gt;And in software, inventory has carrying costs.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>qa</category>
      <category>automation</category>
      <category>ai</category>
    </item>
    <item>
      <title>Most “Flaky Tests” Are Really Architecture Tests</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Sat, 08 Aug 2026 21:26:37 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/most-flaky-tests-are-really-architecture-tests-34na</link>
      <guid>https://dev.to/randomsquirrel802/most-flaky-tests-are-really-architecture-tests-34na</guid>
      <description>&lt;p&gt;There’s a point in almost every browser automation project where someone says:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“The tests are getting flaky.”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And then the team starts fixing the tests.&lt;/p&gt;

&lt;p&gt;Longer waits. More retries. New selectors. Another helper function. Maybe a wrapper around the wrapper around &lt;code&gt;click()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Sometimes that’s the right answer.&lt;/p&gt;

&lt;p&gt;But increasingly, I think “flaky test” is becoming a catch-all phrase for something much broader:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Modern frontends have become surprisingly difficult environments to observe deterministically.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The test isn’t always the problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The DOM Isn’t Really the DOM Anymore
&lt;/h2&gt;

&lt;p&gt;A decade ago, browser automation had a fairly straightforward mental model.&lt;/p&gt;

&lt;p&gt;Load page. Find element. Click element.&lt;/p&gt;

&lt;p&gt;Today you might be dealing with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Shadow DOM&lt;/li&gt;
&lt;li&gt;nested iframes&lt;/li&gt;
&lt;li&gt;third-party widgets&lt;/li&gt;
&lt;li&gt;microfrontends&lt;/li&gt;
&lt;li&gt;virtualized lists&lt;/li&gt;
&lt;li&gt;components that render differently based on their container size&lt;/li&gt;
&lt;li&gt;UI updates arriving over WebSockets&lt;/li&gt;
&lt;li&gt;client-side navigation without full page loads&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That’s a different problem.&lt;/p&gt;

&lt;p&gt;For example, &lt;a href="https://softwaretestingreviews.com/what-to-look-for-in-a-browser-testing-tool-for-shadow-dom-nested-iframes-and-embedded-widgets/" rel="noopener noreferrer"&gt;testing Shadow DOM, nested iframes, and embedded widgets&lt;/a&gt; is less about writing the perfect selector and more about whether your automation system understands the boundaries between those different contexts.&lt;/p&gt;

&lt;p&gt;A perfectly valid selector can still be useless if your test is searching in the wrong DOM context.&lt;/p&gt;

&lt;h2&gt;
  
  
  Responsive Testing Got More Complicated Too
&lt;/h2&gt;

&lt;p&gt;We used to think about responsiveness in terms of viewport width.&lt;/p&gt;

&lt;p&gt;Desktop. Tablet. Mobile.&lt;/p&gt;

&lt;p&gt;CSS container queries changed that assumption.&lt;/p&gt;

&lt;p&gt;A component can now change dramatically while the browser window stays exactly the same size.&lt;/p&gt;

&lt;p&gt;That means testing only a handful of viewport resolutions can miss real regressions. There’s a useful breakdown of &lt;a href="https://testproject.to/how-to-test-css-container-queries-resize-behavior-and-breakpoint-edge-cases-in-browser-automation/" rel="noopener noreferrer"&gt;testing container queries, resize behavior, and breakpoint edge cases&lt;/a&gt; that illustrates how these failures tend to appear.&lt;/p&gt;

&lt;p&gt;This is one of those changes that looks small from the CSS side and surprisingly large from the QA side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then There’s Time
&lt;/h2&gt;

&lt;p&gt;A lot of UI state isn’t created by page load anymore.&lt;/p&gt;

&lt;p&gt;It arrives later.&lt;/p&gt;

&lt;p&gt;Consider an application consuming WebSockets or Server-Sent Events.&lt;/p&gt;

&lt;p&gt;The test opens the page.&lt;/p&gt;

&lt;p&gt;The page says “Connected.”&lt;/p&gt;

&lt;p&gt;The test waits for the transaction.&lt;/p&gt;

&lt;p&gt;Nothing happens.&lt;/p&gt;

&lt;p&gt;Failure.&lt;/p&gt;

&lt;p&gt;Run it again.&lt;/p&gt;

&lt;p&gt;Pass.&lt;/p&gt;

&lt;p&gt;This is where arbitrary sleeps become dangerous. You’re trying to fix a state synchronization problem with a stopwatch.&lt;/p&gt;

&lt;p&gt;A better approach is understanding &lt;a href="https://vibiumlabs.com/how-to-test-websocket-sse-and-live-data-feeds-without-chasing-phantom-browser-failures/" rel="noopener noreferrer"&gt;how WebSocket, SSE, and live-data browser tests fail&lt;/a&gt; and synchronizing against meaningful application state instead of elapsed time.&lt;/p&gt;

&lt;p&gt;The same principle applies to SPAs.&lt;/p&gt;

&lt;p&gt;Browser history, back-button behavior, restored scroll positions, and cached component state can produce bugs that don’t appear when every automated test starts from a clean URL.&lt;/p&gt;

&lt;p&gt;That makes &lt;a href="https://frontendtester.com/how-to-test-client-side-routing-back-button-behavior-and-scroll-restoration-in-spas-without-missing-navigation-bugs/" rel="noopener noreferrer"&gt;client-side routing and scroll restoration&lt;/a&gt; worth treating as first-class behavior rather than navigation trivia.&lt;/p&gt;

&lt;h2&gt;
  
  
  Selectors Still Matter. Just Differently.
&lt;/h2&gt;

&lt;p&gt;I don’t think selectors are going away.&lt;/p&gt;

&lt;p&gt;But I do think teams often spend too much time trying to make one clever selector survive every imaginable frontend change.&lt;/p&gt;

&lt;p&gt;That becomes especially painful with microfrontends.&lt;/p&gt;

&lt;p&gt;One team renames a component.&lt;/p&gt;

&lt;p&gt;Another team introduces a wrapper.&lt;/p&gt;

&lt;p&gt;A shared UI package changes its DOM structure.&lt;/p&gt;

&lt;p&gt;Suddenly 87 tests fail.&lt;/p&gt;

&lt;p&gt;A sane &lt;a href="https://test-automation-experts.com/how-to-build-a-test-selector-strategy-for-microfrontends-dynamic-components-and-shared-ui-libraries/" rel="noopener noreferrer"&gt;selector strategy for microfrontends and shared component libraries&lt;/a&gt; is mostly about establishing contracts between developers and tests.&lt;/p&gt;

&lt;p&gt;That’s much more scalable than XPath archaeology.&lt;/p&gt;

&lt;h2&gt;
  
  
  Virtualized Interfaces Are Another Trap
&lt;/h2&gt;

&lt;p&gt;Infinite scroll and virtualized tables are great examples.&lt;/p&gt;

&lt;p&gt;An item can conceptually exist in the application while not existing in the DOM.&lt;/p&gt;

&lt;p&gt;That distinction matters.&lt;/p&gt;

&lt;p&gt;It’s why comparing approaches such as &lt;a href="https://bugbench.com/endtest-vs-playwright-for-testing-web-apps-with-heavy-virtualized-rendering-and-infinite-scroll/" rel="noopener noreferrer"&gt;Endtest vs Playwright for heavily virtualized interfaces&lt;/a&gt; gets interesting. The question isn’t merely whether the tool can click an element.&lt;/p&gt;

&lt;p&gt;The question is how much machinery you have to build around that click.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Useful Question
&lt;/h2&gt;

&lt;p&gt;When a browser test fails, I’d stop asking:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“How do we make this test pass?”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;At least initially.&lt;/p&gt;

&lt;p&gt;Ask:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;“What assumption did this test make about the application?”&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Maybe it assumed the element was already rendered.&lt;/p&gt;

&lt;p&gt;Maybe it assumed the route caused a page load.&lt;/p&gt;

&lt;p&gt;Maybe it assumed the item existed in the DOM.&lt;/p&gt;

&lt;p&gt;Maybe it assumed the component responded to viewport width.&lt;/p&gt;

&lt;p&gt;Maybe it assumed there was only one document context.&lt;/p&gt;

&lt;p&gt;Once you find that assumption, the failure usually becomes much less mysterious.&lt;/p&gt;

&lt;p&gt;And strangely enough, your tests tend to become much less flaky too.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>automation</category>
      <category>frontend</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Visual Regression Testing Has a Noise Problem</title>
      <dc:creator>Antoine Dubois</dc:creator>
      <pubDate>Fri, 07 Aug 2026 22:07:21 +0000</pubDate>
      <link>https://dev.to/randomsquirrel802/visual-regression-testing-has-a-noise-problem-35bp</link>
      <guid>https://dev.to/randomsquirrel802/visual-regression-testing-has-a-noise-problem-35bp</guid>
      <description>&lt;p&gt;Visual regression testing sounds almost unfairly simple.&lt;/p&gt;

&lt;p&gt;Take screenshot A.&lt;/p&gt;

&lt;p&gt;Take screenshot B.&lt;/p&gt;

&lt;p&gt;Compare them.&lt;/p&gt;

&lt;p&gt;If they’re different, something changed.&lt;/p&gt;

&lt;p&gt;Unfortunately, browsers have spent the last decade making “something changed” a surprisingly complicated concept.&lt;/p&gt;

&lt;p&gt;Fonts render differently.&lt;/p&gt;

&lt;p&gt;Content shifts by a pixel.&lt;/p&gt;

&lt;p&gt;Animations fire.&lt;/p&gt;

&lt;p&gt;Responsive layouts reflow.&lt;/p&gt;

&lt;p&gt;Container queries change components independently of the viewport.&lt;/p&gt;

&lt;p&gt;Web Components hide half the page inside Shadow DOM.&lt;/p&gt;

&lt;p&gt;Suddenly your visual testing system is producing hundreds of differences, and the person reviewing them develops the highly sophisticated QA technique known as:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;clicking Accept repeatedly until the red badges disappear.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;At that point, you technically have visual regression testing.&lt;/p&gt;

&lt;p&gt;You just don’t have much visual regression detection.&lt;/p&gt;

&lt;h2&gt;
  
  
  Signal matters more than sensitivity
&lt;/h2&gt;

&lt;p&gt;The instinct when evaluating screenshot tools is to ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How precisely can it detect pixel differences?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That’s not the question I’d start with.&lt;/p&gt;

&lt;p&gt;A tool that detects every single pixel difference may be worse than one that ignores harmless variation.&lt;/p&gt;

&lt;p&gt;The real question is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How quickly can a human distinguish an actual regression from noise?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those are very different product requirements.&lt;/p&gt;

&lt;p&gt;A useful evaluation should include baseline management, masking, thresholds, rendering consistency, review workflows, and what happens when the expected UI legitimately changes.&lt;/p&gt;

&lt;p&gt;This guide to &lt;a href="https://test-automation-tools.com/how-to-evaluate-a-browser-testing-tool-for-visual-diff-noise-baseline-drift-and-screenshot-review/" rel="noopener noreferrer"&gt;evaluating browser testing tools for visual diff noise, baseline drift, and screenshot review&lt;/a&gt; gets into those details.&lt;/p&gt;

&lt;p&gt;Baseline drift is especially dangerous.&lt;/p&gt;

&lt;p&gt;If developers routinely approve new screenshots without understanding the differences, your baseline slowly stops representing “correct.”&lt;/p&gt;

&lt;p&gt;It represents “whatever the application looked like the last time someone got tired of reviewing screenshots.”&lt;/p&gt;

&lt;h2&gt;
  
  
  Fonts are chaos disguised as typography
&lt;/h2&gt;

&lt;p&gt;Fonts deserve their own section because they cause a ridiculous amount of visual noise.&lt;/p&gt;

&lt;p&gt;A font loads late.&lt;/p&gt;

&lt;p&gt;The fallback font has slightly different metrics.&lt;/p&gt;

&lt;p&gt;A headline becomes one pixel wider.&lt;/p&gt;

&lt;p&gt;A card becomes three pixels taller.&lt;/p&gt;

&lt;p&gt;Everything below it moves.&lt;/p&gt;

&lt;p&gt;Your visual comparison now tells you that half the page changed.&lt;/p&gt;

&lt;p&gt;Technically, it’s right.&lt;/p&gt;

&lt;p&gt;Operationally, it’s useless.&lt;/p&gt;

&lt;p&gt;The same issue shows up with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;dark mode,&lt;/li&gt;
&lt;li&gt;themes,&lt;/li&gt;
&lt;li&gt;localization,&lt;/li&gt;
&lt;li&gt;dynamic content,&lt;/li&gt;
&lt;li&gt;personalization,&lt;/li&gt;
&lt;li&gt;responsive typography.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There’s a good exploration of this in &lt;a href="https://web-developer-reviews.com/what-to-evaluate-in-a-visual-regression-tool-for-dynamic-fonts-theming-and-layout-shifts/" rel="noopener noreferrer"&gt;what to evaluate in a visual regression tool for dynamic fonts, theming, and layout shifts&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A strong visual test system needs to understand that not every difference has equal importance.&lt;/p&gt;

&lt;p&gt;A missing checkout button and a one-pixel font shift should not create identical urgency.&lt;/p&gt;

&lt;h2&gt;
  
  
  Responsive testing isn’t testing three viewport widths
&lt;/h2&gt;

&lt;p&gt;For years, responsive testing meant:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;desktop,&lt;/li&gt;
&lt;li&gt;tablet,&lt;/li&gt;
&lt;li&gt;mobile.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That model is increasingly incomplete.&lt;/p&gt;

&lt;p&gt;CSS Grid and Flexbox often fail at awkward widths that nobody specifically designed.&lt;/p&gt;

&lt;p&gt;A navigation menu might look perfect at 1440px and 375px but wrap badly at 847px.&lt;/p&gt;

&lt;p&gt;A product card might collapse because one translation is longer than the English version.&lt;/p&gt;

&lt;p&gt;These are the bugs users actually see.&lt;/p&gt;

&lt;p&gt;This walkthrough of &lt;a href="https://frontendtester.com/how-to-test-css-grid-flexbox-and-responsive-wrapping-without-missing-layout-breakpoints/" rel="noopener noreferrer"&gt;testing CSS Grid, Flexbox, and responsive wrapping without missing layout breakpoints&lt;/a&gt; makes an important point: meaningful breakpoints often come from the content, not from your CSS framework.&lt;/p&gt;

&lt;p&gt;And container queries push this even further.&lt;/p&gt;

&lt;p&gt;A component’s layout may now depend on its parent container rather than the browser viewport.&lt;/p&gt;

&lt;p&gt;You can have two copies of the same component on the same page behaving differently.&lt;/p&gt;

&lt;p&gt;That makes old-school “set browser width, take screenshot” strategies increasingly inadequate.&lt;/p&gt;

&lt;p&gt;If your frontend uses them heavily, &lt;a href="https://testautomationguide.com/how-to-test-css-container-queries-and-responsive-layouts-without-creating-visual-noise/" rel="noopener noreferrer"&gt;testing CSS container queries without creating visual noise&lt;/a&gt; is worth a look.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then Shadow DOM arrives
&lt;/h2&gt;

&lt;p&gt;Shadow DOM solves legitimate frontend architecture problems.&lt;/p&gt;

&lt;p&gt;It also introduces a fun testing problem:&lt;/p&gt;

&lt;p&gt;The element you can clearly see in the browser may not exist where your automation expects it to exist in the DOM tree.&lt;/p&gt;

&lt;p&gt;That can affect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;selectors,&lt;/li&gt;
&lt;li&gt;interactions,&lt;/li&gt;
&lt;li&gt;accessibility checks,&lt;/li&gt;
&lt;li&gt;screenshots,&lt;/li&gt;
&lt;li&gt;debugging,&lt;/li&gt;
&lt;li&gt;reusable page objects.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your application uses a large component library based on Web Components, this shouldn’t be an edge-case checkbox in your tool evaluation.&lt;/p&gt;

&lt;p&gt;It should be part of the proof of concept.&lt;/p&gt;

&lt;p&gt;There’s a useful checklist for &lt;a href="https://testautomationreviews.com/how-to-evaluate-a-test-automation-platform-for-shadow-dom-web-components-and-encapsulated-ui-libraries/" rel="noopener noreferrer"&gt;evaluating test automation platforms for Shadow DOM and encapsulated UI libraries&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;And don’t just test a single toy component.&lt;/p&gt;

&lt;p&gt;A framework being technically capable of entering a shadow root doesn’t tell you much about how it performs against a real application containing hundreds of components.&lt;/p&gt;

&lt;p&gt;I like the more empirical approach described in &lt;a href="https://bugbench.com/how-to-benchmark-browser-test-stability-on-shadow-dom-heavy-frontends-without-blaming-the-runner/" rel="noopener noreferrer"&gt;benchmarking browser test stability on Shadow DOM-heavy frontends&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Run enough representative tests to discover where the maintenance cost actually appears.&lt;/p&gt;

&lt;h2&gt;
  
  
  Screenshot count is a terrible KPI
&lt;/h2&gt;

&lt;p&gt;One final trap: measuring visual-test maturity by the number of screenshots.&lt;/p&gt;

&lt;p&gt;“We have 4,000 screenshot assertions” sounds impressive.&lt;/p&gt;

&lt;p&gt;It might also mean four engineers are approving meaningless diffs every morning.&lt;/p&gt;

&lt;p&gt;The useful metrics are closer to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;regressions caught before production,&lt;/li&gt;
&lt;li&gt;false-positive rate,&lt;/li&gt;
&lt;li&gt;review time,&lt;/li&gt;
&lt;li&gt;baseline churn,&lt;/li&gt;
&lt;li&gt;percentage of visual checks people still trust.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one is hard to put on a dashboard.&lt;/p&gt;

&lt;p&gt;But it might be the most important.&lt;/p&gt;

&lt;p&gt;Because visual regression testing is only useful as long as humans believe the red pixels mean something.&lt;/p&gt;

&lt;p&gt;The moment they stop believing that, you have a very expensive screenshot generator.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>frontend</category>
      <category>webdev</category>
      <category>automation</category>
    </item>
  </channel>
</rss>
