DEV Community: David Ingraham

Treating the UI as a Contract: Eliminating the Wait in Modern Development

David Ingraham — Mon, 20 Apr 2026 01:07:20 +0000

This blog and concept was a collaboration between Leonardo Lanni & David Ingraham.

Most teams don’t build features.
They wait on them.

Development waits for product.
Then front-end waits for back-end.
Meanwhile QA waits for front-end and back-end.
And finally, testing gets squeezed at the end.

By the time everything connects, bugs show up when context is already gone. Does this sound familiar?

What if you never had to wait in the first place?

This blog introduces a different approach: treating the UI as a contract so everyone can move in parallel from day one. This means instead of work flowing in sequence, everything moves at once. Instead of testing being squeezed at the end, it starts immediately. And instead of integration being discovery, it becomes validation.

Prerequisites for Success

This approach is powerful, but it’s not something you can drop into any team overnight. If your team struggles with communication, this will expose it. If ownership is unclear, this will amplify it. If your process depends on clean handoffs, this will break that expectation entirely.

Because this model replaces handoffs with collaboration.

For it to work, a few things need to be true:

Strong cross-functional collaboration
Front-end, back-end, QA, and product stay aligned continuously, not sequentially.
A shift-left mindset
Teams define behavior, write tests, and validate flows before everything is fully implemented.
Shared ownership of quality
Testing isn’t owned by QA alone. Everyone contributes early.
Comfort with partial implementations
Mocked data, incomplete UIs, and evolving designs are part of the process, not blockers.

Without these, things can get messy fast.

But with the right culture in place, this becomes a force multiplier for both speed and quality. With this in mind, let’s move on to how this is implemented in practice.

Core Idea

As soon as a screen exists, it becomes a contract.

A feature is typically composed of multiple screens. The moment one is defined, we establish how it should behave. This is the UI contract.

This contract can take different forms depending on the team:

A design-driven contract (Figma designs, product requirements, defined behaviors)
A code-driven contract (a UI skeleton with mocked data and interactions)

Both serve the same purpose: defining how the UI should behave before full implementation. Finding the right fit for your team is crucial in gaining buy-in.

In practice, we move quickly from design into a UI skeleton that becomes the executable version of that contract.

This isn’t a prototype or throwaway code. It’s the real UI, just powered by mocked data and behavior instead of live APIs.

This skeleton defines:

All UI controls (inputs, buttons, tables, modals, etc.)
Stable selectors (data-test ids, accessibility roles, labels)
Navigation and validation rules
Realistic mocked behavior instead of API calls

In addition to structure and interactions, the UI contract also defines how the application behaves under different scenarios.

And importantly, it defines how the system behaves under real-world conditions:

Empty results
Permission errors (e.g. 403 Forbidden)
Validation failures (e.g. 422 with field-level errors)
System errors
Slow responses

In our implementation, these behaviors are formalized through structured error types, ensuring that both mocked and real providers expose consistent outcomes to the UI.

Throughout development, the mocked behavior is replaced with real API integrations, but the structure, selectors, and expected behavior remain the same.

How It Works

Once the contract exists, everything changes.

Instead of waiting, the team moves in parallel.

This approach is built around three core pieces:

The UI skeleton (the contract itself)
A mocked behavior layer
Parallel work-streams across the team

1. UI Skeleton (Contract)

Early in development, we implement the UI structure itself:

Page structure and routing
Validation rules
All system states (loading, empty, error, success)
Basic layout and placeholders (UI controls, selectors, structure)

Again, how this definition looks can change between teams. Whether it’s a Figma design, detailed product ticket, real code, or a combination of these, this becomes the shared interface across front-end, back-end, and QA.

2. Mocked Behavior Layer

Rather than calling real services, the UI is powered by a mocked data layer (fixtures, MSW, dependency injection, etc.).

But these aren’t random mocks.

They simulate real-world scenarios like we alluded to earlier, such as successful or empty responses, validation errors, permission issues, or response latency.

Rather than exposing raw HTTP responses, providers return structured outcomes like:

forbidden
validation_error
system_error

This ensures the UI behaves consistently regardless of whether it is connected to mocked data or real back-end services.

The key is that the UI behaves as if it were connected to real systems.

As development progresses, we ensure mocked logic is replaced with real API integrations and the contract always stays stable.

3. Parallel Work-streams

With the contract in place, different roles can fully move independently without waiting on each other.

Ensures proper selectors exist
Starts writing UI/E2E tests immediately
Run the same tests later against real APIs and functioning UI without rewriting

Front-end Developers

Build real logic incrementally
Swap mocks for real services

Back-end Developers

Define and implement APIs
Use the UI and tests as a guide for expected behavior

Designers (or Product)

Refine visuals as feedback arises
Establish visual baselines once designs stabilize

Instead of handing work off between roles, everyone moves forward at the same time against a shared, stable interface.

UI-Level TDD (Test-Driven Development)

This approach enables a form of TDD at the UI level.

This is possible, again, by defining and validating UI behavior upfront:

Define UI behavior and states
Implement UI skeleton with mocks
Write automated UI tests
Replace mocked logic with real implementations
Tests remain unchanged

The key benefit is that tests are written early and remain stable.

They validate behavior, not implementation details, so they continue to pass as the system evolves.

In practice, this is reinforced by running the same automated tests against both mocked and real implementations.

This dual-mode execution acts as a continuous validation that the UI contract holds across different environments and implementations.

Visual Regression Strategy

Visual testing can also be introduced in phases to match the maturity of the UI.

Early on, visual tests are optional or non-blocking and the focus is on behavior, not polish.

Later though, designs are finalized and approved. Visual baselines are then established and visual regression tests become blocking.

At that point, visuals become part of the contract too.

Real-World Example

This all sounds good in theory.

But what does it actually look like in practice?

Let’s pretend we need to build a new order form. Like most features, it starts with a detailed product ticket, business requirements, and a Figma design.

From there, we refine the feature across QA, front-end, back-end, and product. The goal isn’t just to scope the work, but to align on how the UI should behave, what edge cases exist, and what “done” actually looks like. This step is crucial because it ensures the entire team not only understands the upcoming feature, but has the opportunity to contribute to its design. No misunderstandings.

Once the feature is defined, we treat the design and product definition as the contract and start work in parallel. Throughout development, everyone communicates. Daily status updates and raising any additional questions or engineering concerns ensure no one drifts out of the loop.

Eventually, by the time the back-end endpoints are ready, the UI is mostly complete and tests are ~90% done. The last bit of work is just pulling in everyone’s changes and making the small adjustments needed to hook the entire flow up.

Bugs are caught early, feedback is fast, and there are no last-minute scrambles. By the time integration began, everything was already aligned.

What makes this different

At first glance, this can feel familiar — API mocking, shift-left testing, even prototyping. And in some ways, it builds on all of them.

But the difference is what everything is anchored to.

With API mocking, teams simulate backend behavior to unblock development. But those mocks are often temporary and loosely defined, drifting from reality over time.

Here, mocks are part of the contract itself. They don’t just return data, they define expected behaviors like validation errors, permission failures, and edge cases. Both mocked and real implementations are expected to behave the same way.

With shift-left testing, tests are written earlier — but they’re still often reacting to evolving systems and unstable interfaces.

In this approach, tests are written against a stable UI contract from the start. They don’t need to change as implementation evolves, because they validate behavior — not how that behavior is implemented.

And while prototypes are typically throwaway, the UI skeleton here is not. It starts early, supports real testing, and evolves directly into the final product. Nothing gets discarded.

The shift is subtle but important:

The UI isn’t just something we build and test.

It becomes the contract that everything else aligns to and that contract is executable from day one.

GitHub Demo Repository

To make this concept tangible, we built a small demo repository:

ui-contract-testing-demo

It showcases the approach end-to-end, starting with a UI skeleton that defines routes, controls, selectors, and states. This is paired with a provider abstraction that supports both mocked and real implementations, allowing the UI to behave consistently regardless of the data source.

The demo also includes a range of realistic scenarios — from happy paths to empty states, validation errors, permission issues, and even slow responses so behavior can be validated under real-world conditions, not just ideal ones.

On top of that, the same Playwright test suite runs against both mocked and real environments. This means testing can start early, remain stable throughout development, and continuously validate integrations without needing to rewrite tests.

Summary

By treating the UI as a contract, teams unlock a fundamentally different way of building software. Testing starts from day one, not the end. Build the contract first. Let everyone move against it. Replace mocks with reality over time.

This isn’t about making testing faster. It’s about removing the conditions that made you wait in the first place.

No bottlenecks.
No last-minute scrambles.
No rebuilding context when it matters most. Just continuous progress.

Eventually, you realize you were never blocked… you were just waiting.

From both of us, happy testing,
Leonardo Lanni & David Ingraham

Your Cypress Tests Are Slower Than You Think

David Ingraham — Mon, 20 Apr 2026 00:03:45 +0000

If you’ve worked on a decent-sized Cypress suite, you’ve lived this.

You push code. Pipeline kicks off.
Then you wait.
And you wait.
…Still waiting.

You tab over to Slack. Maybe check a PR. Start something else.
Eventually results come back. Green, red, who knows.

But here’s the problem: by the time you see them… you don’t care anymore. That’s the real issue with slow tests. Not time.

It’s that they quietly disconnect themselves from how engineers actually work. And once that happens, the tests don’t just get slower, they stop mattering.

So yeah, we’ll talk about how to speed things up.
But this is really about something else: why slow tests break the feedback loop, and how to fix it before your suite turns into expensive decoration.

The Hidden Cost of Slow Tests

Everyone talks about wait time. That’s the obvious part.

The real cost is behavioral.
Fast tests keep developers in the loop:

write code → run tests → fix problem → move on

It’s tight. Feedback is immediate. You stay in context.
Slow tests break that loop:

write code → push commit → switch task → failure appears later

Now you’re doing archaeology. And unless you secretly enjoy digging through commits like it’s a crime scene, it’s not a great place to be.

Multiply that across a team and something subtle happens.
People stop waiting.

Tests still run. Pipelines still pass or fail.
But they stop influencing decisions.

That’s the line most teams don’t notice crossing.

And if you’ve been around long enough, you know where that ends.

First they’re flaky.
Then they’re ignored.
Eventually, someone suggests deleting or skipping them “just for now.”

And somehow, they never come back.

The 1-Second Problem

Most slow test suites don’t have a single obvious culprit, they bleed out from a hundred small paper cuts.

A second here.
Two seconds there.
A login repeated 100 times.
A page load you didn’t question.
One hard-coded wait that turns into ten next month.

Each one feels harmless. That’s the problem though. Tests don’t run once — they run at scale. And when you scale up a single second, it adds up fast.

For example:

1 second × 300 tests = 5 minutes

Five minutes, from something no one even noticed or questioned. Now imagine what happens when you have ten of these 1-second problems. Now 100. That’s how teams end up with 30-minute pipelines without ever making a “bad” decision. It’s not one accidental mistake.

It’s long-term, unnoticed accumulation.

Test Optimizations

Now that we’ve defined why this matters, let’s actually fix it. We’ll be breaking up the solutions into three core sections, Test, Pipeline and Startup Optimizations. First, let’s start where the damage usually begins, inside the tests.

Replace Hard-Coded Waits

Hard waits are one of the easiest ways to waste time and one of the easiest problems to spot.

cy.wait(5000)

This ensures that your test is sitting there for 5 seconds, regardless if the app actually responded in 200ms. Every time.

Instead, wait for real signals. Use dynamic-waits instead, such as waiting on real endpoint using cy.intercept.

// Define an intercept on a real API
cy.intercept('/api/orders').as('orders')

cy.visit('/')

// Wait on the real load before interacting with the UI
cy.wait('@orders')
cy.get('order-button').click()

You can also add “safe-guard” assertions on the UI too, utilizing Cypress’s built-in retry system.

cy.visit('/')

// Wait for the button to exist in the DOM and be in a clickable state
cy.get('order-button').should('be.visible')
cy.get('order-button').should('be.enabled').click()

The key idea is simple: don’t wait for time, wait for state.

Cache Login Sessions

Login flows are also silent killers and a great example of the 1-second problem. They don’t look expensive but run across hundreds of tests and suddenly you’ve built a login simulator, not a test suite.

This is especially common in suites where every test starts the same way:

cy.visit('/')
cy.get('[data-cy=email]').type('test@test.com')
cy.get('[data-cy=password]').type('password')
cy.get('[data-cy=login]').click()

Again, this looks harmless until you realize you’re doing it 200+ times per run. Instead, use cy.session. Cypress gives you a built-in way to cache authentication and reuse it across tests.

cy.session('user', () => {
  cy.visit('/')
  cy.get('[data-cy=email]').type('test@test.com')
  cy.get('[data-cy=password]').type('password')
  cy.get('[data-cy=login]').click()
})

Now instead of logging in every time, Cypress will:

run this once
cache the session
restore it for the rest of your tests

No extra UI steps. No repeated logins. No wasted time.

Avoid Redundant Checks

To be clear, assertions aren’t the problem. End-to-end tests live or die by the assertions they make. The issue is repeated work.

Cypress actually encourages multiple assertions when they help prove meaningful behavior. The problem shows up when tests keep re-querying the DOM or stacking checks that don’t add much confidence. Every extra cy.get() call isn’t free.

Take a look at this example validating a simple success state.

cy.get('.order').should('exist')
cy.get('.order').should('be.visible')
cy.get('.order').should('contain', 'Order created')
cy.get('.order-title').should('exist')
cy.get('.order-price').should('exist')

Yes, this proves the order exists and contains the expected text but at some point, we’re not improving the test anymore. We’re just making Cypress do more laps, creating a tiny performance bloat.

Most of the time, you can collapse this into something tighter:

cy.get('.order')
  .should('be.visible')
  .and('contain', 'Order created')

Or, if the real goal is proving the user outcome:

cy.contains('Order created')

The goal isn’t fewer assertions for the sake of it. The goal is fewer redundant checks, fewer duplicate queries, and more signal per line of test code.

Mock Expensive Requests

Your app does a lot of things your test does not care about.
Images. Analytics. Recommendation engines. Background noise.

And yet, your tests patiently wait for all of it, especially when running against a prod-like environment. This usually shows up as slow page loads, inconsistent timing, and tests that feel fine locally but drag in CI.

Let’s say your page loads a directory which includes on load:

20 images
3 analytics calls
a recommendation engine hitting some external service

None of that matters if your test is just trying to verify:
“Can a user create an order?”

But your test still pays the cost for it. Every time.

Instead, intercept and stub these requests to skip the overhead entirely, using our trusty friend cy.intercept again.

cy.intercept('/images/*', { fixture: 'placeholder.png' })
cy.intercept('/analytics/*', { statusCode: 200 })
cy.intercept('/recommendations/*', { fixture: 'empty.json' })

Now our test is going to load faster with reduced network noise for a test that has no dependencies on these calls.

That being said, mock responsibly. Don’t mock if:

the request is part of the core user flow
you’re validating real integration behavior
the response directly impacts what the user sees or does

If your test is about creating an order, then test that, not whether your dashboard images finished loading first.

Pipeline Optimizations

Even with clean, efficient tests, your pipeline can still be the bottleneck.

At some point, the slowdown isn’t in your tests anymore. It’s in your infrastructure. Even if your tests are clean, your pipeline can still be the bottleneck.

Run Tests in Parallel

This is usually the biggest win available and pretty straight-forward.

If you’re running everything on a single machine, you’re leaving speed on the table.

Example with 200 tests:

1 machine → ~20 minutes
4 machines → ~5 minutes

Same amount of tests. Same coverage. Same value. Just distributed.

If you’re not parallelizing, you’re choosing to be slow.

Most CI providers support this out of the box and Cypress provides it with integration into Cypress Cloud. Without Cloud there are free plugins that achieve the same thing, such as cypress-split.

Your Pipeline Is Only As Fast As the Slowest Spec

Parallelization helps, until one, pesky spec decides to ruin everything.

spec A → 12 minutes  
spec B → 3 minutes  
spec C → 3 minutes

Your pipeline time? Yeah, 12 minutes.

This is where a lot of teams get tripped up. They split tests by file count instead of runtime.

Knowing your slowest spec lets you target the biggest performance bottlenecks directly instead of guessing.

Focus on outliers. Break them up if possible, or optimize what’s inside them and then track again. You should always know your slowest tests and why they’re slow.

If possible, all specs should execute within a ~1 minute threshold of one another to ensure scaleable balance as the test suite grows and additional resources are to your parallelization.

Cache Cypress and Node Modules

CI loves to pretend it’s starting from scratch every run.

Downloading Cypress. Installing dependencies. Setting up browsers.

But want to know a secret? You don’t need to pay that cost every time.

Cache:

node_modules
Cypress binary
browser dependencies (if applicable)

This is one of those optimizations that feels boring but it shaves minutes off every single run. Always review best practices around CI and your CI provider to ensure your pipeline isn’t dragging unintentionally.

Avoid Persisting Artifacts for Passing Tests

Videos and screenshots are incredibly useful when something breaks.

When everything passes? They’re just expensive souvenirs.

If you’re uploading artifacts for every run, you’re increasing pipeline time, increasing storage costs, and making it harder to find what actually matters.

Turn them off for green runs. Keep them for failures where they’re actually useful.

Startup Optimizations

Some of the worst delays don’t come from the tests themselves, they happen before the first test even runs.

And most teams miss it, because they’re focused on runtime, not startup.

Use APIs for Test Setup

UI data setup is the scenic route. And trust me, I love a gorgeous view just like the next person, but our end-to-end tests don’t need the extra travel.

Let’s pretend we want to test deleting a “project” from our application.

This is what it often looks like, simplified:

cy.visit('/')
cy.get('[data-cy=email]').type('test@test.com')
cy.get('[data-cy=password]').type('password')
cy.get('[data-cy=login]').click()

cy.get('[data-cy=create-project]').click()
cy.get('[data-cy=project-name]').type('Test Project')
cy.get('[data-cy=submit]').click()

Without realizing it, our test is validating the create flow and dependent on that UI for delete functionality that should be isolated. Expand this and a longer UI-setup might take 5–10 seconds just to set state. Multiply that across your suite and the cost becomes obvious.

Instead, utilize cy.request to setup the state programatically.

cy.request('POST', '/api/projects', data)

Now your setup runs in milliseconds instead of seconds. I wrote about this topic in-depth in a blog here and how to easily achieve this pattern at scale in your test suite.

In short, use the UI when you’re testing the UI.
Not when you’re just setting the stage to get to real value.

Avoid Heavy Imports in Support Files

Every Cypress spec loads your support files.
Simply, whatever you import there, runs every time.

If you’re pulling in half your project globally, you’ve created a startup tax.
For clarification, here’s a github issue that explains this very problem.

Just like everything else we’ve mentioned, it adds seconds to every spec file. So only load what you actually need.

move heavy imports into specific tests
lazy-load where possible
keep your support file lean

Here’s an in-depth article from the talented Murat Ozcan that explains this concept in expanded details.

Review Your Preprocessor

Preprocessing is one of those hidden slowdowns most teams don’t think about. Essentially, bundling (or preprocessing) is the step where Cypress compiles and transforms your test files and their dependencies into browser-ready JavaScript before execution.

And in short, if that step is slow, everything is slow.

Cypress uses Webpack by default and for the most part it works. Nearly all teams are using it for this very reason. However, it’s also not exactly famous for being quick in large test setups and might struggle compared to alternative solutions.

If your suite has grown, it’s worth looking at faster preprocessors like esbuild(another fantastic video by Murat here) or Vite (using cypress-vite). In a lot of projects, that switch alone noticeably cuts startup time.

If you are noticing significant performance latency just starting Cypress before tests run, then it might be time to reconsider your preprocessor.

Review Your Config Tip

Finally, sometimes the slowdown isn’t hiding in your tests at all, it’s sitting in your cypress.config quietly adding overhead to every run.

There are a few settings that are worth revisiting with performance in mind: retries, video, screenshots, and numTestsKeptInMemory.

Retries are the easiest one to overlook. They feel helpful, and they also hide flakiness and multiply runtime. A test that passes on the third attempt didn’t just pass, it ran three times.

Video and screenshot settings fall into a similar category. They’re incredibly useful when something breaks, but they don’t always need to be enabled for every passing test.

The point isn’t to turn everything off. It’s to be intentional.

It’s also worth staying reasonably up to date with Cypress itself. Performance improvements do get shipped, and running an older version longer than you need to can quietly hold you back in ways that have nothing to do with your tests.

Final Tip: Be Data-Driven About Your Problem Tests

If you don’t measure it, it will rot.
Test suites don’t stay fast by accident.

You need to know where your time is going: your slowest tests, your slowest specs, and your flakiest tests.

That’s where the real problems live.

Otherwise, you’re guessing — and guessing doesn’t scale.

In most suites, a small number of slow or flaky tests are doing most of the damage. That’s where you start. Fixing your worst offenders will move the needle far more than tweaking 50 “okay” tests. Use data to identify what’s actually hurting you, surface flaky tests causing retries, and keep an eye on runtime drift over time.

Then fix what matters.

That’s how you get faster pipelines, more reliable results, and a test suite people actually trust.

For help identifying where your test flake is occurring, Sebastian Suero has a great plugin that can help with that, cypress-flaky-test-audit. And for how to build your own merge report to track these metrics, you can find my tutorial here.

Final, Final Tip: Does It Need to Be End-to-End?

The hard truth is, after all these other tips, a lot of slow suites are slow because end-to-end tests are carrying work that does not actually belong there. This happens all the time on startup teams, or on teams where QA sits in its own lane and E2E becomes the default place to validate everything.

Cypress supports both end-to-end and component testing, and it's docs are pretty clear that different test types exist for different kinds of confidence. If a check can be proven faster at the API or component level, that is usually the better place for it. That does not mean E2E is less valuable. It just means it should be used where it shines: proving real user flows, not carrying the entire quality strategy on its back.

Cypress can also be used for API-style testing, and community plugins like cypress-plugin-api by Filip Hric, build on that by adding a cy.api command for API-focused workflows in the Cypress runner.

On the UI side, Cypress Component Testing gives you a way to validate rendering, state, and interaction at the component layer, which is often much faster and more targeted than proving the same thing through a full browser journey. A healthy suite usually has a mix of all three.

Summary

Most slow test suites don’t collapse because of one bad decision. They decay over time. A login repeated too often, a wait no one questioned, a setup flow that felt “good enough.” Individually harmless, but together they quietly drag down your feedback loop.

And once that loop breaks, your tests stop shaping behavior. Failures show up too late, context is lost, and fixing issues becomes slower and more painful than it needs to be.

The most valuable moment for a failure isn’t ten minutes later in CI. It’s right after you push, while everything is still fresh.

Fast tests protect that moment. Slow tests erase it.

With that, happy testing.

Lessons I Wish I Knew When I Started in Test Automation

David Ingraham — Wed, 15 Apr 2026 00:26:22 +0000

A Battle-Scarred Senior SDET's Perspective

When I first moved into test automation, I thought I was doing well.

I could write clean code. I built frameworks I felt proud of. Most of my pipelines were green. I was even catching the occasional bug before customers ever saw it. At the time, that felt like success.

Looking back, that view was narrow. I measured myself by output instead of impact. I focused on what I could build, not on how that work helped teams ship safely or make better decisions. "Catching bugs" became the goal, rather than understanding risk.

Over time though, I saw the same problems repeat across teams, tools, and companies. The details changed. The patterns did not.

Automation is not about tests that pass. It is about helping teams understand risk and and ship with confidence. Tests are just one slice of a larger system and if you only focus on the test results, you'll miss the bigger picture entirely like I did early on.

With that in mind, these are the lessons I wish I had learned sooner.

Strong coding skills are necessary, but not sufficient

My path into QA was highly unconventional.

I started as a full-stack developer after graduating with a computer science degree. My mindset was code first, code always. I fell into QA almost by accident. Let's grab a coffee sometime and trade career stories. I'm grateful it happened, but early on, my identity lived entirely in the code. I focused almost obsessively on writing elegant test frameworks. Clean architecture. Reusable helpers. Beautiful setup APIs. Everything pristine. If the framework looked slick, I felt successful.

Now don't get me wrong, that work matters. Strong coding skills are required for test suites that scale, stay readable, and survive change. You cannot build serious automation without them. But code should not be the first focus. It is a tool, not the goal.

Strong code can hide weak thinking, and nobody ever taught me how to think about quality. That is not something you learn in school.

You can write clean tests and still miss the failures that hurt users. Effective testing starts with understanding:

risk
system behavior
where failures are likely to occur
business impact

When you shift from solely focusing on writing code to thinking about the true failure, your impact changes. The code supports the thinking, not the other way around. This is why I believe so many developers struggle to write effective automation tests. They can code just fine, but the thinking can be a bit shallow. And when the thinking lacks depth, you end up with a very nice test that ultimately delivers no business value.

It took me a long time to learn this. I still take pride in my technical skills, but I learned the hard way that strong automation starts with a strong quality mindset. No amount of clever code can compensate for the absence of it.

How flaky tests are handled reflects leadership

The first time someone said "just rerun it," I didn't push back.

It was "one of those tests." A little finicky. We joked that it was having a bad day and needed more coffee. Rerun it, get green, move on. It seems so insignificant at the time but over the years I've realized this mindset is quite damaging.

When teams accept flakes, failures lose meaning. Engineers stop trusting results. Automation turns into noise. You still run the tests, but you stop taking them seriously.

Let's be clear, flaky tests do not reflect weak engineers. They reflect weak decisions about ownership and priorities. Flake will always exist. You cannot eliminate it completely. It is a by-product of distributed systems, async behavior, environments, data, and timing.

Leadership is not just about preventing flake. It is about how you respond when it happens.

Ignoring flakes, hiding them, or rerunning until green are leadership choices. Fixing them requires explicit ownership and real trade-offs. The moment you decide to tolerate noise in your test suite is the moment you accept a brittle quality culture.

At a senior level, this becomes your responsibility. Flakiness must be treated like any other defect, with root cause, follow-up, and prevention. You must protect your green runs and address any failure, of any type, with urgency. Otherwise, over time, if you don't enforce this, you slowly lower your quality bar without realizing it.

This may sound extreme, but pause and think about it. I genuinely believe flaky tests are worse than having no tests at all. Noise distracts teams from work that matters. Being complacent with flakes is a subtle form of accepting failure. One path is easy. The other leads to long-term reliability and trust.

CI pipelines matter as much as the tests

I didn't realize it early in my career but how your tests run is just as important as how they are written.

You can have the most perfectly stable test that runs locally and passes 100/100 times but if you don't put the same effort and thinking into the pipeline, what you will end up seeing is:

slow execution
unclear failures
unreliable test results

Now all of a sudden, a test suite that was valuable becomes noise and a giant liability.

And it only get worse. When pipelines are slow or confusing, teams stop using automation as a decision tool, which is their entire purpose. Now something that used to be valuable turns into a never-ending abyss of tech debt. Tests must be fast and predictable and I learned this the hard way.

In my first role as an SDET, I focused on building a strong local test suite from scratch. Within a month, I had around 50 E2E tests covering most of the application's functionality. They looked great locally. They passed reliably. When I finally hooked them up to CI, everything fell apart. The same tests were now running against different environments, different data, and different performance factors. Instead of building confidence, I suddenly had 50 unstable tests and no pipeline maturity to support them. I ended up retroactively putting out 50 fires instead of starting with a small, reliable subset and building the pipeline correctly from the beginning.

I wish I had spent more time early on learning CI/CD fundamentals and pipeline best practices. Turning a local test suite into a reliable pipeline powerhouse takes skill, and it's a skillset I underestimated for far too long.

Automation is a product and maintenance is the work

This lesson took me longer than I care to admit.

If you start to think about your automation test suite from the same perspective as any product, something starts to change. You quickly realize that you have users just like your business has customers. In this case, those users are developers.

Once you accept that your automation has users, you start designing for them. Readability matters. Supportability matters. Stability matters. Test code is not second-class code. It runs in production pipelines, blocks releases, and influences real decisions. Treating it as "just test code" is how suites rot over time.

Just like a real product, if it is broken, unreliable, or constantly lying, nobody will use it. Treat automation like a real product:

define ownership
classify failure types and severities
track stability and runtime
plan improvements over time with ticketing and planning

Production systems have roadmaps, metrics, and refactoring cycles. Automation needs the same discipline to stay useful.

In QA, we expect the product to meet a high quality bar. If we do not hold our own automation to the same standard, what message are we sending?

Debugging is a skill that must be built on purpose

Debugging is rarely taught.

Most people learn it retroactively through repetition, frustration and mistakes. At this point in my career, troubleshooting is one of the biggest gaps I see between junior and senior engineers.

Debugging is a mindset. While experience helps, you do not need decades to improve it. You can reshape how you debug today.

I like to think of debugging as being equivalent to solving a murder mystery. As a detective, is it easier to solve the case with a hundred suspects or three? The same is true with test failures. How you limit suspects…or variables, matters. This is why starting from a known-good baseline is invaluable. If everything is already broken, you are not debugging. You are firefighting.

Senior engineers practice debugging intentionally. They reflect on their approach. They form hypotheses, rule things out, and trace failures across layers. Debugging deserves the same respect as framework design or test strategy and it's often overlooked until the moment it's needed.

Final thought

It took me years to learn these lessons. In some ways, I'm still learning them.

I've come to appreciate that feeling. It means my assumptions are still being tested. My thinking is still evolving. My feedback loop is still alive.

In a way, it's my own continuous integration. Every failure teaches me something. Every correction makes the system a little stronger.

If I could go back, I wouldn't tell myself to learn another tool.

I mean, I still would. Tools are fun.

But it wouldn't be my first priority.

I would say:

learn how systems fail
learn how teams make decisions
learn how to influence without authority
learn how to think in "quality" rather than thinking in test results

Technical skills grow with time. Impact comes from everything around them.

If you are early in your career and reading this, you are doing better than you think. Feeling uncertain often means you are learning faster than your confidence can keep up.

Even senior engineers feel it.

Especially the ones who say they do not.

Remember, good automation doesn't just catch bugs. It helps teams make better decisions.

Happy testing and learning.

How to Build a QA Team Engineers Actually Want to Work With

David Ingraham — Wed, 15 Apr 2026 00:05:33 +0000

Why quality teams fail, what modern QA actually looks like, and how to build a culture developers trust and want to collaborate with.

If you’ve ever joined a software team where QA was treated like the final boss battle before release, the place where tickets go to die, then you know one of the quiet truths of software development:

Teams don’t struggle with quality because testing is bad.
They struggle because the relationship around testing is broken.

Engineers love quality.
Product managers love quality.
Leadership definitely loves quality.

And yet, despite all that love, “QA” somehow becomes the team people avoid until the very end of the cycle. Not because QA is slow or unhelpful, but because the way we integrate QA into development sets everyone up for friction. When the only time QA enters the conversation is during the final days of a sprint, everything QA finds becomes a roadblock, even if it’s exactly what the team needs to hear.

So the real question isn’t:

“How do we test more, earlier?”

It’s:

“How do we build a QA team engineers actually want to work with?”

That’s a cultural question, not a technical one.
And like any cultural shift, the answer begins long before the first test is ever written.

QA as Partners, Not Police

For years, many QA teams operated under a “late-gate” model, brought in only at the end, tasked with catching what slipped through earlier. That structure often meant QA inherits all risk, all bugs, and all last-minute fire drills.

According to TestRail’s 2025 industry report , late involvement is still one of the top challenges for QA teams, even inside organizations with modern tooling. While nearly 40% of teams report shifting QA earlier in some regards, to address this, a majority still operate inside legacy patterns.

Teams that do shift early see measurable benefits. Those that embed QA early and combine it with automation and CI/CD not only ship faster, they ship with fewer production defects. The improvement isn’t theoretical; it shows up directly in velocity and stability.

That means the problem isn’t just who finds the bug, it’s when the bug is found and whether QA is positioned as a safety net or a strategic collaborator.

Safety nets catch the fall.
Partners prevent the fall.

A Real Story: When Early Collaboration Changes Everything

In a recent talk at CypressConf, I shared a success story that changed the way I think about QA and engineering collaboration.

At my current company, before I joined, QA consisted of a single manual engineer working an eight-hour time difference from the rest of the team. He was excellent, detail-oriented, sharp, and reliable. But the system around him wasn’t. The team constantly faced unpredictable bugs, long regressions, and slow loops of quality feedback. Not because he lacked skill, but because the process wasn’t built for scale or speed. And in startups, time is money. If you can’t learn fast, you can’t survive long.

So my mission was simple:

Bring automation.
Build a Shift Left process.
Create a culture where everyone, not just QA, owns quality

This began with addressing the current team mentality because the hard truth is, the tool you use doesn’t matter if the mindset’s wrong. If the team doesn’t have accountability and buy-in, no framework or tool is going to save you. It sounds simple but it really isn’t. When deadlines tighten, people cut corners, skip conversations, push code as fast as possible and the first thing to diminish is testing and quality. It’s seen as a blocker, something that slows you down instead of helping you move faster.

Sound familiar?

However, Shifting Left isn’t about slowing down at all. It’s about creating the conditions that allow you to sprint consistently long-term. It’s about building a culture where mistakes are safe, where people trust one another, and where failing fast becomes a competitive advantage.

Because if we fail fast, we learn fast.
And if we learn fast, we improve fast.

So that’s where I started. I got to know my coworkers, understood their roles, responsibilities, pain points, and personal definitions of quality. Then together we defined a process everyone agreed on before writing a single line of code or a single test. This early alignment removed future friction, surprises and pushback. People weren’t being forced into a new process; they were excited to build it and be a part of it.

Once that mindset was in place, automation amplified it. Suddenly:

Product and QA collaborated on risks before development began
Developers proactively asked QA for early input
QA wrote automation in parallel with development
Test failures became team conversations
Flaky tests turned into puzzles, not headaches

The tool didn’t fix the culture.
The collaboration did.

And that’s the version of QA developers love working with: the team that helps them ship with confidence, not compliance. Together. But even with a collaborative culture, there’s one moment where QA and engineering collide the hardest…bugs and issues.

Engineers Want Feedback, Not Noise

With all this talk about partnership, we have to address the moment where QA and engineering collide the most: feedback.

There’s a misconception in tech that engineers dislike feedback. Rather, they dislike feedback that is late, vague, incomplete or creates more confusion than clarity.

High-quality QA feedback should be a service, not a verdict. And the way to measure that service is simple:

Does this feedback reduce uncertainty or increase it?

TestRail’s 2025 report found that unclear bug reports remain one of the top workflow inefficiencies for developers, a quiet tax teams pay sprint after sprint. Let’s take a look at a quick example on what bad vs good feedback might look like.

A Bad (Noise) Bug Report:

Title: Checkout broken

Description: Tried to check out — didn’t work.

Steps: Click checkout button

Expected: Should work

Actual: Didn’t

Notes: Happens sometimes.

Ok, this might be a bit of an exaggerated example but the point is hopefully clear. Noise reports don’t look noisy. Noise just means that it lacks everything that matters. The noise happens afterward, in the extensive detective work, ambiguity, Slack threads, and frustration it triggers amongst everyone.

This report doesn’t save time, it creates new work. Compare that to…

A Good (High-Fidelity) Bug Report:

Title: Checkout fails on second payment attempt due to stale session token

Environment: Staging, Safari

Steps:

Add item to cart

Enter payment info

Double-click “Pay” in rapid succession

Expected: Duplicate clicks should be ignored

Actual: Second click sends a stale token → 409 Conflict

Impact: Revenue blocker; affects all Safari users

Artifacts: Screenshot, video, attached

The difference is clear. A High-fidelity feedback builds reliability. Reliability builds trust. Then trust unlocks collaboration. This report allows QA to set the engineering team up for success and mitigate any issues without unnecessary bug clarification.

And this level of clarity shouldn’t just stop at bugs. The best QA teams surface patterns that engineers can’t see from inside the code, like subtle regressions, design inconsistencies, brittle user flows, and scaling issues.

Collaborative teams don’t view this feedback as nitpicking. Rather it’s welcomed as insight. It’s the kind of feedback engineers actively want because it makes their work easier, their decisions safer, and their releases smoother.

The Technical Side of Trust: What Good Automation Looks Like

Up to this point we’ve talked about partnership, and culture, but there’s another dimension that defines whether engineers actually trust (and enjoy working with), your automation.

You can be empathetic, collaborative, and proactive, but if your test suite fails for reasons no one can explain, or breaks every other run, you lose trust immediately. Developers won’t say it out loud, but they think it…

“Why are the tests breaking again?”

“Is this real or flaky?”

A QA team engineers love working with understands that trust isn’t just relational, it’s technical. The relationship sets the foundation, then the technical reinforces it.

Shifting Left is all about thinking ahead; focusing on the high-level so the low-level execution goes smoothly. Test automation is no different and strong automation starts long before a single test is written, the foundation.

Start With the Foundation, Not the Test Count

Many teams rush into automation believing velocity comes from quantity. It doesn’t. A massive test suite built on a weak foundation becomes slow, brittle, and unpredictable. Developers stop trusting failures, and eventually stop looking at them altogether.

Scalable automation begins with intention: reusable helpers, a consistent test structure, clean patterns, solid test data strategy, environment stability, and early conversations about testability. When you invest in the foundation first, reliability stops being an aspiration and becomes the default. Flakes drop. Debugging speeds up. Adding new tests becomes painless. And suddenly, your automation suite becomes what it was always meant to be, a fast, trustworthy source of feedback.

Remember, Shift Left isn’t just about testing earlier; it’s about hearing the truth earlier. And you can only do that when you’re running tests you actually trust.

Protect the Green Baseline

Once you’ve built a foundation you can trust, the next step is protecting it relentlessly. A reliable suite only stays reliable when the team treats every failure with the same urgency, no matter the cause. Flakiness isn’t just an annoyance, it’s a threat to the credibility of your entire pipeline.

Keep failures meaningful. Obsess over the green baseline, over every failure. Treat your test code with the same respect as application code. You wouldn’t knowingly ship broken software to your users; well, with automation, the engineering team is the customer and now you have full control over what they experience.

When QA and engineering hold each other accountable, not merging code with broken tests, not hand-waving flaky failures, not letting noise creep in, the suite becomes something developers rely on instinctively. And when failures mean something, automation becomes one of the most trusted parts of your delivery process.

Make Debugging Effortless to Understand

The next layer of trust comes from visibility. When a test fails, people need to understand what happened without diving into a labyrinth of logs and guesswork. Good automation quickly hands people the answer, or at least the trail of breadcrumbs.

Clear videos. Screenshots. Network logs. A readable stack trace. A test title that sounds like a clear sentence. When context is immediate and available, anyone can help troubleshoot, developers, QA, even that one intern if they’re feeling brave.

Debugging shouldn’t feel like assembling Ikea furniture without the manual so invest in your debugging experience. Transforms failures from individual headaches into shared conversations. When QA empowers developers with clarity, developers feel empowered to help QA in return. It becomes a loop where visibility fuels collaboration.

QA in Every PR

One of the most effective collaboration practices I’ve ever been a part of was a lightweight rule: every PR requires QA approval before merge.

Now, that might raise some eyebrows, so let me clarify what it actually means.

In this model, QA isn’t a gatekeeper, and they’re definitely not blockers. They’re not doing a full engineering-style review or going line by line through the diff (though that can still happen when it’s useful). Instead, QA acts as the team’s final “quality check”, a quick conversation to confirm the change is genuinely ready to become part of the product.

What QA actually looks for is simple:

Are all relevant automated tests passing?
Were any new tests added where they should be?
Has QA pulled the branch and validated the change locally?
Is there anything surprising here that needs to be discussed before merge?

This small practice shifts quality left in the most natural way possible because every change now gets a quality conversation before it lands in the main branch. It eliminates merge surprises. It ensures everyone stays aligned. And it forces QA and engineering to work closely together on everything, creating a genuine partnership.

Most importantly, the conversation is fast. Sometimes it takes a minute. Sometimes it takes a second. Some changes don’t need new tests or deep verification. Others spark deeper discussions, quick brainstorming, or small improvements that would’ve become big problems later. Either way, this tiny ritual prevents hours of rework from untested merges, ensures automation grows with the product instead of becoming tech debt, and creates shared accountability, every change becomes everyone’s change.

And that’s the real shift: quality is no longer something that happens “after development.” It happens with development, on every branch, in every PR, through one simple conversation at a time.

Summary

Across every team and every industry shift, one truth has held steady:

*Quality is not something you do at the end.
It’s something you build into the relationship. *

In my experience, quality cultures emerge from people who care enough to collaborate. Meaning, a great QA team isn’t defined by how many bugs they find or how many tests they automate. They’re defined by how deeply they align with the team around them, how clearly they communicate, how early they collaborate, and how consistently they help people do their best work.

The teams that thrive know how to measure their progress. They look for friction, patterns in failures, breakdowns in communication, blind spots in automation, and when something isn’t working, they pivot early. They adjust, refine, and evolve instead of endure.

And most importantly, they celebrate the wins because quality work is often invisible. They celebrate the first green pipeline. The first stable regression suite. The first time QA and dev debug a failure together. The first release that feels boring… in the best possible way.

Celebration reinforces culture.
Culture reinforces trust.
Trust reinforces quality.

And everything else flows from there.

Be the QA that builds a quality culture. The kind that engineers want to work with.

As always, happy testing and even happier collaborating.

Test Leadership as Change Management: Why Your Real Job Isn’t Testing

David Ingraham — Mon, 03 Nov 2025 16:11:41 +0000

If you’ve ever tried to roll out a new automation framework and been met with blank stares, quiet resistance, or the phrase “we’ve always done it this way”, then you’ve already learned one of the hardest truths about QA leadership:

Testing problems are rarely about testing.
They’re about change.

Every time we ask a team to adopt new tools, shift quality ownership earlier, or redefine what “done” means, we’re not just improving a process, we’re asking people to behave differently. And people don’t resist change because they’re stubborn; they resist it because it threatens what they know, how they work, and sometimes even how they define their value.

As QA leaders, we don’t just test software.
We test how well our teams can handle change.

And now, we’re being tested more than ever because emerging technologies like AI aren’t just changing how we test, they’re changing what it means to be a tester. Especially in an age where a majority of teams don’t view Quality Engineering as strategic.

Meaning, it’s more important than ever to redefine what test leadership really means.

Every QA Initiative Is a Change Initiative

Think about what we actually do as QA leaders today:

Evolve testing practices — Introduce automation over manual testing while defining the overall quality strategy and execution framework for engineers.
Shift quality left — Champion early involvement in design and development so issues are prevented, not just detected.
Bridge teams and perspectives — Act as the liaison between development, product, and leadership, redefining what “quality” means in a culture obsessed with speed.
Coach and elevate others — Develop testers into quality advocates, mentoring teams to own quality beyond their job titles.
Lead cultural change — Create buy-in for new processes and tools, guiding teams through the emotional side of change rather than forcing compliance.

Every one of those moves may involve tools and data but at their core, they’re human.

They influence identity.
They challenge comfort zones.
They redefine what quality means in your organization.

For the first time, tools aren’t just doing the work, they’re thinking alongside us. And that can feel unsettling. One of the most common reactions I hear isn’t fear of the tool, it’s fear of irrelevance.

“If this system can write tests, where do I fit in?”

That’s why the best QA leaders don’t push change harder.
They make it safer.

They understand that leading through new technology isn’t about enforcing adoption, it’s about helping people rediscover their value in a changing landscape. After all, no tool replaces people… but the tester who can lead people through change will always stay ahead of it.

The Emotional Curve of Change

Whether you call it ADKAR, Kübler-Ross, or just “the five stages of rollout panic,” every team goes through a predictable emotional arc during transformation. ADKAR, focuses on the practical side of adoption, what people need to succeed under change. While, Kübler-Ross, originally the grief model, explores the emotional side, how people feel as they adapt.

Together, they remind us that change is both logical and emotion and leaders must guide both.

Here’s what this might look like in testing transformations:

Denial / Awareness — “Our current tests are fine; change won’t actually happen.” People need clarity on why change is necessary — not just what’s changing.
Anger / Desire — “You’re replacing our work with scripts… or worse, an algorithm.” Leaders must turn frustration into motivation by showing personal benefit, not just team goals.
Bargaining / Knowledge — “Maybe this assistant can help a little bit such as flaky test triage.” Curiosity grows once people feel safe to experiment. Training and open demos accelerate this stage.
Depression / Capability — “I’m not sure this is working… maybe this was a mistake.” Fatigue or doubt as the initial excitement wears off. Hands-on support, coaching, and small wins rebuild confidence and capability.
Acceptance / Verification — “Okay, this actually makes our jobs easier.” Hands-on learning builds competence and confidence. This is where adoption begins to stick.

New technologies don’t change the curve, they compress it.
The reactions are the same, but the speed and intensity are higher.

That’s why the best QA leaders treat resistance not as rebellion but as data; signals that show where the team is on the curve and what support they need next.

Instead of fighting resistance, they ask:

“What’s confusing right now?”
“What’s the biggest risk you see in this change?”
“What would make this transition easier for you?”

That empathy builds alignment faster than any roadmap ever will.

The QA Leader’s Change Toolkit

So how do you actually lead through testing change without losing your team’s trust or momentum?

Here’s what separates great change leaders from frustrated ones:

Fear of losing control → Involve people early Co-design rollout plans or test data strategies with the team instead of dictating them. Early ownership creates commitment.
Tooling distrust → Start with pilot wins Don’t announce; experiment. Run small proofs of concept, gather real outcomes, then share those visible successes with the team. Get buy-in one step at a time instead of simply “do this”.
Change fatigue → Normalize iteration Make it clear that every rollout is version 1. Encourage feedback and show that nothing is permanent, it’s all about learning and refining, together.
Lack of communication → Narrate the journey Communicate updates regularly. Share what’s been tried, what failed, and what’s next. Transparency builds confidence faster than perfection.
Tech skepticism → Humanize the role Frame intelligent tools as partners, not replacements. Emphasize that they handle the repetitive parts of testing so humans can focus on strategy, creativity, and context.

Communicate twice as much as you think you need to, then listen twice as hard.

When people feel seen and heard, they’ll follow you anywhere, even through a full AI-driven transformation.

Culture Is Your Test Strategy

At some point, every QA leader realizes: tools are temporary, culture is permanent.

A great testing culture doesn’t happen because of the right frameworks, dashboards, or even our new AI overlords. It happens because people believe in the why behind the practices you lead.

For example, these technologies act as mirrors for your culture. If your team already values curiosity, learning, and experimentation, they’ll thrive with them. If they value control, certainty, and strict boundaries then those cracks will surface immediately.

For example, a 2022 McKinsey Transformation study found that 70% of transformation initiatives fail due to lack of employee adoption and cultural readiness, not strategy or technology. That’s why your biggest success metric isn’t how many AI-powered tests you generate, but rather how confidently your team adopts the next new thing that comes along.

Meaning, when your culture learns to adapt, you stop leading projects; you start leading evolution.

Summary: Testing Change, Not Just Code

The next time you lead a testing transformation, don’t start by asking: “What will we automate?”

Start by asking: “Who will this change impact — and how can I make it safe for them to succeed?”

Because behind every process update or new tool adoption is a person navigating uncertainty. Your role isn’t just to implement technology, it’s to create trust in the transition.

And when it comes to AI or any emerging technology, the real question isn’t “How fast can we use it?”, it’s “How can I help my team trust both the machine — and themselves — as they learn to lead alongside it?”

Every meaningful QA transformation begins with a leader who tests their own assumptions first. Because in the end, change management is quality assurance — for humans, and now, for the intelligent systems learning beside us. So the next time you lead a QA team, take a step back and try the human approach. Test yourself.

And as always, happy testing, through it all.

The Automation Maturity Pyramid

David Ingraham — Wed, 17 Sep 2025 14:11:36 +0000

The Automation Maturity Pyramid

How effective is your automation test suite?
How impactful is it for your product and your team?
Do you know how to grow your test suite without sacrificing quality and performance?

These questions are surprisingly difficult to answer - especially when your entire suite feels like it's constantly on fire, your tests are untrustworthy, and production bugs are popping up like they're going out of style. (Just me?)

To bring some clarity - and because testers love pyramids - I created the Automation Maturity Pyramid as a way to measure automation impact.

First, let's remember why we write automation tests in the first place. At the end-of-the-day, automation tests should support two simple missions:

Increase product quality & confidence
Accelerate development & deployment

So when we think about the pyramid and its phases, everything we do should ultimately align with those missions.

The pyramid has four levels of maturity:

Confidence - Trusting your test results.
Short-Term Impact - Creating value in daily development.
Speed of Development - Scaling automation without slowing down.
Long-Term Impact - Sustaining trust, visibility, and continuous improvement.

Each phase builds on the one below it. Later stages only unlock their benefits once the initial foundation is solid. The pyramid is both tool and type agnostic, meaning you can apply it to any automation suite, framework, or testing type that fits your needs.

Remember, this journey takes time. Think of the pyramid as a compass, not a checklist to rush through. If you're starting fresh, it'll guide you from the beginning. If you already have a suite, it's a framework to measure current impact and decide what to tackle next.

Note: I talked about this topic in-depth (and with more technical examples) in a conference here. If video is more your thing.

Phase 1 - Confidence

A pyramid collapses without a strong base. The same is true with automation. If teams don't trust the test failures (or even successes), everything else becomes meaningless.

When results are unreliable, people stop acting on them. And when tests are ignored, automation loses its purpose. In many ways, unreliable automation is often worse than not having any at all.

The Tests Must Pass

Failures will happen. That's not the issue. The danger is when teams normalize broken tests or flaky failures. Every red test should be taken seriously: investigated, understood, and resolved. While there are exceptions, the default culture must be: stop and fix. Adopt the mindset "all tests must pass", and technical debt will quickly diminish before it starts. A mature automation test suite starts with an accountable mindset.

What Undermines Confidence

Flakiness: Tests that pass or fail inconsistently without code changes. Common causes include race-condition, non-deterministic app behavior, dependent tests or poor test data management.
Environment Instability: Where you will run your tests matter, especially if multiple options are needed. Can you guarantee tests will run reliably across all environments?
Weak Data Strategies: Do tests always have the data they need? Is it static or dynamic? A strong data strategy reduces countless downstream failures.

Phase 1 is about establishing trust. Once failures are credible and environments stable, your suite stops being noise and starts being a safety net. A small, confident test suite is more impactful than a large, unstable one. Some actions items to consider:

Research and implement flake-reduction practices for your tool of choice
Create a culture of accountability: quarantine flaky tests and resolve them quickly
Write tests environment-agnostically
Define a consistent test data strategy that works across environments

If you've done these, you're ready for Phase 2.

Phase 2 - Short-Term Impact

With trust established, the next step is to make automation useful right now. Tests should provide fast feedback and reduce risk during daily development.

If tests only run occasionally or if results arrive too late to act on, they don't influence decision-making. The goal is to make automation an indispensable partner for developers, not a background chore.

This phase is all about defining an initial CI/CD strategy that suites your team's development processes.

CI/CD Strategy

A good rule: the closer tests run to code changes, the more valuable they are. Running suites pre-merge ensures failures tie directly to specific commits, not multiple layers of changes. Fewer variables mean quicker triage.

Nightly or scheduled runs still have a place - especially for full regressions, but the longer the gap between code and results, the harder it is to debug.

Some common strategies:

Pre-merge Tests: Run in under ~10 minutes. Cover critical paths first, then expand with performance in mind.
Full Nightly Regression: Capture broader coverage where speed isn't urgent.
Custom Tag-Based Gates: Sub-groups of tests run based on criteria.

Results Visibility

Running tests is meaningless if no one notices the outcomes. Ensure results are clear, fast, and shared.

Every suite should generate artifacts accessible to all engineers. This includes screenshots, video, error logs and any other additional test information. Without proper artifacts, debugging failures becomes exponentially harder. Additionally, notifications should be immediate and integrated into tools your teams already use.

A professional rule of mine- act like Veruca Salt from Willy Wonka:
"I want those results and I want them now!"

Remember, Phase 2 is about usefulness. Once tests deliver fast, actionable feedback, they directly help teams ship better code, quicker. Developers know within minutes when a real-bug is introduced. Testers know when flake is first introduced, for immediate remediation.

Stick to the mantra: "all tests must pass".

Once you start getting short-term feedback from your tests, it's time to optimize them.

Phase 3 - Speed of Development

Once automation is trusted and embedded in the workflow, the focus shifts to efficiency. The question becomes: how can automation help us move faster without cutting corners?

At small scale, almost any automation adds value. But as suites grow, inefficiency turns automation into a bottleneck. Tests that take hours to run or are painful to debug become blockers instead of enablers. This phase has three areas of focus: writing, debugging and executing tests.

Write Tests Faster

Writing tests faster primarily comes down to test organization and structure. Expanding further:

Standardize Structure: Use any pattern that makes sense to you and don't worry about perfection. Any organization beats spaghetti-code chaos. Optimize over-time.
Reuse Aggressively: Create helpers, builders, and shared libraries for scaleability.
Proactive Test Planning: Review product tickets early to avoid last-minute gaps.
Use AI-assisted Tooling: Just do it. There's no excuse not to use AI anymore. Embrace our new overlords!
Document: Look, we all know it sucks…but providing guides and common gotchas reduce ramp-up time as the team grows. What would past you wish they had when they first onboarded?

Debug Tests Faster

Test failures will happen so response time makes or breaks a suite's value.

Prioritize Readability: Choose clarity over cleverness; smaller, focused tests are easier to diagnose. Always write tests with future you in mind. "Will this make sense to me in six months?".
Reduce Variables: Run tests as close to the change as possible (prioritize pre-merge if not already implemented).
Culture of Accountability: Build a habit of immediate triage: treat all fails with the same urgency so at least some resolution occurs.
Improved Artifact Tools: Interactive runners, browser devtools, and in-depth logs are gold. Improve artifacts as needed.

Run Tests Faster

This one is simple. How fast do our tests run? Repeat after me: "Nobody brags about a three-hour test suite". As the test suite grows, will the team still get quick value without slowing down the process?

Parallelize: Split suites across multiple machines or containers. A must for pre-merge pipelines.
Subset Tests: Run critical paths first; save broader regressions for later. Customize based on need and overall test performance.
Optimize Code: Remove hard-coded waits, reduce unnecessary DOM interactions, apply tool best practices.

Phase 3 is about efficiency. Automation should accelerate delivery, not drag it down. When done well, it enables rapid iteration and frequent, confident releases. All of a sudden our monthly releases can now be reduced to weekly. Then daily. Then maybe even multiple times a day, if you're feeling extra daring. All thanks to your automation test suite.

You deserve a raise.

Phase 4 - Long-Term Impact

The final phase is about sustainability. Once automation is fast, useful, and trusted, it must also deliver long-term value.

Teams and products evolve. Without continuous investment, automation rots: tests get flaky, results get ignored, and the pyramid crumbles. Which is all super sad. Professional advice, don't be sad.

Long-term impact ensures automation remains a source of truth while showcasing just how cool your team is.

Metrics Inform, Not Punish

This phase is purely about responding to metrics, but use them wisely. Metrics should guide investment, not assign blame. Focus on impactful metrics that guide your automation roadmap. Simply, you don't know what to improve if you don't know what's ineffective.

Some Suggestions:

Test Coverage: Directional, not definitive. Pair with quality checks.
Pass/fail and flake rates: Indicators of credibility.
Execution time: Is the suite scaling with the team?
Time-to-resolution (TTR): How quickly do teams fix failures?
Defect detection efficiency (DDE): Percentage of bugs caught by automation.

If possible, consider augmenting these with a dashboard where visibility is further increased. Visual trends make it easier to consume historical trends and identify weaknesses. Plus bar graphs are fun and line graphs always look convincing. Don't even threaten me with a good time and bring up pie charts.

This phase is small but important. It's the culmination of all the previous phases, and purely intended to bring visibility into how well things went in the previous phases. It drives future revisions and ensures the test suite is never stagnant in it's impact.

Phase 4 is all about trust at scale. Mature automation creates transparency, informs investment, and continues to improve over time.

Putting It All Together

The Automation Maturity Pyramid is a lot smaller than the Pyramids of Giza but much more relatable since those are real and in Egypt and this is thought-leadership and about testing. Just to clarify any confusion to this point.

But seriously, it's about measuring your impact, one phase at a time. Building a successful automation test suite is hard without proper guidance. There's many technical steps and failures can quickly become overwhelming and frustrating.

To recap:

Confidence First: You have to trust your tests, always. The rest will follow.
Early Wins: No matter the test suite size, obtain value. Start catching real issues.
Take small steps: Steady improvements compound into big gains. Efficiency is a learning curve and only obtained through experience.
Welcome Failures: Hello failures, come on it. Have a seat. Let's talk about how you're making my current life bad so we can make my future life good.
Celebrate Progress: Building a reliable, impactful suite is a team achievement. Be proud of that green test run, those first 100 tests, or the first real-bug your suite caught. You're a rockstar, genuinely.

Done well, automation isn't overhead - it's a strategic advantage. Build a base of trust, create fast feedback loops, optimize for speed, and commit to long-term transparency. That's how you turn test automation into a driver of product success.

Best of luck in your climb. And as always, happy testing.

Cypress — How to Create Automatic Weekly Flake Alerting

David Ingraham — Fri, 15 Aug 2025 14:18:45 +0000

Flaky tests waste time, erode confidence, and make debugging a nightmare — especially when running in parallel across multiple CI machines.

Cypress Cloud provides fantastic flake detection and historical insights, but what if you don’t have access to it? You’re not out of luck.

In this guide, I’ll show you how to build your own automated flake alerting system that runs weekly, requires no manual effort, and integrates with your existing workflow.

By the end, you’ll have a setup that:

Tracks flake across different environments
Detects trends over time
Automatically alerts your team when thresholds are exceeded

Let’s dive in.

Prerequisite: Merge Report Infrastructure

This tutorial builds upon an earlier post How to Create a Custom Merge Report in Cypress. If you haven’t implemented that yet, start there — we’ll extend that implementation to save, merge and analyze flake data.

1. Save the Flake Results

Once your custom merge-report script is running, let’s expand it to track flake history over time

Add this at the bottom of your merge script:

  // Flake Detection
  const flakeHistoryPath = path.join(resultsDir, 'flake-history.json')
  const flakeHistory = {}

  // Optional: Differentiate by environment
  const env = mergedResults.baseUrl.includes('localhost') ? 'local' : 'dev'
  delete mergedResults.baseUrl

  for (const { test } of flakyTests) {
  flakeHistory[test] = {
    env,
    lastSeen: new Date().toISOString(),
  }

  // Save file
  fs.writeFileSync(flakeHistoryPath, JSON.stringify(flakeHistory, null, 2))
  console.log(`Flake history created at ${flakeHistoryPath}`)

💡 Tip: Tracking the env (like dev or local) helps you isolate which environments are producing flaky results.

Now each test run will produce a flake-history.json file.

Let’s persist it for future aggregation.

2. Add a Weekly Aggregator Pipeline

We’ll now persist the flake history files and process them weekly.

2.1 Persist Flake Files

In your merge workflow, upload the artifact after the script runs. The following example uses Github Actions but any CI/CD tool works.

- name: Upload Cypress Flake Results
  uses: actions/upload-artifact@v4
  with:
    name: flake-history-${{ github.run_id }}
    path: cypress/results/flake-history.json
    retention-days: 7

2.2 Weekly Aggregation Workflow

Next, set up a new pipeline workflow that runs weekly. It will fetch all flake files from our recent runs and prepare them for merging

name: Cypress - Flake Aggregator

on:
schedule:
- cron: '0 9 * * MON' # 3am MDT every Monday, tweak as desired
workflow_dispatch:

jobs:
  aggregate-flake-files:
  runs-on: your-runner # Your runner
  steps:
    - name: Checkout
      uses: actions/checkout@v4

  # 1. Get Workflow Run IDs from past 7 days
    - name: Get Recent Test Workflow Runs
      id: get-cypress-runs
      uses: actions/github-script@v7
      with:
          script: |
            const workflowNames = ['Cypress Tests'] // Your Cypress GH Workflow names
            const sinceDate = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) // 7 days ago
            const allRunIds = []

            const workflowsResp = await github.rest.actions.listRepoWorkflows({
              owner: context.repo.owner,
              repo: context.repo.repo,
            })

            for (const name of workflowNames) {
              const wf = workflowsResp.data.workflows.find(wf => wf.name === name)
              if (!wf) continue

              let page = 1
              let keepFetching = true

              while (keepFetching) {
                const runsResp = await github.rest.actions.listWorkflowRuns({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  workflow_id: wf.id,
                  per_page: 100,
                  page,
                })

                // Fetch all workflows within the last week and persist run ids
                const recentRuns = runsResp.data.workflow_runs.filter(run =>
                  new Date(run.created_at) >= sinceDate
                )

                recentRuns.forEach(run => allRunIds.push(run.id))

                // Stop if there are no more recent runs or we hit end of pages
                if (
                  recentRuns.length < runsResp.data.workflow_runs.length ||
                  runsResp.data.workflow_runs.length === 0
                ) {
                  keepFetching = false
                } else {
                  page++
                }
              }
            }
            return allRunIds
          result-encoding: string

    # 2. Download Artifacts
    - name: Download Flake Artifacts
      run: |
        mkdir -p cypress/flakes // Wherever you stored the results
        echo '[${{ steps.get-cypress-runs.outputs.result }}]' | jq -r '.[]' | while read RUN_ID; do
          artifact_name="flake-history-$RUN_ID"
          echo "Attempting to download $artifact_name from run $RUN_ID"
          if gh run download $RUN_ID -n $artifact_name -D cypress/flakes/; then
            // Rename each file so it's unique
            mv "cypress/flakes/flake-history.json" "cypress/flakes/flake-history-$RUN_ID.json"
          else
            echo "Missing: $artifact_name"
          fi
        done

    # 3. Merge All Flake Files
    - name: Merge Flake History Files
      run: node cypress/merge-flake-history.js

    # 4. Optionally Upload Aggregated Report
    - name: Upload Aggregated Report
      uses: actions/upload-artifact@v4
      with:
        name: aggregated-flake-history
        path: cypress/merged-flake-history.json
        retention-days: 7

Now, every Monday this job collects all flake history files from the past week and merges them.

Here’s the complete process at a glance, from Cypress test runs to automated alerts:

3. Merge the Flake Results

With the workflow in place, create the new merge-flake-history.js script. This script will loop through all the flake files from the past week and aggregate counts by test name and environment.

/* eslint-disable no-undef */
const fs = require('fs')
const path = require('path')

const mergedFlakeHistoryPath = path.join(__dirname, 'merged-flake-history.json')

const mergeFlakeHistories = async () => {
  const flakesDir = path.join(__dirname, 'flakes')
  const flakeFiles = fs.readdirSync(flakesDir).filter((f) => f.endsWith('.json'))

  console.log(`Found ${flakeFiles.length} flake files to analyze`)

  const merged = {}

  // Loop through all flake files
  for (const file of flakeFiles) {
    const contents = fs.readFileSync(path.join(flakesDir, file), 'utf-8')
    const flakeData = JSON.parse(contents)

    // For each flaky test, add to a new merged object 
    for (const testName in flakeData) {
      const { env, lastSeen } = flakeData[testName]

      if (!merged[testName]) {
        merged[testName] = {}
      }

      if (!merged[testName][env]) {
        merged[testName][env] = {
          count: 1,
          lastSeen,
        }
      } else {
        // Count amount of times the test has flaked across all files
        merged[testName][env].count += 1

        const existingDate = new Date(merged[testName][env].lastSeen)
        const newDate = new Date(lastSeen)

        // Use the earliest lastSeen timestamp
        if (newDate < existingDate) {
          merged[testName][env].lastSeen = lastSeen
        }
      }
    }
  }

  // Persist merge flake report
  fs.writeFileSync(mergedFlakeHistoryPath, JSON.stringify(merged, null, 2))
  console.log(`Merged flake history written to ${mergedFlakeHistoryPath}`)
}

mergeFlakeHistories()

Example output:

{
  "Checkout Flow - should display cart": {
    "local": { "count": 2, "lastSeen": "2025-08-05T14:58:03.908Z" }
  },
  "Login - valid credentials": {
    "local": { "count": 10, "lastSeen": "2025-08-05T14:58:03.913Z" },
    "dev":   { "count": 4,  "lastSeen": "2025-08-07T12:40:06.156Z" }
  }
}

4. Alert When Thresholds are Exceeded

Finally, with our merged report, we can trigger alerts when flake counts pass defined thresholds.

Here’s an example using an integration using Linear.

// Define thresholds based on env
const LOCAL_FLAKE_THRESHOLD = 7
const DEV_FLAKE_THRESHOLD = 3
// Any integration secrets
const LINEAR_API_KEY = process.env.LINEAR_API_KEY
const LINEAR_TEAM_ID = process.env.LINEAR_TEAM_ID

// Linear integration
const createLinearTicket = async (testName, count, env, lastSeen) => {
  const body = {
    query: `
      mutation {
        issueCreate(input: {
          title: "Cypress Flaky Test: ${testName}",
          description: "This test flaked ${count} times in the last week on ${env}. First observed: ${lastSeen}",
          teamId: "${LINEAR_TEAM_ID}"
        }) {
          success
        }
      }
    `,
  }

  try {
    const response = await fetch('https://api.linear.app/graphql', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: LINEAR_API_KEY,
      },
      body: JSON.stringify(body),
    })

    const json = await response.json()
    if (json.data.issueCreate.success) {
      console.log(`Linear ticket created for "${testName}"`)
    } else {
      console.error(`Failed to create Linear ticket for "${testName}"`, json)
    }
  } catch (err) {
    console.error(`Error creating Linear ticket for "${testName}"`, err)
  }
}

// After merging results in mergeFlakeHistories 
for (const testName in merged) {
  for (const env in merged[testName]) {
    const { count, lastSeen } = merged[testName][env]
    // Optionally evaluate threshold based on env
    const threshold = env === 'local' ? LOCAL_FLAKE_THRESHOLD : DEV_FLAKE_THRESHOLD

    // Create Linear ticket if flake detected
    if (count >= threshold) {
      console.log(`Flake threshold exceeded: ${testName} (${count} on ${env})`)
      await createLinearTicket(testName, count, env, lastSeen)
    }
  }
}

💡 Tip: Swap out Linear for Jira, Slack, or any other alerting tool your team uses.

A simple, example Linear ticket that’s automatically created might look like this:

Conclusion

With this setup, automatically every week your team gets actionable alerts about the flakiest tests — without anyone manually digging through logs.

You’ll:

Automatically collect and merge flake data
Detect patterns per environment
Alert only when thresholds are breached
Keep your test suite healthier over time

You don’t need a full observability platform to track flake — just a clever CI workflow and a few scripts. For teams needing more depth, Cypress Cloud offers free trials.

Happy testing!

Cypress - Real-Time Multi-User Testing Made Easy

David Ingraham — Mon, 02 Jun 2025 14:00:41 +0000

When building applications with real-time features or collaborative workflows, you'll often encounter scenarios where actions by one user must trigger updates for another. For example:

Testing a chat app where a message from one user should instantly appear for another
Verifying permission boundaries in admin/user portals
Validating notifications or assignments triggered by another account
Testing multiplayer games where player actions affect others

Traditionally, automating these scenarios can seem daunting - you might imagine needing multiple browser tabs, separate user sessions, or even multiple parallel test runners. However, Cypress was designed with a single-browser, single-tab architecture, meaning it doesn't natively support true multi-tab or multi-window testing. Fortunately, there's a much simpler and more reliable way: by leveraging Cypress's API commands, you can simulate multi-user behavior efficiently, all within a single test.

Let's break down how this works using a simple, chat application scenario: we have two users and want to verify that when User 2 sends a message, User 1 receives it instantly. The pattern follows three main steps:

Authenticate both users and store sessions
Track the active API request user
Dynamically set API request headers for any API request

Step 1: Authenticate Both Users and Store Sessions

The foundation of multi-user testing is the ability to switch between users on demand. To do this, you'll first need to authenticate all relevant users up front and securely store their authentication details for later use.

Below is an example login custom command. It logs in as a specific user and saves the necessary authentication cookies. It utilizes cy.session command for session persistence across tests (this is optional but highly recommended for a performative test suite long-term).
Additionally, this setup uses the cypress-wait-until plugin to extend dynamic waiting on the cy.getCookie command which natively doesn't retry. This is also optional but without, it's possible the cy.getCookie command will be performed before the application has created all the appropriate tokens.

import 'cypress-wait-until'

// cy.loginViaAuth0UI custom command
Cypress.Commands.add('loginViaAuth0UI', (user: LoginUsers) => {
  Cypress.log({ message: `Logging in as ${user}` })

  // Set initial value if undefined
  !Cypress.env('userSessionCookies') && Cypress.env('userSessionCookies', {}) 

  // This variable will indicate if we are creating or restoring a session
  // Either way we will be able to store off the current user's cookies
  let initialSessionToken = null

  // Your user credentials
  // Here we assume they are stored in cypress.config.js as an env var
  const { username, password } = Cypress.env('USERS')[user] 

  cy.session(username, () => {
    cy.visit('/') // App landing page redirects to Auth0.

    // Replace with your actual Auth0 or authentication provider URL
    cy.origin(
      'https://your-auth0-url/', 
      { args: { username, password } },
      ({ username, password }) => {
        cy.get('input#username').type(username)
        cy.get('input#password').type(password, { log: false })
        cy.contains('button', 'Continue').click()
      },
    )

    // Ensure we are redirected to the homepage after login
    cy.url().should('include', Cypress.env('HOMEPAGE_URL'), { timeout: 15_000 }) 

    // After initial session creation, store cookies
    waitUntilTokensExist(user)
    initialSessionToken = 'initialSessionToken'
  })

  // If restoring a session, store off restored cookies
  if (!initialSessionToken) {
    waitUntilTokensExist(user)
  }

  // Switch the current user to be the api request user
  Cypress.env('apiRequestUser', user)
}

You'll notice that whether a brand-new session or existing session is restored, the login commands waitUntilTokensExist. Let's check out this helper below

const addTokenIfNotExists = ({ user, cookies }: { user: LoginUsers; cookies: any }) => {
  const tokenStore = Cypress.env('userSessionCookies')

  // Add our cookies to the Cypress.env('userSessionCookies') env var
  if (!tokenStore[user]) {
    tokenStore[user] = {
      cookies,
    }
  }
}

// Helper function that waits until cookies are populated for the current user
const waitUntilTokensExist = (user: LoginUsers) => {
  // Utilizes the `cypress-wait-until' plugin 
  // This guarantees the authenticated cookies exist before storing them
  cy.waitUntil(() => cy.getCookie('XSRF-TOKEN').then((cookie) => Boolean(cookie && cookie.value)), {
    timeout: 25_000,
  }).then(() => {
    cy.getAllCookies().then((cookies) => {
      // Save off cookies if not already added
      addTokenIfNotExists({ user, cookies })
    })
  })
}

waitUntilTokenExists waits until Cypress detects a key authentication cookie (XSRF-TOKEN, for example) after login. Only then does it grab all cookies for the user and save them. This avoids race conditions where Cypress would try to use cookies before they're ready.

Finally, it saves off each user's authentication cookies to a shared storage (userSessionCookies) with a final addTokenIfNotExist helper. By indexing cookies by user, you ensure each user's credentials are available whenever you need to act as that user.

Step 2: Track the Active API Request User

Once you have multiple users authenticated, you need a mechanism to decide which user is "sending" an API request at any given time. This is accomplished by keeping track of the "active" API request user via an environment variable (like Cypress.env('apiRequestUser')).

By creating a simple custom command, cy.switchAPIRequestUser(), which just updates the apiRequestUser variable, switching between users is trivial. Additionally, since this is a custom command, it's accessible throughout all our specs and tests.

Cypress.Commands.add('switchAPIRequestUser', (user) => {
  Cypress.log({ message: `switchAPIRequestUser: ${user}` })

  Cypress.env('apiRequestUser', user)
})

Step 3: Dynamically Set API Request Headers

Lastly, now that our user cookies are stored, whenever we need to send a authenticated API request (e.g., simulating User 2 sending a message), we need to set up each API request with the right authentication headers. These two helpers do the heavy lifting:

const getAuthenticationHeaders = () => {
  // Retrieve the XSRF token and cookies from the environment variables
  // Uses the current request user to get the correct tokens
  const { cookies } = Cypress.env('userSessionCookies')[Cypress.env('apiRequestUser')]

  // Pull key authentication tokens from cookies
  // This might change based on your application's authentication mechanism
  const token = cookies.find((cookie: any) => cookie.name === 'XSRF-TOKEN')?.value
  const session = cookies.find((cookie: any) => cookie.name === 'SESSION')?.value

  // Example of setting authentication headers
  return {
    'Content-Type': 'application/json',
    'X-XSRF-TOKEN': token,
    SESSION: session,
    cookie: `XSRF-TOKEN=${token}; SESSION=${session}; JSESSIONID=${session}`,
  }
}

This function fetches the cookies for the currently selected user (from Step 2) and formats them as headers for an authenticated API request. It abstracts away the manual management of tokens for every request.

const sendRequest = (
  options: Partial<Cypress.RequestOptions>,
): Cypress.Chainable<Cypress.Response<any>> =>
  cy.request({
    ...options,
    headers: {
      ...getAuthenticationHeaders(), // Set authentication headers using the current api request user
      ...options.headers,
    },
    url: `your base api url/${options.url}`, // Replace with your actual base API URL
  })

Lastly, this is a wrapper around cy.request that automatically injects the correct authentication headers. Now, every API call you make can easily impersonate whichever user you've selected with cy.switchAPIRequestUser.

Step 4: Simulate Multi-User Interactions in Your Test

Here's how it all fits together in a real-world scenario - simulating two users in a chat app test:

it('should receive a real-time message from another user', () => {
  cy.intercept('GET', '/api/v1/notifications/incoming').as('interceptGETIncomingMessage')

  // Authenticate with BOTH users first
  cy.loginViaAuth0UI(LoginUsers.user2)
  cy.loginViaAuth0UI(LoginUsers.user1)
  cy.visit('/') // Your app url

  // Stay logged in as user1 but switch the API request user to user2
  cy.switchAPIRequestUser(LoginUsers.user2)

  // Send a message using the API from user2
  cy.sendMessage({ message: 'test message' }).then(() => {
    // Wait on some received API call from user1's perspective
    // Optional based on your application
    cy.wait('@interceptGETIncomingMessage')

    // Validate as user1 we get a real-time incoming message
    cy.get('.message-box').should('be.visible').and('contain', 'test message')
  })
})

Remember these steps:

Authenticate and store both users using your login helper.
Visit your app as User 1 (the main UI context).
Switch API context to User 2 and perform some action via the API.
Wait for the real-time message event and validate it appears for User 1 in the UI.

Using this pattern, you can reliably test real-time, cross-user features - all within one Cypress runner.

Conclusion

Multi-user end-to-end testing in Cypress doesn't have to be complicated. By authenticating all necessary users up front, securely storing their session tokens, and dynamically switching API request context, you can confidently validate real-time and collaborative features.

This approach is faster, more maintainable, and less brittle than juggling multiple browser sessions or tabs. It can easily be extended to more than two users or more complex workflows.

If you're building applications where user-to-user interactions matter, this pattern belongs in your Cypress toolkit.

As always, happy testing!

Cypress — How to Create a Merge Report in your Pipeline

David Ingraham — Tue, 15 Apr 2025 21:46:59 +0000

Cypress Cloud is a powerful tool for tracking test health and identifying long-term patterns. However, for teams without the financial bandwidth to use Cypress Cloud, having access to similar analytics — such as flake percentage, test performance, and failure data — remains essential, especially as test suites grow in size and complexity.

Combine that with a test suite running in parallel, and tracking health across multiple runners can quickly become a time-consuming chore.

In this blog, I’ll break down an easy pattern you can implement to generate a JSON merge report, providing a single place to track test health and custom analytics.

This tutorial assumes your test suite runs in parallel. While it will still work for a single-runner setup, the benefits are magnified when test results are spread across jobs. If you need parallelization, check out the free cypress-split plugin.

To create the test report, follow these four steps:

Save the test results
Merge the test results
Add the script to the pipeline
Use the report

Save the Test Results

To begin, we need to save the results of each individual test run. This can be achieved using the after:run event.

Here’s an example of how to capture and save results, with each step commented along the way.

setupNodeEvents(on, config) {
  on('after:run', (results) => {
    // 1. Define results directory
    const resultsDir = path.join(__dirname, 'cypress/results')

    // 2. Create the results directory, if it doesn't exist
    if (!fs.existsSync(resultsDir)) {
      fs.mkdirSync(resultsDir, { recursive: true })
    }

    // 3. Write the results to a file 
    // Each parallel runner need its own results file. 
    // Use randomUUID to ensure uniqueness
    const fileName = `test-results-${randomUUID()}.json`

    // 4. Write the result to the results directory
    const filePath = path.join(resultsDir, fileName)
    fs.writeFileSync(filePath, JSON.stringify(results, null, 2))
  })

  return config
}

Each parallel runner will now save its own test result file to a shared results folder.

Merge the Test Results

Next, let’s combine those individual files into a single report. The script below will read all result files, analyze them, and generate a final merged report.

Create a new file: merge-results.js

const fs = require('fs')
const path = require('path')
const process = require('process')

// 1. Define the results directory and the merged results file path
const resultsDir = path.join(__dirname, 'results')
const mergedResultsFilePath = path.join(resultsDir, 'merged-results.json')

// 2. Create your custom result format JSON
// Below is an example of example metrics, you can customize it to suite your needs
const overallResults = {
  totalDuration: 0,
  totalTests: 0,
  totalPassed: 0,
  totalFailed: 0,
  totalPending: 0,
  totalSkipped: 0,
  flakePercentage: 0,
}
const flakyTests = []
const slowestTests = []
const errors = []

const mergedResults = {
  ...overallResults,
  flakyTests,
  slowestTests,
  errors,
}

// 3. Helper methods 
// Mine convert milliseconds to minutes for better readability
const convertMSToSeconds = (ms) => (ms / 1000).toFixed(2)
const convertSecondsToMinutes = (ms) => (convertMSToSeconds(ms) / 60).toFixed(2)

// Core Method
const mergeResults = () => {
  // 4. Check if the results directory exists, exit if not
  if (!fs.existsSync(resultsDir)) {
    console.error('❌ No results directory found.')
    process.exit(1)
  }

  const resultFiles = fs
    .readdirSync(resultsDir)
    .filter((file) => file.endsWith('.json') && file !== 'merged-results.json')

  // 5. Loop through and read each individual result file
  // Save off the results into a JSON object
  resultFiles.forEach((file) => {
    const filePath = path.join(resultsDir, file)
    const data = JSON.parse(fs.readFileSync(filePath, 'utf8'))

    // 6. Loop through each test run and extract the relevant metrics
    data.runs.forEach((run) => {
      run.tests.forEach((test) => {
        // Flaky Tests
        // A test is considered flaky if it has multiple attempts and at least one of them passed
        if (
          test.attempts.length > 1 &&
          test.attempts.some((attempt) => attempt.state === 'passed')
        ) {
          flakyTests.push({ test: test.title.join(' - ') })
        }

        // Errors
        // A test is considered to have failed if its state is 'failed' and it has a displayError
        if (test.state === 'failed') {
          const error = test.displayError
          errors.push({
            test: test.title.join(' - '),
            error,
          })
        }

        // Slowest Tests
        // Save off all test regardless of their state
        slowestTests.push({
          test: test.title.join(' - '),
          duration: convertMSToSeconds(test.duration),
        })
      })
    })

    // Overall Results
    mergedResults.totalDuration += data.totalDuration
    mergedResults.totalTests += data.totalTests
    mergedResults.totalPassed += data.totalPassed
    mergedResults.totalFailed += data.totalFailed
    mergedResults.totalPending += data.totalPending
    mergedResults.totalSkipped += data.totalSkipped
  })

  // 7. Now that all files have been processed, we can calculate overall metrics
  // Convert total duration into a readable format
  mergedResults.totalDuration = `${convertSecondsToMinutes(mergedResults.totalDuration)} min`
  // Sort the slowest tests by duration
  slowestTests.sort((a, b) => b.duration - a.duration)
  // Create a overall flake percentage for the run
  mergedResults.flakePercentage = (
    (flakyTests.length / mergedResults.totalTests) *
    100
  ).toFixed(2)

  // 8. Save the merged results to a file
  fs.writeFileSync(mergedResultsFilePath, JSON.stringify(mergedResults, null, 2))
  console.log(`✅ Merged test results saved at ${mergedResultsFilePath}`)
}

// 9. Call the mergeResults function to execute the script
mergeResults()

Add the Script to the Pipeline

Now that the merge-results.js script is created, let’s utilize it in the CI/CD pipeline. Add a new merge job and ensure it runs after all Cypress run jobs have completed. Below is an example using GitHub Actions.

 # Run Cypress Test Job
  cypress-run:
    runs-on: your-runner # Your runner
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4] # Number of parallel jobs, uses cypress-split
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      # 1. Run Cypress
      - name: Run Cypress Tests
        run: npm run cy:run # Your run command
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }} 
      # 2. Upload Cypress Results
      - name: Upload Cypress Results
        uses: actions/upload-artifact@v4
        if: success() || failure()
        with:
          name: cypress-results-${{ strategy.job-index }}
          path: your-folder/cypress/results # Your results folder path
          retention-days: 1

  # Merge Cypress Test Results Job
  merge-test-results:
    runs-on: your-runner # Your runner
    needs: [cypress-run] # Ensure this job runs after all cypress run jobs have finished
    if: success() || failure()
    strategy:
      fail-fast: false
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      # 3. Download Cypress Results, since they're saved from the previous job
      - name: Download Cypress Results
        uses: actions/download-artifact@v4
        with:
          pattern: cypress-results-*
          path: your-folder/cypress/results # Your results folder path
          merge-multiple: true
      # 4. Merge Cypress Results - Call the merge script
      - name: Merge Test Results
        run: node your-folder/cypress/merge-results.js # Your merge script path
      # 5. Upload Merged Results 
      - name: Upload Merged Report
        uses: actions/upload-artifact@v4
        with:
          overwrite: true
          path: your-folder/cypress/results/merged-results.json # Your merged results output file
          if-no-files-found: ignore
          retention-days: 7

The high-level job structure should look something like this:

Use the Report

Each time your pipeline runs, it will now generate a merged JSON file as a pipeline artifact to be downloaded. Here’s a sample output:

// merged-results.json
{
  "totalDuration": "45.21 min",
  "totalTests": 120,
  "totalPassed": 117,
  "totalFailed": 3,
  "totalPending": 0,
  "totalSkipped": 0,
  "flakePercentage": "1.25",
  "flakyTests": [
    {
      "test": "Test Number 1 - should create a new user"
    },
        {
      "test": "Test Number 12 - should delete an existing user"
    },
  ],
  "slowestTests": [
    {
      "test": "Test Group 1 - Test Number 14 - should edit an existing user",
      "duration": "214.14"
    },
    {
      "test": "Test Group 3 - Test Number 2 - should paginate the user list",
      "duration": "174.29"
    }
  ],
  "errors": [
    {
      "test": "Test Group 21 - Test Number 3 - should submit a new bill for an existing user",
      "error": "AssertionError: Timed out retrying after 4000ms: expected '27251' to equal '27,251'"
    },
    {
      "test": "Test Number 13 -  should invoice a user",
      "error": "CypressError: Timed out retrying after 7500ms: `cy.wait()` timed out waiting `7500ms` for the 1st request to the route: `interceptMyUser`."
    }
  ]
}

With this data in a single file, your team can quickly identify flaky tests, bottlenecks, and failures without digging through multiple CI job logs. This report is easy to share and provides direct and clean insight making it crucial as part of the CI/CD troubleshooting process.

Conclusion

By generating your own test analytics, you unlock insight into test reliability and performance — without relying on third-party tools. This kind of visibility is essential to keeping a large test suite healthy and maintainable.

Now that this initial data is available, for future iterations one could consider expanding the logic to:

Integrate alerts for flake thresholds or long runtimes
Track results across multiple runs and upload into an internal dashboard
Identify differences between previous result health to spot immediate regressions

You don’t need a full observability platform to spot trends — just a clever script and a bit of CI magic. For those looking for a bit more or want to experience it's value, Cypress Cloud offers free trials.

Happy testing!

Cypress — How to Create a Merge Report in your Pipeline

David Ingraham — Tue, 15 Apr 2025 21:46:59 +0000

Combine that with a test suite running in parallel, and tracking health across multiple runners can quickly become a time-consuming chore.

In this blog, I’ll break down an easy pattern you can implement to generate a JSON merge report, providing a single place to track test health and custom analytics.

To create the test report, follow these four steps:

Save the test results
Merge the test results
Add the script to the pipeline
Use the report

Save the Test Results

To begin, we need to save the results of each individual test run. This can be achieved using the after:run event.

Here’s an example of how to capture and save results, with each step commented along the way.

setupNodeEvents(on, config) {
  on('after:run', (results) => {
    // 1. Define results directory
    const resultsDir = path.join(__dirname, 'cypress/results')

    // 2. Create the results directory, if it doesn't exist
    if (!fs.existsSync(resultsDir)) {
      fs.mkdirSync(resultsDir, { recursive: true })
    }

    // 3. Write the results to a file 
    // Each parallel runner need its own results file. 
    // Use randomUUID to ensure uniqueness
    const fileName = `test-results-${randomUUID()}.json`

    // 4. Write the result to the results directory
    const filePath = path.join(resultsDir, fileName)
    fs.writeFileSync(filePath, JSON.stringify(results, null, 2))
  })

  return config
}

Each parallel runner will now save its own test result file to a shared results folder.

Merge the Test Results

Next, let’s combine those individual files into a single report. The script below will read all result files, analyze them, and generate a final merged report.

Create a new file: merge-results.js

const fs = require('fs')
const path = require('path')
const process = require('process')

// 1. Define the results directory and the merged results file path
const resultsDir = path.join(__dirname, 'results')
const mergedResultsFilePath = path.join(resultsDir, 'merged-results.json')

// 2. Create your custom result format JSON
// Below is an example of example metrics, you can customize it to suite your needs
const overallResults = {
  totalDuration: 0,
  totalTests: 0,
  totalPassed: 0,
  totalFailed: 0,
  totalPending: 0,
  totalSkipped: 0,
  flakePercentage: 0,
}
const flakyTests = []
const slowestTests = []
const errors = []

const mergedResults = {
  ...overallResults,
  flakyTests,
  slowestTests,
  errors,
}

// 3. Helper methods 
// Mine convert milliseconds to minutes for better readability
const convertMSToSeconds = (ms) => (ms / 1000).toFixed(2)
const convertSecondsToMinutes = (ms) => (convertMSToSeconds(ms) / 60).toFixed(2)

// Core Method
const mergeResults = () => {
  // 4. Check if the results directory exists, exit if not
  if (!fs.existsSync(resultsDir)) {
    console.error('❌ No results directory found.')
    process.exit(1)
  }

  const resultFiles = fs
    .readdirSync(resultsDir)
    .filter((file) => file.endsWith('.json') && file !== 'merged-results.json')

  // 5. Loop through and read each individual result file
  // Save off the results into a JSON object
  resultFiles.forEach((file) => {
    const filePath = path.join(resultsDir, file)
    const data = JSON.parse(fs.readFileSync(filePath, 'utf8'))

    // 6. Loop through each test run and extract the relevant metrics
    data.runs.forEach((run) => {
      run.tests.forEach((test) => {
        // Flaky Tests
        // A test is considered flaky if it has multiple attempts and at least one of them passed
        if (
          test.attempts.length > 1 &&
          test.attempts.some((attempt) => attempt.state === 'passed')
        ) {
          flakyTests.push({ test: test.title.join(' - ') })
        }

        // Errors
        // A test is considered to have failed if its state is 'failed' and it has a displayError
        if (test.state === 'failed') {
          const error = test.displayError
          errors.push({
            test: test.title.join(' - '),
            error,
          })
        }

        // Slowest Tests
        // Save off all test regardless of their state
        slowestTests.push({
          test: test.title.join(' - '),
          duration: convertMSToSeconds(test.duration),
        })
      })
    })

    // Overall Results
    mergedResults.totalDuration += data.totalDuration
    mergedResults.totalTests += data.totalTests
    mergedResults.totalPassed += data.totalPassed
    mergedResults.totalFailed += data.totalFailed
    mergedResults.totalPending += data.totalPending
    mergedResults.totalSkipped += data.totalSkipped
  })

  // 7. Now that all files have been processed, we can calculate overall metrics
  // Convert total duration into a readable format
  mergedResults.totalDuration = `${convertSecondsToMinutes(mergedResults.totalDuration)} min`
  // Sort the slowest tests by duration
  slowestTests.sort((a, b) => b.duration - a.duration)
  // Create a overall flake percentage for the run
  mergedResults.flakePercentage = (
    (flakyTests.length / mergedResults.totalTests) *
    100
  ).toFixed(2)

  // 8. Save the merged results to a file
  fs.writeFileSync(mergedResultsFilePath, JSON.stringify(mergedResults, null, 2))
  console.log(`✅ Merged test results saved at ${mergedResultsFilePath}`)
}

// 9. Call the mergeResults function to execute the script
mergeResults()

Add the Script to the Pipeline

 # Run Cypress Test Job
  cypress-run:
    runs-on: your-runner # Your runner
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4] # Number of parallel jobs, uses cypress-split
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      # 1. Run Cypress
      - name: Run Cypress Tests
        run: npm run cy:run # Your run command
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }} 
      # 2. Upload Cypress Results
      - name: Upload Cypress Results
        uses: actions/upload-artifact@v4
        if: success() || failure()
        with:
          name: cypress-results-${{ strategy.job-index }}
          path: your-folder/cypress/results # Your results folder path
          retention-days: 1

  # Merge Cypress Test Results Job
  merge-test-results:
    runs-on: your-runner # Your runner
    needs: [cypress-run] # Ensure this job runs after all cypress run jobs have finished
    if: success() || failure()
    strategy:
      fail-fast: false
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      # 3. Download Cypress Results, since they're saved from the previous job
      - name: Download Cypress Results
        uses: actions/download-artifact@v4
        with:
          pattern: cypress-results-*
          path: your-folder/cypress/results # Your results folder path
          merge-multiple: true
      # 4. Merge Cypress Results - Call the merge script
      - name: Merge Test Results
        run: node your-folder/cypress/merge-results.js # Your merge script path
      # 5. Upload Merged Results 
      - name: Upload Merged Report
        uses: actions/upload-artifact@v4
        with:
          overwrite: true
          path: your-folder/cypress/results/merged-results.json # Your merged results output file
          if-no-files-found: ignore
          retention-days: 7

The high-level job structure should look something like this:

Use the Report

Each time your pipeline runs, it will now generate a merged JSON file as a pipeline artifact to be downloaded. Here’s a sample output:

// merged-results.json
{
  "totalDuration": "45.21 min",
  "totalTests": 120,
  "totalPassed": 117,
  "totalFailed": 3,
  "totalPending": 0,
  "totalSkipped": 0,
  "flakePercentage": "1.25",
  "flakyTests": [
    {
      "test": "Test Number 1 - should create a new user"
    },
        {
      "test": "Test Number 12 - should delete an existing user"
    },
  ],
  "slowestTests": [
    {
      "test": "Test Group 1 - Test Number 14 - should edit an existing user",
      "duration": "214.14"
    },
    {
      "test": "Test Group 3 - Test Number 2 - should paginate the user list",
      "duration": "174.29"
    }
  ],
  "errors": [
    {
      "test": "Test Group 21 - Test Number 3 - should submit a new bill for an existing user",
      "error": "AssertionError: Timed out retrying after 4000ms: expected '27251' to equal '27,251'"
    },
    {
      "test": "Test Number 13 -  should invoice a user",
      "error": "CypressError: Timed out retrying after 7500ms: `cy.wait()` timed out waiting `7500ms` for the 1st request to the route: `interceptMyUser`."
    }
  ]
}

Conclusion

Now that this initial data is available, for future iterations one could consider expanding the logic to:

Integrate alerts for flake thresholds or long runtimes
Track results across multiple runs and upload into an internal dashboard
Identify differences between previous result health to spot immediate regressions

You don’t need a full observability platform to spot trends — just a clever script and a bit of CI magic. Looking forward to hearing your metric implementations.

Happy testing!

Cypress - 7 Features I Wish the Native Test Runner Had (And How to Work Around Them)

David Ingraham — Fri, 07 Mar 2025 03:38:50 +0000

Cypress is one of the best tools for end-to-end testing, offering a fast, developer-friendly experience with built-in waiting, an intuitive API, and great debugging capabilities. However, like any tool, it has its limitations. Some of these are intentional design decisions, while others feel like omissions that would significantly improve the native test runner.

Fortunately, the strong Cypress community has developed plugins, workarounds, and best practices to help overcome these challenges. Below, I’ll outline seven features I wish Cypress had natively and the ways to work around these shortcomings.

1. Parallel Test Execution (without Cypress Cloud)

Cypress supports parallel test execution, but only if you use the Cypress Cloud Service. While Cypress Cloud is a fantastic, paid feature that provides a slick visual dashboard for improved debugging, analytics, and overall test health, parallelization bundled with the service feels like an unnecessary variable in today’s testing world. With other major testing competitors offering this feature out-of-the-box and considering how high-importance executing tests simultaneously is, this limitation is a big blocker for many teams. Thankfully, there are plugin workarounds to achieve parallel test execution without Cloud.

Workarounds:

cypress-split — easy installation and use. Updated frequently.
cypress-parallel — similar to split but frequently less updates.

2. Extended Test Filtering

Native Cypress CLI commands only allow filtering by spec files, meaning there is no built-in way to tag or filter tests based on test name or labels like @smoke or @regression. This is limiting for those that want to avoid organizing their test specs and folders entirely based off of test runs. Additionally, teams may need to run subsets of tests within the same spec file at different points in the Software Development Lifecycle. Once again, there are plugins to help achieve this.

Workarounds:

Install the @cypress/grep plugin to enable filtering on titles or tags
Install the cypress-grep plugin. Similar to @cypress/grep but more frequently updated.
Install the cypress-cli-select plugin to easily execute specific specs by file, name or tag in the cli. Built off of cypress-grep.

3. Test Runner Filtering (Live Filtering in UI)

Continuing the filtering trend, there’s no easy way to execute a single test or group of tests within the Test Runner itself. The workaround to this is to go into your code and add a .only to the it or describe methods but this requires back and forth between the test runner and the code. For those encouraging non-QA engineers to run Cypress tests locally, having a way to filter tests without knowing the test code is useful. Simply, the Test Runner does not offer a UI-based way to filter tests dynamically while running them.

Workarounds:

Install the cypress-plugin-grep-boxes plugin to start filtering within the test runner without code changes. Built off of cypress-grep.
Install the cypress-plugin-last-failed plugin to easily re-run the last failed test(s). Also supports CLI. Built off of cypress-grep.

4. Official Safari/WebKit Support

Cypress officially supports Chrome, Edge, and Firefox, but not Safari or WebKit, making it difficult to test apps in an environment closer to Apple’s ecosystem. However, despite a lack of official support, there’s experimental support built on top of the playwright-webkit plugin.

This is a great option that is better than nothing but there are still a handful of known issues that could be blockers for a complicated test suite, such as missing cy.origin support.

Workarounds:

Install playwright-webkit and add experimentalWebKitSupport: true to your config file. Run Cypress as usual using the --browser webkit cli param.
Pray you don’t need to test against Safari…kidding but sometimes hard-blockers require alternative testing tools as a last result.

5. Multi-Browser Support

Cypress is designed to run within a single browser tab, making it difficult to test workflows that involve multiple tabs or popups. There’s no easy command to switch between multiple tabs or browser contexts to accomplish multi-user functionality such as a chat box. That being said, Cypress has a ton of workarounds that still provide full-test confidence without needing to truly simulate two browser context.

Workarounds:

Use cy.request to simulate receiving requests from another user.
Use @cypress/puppeteer plugin to support switching to a new tab
Validate the new tab without truly redirecting to the new tab

6. Real User Interactions

Cypress primarily relies on synthetic events for interacting with the DOM, which typically achieve most common, user interactions. However some events, such as simulating hover states and drag-and-drop actions, aren’t easily achievable with this default behavior. Once again, the community has provided a handful of useful plugins to make user interactions a bit easier.

Workarounds:

Utilize the cypress-real-events plugin to fire real, native events such as realHover and realClick. Only works for chromium-based browsers.
For drag and drop, use cypress-drag-drop or manually dispatch mousedown/mousemove/mouseup events.

7. Command Retry Logic

Cypress has fantastic built-in assertion retry-ability but currently does not support retries for non-query actions, such as commands. On theme, there are a handful of workarounds to extend the default Cypress functionality including a must-use plugin.

Workarounds:

Use the cypress-wait-until plugin for retrying non-query logic. Adds waiting power to literally everything.
Utilize the cypress-recurse plugin to retry custom commands until a predicate function returns true. Another great alternative.
Convert your custom command into a custom query instead which supports retrievability. Cypress v12 and above, not recommended for every command.

Conclusion

While Cypress is an incredibly powerful testing tool, it has limitations that can sometimes slow down test automation efforts. Thankfully, the strong community and ecosystem of plugins provide workarounds for most missing features. It just requires a bit of searching and additional setup.

As Cypress continues to evolve, I hope to see more of these features become natively supported. Until then, hopefully this list serves as a useful conglomeration of workarounds to help make the most of your Cypress test automation efforts.

As always, happy testing!🚀

Cypress: (Help)ers Wanted — When and Where to Use Them

David Ingraham — Fri, 07 Mar 2025 03:24:16 +0000

When organizing your Cypress helpers, should they go in a utility file, become a custom command or remain inline? Do I even need a helper, or can I repeat the functionality in multiple specs? These questions might seem small, but they’re crucial as your test suite grows.

As your automation test suite grows, organizing helpers becomes critical. Without proper structure, you risk creating inconsistent implementations, duplicated code, and increased maintenance overhead. These challenges lead to slower development cycles and decrease confidence in the test suite’s reliability. A well-structured and predictable framework enables multiple engineers to work collaboratively, ensuring consistency across tests. But how do you know where to define a test helper?

Before we get to the where, let’s discuss the why. Helpers in automation testing service two primary purposes:

Encapsulating common or repeated operations to reduce duplicated code.
Improving readability by abstracting complex or lengthy logic.

As a result of this, all helpers you write should address one or both of these benefits. Reducing duplication — following the Don’t Repeat Yourself(DRY) principle — is a common software development practice that should carry over into automation testing. While DRY can be taken to extremes (where maintaining the abstraction becomes more work than its value), automation testers should focus on removing duplicated code to improve maintainability, scalability, and predictability.

Additionally, some helpers might just be useful for readability. For example, instead of populating a form with ten (or more) fields directly in a test, you might prefer a single call to populateNewUserForm(). This abstraction makes the test cleaner and easier to understand at a glance. In the example below the test is easy to read at a high-level due to the helper but still provides the user easy access to it if needed.

const populateNewUserForm = () => {
  cy.get('first-name-input').type('David');
  cy.get('last-name-input').type('Ingraham');
  cy.get('gender-select').select('male');
  cy.get('email-input').type('test@gmail.com');
  cy.get('password-input').type('happy testing');
  cy.get('address-input').type('123 Testing Street');
  cy.get('address-2-input').type('Unit 1');
  cy.get('city-input').type('Denver');
  cy.get('state-select').select('Colorado');
  cy.get('zip-input').type('12345');
};

describe('Create User', () => {
  it('should create a user', () => {
    cy.intercept('user', 'POST').as('postUser');
    cy.visit('users');

    // Populate new user form
    cy.get('create-user-button').click();
    populateNewUserForm();

    // Submit User
    cy.get('submit-button').click();
    cy.wait('@postUser').then(({ response }) => {
      expect(response?.statusCode).to.eq(201);
    });
  });

Once you’ve identified the use for a helper, the next step is deciding where to define it. In Cypress, there are three primary options, each suited for different scenarios.

Single Spec

Helpers that are only relevant to one test or spec file should be defined within that file. This keeps the scope limited and avoids unnecessary abstraction.

An example of this might be a populate helper unique to a single spec or a re-used piece of logic unique to that file. We can expand our example above and add another single, spec helper called waitForUserTableToLoad since it’s used multiple times only for this one test.

const populateNewUserForm = () => {
  cy.get('first-name-input').type('David');
  cy.get('last-name-input').type('Ingraham');
  cy.get('gender-select').select('male');
  cy.get('email-input').type('test@gmail.com');
  cy.get('password-input').type('happy testing');

  cy.get('address-input').type('123 Testing Street');
  cy.get('address-2-input').type('Unit 1');
  cy.get('city-input').type('Denver');
  cy.get('state-select').select('Colorado');
  cy.get('zip-input').type('12345');
};

const waitForUserTableToLoad = () => {
  cy.wait('getUsers');
  cy.get('user-table-loading-spinner').should('not.exist');
  cy.get('user-table').should('be.visible');
};

describe('Create User', () => {
  it('should add a user', () => {
    cy.intercept('user', 'GET').as('getUsers');
    cy.intercept('user', 'POST').as('postUser');

    cy.visit('users');
    waitForUserTableToLoad();

    // Populate new user form
    cy.get('create-user-button').click();
    populateNewUserForm();

    // Submit User
    cy.get('submit-button').click();
    cy.wait('@postUser').then(({ response }) => {
      expect(response?.statusCode).to.eq(201);

      waitForUserTableToLoad();
      cy.get('user-table').contains('David Ingraham');

      // Validate user persists after reload
      cy.reload();
      waitForUserTableToLoad();
      cy.get('user-table').contains('David Ingraham');
    });
  });

The helper reduces duplicated code while keeping the test readability, two traits that are once again crucial for maintaining a test suite as it grows. If either one of these helpers are needed across multiple specs, then you might want to consider moving it to an external helper file instead.

Helper File

If a helper is used across multiple spec files, it should be defined or moved to a shared helpers or utils folder within your Cypress project. This centralizes logic, making it easier to maintain and update. An example of a shared helper could include anything ranging from re-used setup code, validations or calculations.

// Helper.ts

// Shared Navigation Helper
export const navigateToUserPage = () => {
  cy.intercept('user', 'GET').as('getUsers');

  cy.login() // Login - Custom Command
  cy.visit('users');
  cy.wait('@getUsers');
};

// Shared Validation Helper
export const validateUserTable = (user: User) => {
  const { firstName, lastName } = user;

  cy.get('user-table').contains(`${firstName} ${lastName}`);
};

// Shared Calculation Helper
export const calculateTotal = (price: number) => {
  const tax = 0.12;
  const shipping = 5.99;
  const priceTax = tax * price;
  return price + shipping + priceTax;
};

All three of these examples are now available to be imported into multiple specs across our test suite. If any issues or updates occur, updating them is a breeze as we only have to make changes in a single place without worrying about touching multiple test files.

Custom Commands

Finally, Cypress is unique because it provides a built-in helper called a Custom Command. They are ideal for helpers that represent common actions, validations or setup that’s shared across a majority or all specs and tests. Cypress Commands offer several advantages over plain javascript helper files including:

Global Availability: No need to import them into every spec file.
Chaining Support: Ability to chain off other Cypress Commands.
Return Values: Able to return a value using .then

If I have the following login function and defined it as both a Cypress Command and a Javascript helper, it might look like this.

// Custom Command 
Cypress.Commands.add('login', (user) => {
  const { username, password } = user;
  cy.session(username, () => {
    cy.request({
      method: 'POST',
      url: '/login',
      body: { username, password },
    }).then(({ body }) => {
      window.localStorage.setItem('authToken', body.token);
    });
  });
});

// JS Helper
export const login = (user) => {
  cy.session(username, () => {
    cy.request({
      method: 'POST',
      url: '/login',
      body: { username, password },
    }).then(({ body }) => {
      window.localStorage.setItem('authToken', body.token);
    });
  });
};

While the code is nearly identical, utilizing this code in a spec is a bit different. For a Cypress Custom Command, I have access to my login command by calling cy.login as it’s now globally available to all my files versus having to import my login JS helper across all specs. Further, if my login helper returns user information, I can easily access it by doing:

it('some test', () => {
  // You CAN do this
  cy.login((userInfo) => {
      // If I need to use `userInfo` in my tests, I can do so
  })

  // You CANT do this
  const userInfo = login() 
})

Accessing a returned value isn’t possible with a regular helper due to the asynchronous nature of how Cypress works.

Note: Cypress also has the option to create Custom Queries. These are type of command that follow a slightly different set of rules than Custom Commands. These won’t be discussed specifically, but they are available as needed.

The Custom Commands I define fall into one of the following three categories API, Validation and Action.

API Custom Commands

These are helpers that simply wrap API calls which are needed for test setup. Programmatically controlling test data using cy.request is a best practice for maintaining test data reliably. Defining this setup in a custom command allows any tests that utilize it to access any created API data as needed. For example, let’s define a Custom Command that programmatically creates a user for our test and allows us to access the created user response for further validation and use.

// Create User POST Custom Command
Cypress.Commands.add('createUser', () => {
    Cypress.log({ message: 'createUser' })

    return cy.request({
      method: 'POST',
      url: '/user',
      body: { username, password },
    }).then(({ response }) => response.body
});

// Test Example
it('should edit an existing user', () => {
  // Programatically create the user and fetch the response 
  cy.createUser().then((createdUser) => {
    const { firstName } = createdUser

     // Use the real response to validate application data
     cy.get('edit-user-button').click()
     cy.get('first-name-input').should('have.value', firstName)
  })
})

For more information on this pattern, check out my article here.

Validation Custom Commands

These type of custom commands are shared assertions reused across all test files. The benefit of a custom command is that these might be validations that can be chained off a previous cypress selector command.

An example of this could be validateMenuTabLists which is a helper that validates a particular page has the correct tab list and text. The implementation might look something like this:

// Validation Custom Command 
Cypress.Commands.add('validateMenuListTabs', { prevSubject: 'element'}, (subject, tabs) => {
  // Validate list tab length
  cy.wrap(subject).find('[role="tab"']).should('have.length.of', tabs.length)
  // Validate individual tab texts
  cy.wrap(subject)
    .find('[role="tab"]')
    .each((tab, index) => {
      cy.wrap(tab).should('contain', tabs[index])
    })
})

// Test Example
it('should validate user page', () => {
  cy.get('user-page-menu-list').validateMenuListTabs(['My Info', 'Billing', 'Settings'])
})

This is extremely valuable as validating any tab list in my application is now simple and accessible. Defining common validations is powerful because you additionally gain confidence that all your elements are verified the same.

Action Custom Commands

Finally, action command are those that do something throughout the test.

A popular example might be defining a custom, cy.get command such as getByTestId. In this example, the command forces the user to grab an element using a custom, data-test attribute instead of other methods. Another example could be defining a custom dropdown, select helper for a particular framework such as selectPrimeVueOption for the PrimeVue component library. Defining custom commands that slightly tweak the default Cypress command behavior to work with your preferred framework and patterns provides a huge benefit across all of your tests.

// Action Custom Command - Customized Get
Cypress.Commands.add('getByDataTest', (dataTest, options) => {
  return cy.get(`[data-test=${dataTest}]`, options)
})

// Action Custom Command - Customized Dropdown Select
Cypress.Commands.add('selectPrimeVueOption', { prevSubject: 'element'}, (subject, option) => {
  cy.wrap(subject).scrollIntoView()
  cy.wrap(subject).find('[data-pc-section="dropdown"]').as('dropdownButton')
  cy.get('@dropdownButton').click() // expand dropdown

  cy.get('.p-select-list').within(() => {
      cy.contains(option).click() // Select dropdown option
  })
})

// Test Example
it('should select user data', () => {
  cy.getByDataTest('user-age-select').selectPrimeVueOption('30')
})

Conclusion

Wherever you put your helper, it should always enhance your test suite’s maintainability, scaleability and readability. Good helpers provide these benefits while poorly structured helpers add unnecessary complexity. Ultimately though, there’s not one right answer, just suggested organization. Use your best judgement based on the scope of impact and level of effort to maintain the helper if something changes. Having this awareness and purpose for all your Single Spec helpers, Helper Files, or Custom Commands, creates a robust and organized Cypress test suite that grows gracefully with your application and sets your team up for long-term success.

const closingMessage = () => {
  console.log('Thanks for Reading, Happy Testing.')
}

it('connect with me on LinkedIn and say hi', () => {
   closingMessage()
})