DEV Community: Murat K Ozcan

The testing meta most teams have not caught up to yet

Murat K Ozcan — Sat, 10 Jan 2026 21:22:19 +0000

A QA operating system for AI testing

This post is for anyone shipping Playwright tests and watching AI output rot in review. I built this to reduce review bandwidth and stop AI from guessing - it is a QA operating system built from real test architecture, not prompt luck.

Quick links:

@seontechnologies/playwright-Utils: Playwright DX and reusable utilities
BMAD Method: agentic development framework
TEA overview: BMAD Test Architect workflows

TL;DR: TEA (Test Architect) encodes test strategy and release gates, Playwright-Utils fixes Playwright DX, and Playwright MCPs give AI live verification. Together, they turn AI testing from promptware into repeatable engineering by standardizing patterns and verifying against the real system.

If your AI tests look productive but rot in review, you do not have a test suite, you have a slop factory.
By slop I mean redundant coverage, wrong assertions, nondeterministic flows, and unreviewable diffs.

I stopped trusting AI test generation the day it could not explain my own quality standards, so I embedded them.

Fixing DX first: Playwright-Utils

The first thing I addressed was Playwright DX. Playwright-Utils bridges the gap between Cypress-like ergonomics and Playwright's power with a functional core and fixture shell design. Many capabilities work as a pure function or as a fixture, for UI as well as API/backend flows. It standardizes the primitives teams drift on: auth, API, retries, logging, files, etc.

The brain: TEA (Test Architect agent from BMAD)

BMAD TEA (Test Architect) is a quality operating model as an agent: risk-based test design, fixture architecture, CI/CD gates, and release readiness packaged as workflows. I'm a certified Siemens test architect. TEA is how I made my quality standards executable - workflows instead of tribal memory.

TEA spans eight workflows: *framework, *ci, *test-design, *atdd (acceptance test driven design), *automate, *test-review, *trace, and *nfr-assess - everything from strategy to gates.

The hands and eyes: Playwright MCPs

Playwright MCPs arrived and turned diagnosis, execution, and fixing into a live loop. MCPs close the loop: the agent can run the flow, confirm the DOM against the accessibility tree / the network responses against the API spec, and use Playwright's Trace Viewer/UI mode as the verification canvas. For backend behavior, API flows are validated via api-request plus recurse, with an optional UI mode. Later, Playwright shipped its agent prompt packs, and I folded that guidance into TEA as an optional accelerator.

How it works in practice

TEA produces the test strategy and gate criteria for the feature.
TEA generates tests using Playwright-Utils patterns (auth, retries, API, logging, etc.).
MCPs verify UI flows / API behavior against the live system.
Failures feed back into TEA to repair, using the same standards.
Output: PR-ready tests plus traceability and gate artifacts (risk table, requirements-to-tests traceability matrix, release gate checklist).

Before example: 20 AI tests -> 12 flaky, 5 redundant, 3 wrong assertions
After: risk plan + gates -> P0/P1 only -> selectors/API calls verified -> behavior validated -> fewer tests, higher signal

Why this stack matters

Playwright is intentionally unopinionated, which means no two projects look the same and drift happens fast, even inside the same org. It optimizes for stability and speed over standardized DX, so teams reinvent the same primitives. Every team reinvents auth, API calls, retries, and logging. Then the AI arrives at a mess, and a mess produces slop.

Meanwhile, prompt-driven testing paths like cy.prompt lean into nondeterminism, which is the exact opposite of what testing exists to protect. The fix is not a better prompt; it is a better system: TEA to encode the standards, Playwright-Utils to enforce the patterns, and MCPs to verify UI flows against the live system, with api-request plus recurse covering backend behavior.

What you get out of the box

Playwright-Utils provides nine utilities. Six are backend or frontend agnostic: api-request (with schema validation), recurse, log, file-utils, burn-in, and auth-session. Three are UI-focused: network-error-monitor, network-recorder, and intercept-network-call.

TEA uses those patterns by default when you enable the integration in the BMAD installer, so the AI does not have to relearn the same documentation every time. That is context engineering in practice.

Call to action

This stack is already open source and being used in production at SEON technologies. If you want to level up your craft, use it. If you want to push it further, contribute. If you want to build with people who care about quality at this level, reach out.

Turkish and Functional Programming: A Surprisingly Perfect Match

Murat K Ozcan — Sat, 22 Feb 2025 19:56:02 +0000

While watching Zoe.languages' "10 Fun Facts about the Turkish Language", I had an unexpected realization—Turkish feels a lot like functional programming. It’s concise, structured, and beautifully logical. Both Turkish and functional programming avoid unnecessary elements, embrace efficiency, and favor a streamlined, harmonious approach. Let’s break down why Turkish and functional programming are a perfect match.

1. No Redundant Elements (Minimalism)

Functional programming aims to eliminate unnecessary complexity, and Turkish does the same by avoiding redundant words.

Turkish: Instead of explicitly stating pronouns, Turkish encodes them in the verb itself.
- Geliyorum = I am coming (no separate "I" needed because "-yorum" already implies it).
Functional Programming: Functions are "pure" and avoid external dependencies, making them self-contained and efficient.
- Example:
```
const greet = name => `Hello, ${name}!`;
console.log(greet("Ali"));
```
No extra variables, no global state—just clean, functional style.

2. Agglutination = Function Composition

In both Turkish and functional programming, smaller building blocks are combined to create more powerful expressions.

Turkish: Words are built by adding suffixes in a modular way.
- Ev (house) → Evler (houses) → Evlerimiz (our houses) → Evlerimizde (in our houses).
Functional Programming: Functions are composed instead of relying on step-by-step mutations.
- Example:
```
const add = x => y => x + y;
const double = x => x * 2;
const addAndDouble = x => double(add(2)(x));
```
Just like Turkish suffixes, functions stack elegantly without breaking the structure.

3. Harmony & Consistency

Turkish: Has vowel harmony, meaning suffixes adapt to the root word, ensuring smooth pronunciation.
- Evimizde (in our house), Köyümüzde (in our village) — the suffixes follow a predictable rule.
Functional Programming: Ensures consistency through immutability and declarative logic.
- If a function works in one context, it works everywhere—it doesn’t mutate unexpectedly.

4. No Extra Noise (Declarative Style)

Turkish: Instead of unnecessary words like "is," "are," or "do," Turkish expresses meaning through the verb structure.
- Geliyorum (I am coming) instead of "Ben geliyorum." The subject is already implied.
Functional Programming: Uses declarative expressions instead of step-by-step instructions.
- Example:
```
const numbers = [1, 2, 3, 4, 5];
const squared = numbers.map(x => x * x);
```
No for loops, no manual mutations—just pure transformation.

Final Thoughts

Turkish and functional programming share an appreciation for elegance, efficiency, and logic. Both remove unnecessary clutter, rely on structured patterns, and create fluid, harmonious expressions. Whether you’re learning Turkish or diving into functional programming, you might find that one helps you appreciate the other.

Who knew that a language spoken by millions and a programming paradigm could have so much in common?

Page Objects vs. Functional Helpers

Murat K Ozcan — Sat, 22 Feb 2025 12:44:13 +0000

Page Objects vs. Functional Helpers

A while ago, I published Functional Programming Test Patterns with Cypress, where I went in-depth on why page objects are unnecessary in modern test automation. Back then, I didn’t realize just how ahead of its time that take was—since many people still insist on using page objects today.

This post is a simpler, updated version of that argument.

🚀 Why Functional Helpers > Page Objects?

The Page Object Model (POM) follows inheritance, while functional helpers follow composition.

But modern web apps are built with component-based architecture, where components are the real building blocks—not pages.

❓ If components compose and pages are just collections of them, does it really make sense to abstract pages with classes? Or does it introduce unnecessary duplication and over-abstraction?

In modern testing frameworks like Playwright and Cypress, strict Page Object Model (POM) is often overkill, especially when:

✅ You’re using data selectors (data-qa, data-cy) for stable locators.
✅ The tools already offer powerful built-in utilities for UI interactions.
✅ POM introduces extra complexity that makes debugging harder.

❌ Why Page Objects No Longer Make Sense

🔙 Why POM Needed Classes in the Past

It’s worth acknowledging that POM wasn’t originally a bad idea—in the Selenium/WebDriver days, classes were necessary to manage workflows across different pages because test frameworks lacked built-in ways to handle:
• State → A base page maintained shared properties like driver (in Selenium) or page (in Playwright).
• Inheritance → Page classes extended a BasePage to inherit common navigation, helper methods, or interactions.
• Encapsulation → Pages wrapped UI interactions to abstract implementation details and expose only relevant actions.

At the time, this structure solved real problems—but modern frameworks already provide solutions for these things, making POM mostly redundant.

For example, Playwright already manages state with context.newPage(), provides a clean API for interactions, and supports direct functional composition without forcing abstraction layers.

📌 POM was useful when test frameworks were low-level, but today? The problems it solved either no longer exist or are solved better without it.

1️⃣ Unnecessary Abstraction

POM adds an extra layer that often doesn’t provide feasible value.
Modern test frameworks are already powerful enough without it.

2️⃣ Base Page Inheritance is Overkill

Having a BasePage class with generic methods (click(), fill()) just to wrap Playwright’s API (or Cypress) makes no sense.
Playwright (or Cy) already has page.locator(), page.click(), page.fill(), etc.

3️⃣ Harder Debugging

With POM, if a test fails, you have to jump between multiple files to figure out what went wrong.
With direct helper functions, you see exactly what’s happening.

🔴 Traditional Page Object Model (POM)

🚨 Problems with POM:
❌ Unnecessary complexity → Extra class & inheritance
❌ Harder debugging → Need to jump between files
❌ Wrapping Playwright’s own API for no reason

🔹 Example (LoginPage.js - POM Approach)

class LoginPage {
  constructor(page) {
    this.page = page;
    this.usernameField = page.locator('[data-testid="username"]');
    this.passwordField = page.locator('[data-testid="password"]');
    this.loginButton = page.locator('[data-testid="login-button"]');
  }

  async login(username, password) {
    await this.usernameField.fill(username);
    await this.passwordField.fill(password);
    await this.loginButton.click();
  }
}

export default LoginPage;

🔹 Usage in a Test

import { test, expect } from "@playwright/test";
import LoginPage from "./LoginPage.js";

test("User can log in", async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.login("testUser", "password123");

  await expect(page.locator('[data-testid="welcome-message"]')).toHaveText(
    "Welcome, testUser"
  );
});

✅ Functional Helper Approach (Better)

📌 Why is this better?

✅ No extra class → Directly use Playwright API
✅ No unnecessary this.page assignments
✅ Much easier to maintain & debug

🔹 Example (loginHelpers.js - Functional Helper Approach)

export async function login(page, username, password) {
  await page.fill('[data-testid="username"]', username);
  await page.fill('[data-testid="password"]', password);
  await page.click('[data-testid="login-button"]');
}

🔹 Usage in a Test

import { test, expect } from "@playwright/test";
import { login } from "./loginHelpers.js";

test("User can log in", async ({ page }) => {
  await login(page, "testUser", "password123");

  await expect(page.locator('[data-testid="welcome-message"]')).toHaveText(
    "Welcome, testUser"
  );
});

🔥 Final Thoughts

Helper functions are simpler, faster to debug, and scale better in component-driven apps.

💡 POM was useful in Selenium/WebDriver days, but today? Just use functions.

🔥 What do you think? Are you still using POM? Have you already switched to functional helpers?
💬 Drop a comment below—I’d love to hear your take on this!

Addendum

Thanks everyone for their contributions!

One thing I identified is that Component in testing isn’t used like in UI frameworks.

For frontend devs, a component is a reusable UI unit with state, props, and lifecycle hooks.

For QA, a component is often just a logical grouping of elements & interactions within a larger page.

🔹 ex: A checkout form in testing might represent the entire checkout UI, while in React, it might be broken into FormField, Button, AddressInput.

🔹 In POM, subcomponents are often instantiated as properties within a page object, mirroring the UI structure but without true reusability.

While I understand this approach, I don’t fully agree with the terminology.

I advocate for UI-component-driven testing, as I discuss in my book: https://muratkerem.gitbook.io/cctdd.

With tools like PW Component Testing, Cy Component Testing & Vitest with UI, we can now test at a lower level, reducing the need for full-page interactions.

Though still uncommon in QA, this shift solves many POM complexities:

✅ test components in isolation before e2e
✅ many cases covered at the UI component level
✅ smaller tests = quicker execution

We only move up the testing pyramid when we must: routing, user flows, backend calls.

I see that even the strongest advocates of POM now acknowledge its role has shifted—it’s mainly used to organize and store locators. But with AI-assisted test debugging advancing rapidly, it’s time to rethink this approach entirely.

Watch this Playwright demo ~13 min mark by Debbie O'Brien & Simon Knott

They showcase an experimental Playwright config that outputs the DOM in a structured way, similar to aria snapshots:

This creates a clean, machine-readable format (like YML) that represents the DOM in a concise way. The AI then analyzes the DOM snapshot, diagnoses the failure, and suggests stable ARIA selectors.

What does this mean for QA?

Instead of manually managing locators in a POM abstraction, we should adopt ARIA selectors as the standard for DOM interaction.

1.ARIA selectors provide a shared understanding between screen readers, test tools like Playwright, and AI.

2.Debugging can be AI-powered—rather than relying on a static POM file, AI can analyze the ARIA selector tree and suggest better selectors dynamically.

3.The DOM becomes the source of truth, not a manually maintained POM abstraction

If the argument is still “store all selectors in a POM,” I disagree. The future of test automation is ARIA-driven, where the source code and AI-powered ARIA snapshots become the true source of truth—not a manually curated selectors file.

This post took off, didn’t it?

The conversation in the comments has revealed perspectives I initially overlooked. One key takeaway is that many teams still use non-JS/TS languages (Java, C#) for UI testing, meaning they don’t even have the option of functional helpers—they are locked into POM and class-based structures by default.

I didn’t consider this because, for me, the core unwritten rule is that UI tests should be written in the same language as the source code. If the app is built in JavaScript/TypeScript, it makes sense to test it in JS/TS.

For teams transitioning from non-JS/TS to modern test frameworks, I now see how this post could feel completely foreign. In that case, the first step isn’t functional helpers vs. POM—it’s reevaluating whether Java/C# are the right tools for UI testing at all.

Solving Cross-Execution Issues in Pact Testing with Kafka and Message Queues

Murat K Ozcan — Wed, 16 Oct 2024 14:56:26 +0000

Solving Cross-Execution Issues in Pact Testing with Kafka and Message Queues

In our integration tests, Pact, a consumer-driven contract testing tool, has been instrumental in verifying interactions between services. However, when integrating Kafka and message queues, we encountered a perplexing issue where tests were cross-executing between different provider-consumer pairs. In this blog post, we'll delve into the problem we faced, why it was critical to resolve, and how we ultimately solved it.

You can find the solution in this PR.

The Problem: Cross-Execution of Tests Between Provider-Consumer Pairs

We had two repositories representing two distinct provider-consumer pairs:

Provider (MoviesAPI)

Consumer (WebConsumer)

Where The 2 repos have 2 unique contracts between them:

MoviesAPI ↔ WebConsumer (HTTP-based interaction)
MoviesAPI-event-producer ↔ WebConsumer-event-consumer (Kafka message-based interaction)

Each pair had its own Pact contracts and verification tests. Locally, everything worked flawlessly, in CI PRs as well.

However, when the consumer triggered a webhook in the provider repository (as part of the CI/CD pipeline), we encountered a strange issue:

The Kafka test (provider-kafka.pacttest.ts), intended for the MoviesAPI-event-producer and WebConsumer-event-consumer, was inadvertently executing tests meant for the HTTP-based pair (MoviesAPI and WebConsumer).
This cross-execution led to test failures and confusion, especially since the tests were running in the context of the wrong provider-consumer pair.

Root Cause Analysis

The crux of the problem was the handling of the PACT_PAYLOAD_URL environment variable:

The PACT_PAYLOAD_URL was being set globally, causing both test suites to pick it up, regardless of whether it was relevant to them.

This is the way things have to be, because during a webhook trigger, which is caused by a repository_dispatch, we want an HTTP request from the consumer repo to the provider repo instructing GitHub to start the webhook action.

A repository_dispatch in GitHub is basically an HTTP request to your GitHub project instructing GitHub to start any action or webhook. In our example, a repository_dispatch with the event type contract_requiring_verification_published will trigger the workflow.

In our local tests and in PRs, PACT_BROKER_BASE_URL is used, and during a web hook trigger it is replaced with PACT_PAYLOAD_URL. This is because we want to verify the newly published contract from the consumer repo PR that caused the trigger.

Recall the consumer and provider flow.

The key is that, when there are multiple repos, the provider has to run test:provider-ci (#3) after the consumer runs publish:pact (#2) but before the consumer can run can:i:deploy:consumer (#4) . The trigger to run test:provider-ci (#3) has to happen automatically, webhooks handle this.
# Consumer
npm run test:consumer # (1)
npm run publish:pact  # (2)
npm run can:i:deploy:consumer # (4)
# only on main
npm run record:consumer:deployment --env=dev # (5) change the env param as needed

# Provider
npm run test:provider-ci # (3) triggered by webhooks
npm run can:i:deploy:provider # (4)
# only on main
npm run record:provider:deployment --env=dev # (5) change the env param as needed

Why Was the Problem Worth Solving

Ensuring that each test suite only executes its intended tests is crucial for several reasons:

Accuracy: Tests should only verify the contracts relevant to their specific provider-consumer pair.
Reliability: Cross-execution can lead to false negatives, causing CI/CD pipelines to fail unnecessarily.
Scalability: As more provider-consumer pairs are added, the problem would compound, leading to more significant issues down the line.

By resolving this issue, we aimed to improve the reliability of our test suites, streamline our CI/CD processes, and set a foundation for scalable contract testing.

How We Solved It

To address the issue, we undertook a systematic approach:

1. Refactoring the URL Handling Logic

We focused on the handlePactBrokerUrlAndSelectors function, which was responsible for configuring the verifier options based on the presence of a PACT_PAYLOAD_URL.

Original Issues:

Complexity: The function had a high cyclomatic complexity, making it hard to read and maintain.
Global Variable Usage: It relied on globally set environment variables, leading to unintended side effects.
Lack of Conditional Handling: There was no check to ensure that the PACT_PAYLOAD_URL matched the expected provider and consumer before using it.

Refactored Solution:

We broke down the handlePactBrokerUrlAndSelectors function into smaller, focused functions:

parseProviderAndConsumerFromUrl: Extracts the provider and consumer names from the PACT_PAYLOAD_URL.
processPactPayloadUrl: Determines whether the PACT_PAYLOAD_URL should be used based on whether it matches the expected provider and consumer.
usePactPayloadUrl Configures the verifier options to use the PACT_PAYLOAD_URL for verification.
usePactBrokerUrlAndSelectors: Configures the verifier options to use the Pact Broker URL and appropriate consumer version selectors.

By modularizing the code, we reduced complexity and made it easier to understand and maintain.

2. Implementing Conditional Logic

We introduced logic to ensure that a test suite only uses the PACT_PAYLOAD_URL if it matches its specific provider and consumer:

if (providerMatches && consumerMatches) {
  usePactPayloadUrl(pactPayloadUrl, options);
  return true; // Indicate that the Pact payload URL was used
} else {
  console.log(
    `PACT_PAYLOAD_URL does not match the provider (${options.provider}) and consumer (${consumer || 'all'}), ignoring it`
  );
}

This check prevents a test suite from inadvertently using a PACT_PAYLOAD_URL intended for a different provider-consumer pair.

3. Ensuring Proper Fallback

If the PACT_PAYLOAD_URL is not provided or doesn't match, the verifier options are configured to use the Pact Broker URL and consumer version selectors:

// If pactPayloadUrl is not provided or doesn't match, use the Pact Broker URL and selectors
usePactBrokerUrlAndSelectors(
  pactBrokerUrl,
  consumer,
  includeMainAndDeployed,
  options
);

This ensures that the test suite can still run and verify contracts appropriately, even in the absence of a matching PACT_PAYLOAD_URL.

4. Updating Test Files

In our test files (provider-contract.pacttest.ts and provider-kafka.pacttest.ts), we made sure to specify the correct provider and consumer when building the verifier options. This information is crucial for the processPactPayloadUrl function to correctly determine whether to use the PACT_PAYLOAD_URL.

const options = buildVerifierOptions({
  provider: 'MoviesAPI',
  consumer: 'WebConsumer',
  // ... other options
})

///

const options = buildMessageVerifierOptions({
  provider: 'MoviesAPI-event-producer',
  consumer: 'WebConsumer-event-consumer',
})

5. Verifying the Solution

Trigger a push from the consumer repo, and we had a CI failure at the provider web hook job.

Trigger a push from the consumer repo again, and no more CI failure.

Conclusion

By refactoring our code and introducing conditional logic to handle the PACT_PAYLOAD_URL appropriately, we resolved the cross-execution issue in our Pact tests involving Kafka and message queues. This solution not only fixed the immediate problem but also enhanced the maintainability and scalability of our testing framework.

Key Takeaways:

Modular Code Improves Maintainability: Breaking down complex functions into smaller, focused ones makes the codebase easier to understand and work with.
Conditional Logic Prevents Cross-Execution: Implementing checks to ensure that only the intended tests are executed prevents cross-execution issues.
Testing Various Scenarios: Verifying the solution under different conditions ensures robustness and reliability.

Final Thoughts:

Addressing this issue was crucial for maintaining the integrity of our testing process. As microservices architectures continue to grow in complexity, having reliable and maintainable testing practices becomes ever more important. We hope that sharing our experience will help others facing similar challenges in their testing journeys.

Building Custom Request Filters for PactJs Verifications in Express and Non-Express Environments

Murat K Ozcan — Tue, 08 Oct 2024 12:08:51 +0000

When working with PactJs contract testing, request filters are essential for modifying HTTP requests during the verification process between a consumer and a provider. Request filters allow you to add headers, modify request bodies, or handle authentication tokens before requests are sent to the provider. However, implementing these filters can be challenging when working across non-Express setups.

Pact docs recommend to only use requestFilter feature for things that cannot be persisted in the pact file. Auth tokens are a common use case.

This blog post will walk through how to create a custom request filter that adds an Authorization header, which works both in Express environments (where middleware functions handle requests) and non-Express environments such as lambdas. Additionally, we'll explore why the solution is designed as a higher-order function and how it accommodates Pact's express-like type requirements.

Here is a link to the PR with the specific changes & the source code.

The Problem:

In certain contract testing scenarios with Pact, you may need a mechanism to modify HTTP requests, such as injecting an Authorization token into headers before the requests are sent to the provider for verification. When testing in non-Express environments, the issue lies with Pact types requiring an Express-like shape for request handling.

Express middleware typically requires three arguments: req, res, and next. In non-Express environments, only the request object might be available, and the absence of the next function (used to pass control to the next middleware) can break the flow of the request-handling logic. Thus, the solution must accommodate both cases while ensuring the logic remains flexible, especially for custom token generation.

The Solution: A Higher-Order Function

To solve this issue, the request filter is implemented as a higher-order function. This allows flexibility in how token generation logic is injected into the request handler. Additionally, it conforms to the Pact verifier's expectations of handling the req, res, and next arguments. This way, the filter can work seamlessly with both Express and non-Express environments.

Let's break down the code:

// generic HttpRequest structure to accommodate both Express and non-Express environments
type HttpRequest = {
  headers: Record<string, string | string[] | undefined>
  body?: unknown
}

type NextFunction = () => void | undefined

// allows customization of token generation logic
type RequestFilterOptions = {
  tokenGenerator?: () => string
}

Here, HttpRequest is a generic structure that can represent both Express requests and non-Express requests. The RequestFilterOptions allows for customizable token generation by providing an optional tokenGenerator function.

const handleExpressEnv = (
  req: HttpRequest,
  next: NextFunction
): HttpRequest | undefined => {
  // If this is an Express environment, call next()
  if (next && typeof next === 'function') {
    next()
  } else {
    // In a non-Express environment, return the modified request
    return req
  }
}

The handleExpressEnv function is the key to managing both Express and non-Express environments. It checks if next exists and, if so, it assumes the environment is Express and passes control to the next middleware. Otherwise, it simply returns the modified request for non-Express environments. The else clause can be modified to suite your needs.

const createRequestFilter =
  (options?: RequestFilterOptions): ProxyOptions['requestFilter'] =>
  (req, _, next) => {
    const defaultTokenGenerator = () => new Date().toISOString()
    const tokenGenerator = options?.tokenGenerator || defaultTokenGenerator

    // add an authorization header if not present
    if (!req.headers['Authorization']) {
      req.headers['Authorization'] = `Bearer ${tokenGenerator()}`
    }

    return handleExpressEnv(req, next)
  }

The createRequestFilter is a higher-order function because it returns a function that will be used to filter requests. It allows for the optional injection of a custom token generator. Inside the function, if the Authorization header is missing, a token is generated and added to the request headers. After modifying the headers, it hands off the request to handleExpressEnv for environment-appropriate handling.

// if you have a token generator, pass it as an option
// createRequestFilter({ tokenGenerator: myCustomTokenGenerator })
export const requestFilter = createRequestFilter()

export const noOpRequestFilter: ProxyOptions['requestFilter'] = (
  req,
  _,
  next
) => handleExpressEnv(req, next)

Here, we define two exports:

requestFilter is the default filter that adds the Authorization header.
noOpRequestFilter is a no-operation filter that doesn’t modify the request but still handles the environment appropriately.

These two exports are used to build verifier options for Pact tests. The noOpRequestFilter can be used as a default value for the buildVerifierOptions function. requestFilter can be used directly in our tests to modify the http request

Key Takeaways:

Higher-Order Functions: By using a higher-order function, we provide flexibility for future customization, like custom token generation. This design pattern is crucial in ensuring reusable and customizable logic.
Environment Agnosticism: The combination of handleExpressEnv and higher-order functions allows the filter to work in both Express and non-Express environments. This makes the code more robust and versatile across different contexts.
Pact's express-like type requirements: The filter satisfies Pact's need to handle three arguments (req, res, and next), even if the environment doesn't use Express, ensuring compatibility during the contract testing process.

Conclusion:

In summary, this custom request filter solves the challenge of modifying HTTP requests in both Express and non-Express environments while allowing for future customization. Using a higher-order function ensures that we can inject different token generation strategies, providing flexibility and maintaining compatibility with Pact’s express-like type requirements.

Handling Pact Breaking Changes Dynamically in CI/CD

Murat K Ozcan — Sun, 15 Sep 2024 14:54:18 +0000

When working with consumer-driven contract testing using Pact, ensuring compatibility between provider and consumer services can be challenging, especially when introducing breaking changes. Providers often need to make changes that might not be immediately compatible with consumers, and coordinating these changes can be a hassle. To streamline this process, we can leverage environment variables and GitHub Actions to dynamically handle breaking changes in Pact tests with minimal developer intervention.

In this blog post, I'll walk you through a method to manage breaking changes in Pact verification tests, ensuring a smoother development workflow and reducing the friction often associated with such changes.

You can find the final code at the repository https://github.com/muratkeremozcan/pact-js-example-provider. It is continuously battle tested and improved; you may see mismatches between the post and the code.

The Problem: Breaking Changes in Pact Verification

By default, when verifying consumer contracts, we might test against multiple consumer versions:

matchingBranch: Verifies against feature branches that match the provider's branch, facilitating coordinated development.
mainBranch: Verifies against the consumer’s main branch, which should be stable.
deployedOrReleased: Verifies against consumer versions that are currently deployed or released into production.

However, when a provider introduces a breaking change, these default checks can cause unnecessary failures. Developers want to focus on verifying against the matchingBranch, where compatible changes are expected, without being bogged down by failures from the mainBranch or deployedOrReleased versions.

The Solution: Managing Breaking Changes with Environment Variables

To address this, we introduce an environment variable PACT_BREAKING_CHANGE. When set to true, it modifies the verification process to focus solely on the matchingBranch, skipping the checks against mainBranch and deployedOrReleased. This allows developers to proceed with breaking changes without the CI pipeline failing due to incompatibilities with consumer versions that haven't yet been updated.

Implementation Details

Verifier Options

We adjust the verifier options to include or exclude certain consumer version selectors based on the PACT_BREAKING_CHANGE variable:

const options: VerifierOptions = {
  providerBaseUrl: `http://127.0.0.1:${port}`,
  provider: 'MoviesAPI',
  publishVerificationResult: true,
  providerVersion,
  providerVersionBranch: providerBranch,
  pactBrokerUrl: process.env.PACT_BROKER_BASE_URL,
  pactBrokerToken: process.env.PACT_BROKER_TOKEN,
  stateHandlers,
  beforeEach,
  afterEach,
  enablePending: PACT_BREAKING_CHANGE === 'true',
};

// Determine which consumer versions to verify against
options.consumerVersionSelectors = [
  { matchingBranch: true },
  ...(includeMainAndDeployed ? [
    { mainBranch: true },
    { deployedOrReleased: true },
  ] : []),
  ...(includeAllPacts ? [{ all: true }] : []),
];

includeMainAndDeployed: Set to false when PACT_BREAKING_CHANGE is 'true', excluding mainBranch and deployedOrReleased from verification.
enablePending: Enables pending pacts when introducing breaking changes, allowing the provider to accept pending pacts without failing the build.
includeAllPacts: When set, includes all pacts in the verification process, useful for broader testing during breaking changes.

Building Consumer Version Selectors

We encapsulate the logic for determining which consumer versions to verify in a helper function:

import type { ConsumerVersionSelector } from '@pact-foundation/pact-core';

function buildConsumerVersionSelectors(
  consumer: string | undefined,
  includeMainAndDeployed: boolean,
  includeAllPacts: boolean
): ConsumerVersionSelector[] {
  const baseSelector: Partial<ConsumerVersionSelector> = consumer ? { consumer } : {};

  const selectors: ConsumerVersionSelector[] = [
    { ...baseSelector, matchingBranch: true },
    ...(includeMainAndDeployed ? [
      { ...baseSelector, mainBranch: true },
      { ...baseSelector, deployedOrReleased: true },
    ] : []),
    ...(includeAllPacts ? [{ ...baseSelector, all: true }] : []),
  ];

  return selectors;
}

Integrating with CI/CD

To make this process seamless in CI, we use GitHub Actions to set the PACT_BREAKING_CHANGE environment variable based on the PR description. Developers can indicate a breaking change by including a checkbox in the PR description. This automates the process, eliminating the need for manual intervention.

GitHub Actions Workflow

Here's how we set up the workflow:

name: Run contract tests

on:
  pull_request:
    types: [opened, synchronize, reopened, edited]
  push:
    branches:
      - main

jobs:
  contract-test:
    runs-on: ubuntu-latest
    env:
      PACT_BROKER_BASE_URL: ${{ secrets.PACT_BROKER_BASE_URL }}
      PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}
      GITHUB_SHA: ${{ github.sha }}
      GITHUB_REF_NAME: ${{ github.ref_name }}
      DATABASE_URL: 'file:./dev.db'
      PORT: 3001
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref || github.ref_name }}
          fetch-depth: 0

      - name: Read Node version from .nvmrc
        id: node_version
        run: echo "NODE_VERSION=$(cat .nvmrc)" >> $GITHUB_ENV

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      # Set PACT_BREAKING_CHANGE based on PR description during pull_request events
      - name: Set PACT_BREAKING_CHANGE based on PR description (PR event)
        if: ${{ github.event_name == 'pull_request' }}
        uses: actions/github-script@v7
        with:
          script: |
            const prBody = context.payload.pull_request.body || '';
            if (prBody.includes('[x] Pact breaking change')) {
              core.exportVariable('PACT_BREAKING_CHANGE', 'true');
              console.log('PACT_BREAKING_CHANGE set to true based on PR description checkbox.');
            } else {
              core.exportVariable('PACT_BREAKING_CHANGE', 'false');
              console.log('PACT_BREAKING_CHANGE remains false.');
            }

      # Set PACT_BREAKING_CHANGE based on merged PR description during push events
      - name: Set PACT_BREAKING_CHANGE based on merged PR description (push to main)
        if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
        uses: actions/github-script@v7
        with:
          script: |
            const commitSha = context.sha;
            const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
              owner: context.repo.owner,
              repo: context.repo.repo,
              commit_sha: commitSha,
            });
            const mergedPr = prs.find(pr => pr.merged_at !== null && pr.merge_commit_sha === commitSha);
            if (mergedPr) {
              const prBody = mergedPr.body || '';
              if (prBody.includes('[x] Pact breaking change')) {
                core.exportVariable('PACT_BREAKING_CHANGE', 'true');
                console.log('PACT_BREAKING_CHANGE set to true based on merged PR description checkbox.');
              } else {
                core.exportVariable('PACT_BREAKING_CHANGE', 'false');
                console.log('PACT_BREAKING_CHANGE remains false.');
              }
            } else {
              core.exportVariable('PACT_BREAKING_CHANGE', 'false');
              console.log('No merged PR found for this commit. PACT_BREAKING_CHANGE remains false.');
            }

      - name: Install dependencies
        run: npm ci

      - name: Run provider contract tests
        run: |
          echo "Running provider contract tests with PACT_BREAKING_CHANGE=$PACT_BREAKING_CHANGE"
          npm run test:provider-ci

      - name: Can I deploy provider?
        if: env.PACT_BREAKING_CHANGE == 'false'
        run: npm run can:i:deploy:provider

      - name: Record provider deployment
        if: github.ref == 'refs/heads/main'
        run: npm run record:provider:deployment --env=dev

PR Description Checkbox

Developers can indicate a breaking change by including the following in their PR description:

### Pact Breaking Change

- [x] Pact breaking change (check if this PR introduces a breaking change)

When the checkbox is checked, the PACT_BREAKING_CHANGE variable is set to true in the CI environment.

Handling Verification Failures Due to Breaking Changes

In scenarios where verification failures occur due to breaking changes, we can adjust our test code to handle these failures gracefully. Here's how we modify the test:

const PACT_BREAKING_CHANGE = process.env.PACT_BREAKING_CHANGE || 'false'
const GITHUB_BRANCH = process.env.GITHUB_BRANCH || 'local'

describe('Pact Verification', () => {
  const port = process.env.PORT || '3001'
  const options = buildVerifierOptions({
    provider: 'MoviesAPI',
    consumer: process.env.PACT_CONSUMER, // filter by the consumer, or run for all if no env var is provided
    includeMainAndDeployed: PACT_BREAKING_CHANGE !== 'true', // if it is a breaking change, set the env var
    enablePending: PACT_BREAKING_CHANGE === 'true',
    port,
    stateHandlers,
    beforeEach: () => {
      console.log('I run before each test coming from the consumer...')
      return Promise.resolve()
    },
    afterEach: () => {
      console.log('I run after each test coming from the consumer...')
      return Promise.resolve()
    }
  })
  const verifier = new Verifier(options)

  it('should validate the expectations of movie-consumer', async () => {
    try {
      const output = await verifier.verifyProvider()
      console.log('Pact Verification Complete!')
      console.log('Result:', output)
    } catch (error) {
      console.error('Pact Verification Failed:', error)

      if (PACT_BREAKING_CHANGE === 'true' && GITHUB_BRANCH === 'main') {
        console.log(
          'Ignoring Pact verification failures due to breaking change on main branch.'
        )
      } else {
        throw error // Re-throw the error to fail the test
      }
    }
  })
})

Error Handling: We wrap the verification in a try-catch block. If verification fails and PACT_BREAKING_CHANGE is 'true' on the main branch, we log a message and prevent the test from failing the CI pipeline.
Local Development: When running tests locally, PACT_BREAKING_CHANGE defaults to 'false', ensuring that tests fail on verification failures, prompting developers to address issues promptly.

Workflow Summary

This approach allows us to manage breaking changes effectively by:

Automating Environment Variable Setting: Using GitHub Actions to set PACT_BREAKING_CHANGE based on the PR description reduces manual steps.
Adjusting Verification Behavior: Modifying the consumer version selectors dynamically based on PACT_BREAKING_CHANGE ensures that we only verify relevant consumer versions during breaking changes.
Handling Test Failures Gracefully: By adjusting the test code to conditionally handle verification failures, we prevent unnecessary CI failures while maintaining test integrity.

Breaking Change Flows

Consumer Flow

For consumers adapting to breaking changes:

# 1. Update the consumer tests
npm run test:consumer  # Execute the updated tests
npm run publish:pact    # Publish the updated pact
npm run can:i:deploy:consumer  # Check if it's safe to deploy
# Only on main
npm run record:consumer:deployment --env=dev  # Record the deployment

Provider Flow

For providers introducing breaking changes:

# 1. Create a branch with the breaking change
PACT_BREAKING_CHANGE=true npm run test:provider-ci  # Run the provider tests with the breaking change flag
# Note: The can:i:deploy:provider step is skipped because we're introducing a breaking change
# 2. Merge to main

Conclusion

By introducing this environment-driven mechanism, we now have a flexible way to manage breaking changes in Pact tests. This approach allows us to:

Maintain confidence in our provider contracts while rolling out changes incrementally.
Simplify the developer experience by dynamically handling the changes via an env var and a checkbox in the PR description.

Moreover, the addition of the buildConsumerVersionSelectors function significantly DRYs up the code by centralizing the logic for determining which consumer versions to verify. By encapsulating this logic, it not only makes the code more maintainable but also provides a cleaner developer experience when managing consumer filtering and selector behavior.

This approach can be applied to any project using Pact and provides a smooth, automated way to handle breaking changes without manual intervention.

Strengthening Pact Contract Testing with TypeScript and Data Abstraction

Murat K Ozcan — Mon, 02 Sep 2024 12:47:25 +0000

Working on a pair of sample repositories for consumer-driven contract testing with Pact, I realized that a major potential breaking point when setting up state handlers is when the state parameters are being passed from the consumer to the provider.

Let the code talk; here are the sample repos and PRs the blog post is about:

https://github.com/muratkeremozcan/pact-js-example-consumer/pull/14/files

https://github.com/muratkeremozcan/pact-js-example-provider/pull/7/files

What are State Handlers?
The Problem: Type Mismatch in Provider State Handling
Introducing createProviderState: A Type-Safe Abstraction
- What is JsonMap?
- So, how do we start fixing things?
- The abstraction we want: createProviderState
- Example: Consumer Side Tests
Aligning Provider State Handlers with Strong Typing
Conclusion

What are State Handlers?

In Pact testing, state handlers are functions on the provider side that prepare the provider's system under test to be in a specific state before verifying the interactions described in the Pact. These states are described by the consumer during the contract test. For example, a state might describe a scenario where "a movie with a specific ID exists in the database." The consumer will call this state, and the provider's state handler will set up the necessary conditions to meet this requirement.

The Problem: Type Mismatch in Provider State Handling

As it is, on the consumer side we are allowed to pass in anything. The JS and TS versions of the test are the same.

// consumer-contract.pacttest.ts
it('should return a specific movie', async () => {
  const testId = 100
  const EXPECTED_BODY = { id: testId, name: 'My movie', year: 1999 }
  const params = { id: testId }

  provider
    .given('Has a movie with a specific ID', params)
    // ...

// another consumer side test
it('should not add a movie that already exists', async () => {
  const { name, year } = {
    name: 'My existing movie',
    year: 2001
  }
  const params = { name, year }

  provider
    .given('An existing movie exists', params)
    // ...

The given method accepts a type of (providerState: string, parameters?: JsonMap). As users, we care about the state and parameters we are passing in for the provider to use. The provider will have to setup the stateHandlers, and run the test suite / pact from the consumer against its own locally served server. There comes the main potential breaking point; what if the state/params passed in are not correct? This can become a significant issue, especially when dealing with complex data structures - in the PRs I had much trouble working with objects with 1-2 properties!

Here’s the working provider-side state handler in JavaScript. It worked fine, but TypeScript revealed issues:

// provider-contract.pacttest.ts

// @ts-nocheck - the only way this would work with TS

const stateHandlers = {
  "Has a movie with a specific ID": (params) => {
    movies.getFirstMovie().id = params.id;
    return Promise.resolve({
      description: `Movie with ID ${params.id} added!`,
    });
  },
  "An existing movie exists": (params) => {
    movies.addMovie(params);
    return Promise.resolve({
      description: `Movie with ID ${params.id} added!`,
    });
  },
};

// and then it's used at options
const options: VerifierOptions = {
  // ...
  stateHandlers,
};

Introducing `createProviderState`: A Type-Safe Abstraction

The core of the problem is how we pass the parameters from the consumer. There is no straightforward way to make the provider types happy without changing that. The challenge lies in the fact that Pact accepts JsonMap during its communication between the consumer and provider.

What is `JsonMap`?

JsonMap is a type used by Pact that ensures the parameters passed between the consumer and provider are in a format compatible with JSON. The keys must be strings, and the values must be JSON-serializable. Failing to convert parameters properly can lead to issues, particularly with complex nested structures. Without proper conversion, you may encounter runtime errors or subtle bugs where the provider's state doesn't match what the consumer expects. This becomes especially problematic when dealing with objects containing nested objects, arrays, or dates, as these might not be automatically converted into the format expected by Pact, leading to unexpected behavior or test failures.

So, how do we start fixing things?

Well, if pact expects JsonMap, we have to pass in JsonMap. That means we have to convert all key values to strings.

it('should return a specific movie', async () => {
  const testId = 100
  const EXPECTED_BODY = { id: testId, name: 'My movie', year: 1999 }
  const params = { id: String(testId) } // PLEASE NO

  provider
    .given('Has a movie with a specific ID', params)
    // ...
})

it('should not add a movie that already exists', async () => {
  const { name, year } = {
    name: 'My existing movie',
    year: 2001
  }
  const params = { name, String(year) } // PLEASE NO

  provider
    .given('An existing movie exists', params)
    // ...
})

That is a hassle to remember, and can get very error prone and painful when the data structure is complex. Why don't we have some nice utility, where we pass in what we need as the user, and Pact takes care of whatever it needs for its communication without us needing to know about it.

The abstraction we want: `createProviderState`

To address this, we developed the createProviderState function. This utility abstracts the complexity of passing data between the consumer and provider by converting parameters into a JsonMap that complies with Pact's expectations. It ensures that all data passed to the provider is properly formatted, handling everything from null values to complex objects.

import type { JsonMap } from "@pact-foundation/pact/src/common/jsonTypes";

const toJsonMap = (obj: Record<string, unknown>): JsonMap =>
  Object.fromEntries(
    Object.entries(obj).map(([key, value]) => {
      if (value === null || value === undefined) {
        return [key, "null"];
      } else if (
        typeof value === "object" &&
        !(value instanceof Date) &&
        !Array.isArray(value)
      ) {
        return [key, JSON.stringify(value)];
      } else if (typeof value === "number" || typeof value === "boolean") {
        return [key, value];
      } else if (value instanceof Date) {
        return [key, value.toISOString()];
      } else {
        return [key, String(value)];
      }
    })
  );

type ProviderStateInput = {
  name: string;
  params: Record<string, unknown>;
};

export const createProviderState = ({
  name,
  params,
}: ProviderStateInput): [string, JsonMap] => [name, toJsonMap(params)];

With createProviderState, we not only abstract the necessary conversion but also ensure a clean API that communicates exactly what the right approach is.

Example: Consumer Side Tests

Here are the consumer-side tests before and after using createProviderState:

// consumer-contract.pacttest.ts

// before
it("should return a specific movie", async () => {
  const testId = 100;
  const EXPECTED_BODY = { id: testId, name: "My movie", year: 1999 };
  const params = { id: String(testId) }; // PLEASE NO

  provider.given("Has a movie with a specific ID", params);
  // ...
});

// after
it("should return a specific movie", async () => {
  const testId = 100;
  const EXPECTED_BODY = { id: testId, name: "My movie", year: 1999 };

  const [stateName, stateParams] = createProviderState({
    name: "Has a movie with a specific ID",
    params: { id: testId },
  });

  provider.given(stateName, stateParams);
  // ..
});

// another consumer side test

// before
it('should not add a movie that already exists', async () => {
  const { name, year } = {
    name: 'My existing movie',
    year: 2001
  }
  const params = { name, String(year) } // PLEASE NO

  provider
    .given('An existing movie exists', params)
    // ...
})

// after
it("should not add a movie that already exists", async () => {
  const movie: Movie = {
    name: "My existing movie",
    year: 2001,
  };

  const [stateName, stateParams] = createProviderState({
    name: "An existing movie exists",
    params: movie,
  });

  provider.given(stateName, stateParams);
  // ...
});

It seems like a minor change, but without createProviderState, you would have to manually convert parameters into a format compliant with Pact's JsonMap, leading to confusing and error-prone code, especially with complex data structures.

Aligning Provider State Handlers with Strong Typing

On the provider side, we need to ensure that the types of incoming parameters match what we expect while still making sure the tests work. One of the hard parts of consumer driven contract testing is diagnosing why the provider side test fails. When the types aren't working correctly, it's much harder to diagnose why the provider-side test fails. By ensuring strong typing, we can isolate issues more effectively simply because the surface area of the failure is smaller.

We start by using the existing types we already have, like MovieType. In a larger organization, these types could be defined in shared libraries, ensuring consistency across different projects and teams. By using a common set of types, everyone can agree on and rely on, we reduce duplication and potential errors. The params being passed in from the consumer are then just a subset of those known and trusted types, promoting reusability and consistency. This approach also facilitates collaboration and code maintenance, as developers across the organization can confidently work with the same data structures without having to redefine them.

// provider-contract.pacttest.ts
// import from wherever
type MovieType = {
  id: number;
  name: string;
  year: number;
};

// the types we will use in the state handler
// matching the parameters coming from the consumer
type ExistingMovieParams = Omit<MovieType, "id">;
type HasMovieWithSpecificIDParams = Omit<MovieType, "name" | "year">;

The below is what the state handler looks like before and after.

We have to match the name of the state versus the consumer side.

We have to use AnyJson as the params type, because that is what Pact communicates with and expects for the types.

We cast params into what we actually need (HasMovieWithSpecificIDParams, ExistingMovieParams) and destructure it.

// provider-contract.pacttest.ts

// before
// @ts-nocheck - the only way this would work with TS
const stateHandlers = {
  "Has a movie with a specific ID": (params) => {
    movies.getFirstMovie().id = params.id;
    return Promise.resolve({
      description: `Movie with ID ${params.id} added!`,
    });
  },
  "An existing movie exists": (params) => {
    movies.addMovie(params);
    return Promise.resolve({
      description: `Movie with ID ${params.id} added!`,
    });
  },
};

// after
type HasMovieWithSpecificIDParams = Omit<MovieType, "name" | "year">;
type ExistingMovieParams = Omit<MovieType, "id">;

const stateHandlers: StateHandlers & MessageStateHandlers = {
  "Has a movie with a specific ID": (params: AnyJson) => {
    const { id } = params as HasMovieWithSpecificIDParams;
    const movie = movies.getFirstMovie();

    if (!movie) {
      return Promise.reject(new Error("No movie found to update"));
    }

    movie.id = id;
    return Promise.resolve({
      description: `Movie with ID ${id} added!`,
    });
  },
  "An existing movie exists": (params: AnyJson) => {
    const { name, year } = params as ExistingMovieParams;
    const movie = { name, year };

    movies.addMovie(movie);
    return Promise.resolve({
      description: `Movie with name ${movie.name} added!`,
    });
  },
};

This approach allows us to enforce type safety while ensuring the tests are robust and maintainable. By aligning the provider-side state handlers with the types expected from the consumer, we minimize the risk of errors and promote consistency across the codebase.

Another benefit of this approach is that the types defined for state handlers can be shared across the provider repo, or even the organization. After all, the exact state is shared, no matter where the test is being run. This promotes consistency and reduces duplication of effort, as developers can rely on a common set of types and patterns when writing their tests.

Conclusion

By introducing createProviderState on the consumer side, we improved the developer experience with a desired abstraction, and on the provider side, we ensured type alignment with Pact's communication and our actual domain types. This approach not only makes our tests more reliable but also easier to understand and maintain. The clear separation of concerns and robust type handling ensures that both consumer and provider speak the same language, reducing the likelihood of miscommunication and difficult-to-diagnose test failures. Moreover, this aligns with broader TypeScript best practices, where type safety and clarity are paramount. By encapsulating complexity and ensuring that data structures conform to expected types across the entire testing process, we promote a more predictable and maintainable codebase. This abstraction helps teams focus on their business logic while letting the utility handle the complexities of data transformation.

My thoughts and notes about Consumer Driven Contract Testing

Murat K Ozcan — Fri, 30 Aug 2024 12:19:01 +0000

My thoughts and notes about Consumer Driven Contract Testing

I've recently been diving deep into the world of contract testing, guided by the excellent resource that is the Contract Testing in Action book by Marie Cruz and Lewis Prescott. As someone who has spent time navigating the challenges of modern software development & testing particularly in environments that rely heavily on microservices and API-driven architectures, this book has provided invaluable insights.

This post encapsulates my notes, interpretations, reflections, and thoughts on how the principles from the book align with my own experiences, and how they can be effectively integrated into a real-world testing and development workflow.

Contract Testing
Consumer side
Provider side
The Pact Broker
- Example new release scenario:
- Environments:
- Pact Broker's Role:
- SaaS (PactFlow) vs. Self-Hosted Pact Broker
PactBroker CI/CD & key features
- The matrix:
- can-i-deploy
- Webhooks:
Provider-driven (bi-directional) contract testing
- Cypress adapter in a nutshell
CDCT vs e2e vs integration vs schema testing
- Testing Lambda Handlers (Unit or Integration Tests)
- Testing Deployments via HTTP (Temporary Stack, Dev, Stage)
- Testing Local Endpoints via HTTP (Local)
- CDCT (Contract-Driven Contract Testing)
- What kind of e2e testing would you replace with contract testing?
- How does CDCT fit with schema testing (Optic)?
Message Queue testing
- Purpose of Message Queue Testing
- Example Scenario:
- Message queue consumer tests in short
- Message queue provider tests in short
- Execution (Same as traditional CDCT)
- messageProviders vs stateHandlers on the producer/provider side
Breaking changes
- Real-World Scenarios
- Workarounds (to blasphemy)
- Pending vs. WIP Pacts
- Sample repos

Contract Testing

Contract testing ensures that two systems (e.g., a web application and an API, or 2 APIs) have a shared understanding of expectations, verified through a "contract", usually a json file that captures the interactions between the systems. It's particularly useful in microservices architectures.

How Contract Testing Works:

The consumer defines the expected interactions and stores them in a contract. During this process the consumer creates and tests against a mock provider, generating a contract. They publish the contract to the PactBroker.
The provider (e.g., an API) verifies that it can meet these expectations by running tests against the contract.
Both sides interact through this contract, ensuring compatibility before deploying changes. The contract is what binds the systems together, without needing to be on the same machine or deployment.

Why Contract Testing Matters:

You can test it up front, even when you're doing local development of the client or the service; fast feedback.
You can deploy changes independently and with confidence.
Reduces some of the need for comprehensive end-to-end tests.

How to start

Identify consumers and providers.
Define the contract: establish clear expectations between consumer and provider.
Write tests: implement basic consumer and provider contract tests.
Integrate into CI/CD: add the tests to pipelines and continuously improve the process.

Vocabulary:

Consumer: The service user that makes requests to another service, driving the contract in CDCT.
Provider: The service that responds to the consumer's requests, verifying the contract against its implementation.
Contract: A JSON object detailing the interactions between the consumer and provider, including request details and expected responses.
Contract Broker: A central location for storing and managing contracts, facilitating communication between consumers and providers.

Consumer Driven Contract Testing Lifecycle (CDCT):

generating a contract from the consumer's test code
publishing it to a broker
having the provider pull, verify, and update the contract’s status
Both sides then check the verification status before deploying changes (can-i-deploy).

Contract Testing Approaches:

Consumer-Driven Contract Testing (CDCT): Focuses on the consumer creating the contract, ideal for greenfield projects and organizations committed to contract testing.
Provider-Driven Contract Testing (PDCT): Uses existing OpenAPI specifications of the provider and executes the consumer tests against that, instead of the traditional provider side test execution (of the contract) against the locally served provider service.
Message Queue Testing: meant for systems that communicate asynchronously using message queues instead of direct HTTP requests. Examples of message-based systems include RabbitMQ, Kafka, AWS SQS or EventBridge.

Communication Types Supported:

GraphQL: Supported through a specific wrapper in Pact, allowing flexible data requests.
Event-Driven Systems: Supported by abstracting the message body from the messaging technology, enabling contract testing in systems using asynchronous messaging.

Consumer side

The consumer can be any client that makes API calls. Can be an API service, can be a web app (using Axios for example); it does not make a difference.

Focus: Consumer contract tests should focus solely on ensuring that the consumer’s expectations match the provider’s responses, without delving into the provider’s internal functionality.
Loose Matchers: Use loose matchers to avoid high coupling, allowing flexible assertions.
Isolated Tests: stateless, order independent.
Using can-i-deploy: The can-i-deploy tool in Pact helps verify whether changes are safe to deploy, ensuring that the provider has successfully verified the contract before deployment.

Here is how it works:

Write the consumer test.
Execute the and generate a contract / pact / json file.

The contract specifies how the provider should respond upon receiving requests from the consumer.

Once the contract is created, from then on the Pact mockProvider takes over as if we are locally serving the provider API and executing the tests against that.
That means, there is no need to serve the client api or the provider api at the moment, the consumer tests and mockProvider cover that interaction.
The consumer test can only fail at the executeTest portion, if / when the assertions do not match the specifications. Any changes to the provider section makes updates to the contract.

Here's how a test generally looks:

// consumer test

it('...', () => {
  await pact
    // simulate/specify how the provider should respond
    .addInteraction(...)
    .given(some state name) // optional
    .uponReceiving(<the name of the test>)
    .withRequest(http verb, path)
    .willRespondWith({ this is the meat and bones of the response })

    .executeTest(async(mockProvider) => {
    // call the source code &
    // make assertions against the mockProvider/contract
  })
})

Run the consumer tests:

npm run test:consumer

The pact gets recorded, the consumer tests (executeTest) are verified against the contract.

Now, for the provider to know about it all, we need to publish the contract

Publish the contract to your Pact Broker:

npm run publish:pact

The consumer side uses a mock provider. The specifics of the mock provider are defined in the it blocks.

What is the difference? TL, DR; a mock is a smarter stub.

Stub: Provides a simple, fixed response based on a predefined script. It doesn’t validate how it's used.

Mock: More advanced than a stub; it imitates a real object’s behavior and validates interactions, such as method calls and arguments, making it more suitable for scenarios like contract testing.

// Define the ApiClient interface
interface ApiClient {
fetchUser(userId: number): { id: number; name: string } | null;
}

// The UserService depends on the ApiClient to fetch user data
class UserService {
private apiClient: ApiClient;

constructor(apiClient: ApiClient) {
 this.apiClient = apiClient;
}

getUser(userId: number): { id: number; name: string } | null {
 return this.apiClient.fetchUser(userId);
}
}

////// STUB
const apiClientStub: ApiClient = {
fetchUser: (userId: number) => {
 return { id: userId, name: "John Doe" }; // Predetermined response
},
};

// same in both
const userService = new UserService(apiClientStub);
const user = userService.getUser(1);
console.log(user); // { id: 1, name: 'John Doe' }

/////// MOCK
const apiClientMock: ApiClient = {
fetchUser: jest.fn((userId: number) => {
 if (userId === 1) {
   return { id: 1, name: "John Doe" };
 }
 return null;
}),
};

// same in both
const userService = new UserService(apiClientMock);
const user = userService.getUser(1);
console.log(user); // { id: 1, name: 'John Doe' }
// KEY: verify that fetchUser was called with the correct argument
expect(apiClientMock.fetchUser).toHaveBeenCalledWith(1);

Provider side

The main goal is to verify that the provider API can fulfill the contract expectations defined by the consumer(s). This ensures that any changes made to the provider won't break existing consumer integrations.

Here is how it works

The consumer already generated the contract and published it.
The provider has one test per consumer to ensure all is satisfactory. Most of the file is about setting up the options.
We ensure that the provider api is running locally.
The consumer tests execute against the provider api, as if they are a regular API client running locally.

Here is how the test generally looks:

const options = {..} // most the work is here (ex: provider states)
const verifier = new Verifier(options)

it('should validate the expectations..', async () => {
  const output = await verifier.verifyProvider()
  console.log('Pact Verification Complete!')
  console.log('Result:', output)
})

The provider API has to be running locally for the provider tests to be executed. Then we can execute the provider test.

npm run start:provider

# another tab

npm run test:provider

Provider States: We can simulate certain states of the api (like an empty or non-empty db) in order to cover different scenarios

Provider states help maintain the correct data setup before verification.
State handlers must match the provider states defined in consumer tests.

Can-I-Deploy Tool: Before deploying to an environment, we verify if the consumer and provider versions are compatible using the can-i-deploy tool. This step ensures that any changes made to the consumer or provider do not break existing integrations across environments.

Verify the provider:

npm run can:i:deploy:provider

Verify the consumer:

npm run can:i:deploy:consumer

The Pact Broker

Why do we need a pact broker?

Sharing contracts: it is what binds the repositories together, without needing a common deployment.
Coordination: it can coordinate contracts and releases between branches, environments and teams.
CI: the pact broker provides a CLI which offers easy access to provider verification status and webhooks to trigger dependent CI pipelines.

Pact broker flow:

The Consumer pushes the consumer contracts to the Pact broker.
The provider pulls the contracts from the Pact broker.
The provider publishes the verification status of the contract to the Pact broker.
The consumer & the provider pull the verification status from the Pact broker to check if they can deploy their service/app.

Example new release scenario:

Imagine you have a Movies API that serves different clients, such as a web application and a mobile app. These clients are at different stages of development and are deployed in different environments (e.g., Staging and Production).

Environments:

Staging Environment: Running version v1.2.0 of the Movies API.
Production Environment: Running version v1.1.0 of the Movies API.

Pact Broker's Role:

A new version of the API, v2.0.1, has been developed to fix some bugs.
Before this version can be deployed, the Pact Broker checks it against the existing consumer contracts (which specify how clients interact with the API) to ensure it won't break any current implementations.
The Pact Broker verifies the contract against v1.2.0 in the Staging environment and v1.1.0 in the Production environment.
The checkmarks indicate that both environments successfully verified the new contract, meaning v2.0.1 can be safely integrated into either environment.

SaaS (PactFlow) vs. Self-Hosted Pact Broker

SaaS (PactFlow) Solution:

Cost: SaaS cost, but no setup or maintenance effort.
Maintenance: none.
Features: Offers advanced features like Can I Deploy, API token-based security, and support for Provider driven contracts (e.g., OpenAPI).
Ease of Use: Easy to set up, with a user-friendly interface and seamless third-party integrations.
Support: Dedicated support from the PactFlow team.

Self-Hosted Pact Broker:

Cost: Lower running costs, but requires infrastructure setup and residual maintenance.
Maintenance: Requires ongoing management of security patches and updates.
Features: Provides core functionality with basic username/password security.
Setup Complexity: More complex setup, especially when using Docker or Kubernetes.
Support: Relies on community support, which may be less consistent.

PactBroker CI/CD & key features

Environment Variables: Why GITHUB_SHA and GITHUB_BRANCH?

GITHUB_SHA: This variable represents the unique commit ID (SHA) in Git. By using the commit ID as the version identifier when publishing the contract or running tests, you can precisely trace which version of your code generated a specific contract. This traceability is crucial in understanding which code changes correspond to which contract versions, allowing teams to pinpoint when and where an issue was introduced.
GITHUB_BRANCH: Including the branch name ensures that contracts and deployments are correctly associated with their respective branches, supporting scenarios where different branches represent different environments or features under development. It helps prevent conflicts or mismatches in contracts when multiple teams or features are being developed simultaneously.

TL,DR; best practice, do it this way.

On the provider side Pact verifies all the relevant versions that are compatible:

// provider spec file

const options = {
  // ...
  providerVersion: process.env.GITHUB_SHA,
  providerVersionBranch: process.env.GITHUB_BRANCH, // represents which contracts the provider should verify against
  consumerVersionSelectors = [
      { mainBranch: true },  // tests against consumer's main branch
      { matchingBranch: true }, // tests against consumer's currently deployed and currently released versions
      { deployedOrReleased: true } // Used for coordinated development between consumer and provider teams using matching feature branch names
    ]
}

The matrix:

The Pact Matrix is a feature within Pactflow (or other Pact brokers) that visualizes the relationships between consumer and provider versions and their verification status across different environments. The matrix shows:

Which versions of consumers are compatible with which versions of providers.
The verification results of these interactions across various environments (e.g., dev, stage, prod).

By using GITHUB_SHA and GITHUB_BRANCH in your CI/CD workflows, you ensure that the matrix accurately reflects the state of your contracts and their verifications. This makes it easier to determine if a particular consumer or provider version is safe to deploy in a specific environment, ultimately enabling seamless integration and deployment processes.

Example matrix:

Consumer Version (SHA)	Provider Version (SHA)	Branch	Environment	Verification Status	Comments
`abc123`	`xyz789`	`main`	`production`	Passed	The consumer and provider are both verified and deployed in production.
`def456`	`xyz789`	`main`	`staging`	Passed	The same provider version is compatible with a newer consumer version in staging.
`ghi789`	`xyz789`	`feature-x`	`development`	Failed	The consumer from a feature branch failed verification with the provider in the development environment.
`jkl012`	`uvw345`	`main`	`production`	Pending	A new provider version is pending verification against the consumer in production.

can-i-deploy

The can-i-deploy tool queries the Pact Broker to ensure that the consumer and provider versions are compatible before deployment. This tool provides an extra layer of safety, preventing the deployment of incompatible versions.on.

Webhooks:

Recall the consumer and provider flow.

The key is that, when there are multiple repos, the provider has to run test:provider (#3) after the consumer runs publish:pact (#2) but before the consumer can run can:i:deploy:consumer (#4) . The trigger to run test:provider (#3) has to happen automatically, webhooks handle this.

# Consumer
npm run test:consumer # (1)
npm run publish:pact  # (2)
npm run can:i:deploy:consumer # (4)
# only on main
npm run record:consumer:deployment # (5)

# Provider
npm run test:provider # (3) triggered by webhooks
npm run can:i:deploy:provider # (4)
# only on main
npm run record:consumer:deployment # (5)

Provider-driven (bi-directional) contract testing

Consumer-Driven Contract Testing is highly effective when the consumer has a close relationship with the provider, typically within the same organization. However, when dealing with third-party providers, especially those who might not be aware of or prioritize the specific needs of any single consumer, this approach becomes unfeasible. It might also happen that the provider is not testable locally, which makes running the tests from the consumer impossible on the provider side.

Provider-Driven Contract Testing makes more sense in these situations. The provider defines the contract (OpenAPI specification), and it’s up to the consumers to align with this specification. This shifts the responsibility to the consumer to ensure that they are compatible with the provider’s API, rather than the other way around.

The flow:

The provider uploads their OpenAPI specification to PactFlow.
The consumer uploads their contract, generated by their consumer-driven contract tests (CDCT).
PactFlow cross-validates the contracts to ensure compatibility between the provider's capabilities and the consumer's expectations.
Both the provider and the consumer use the can-i-deploy tool to verify whether it's safe to deploy their changes.

The key difference in this approach is that instead of the provider running the contract tests / verifying the contract on their end, they upload their OpenAPI specification to PactFlow. PactFlow then takes care of the verification by comparing the consumer's expectations with the provider's actual capabilities. When working with a 3rd party, the teams can upload a copy of their OpenAPI to PactFlow, and verify their contract testing.

Caveat: Risk of False Confidence: Since the testing is based on the OpenAPI spec of the provider rather than the actually running the consumer tests (the contract) on the provider side, there's a risk that the contract might not fully capture the nuances of the provider's implementation. This could lead to scenarios where a contract is deemed compatible even though the actual service could fail in production. This risk emphasizes the importance of maintaining up-to-date and accurate contracts.

This gap could be addressed with generating OpenAPI docs from types, or Zod-to-openapi (@asteasolutions/zod-to-openapi) or generating OpenAPI spec from e2e (an Optic feature)). We can also test the schema via a Cypress plugin cypress-ajv-schema-validator chaining on to already existing api calls.

With bi-directional contract testing, the consumer and the provider workflows are detached; the provider independently publishes their OpenAPI spec (to main) and the consumer tests against it.

Here is how it goes:

Generate the OpeAPI spec at the provider

Automate this step using tools like zod-to-openapi, swagger-jsdoc, generating OpenAPI documentation directly from TypeScript types, or generating the OpenAPI spec from e2e tests (using Optic). Manual spec writing is the last resort..

Ensure that the spec matches the real API

cypress-ajv-schema-validator: if you already have cy e2e and you want to easily chain on to the existing calls.

Optic: lint the schema and/or run the e2e suite against the OpenAPI spec through the Optic proxy.

Dredd: executes its own tests (magic!) against your openapi spec (feeds your local server, has hooks for things like auth.)

Publish the OpenAPI spec to the pact broker.

   pactflow publish-provider-contract
     src/api-docs/openapi.json # path to OpenAPI spec
     # if you also have classic CDCT in the same provider,
     # make sure to label the Bi-directional provider name differently
     --provider MoviesAPI-bi-directional
     --provider-app-version=$GITHUB_SHA # best practice
     --branch=$GITHUB_BRANCH # best practice
     --content-type application/json # yml ok too if you prefer
     --verification-exit-code=0 # needs it
      # can be anything, we just generate a file on e2e success to make Pact happy
     --verification-results ./cypress/verification-result.txt
     --verification-results-content-type text/plain # can be anything
     --verifier cypress # can be anything

Note that verification arguments are optional, and without them you get a warning at Pact broker that the OpenAPI spec is untested.

Execute the consumer contract tests

As you can notice, there is nothing about running the consumer tests on the provider side ( test:provider), can-i-deploy checks (can:i:deploy:provider), or recording the pact deployment (record:provider:deployment).

We have a sample consumer repo for BDCT pact-js-example-react-consumer.

The api calls are the same as the plain, non-UI app used int CDCT (link).

We cannot have CDCT and BDCT in the same contract relationship. Although, we can have the provider have consumer driven contracts with some consumers and provider driven contracts with others.

Consumer        -> CDCT  -> Provider

Consumer-React  <- BDCT  <- Provider

BDCT Scripts on the provider:

npm run generate:openapi # generates an OpenAPI doc from Zod schemas
npm run publish:pact-openapi # publishes the open api spec to Pact Broker for BDCT

Cypress adapter in a nutshell

UI Tests: You have your regular Cypress UI tests where you test the behavior of your application.

Stubbing the Network: You use cy.intercept to stub network requests. This allows you to mock responses from APIs, ensuring that your UI tests are consistent and isolated from backend fluctuations.

Generating a Pact Contract: When you use the cy.setupPact command in conjunction with cy.intercept, during the test execution, the Cypress Pact adapter generates a Pact contract. This contract captures the expectations set by the stubbed network requests.

CDCT-Like Use: The generated contract can be used in a similar fashion to a Consumer-Driven Contract Testing (CDCT) setup. You can publish this contract to a Pact Broker and have your provider verify it, ensuring that the provider meets the consumer's expectations.

CDCT vs e2e vs integration vs schema testing

Testing Lambda Handlers (Unit or Integration Tests)

Fast Feedback Loop:
- Quick Iterations: Enables rapid feedback through tests directly on the Lambda handler.
- Easier Debugging: Isolates the Lambda function, simplifying the debugging process.
Partial Coverage:
- Integration Points: Can test Lambda interactions with services like DynamoDB Local, KMS, Kafka, etc., within a Docker environment. (This is the distinction between unit and integration.)
- Error Simulation: Allows for controlled simulations of various edge cases and failure conditions.
Drawbacks:
- Less Realistic: Does not test the entire request-response lifecycle, potentially missing configuration or integration issues outside the Lambda function.
- Requires Mocks for Some Scenarios: Dependency isolation may necessitate mocks, which could lead to false positives.
- Other: Lacks testing for build & configuration, deployments, IAM, or interactions with external services.

Testing Deployments via HTTP (Temporary Stack, Dev, Stage)

Build & Configuration: Validates that all cloud resources, including CDK/Serverless Framework / infra as code, environment variables, secrets, and services (API Gateway, Lambda, S3), are correctly operational.
IAM Permissions: Ensures that the IAM roles and permissions are correctly configured.
Comprehensive Coverage: Tests the entire system, including authentication mechanisms and integrations with other services.
Drawbacks:
- Requires deployment, which adds complexity.
- Results in a slower feedback loop due to the deployment process.

Testing Local Endpoints via HTTP (Local)

Consistency: Same tests can be run locally as in deployment environments, offering low-cost testing.
Ease of Implementation: Simple to test API interactions before development using tools like REST clients or Postman.
Network Mocking: Easy to configure network mocks with tools like Mockoon, without code changes.
No Deployment Drawbacks: Avoids deployment, providing fast feedback.
Caveats: Lacks testing for build & configuration, deployments, IAM, or interactions with external services.

CDCT (Contract-Driven Contract Testing)

External Service Interaction Coverage: Addresses a major drawback of local testing by ensuring compatibility with external services.
Fast Feedback: Similar to local testing, CDCT avoids deployment and provides rapid feedback.
Caveats: Does not cover build & configuration, deployments, IAM, or full end-to-end interactions.

What kind of e2e testing would you replace with contract testing?

Replicate what contract tests cover or are overly simple: If an e2e test simply checks basic API functionality—like whether an API responds with the correct status code or data structure—and this is already thoroughly covered by a contract test, the e2e test may be redundant. This is especially true if the e2e test doesn't involve multi-step interactions, such as getting a response and then performing additional actions based on that response. In such cases, contract tests might provide the necessary coverage, making the e2e test unnecessary.
Have low value for effort: Tests that are complex to maintain but provide little value, such as those that test non-critical paths or scenarios unlikely to change, could be candidates for removal.
Streamline Redundant Coverage for build, config, deployment, IAM etc.: If your contract tests are comprehensive and your existing e2e tests already cover key build & configuration, deployment, infrastructure, and IAM, you consider trimming or removing overlapping e2e tests.
Optimize Deployment Strategy: You can consider skipping temporary stacks or ephemeral environments on PRs and instead focus on deployment testing in the dev environment. This strategy can save costs associated with deployments from scratch. However, be mindful that this may delay the discovery of issues related to build & configuration, deployment, or IAM until later stages. It's a tradeoff between cost efficiency and early detection.
Minimize the need for synchronized service versions covered bye 2e: In traditional testing setups, you might rely on running e2e tests across multiple services in later environments like dev or staging to ensure that changes to Service A don't break Services B, C; deploy A (run e2e for A), but also run e2e for B & C although they didn't deploy. With contract tests, this need is significantly reduced, as they ensure compatibility between services at the contract level.

How does CDCT fit with schema testing (Optic)?

There is a potential gap in Provider-Driven Contract Testing where the OpenAPI spec provided by the API might not accurately reflect the current implementation of the code. This gap can be mitigated by generating OpenAPI documentation directly from TypeScript types, or generating the OpenAPI spec from e2e tests (using Optic), ensuring that the specification remains in sync with the actual codebase.

Optic offers a low-cost, quick solution for schema validation by focusing solely on the OpenAPI specification, catching discrepancies early in the development process. On the other hand, Provider-Driven Contract Testing with Pact validates the interactions between consumers and the provider's OpenAPI spec, ensuring that the API's behavior aligns with consumer expectations.

By combining Optic for upfront schema validation and Pact for deeper contract verification, you can achieve a more comprehensive approach to API testing.

Message Queue testing

Message-based testing, like the one described in the Pact documentation on message queues, is meant for systems that communicate asynchronously using message queues instead of direct HTTP requests. Examples of message-based systems include RabbitMQ, Kafka, AWS SQS or EventBridge.

Here’s the difference:

Consumer-Driven Contract Testing (Standard Pact):

Focuses on direct HTTP requests (e.g., REST API).
The consumer sends a request and expects the provider to respond directly.
The contract defines these interactions (e.g., "When I make a GET request to /movies, I expect a response with a list of movies").

Message Queue Testing (Pact with Messaging):
- Instead of HTTP requests, services exchange messages via a message queue (asynchronous).
- Consumer in this context is the service consuming messages from a queue (not making requests to the provider).
- Provider is the service publishing messages to the queue.
- The contract defines what messages the provider is expected to send or receive, but there is no direct request-response. It’s about verifying the structure and content of the messages flowing through the queue.
- Example: Instead of sending a request to get data, the consumer might subscribe to a message topic or queue, and the provider sends messages through that queue.

Purpose of Message Queue Testing

You care about message queues when:

Asynchronous Communication: The consumer and provider don’t talk directly through HTTP requests but communicate through a third party (e.g., RabbitMQ or Kafka).
Event-Driven Architecture: Systems using event-driven patterns often use queues. The provider might broadcast events like “Order Placed” or “Movie Created,” and the consumer listens for those events.
Ensuring Decoupled Communication: Just as with direct HTTP contract testing, message queue testing ensures that the consumer and provider agree on the format and content of messages exchanged, even though they’re not communicating synchronously.

Example Scenario:

Imagine a system where:

Provider (e.g., a payment service) doesn’t respond to HTTP requests directly but publishes a message to a queue saying, “Payment Processed.”
Consumer (e.g., an order system) listens to this queue for a message saying, “Payment Processed,” and takes action when it receives that message.

In this case, you'd want to test that:

The provider correctly publishes the "Payment Processed" message in the right format.
The consumer knows how to handle that message and react accordingly.

This is where Pact’s message testing comes in handy.

Message queue consumer tests in short

Write the consumer test
Message q consumer test: simulates receiving a message from the queue.
Traditional consumer test: simulates receiving a response from the provider.
Execute the and generate a contract / pact / json file. (same)
The message queue contract specifies the expected structure of the message from the producer.
Traditional consumer test contract specifies the expected structure of the response from the provider.
Once the contract is created, Pact mockProducer takes over as if we are locally pushing messages to the queue (expectsToReceive) and verifying the message agains our src code event-message-consumer-handler.
This is similar to traditional consumer test where the mockProvider takes over as if we are locally serving the provider API and executing the tests against that.
The test can fail at the verify portion, if / when the simulated message does not match our src code event-message-consumer.
This is similar to a traditional consumer test where the test can only fail at executeTest portion, if / when the assertions do not match the specifications.

Here is a contract test vs message queue test side by side:

// message queue consumer test

it('...', () => {
  await messagePact
    // simulate/specify the expected message
    .expectsToReceive('some text for event name')
    .withContent({ this is the meat and bones of the event })
    .withMetaData({ contentType: 'application/json' })

    // feed the message into the event consumer
    .verify(asynchronousBodyHandler(yourEventConsumerHandler))
  })
})

// consumer test

it('...', () => {
  await pact
    // simulate/specify how the provider should respond
    .addInteraction(...)
    .given(some state name) // optional
    .uponReceiving(the name of the test)
    .withRequest(http verb, path)
    .willRespondWith({ this is the meat and bones of the response })

    .executeTest(async(mockProvider) => {
    // call the source code &
    // make assertions against the mockProvider/contract
  })
})

The flow is the same as a traditional consumer test, in fact if the repo has both, both tests are executed

npm run test:consumer
npm run publish:pact

Message queue provider tests in short

These tests verify that the messages the provider produces match the structure the consumer expects.

In contrast, in traditional CDCT, the tests verify that the responses of the provider API match the structure the consumer expects.

The message consumer already generated the contract and published it. (Same thing as the traditional CDCT)
The provider has one test per consumer to ensure all is satisfactory. Most of the file is about setting up the options. (Again, same)
We ensure that the provider api is running locally. (Again, same)
The message queue consumer tests execute against the provider/message-producer, as if they are a regular message-consumer running locally.

In contrast, in traditional CDCT, the consumer tests execute against the provider api, as if they are a regular API client running locally.

Execution (Same as traditional CDCT)

Run the server/service.

npm run start

The provider/producer server/service has to be running locally for the provider message queue tests to be executed.

Run the provider/producer message queue test

npm run test:provider

Two in one:

npm run test:provider:ci

`messageProviders` vs `stateHandlers` on the producer/provider side

In message queue testing, messageProviders define the messages the provider should produce.

Each key is an event (e.g., 'movie-created') linked to a handler (providerWithMetadata from Pact) that generates the message.

Metadata like contentType ensures correct message format interpretation.

In traditional CDCT, stateHandlers (although optional) are the primary way of setting the test state prior to a consumer test executing against the provider.

// messageProvider example

import { providerWithMetadata } from "@pact-foundation/pact";
import { produceEvent } from "your-event-producer";

export const messageProviders = {
  "a create event": providerWithMetadata(
    () => produceEvent(entity, "created"),
    { contentType: "application/json" }
  ),
  "an update event": providerWithMetadata(
    () => produceEvent(entity, "updated"),
    { contentType: "application/json" }
  ),
  "a movie-deleted event": providerWithMetadata(
    () => produceEvent(entity, "deleted"),
    { contentType: "application/json" }
  ),
};

// stateHandler example
import type { AnyJson } from '@pact-foundation/pact/src/common/jsonTypes'

export const stateHandlers: = {
  'Has an entity with a specific ID': async (params: AnyJson) => {
    const { id } = params // params are passed in from the consumer test

    // some logic to create the entity

    return {
      description: `Entity with ID ${id} is set up.`
    }
  },
  'An existing entity exists': async (params: AnyJson) => {
    const { name, year } = params // whatever params

    // some logic to create the entity

    return {
      description: `Movie with name "${name}" is set up.`
    }
  },
  'No entitiess exist': {
    // with state handlers we also have granular control
    // of the state before and the after the test from the consumer
    setup: async () => {
      await truncateTables() // clean up the db, however you do it
    },
    teardown: async () => {
      console.log('Teardown of state No movies exist ran...')
      // Logic to restore default entities or clean up further can go here.
    }
  }

Breaking changes

You have really two good options, and additional workarounds, depending on the coordination and maturity of your teams.

Coordinating Matching PRs: When a breaking change is needed, both teams (provider and consumer) collaborate in tandem. They create matching feature branches or PRs, ensuring that the consumer can adopt the new changes as the provider introduces them. This prevents downtime or breaking of functionality and ensures that both services are always compatible.

Backward Compatibility: If synchronizing changes isn't feasible, the provider introduces a backward-compatible update that supports both the old and the new versions of the contract. This allows the consumer to gradually adopt the new behavior while the old functionality is deprecated over time. Once the consumer has fully migrated to the new behavior, the old behavior can safely be removed.

Real-World Scenarios

Coordinated matching PRs (preferred): If both consumer and provider teams communicate and align changes in their respective services, this should be your go-to approach. It's clean, avoids breaking things, and is more controlled.
Backward compatibility (fallback): If immediate synchronization isn't possible, backward compatibility ensures the system continues to function as expected until the consumer adopts the new features.

Workarounds (to blasphemy)

Pending pacts (safety net) enablePending: It's there to prevent unnecessary test failures if a consumer inadvertently introduces a breaking change without proper alignment. It essentially provides a "grace period" to allow providers time to adapt without blocking deployment, particularly useful in larger, less-coordinated teams or projects with complex dependencies.
WIP pacts (experimental features) includeWipPactsSince : WIP pacts come into play when consumers are working on experimental or long-running feature branches that haven't yet been finalized. WIP pacts give providers a heads-up that the consumer is working on something new, but won't block the provider's build if they don't support it yet. Think of it as a way to handle feature branches that are in progress but haven’t been fully integrated. This is useful when teams are working on new features in parallel but aren't ready to fully verify everything against the provider yet. The provider can continue working without failing for pacts that aren't fully baked. It gets removed before the merge to main.

Pending vs. WIP Pacts

Pending Pacts are for unexpected or uncoordinated breaking changes. They provide a temporary buffer where the provider tests won’t fail immediately if a consumer introduces breaking changes, allowing the provider time to catch up.
WIP Pacts are for work-in-progress or experimental features in long-running feature branches. The provider acknowledges that the consumer is working on something but doesn’t yet block the provider from progressing with their build if those changes aren’t supported yet.

Both are "workarounds" but solve slightly different problems in chaotic or asynchronous environments. In well-coordinated environments, you should not need either of these.

Sample repos

Based on the excellent Contract Testing in Action book by Marie Cruz and Lewis Prescott, these are my interpretations of the examples.

https://github.com/muratkeremozcan/pact-js-example-consumer

https://github.com/muratkeremozcan/pact-js-example-provider

https://github.com/muratkeremozcan/pact-js-example-react-consumer

Automating API Documentation: A Journey from TypeScript to OpenAPI and Schema Governence with Optic

Murat K Ozcan — Fri, 01 Dec 2023 20:39:27 +0000

In the previous blog post Documenting and Testing Schemas of Serverless Stacks with Optic & Cypress, we focused on the benefits of schema testing and governance. Briefly, some of the core problems addressed were:

Effortless API Documentation: Automating the API documentation creation and update, ensuring that our API documentation evolves in tandem with our schema, courtesy of Optic's forward-governing capabilities.
Detecting Cross-Service Integration Issues Up Front: Having the ability to test and detect cross-service integration issues locally; long before we have to deploy services to a common environment and test them with e2e.
Making the Process Painless: Re-using existing HTTP/e2e suites (or their subsets) to update and/or verify our OpenAPI schema; no additional work for team-level service owners.

In the previous approach, we relied on Optic's key feature to capture the HTTP traffic using the Optic proxy. Henceforth in the text, we will refer to that as HTTP-capturing Approach:

Capture HTTP traffic during e2e tests.
Generate or update OpenAPI spec based on captured traffic.
Perform schema governance with Optic.

In this post, we want to evaluate an alternative approach of generating our OpenAPI documentation from our TypeScript types. Henceforth in the text, we will refer to that as TypeScript-based Approach:

Organize our TypeScript request and response types into a certain folder.
Use these TypeScript request & response types in our lambda code.
Generate JSON schemas with the ts-json-schema-generator package.
Generate the OpenAPI spec with the openapi-types package.
Perform schema governance with Optic.

For our case study, we utilize the same comprehensive repository that includes a TypeScript-based backend and frontend, AWS Lambdas, and temporary stacks managed with AWS CDK. These components are tested in PRs through backend and frontend e2e tests targeting temporary deployments, as well as consistent deployments in development, staging, and production environments.

Here is the link to the repo and initial Optic PR.
Here is the new PR with OpenAPI generation from types.

Note: This case study results in two different OpenAPI spec files, each representing a distinct approach. The traditional method involving HTTP tests and Optic proxy generates an openapi.yml file, whereas the new type-based method produces an openapi.json file. Keep in mind that in practical applications, the choice of file format (YAML or JSON) depends on your specific needs. This case study also includes two sets of Optic diff and lint scripts, which is unusual for most real-world applications but was necessary here to clearly differentiate between the two approaches within a single repository."

1. Organize our TypeScript request and response types into a certain folder
- Create a TS file per handler to house the request and response types
2. Use these TypeScript request & response types in our lambda code
3. Generate JSON schemas with the ts-json-schema-generator package
4. Generate the OpenAPI spec with the openapi-types package.
- Create the openapi.ts file
- Create a script to execute the openapi.ts file and generate the OpenAPI spec.
5. Perform schema governance with Optic
Comparison of the TS approach vs http-capture approach
- TypeScript-based Approach with Optic: Use Case Scenario:
- HTTP-based Approach with Optic: Use Case Scenario
Conclusion
Addendum: Using the Types at the Lambdas as the Source of Truth (Recommended)

1. Organize our TypeScript request and response types into a certain folder

#(We are here)
Using types -> JSON schemas -> OpenAPI spec -> schema diffing with Optic

In the source code, let's examine our lambda handlers. Currently, they return HTTP responses but do not utilize specific request or response types.

// ./infrastructure/Lambda/delete/lambda/index.ts

return httpResponse(
  200,
  JSON.stringify({ message: "Todo deleted successfully." })
);

// ./infrastructure/Lambda/get/lambda/index.ts

const { Items }: DynamoDB.ScanOutput = await dynamoDB
  .scan({ TableName: tableName })
  .promise();

return httpResponse(200, JSON.stringify({ todos: Items }));

// ./infrastructure/Lambda/post/lambda/index.ts

const todo: Todo = {
  id: uuidv4(),
  todo_completed,
  todo_description,
  todo_name,
};

await dynamoDB.put({ TableName: tableName, Item: todo }).promise();

return httpResponse(200, JSON.stringify({ todo }));

// ./infrastructure/Lambda/put/lambda/index.ts

const updatedTodo: Todo = {
  id,
  todo_name,
  todo_description,
  todo_completed,
};

return httpResponse(200, JSON.stringify({ todo: updatedTodo }));

If we define request and response types centrally and use them across our lambda functions, we can leverage them to automatically generate an OpenAPI specification. This method ensures that our OpenAPI documentation remains synchronized with our codebase, reflecting any changes in our types seamlessly. The cost is the requirement of a one-time setup investment. This approach not only streamlines our documentation workflow but also enhances the accuracy and reliability of our API specifications.

The distinction from our previous approach, which involved capturing HTTP tests via the Optic proxy to verify or update our OpenAPI schema, lies in our newfound reliance on TypeScript types. In the earlier method, we depended on capturing real HTTP traffic to reflect our API's behavior. Now, we pivot to a more proactive approach, using our TypeScript request and response types as the primary source of truth.

Create a TS file per handler to house the request and response types

This folder can be anywhere in our repository. For the example's sake, let's assume it is under ./infrastructure/api-specs. We will also make a possible real world assumption that there may be multiple versions of the API, and we are working with v1.

├── api-specs
  └── v1
      ├── deleteTodo.ts
      ├── getTodos.ts
      ├── postTodo.ts
      ├── putTodo.ts

Define the response body and if needed the request body per handler. Here we get a clue from our already existing lambda code.

// ./infrastructure/api-specs/v1/deleteTodo.ts

export type ResponseBody = {
  message: string;
};

Import any types used in these responses from anywhere. Example: type Todo.

// ./infrastructure/api-specs/v1/getTodos.ts

// import any types used in these responses from anywhere
import type { Todo } from "customTypes/index";

export type ResponseBody = {
  todos: Todo[];
};

If we have a possible request body, define it here. Although it may or may not be used in the lambda function, we want to define these here so that our OpenAPI doc is complete for the benefit of our API's consumers.

// ./infrastructure/api-specs/v1/postTodo.ts

import type { Todo } from "customTypes/index";

export type ResponseBody = {
  todo: Todo;
};

export type RequestBody = {
  todo: Todo;
};

// ./infrastructure/api-specs/v1/putTodo.ts

import type { Todo } from "customTypes/index";

export type ResponseBody = {
  todo: Todo;
};

export type RequestBody = {
  todo: Todo;
};

2. Use these TypeScript request & response types in our lambda code

#(We are here still)
Using types -> JSON schemas -> OpenAPI spec -> schema diffing with Optic

Now that we have the request and response types in a central location, use them in our lambda code. Here are the key changes in the repo example. You can find the full code in the repo as well as the PR.

Import the type from the central location, and use it in the handler. All we are doing differently here is making an assignment to a const with a type and using it in the http response.

// ./infrastructure/Lambda/delete/lambda/index.ts

import { ResponseBody } from "api-specs/v1/deleteTodo";

// the assignment
const response: ResponseBody = {
  message: "Todo deleted successfully.",
};

// before:
// JSON.stringify({message: 'Todo deleted successfully.'}),
// after:
return httpResponse(200, JSON.stringify(response));

Disclaimer; we may need to do some additional work to make TypeScript happy, but this will also be a one-time effort. We foresee that most the time the code change will be minimal, and they will be future proof because of the type protection.

Check out the Addendum section for an alternative approach where the types in the lambdas are the source of the truth, which can mediate such issues.

// ./infrastructure/Lambda/get/lambda/index.ts

const { Items }: DynamoDB.ScanOutput = await dynamoDB
  .scan({ TableName: tableName })
  .promise();

const response: ResponseBody = { todos: Items };

// before:
// return httpResponse(200, JSON.stringify({todos: Items}))
// after:
return httpResponse(200, JSON.stringify(response));

// ./infrastructure/Lambda/post/lambda/index.ts

import type { ResponseBody } from "api-specs/v1/postTodo";

// this part is the same
const todo: Todo = {
  id: uuidv4(),
  todo_completed,
  todo_description,
  todo_name,
};
await dynamoDB.put({ TableName: tableName, Item: todo }).promise();

// Use the Response Type in the Lambda Handler
const response: ResponseBody = { todo };

// before:
// return httpResponse(200, JSON.stringify({todo}))
// after:
return httpResponse(200, JSON.stringify(response));

// ./infrastructure/Lambda/put/lambda/index.ts

const updatedTodo: Todo = {
  id,
  todo_name,
  todo_description,
  todo_completed,
};

// Use the Response Type in the Lambda Handler
const response: ResponseBody = { todo: updatedTodo };

// before:
// return httpResponse(200, JSON.stringify({todo: updatedTodo}))
// after:
return httpResponse(200, JSON.stringify(response));

Note: In the following sections, the files can be named and placed anywhere of your preference. The scripts may need adjusting. It is only important that if you plan to use them ubiquitously, every service repo follows the set pattern.

3. Generate JSON schemas with the `ts-json-schema-generator` package

############# (We are here)
Using types -> JSON schemas -> OpenAPI spec -> schema diffing with Optic

The library is used to generate json schemas, which in turn will get used in the final open api spec. We have to use an elaborate script here, but it should not require modification and should be easy to reuse. Consider creating a package for these scripts for repeated usage.

You can find the final code in the repo and the PR.

// ./infrastructure/api-specs/generate-json-schemas.ts

import * as tsj from "ts-json-schema-generator";
import * as fs from "fs";
import * as path from "path";

// Function to recursively find all .ts files in subdirectories and exclude 'openapi.ts'
function findTsSchemaFiles(
  dir: string,
  fileList: string[] = [],
  isRoot = true
): string[] {
  fs.readdirSync(dir, { withFileTypes: true }).forEach((dirent) => {
    const fullPath = path.join(dir, dirent.name);
    if (dirent.isDirectory()) {
      // Process subdirectories; skip processing the api-specs folder root
      if (!isRoot) {
        fileList = findTsSchemaFiles(fullPath, fileList, false);
      }
    } else if (
      dirent.isFile() &&
      dirent.name.endsWith(".ts") &&
      dirent.name !== "openapi.ts"
    ) {
      // Add only .ts files that are not named 'openapi.ts', and only if it's not in the root directory
      if (!isRoot) {
        fileList.push(fullPath);
      }
    }
  });

  // If it's the root directory, proceed to its subdirectories
  if (isRoot) {
    fs.readdirSync(dir, { withFileTypes: true }).forEach((dirent) => {
      if (dirent.isDirectory()) {
        fileList = findTsSchemaFiles(
          path.join(dir, dirent.name),
          fileList,
          false
        );
      }
    });
  }

  return fileList;
}

// Function to generate JSON schema from a TypeScript file
function generateSchema(tsFilePath: string): void {
  const schemaFilePath = tsFilePath.replace(".ts", ".schema.json");
  const config = {
    path: tsFilePath,
    tsconfig: path.join(__dirname, "../tsconfig.json"),
    noTypeCheck: true,
    // generate schema for all types;
    // RequestBody, ResponseBody and all the imported types they need
    type: "*",
    // avoid creating shared $ref definitions (which is not valid in OpenAPI)
    // this e results in JSON schema files that directly embed the type definitions,
    // instead of referring to them via $ref
    expose: "none" as const,
  };

  try {
    const schema = tsj.createGenerator(config).createSchema(config.type);
    fs.writeFileSync(schemaFilePath, JSON.stringify(schema, null, 2));
    console.log(`Generated JSON schema for ${tsFilePath}`);
  } catch (error) {
    console.error(`Error generating JSON schema for ${tsFilePath}:`, error);
  }
}

// Main execution
const openApiFiles = findTsSchemaFiles(__dirname);
openApiFiles.forEach(generateSchema);

├── api-specs
  ├── generate-json-schemas.ts
  └── v1
      ├── deleteTodo.ts
      ├── getTodos.ts
      ├── postTodo.ts
      ├── putTodo.ts

4. Generate the OpenAPI spec with the `openapi-types` package

############################# (We are here)
Using types -> JSON schemas -> OpenAPI spec -> schema diffing with Optic

We need to utilize this library and make a one time investment to create an openapi.ts file. This is the file that will generate the OpenAPI spec. Note that if we add new endpoints to our api, we will need to add it to this file as well. If we have different versions of the API, we might need multiple files, but since these are all in code, using helper modules is a possibility albeit at the cost of abstraction.

Create the `openapi.ts` file

The general pattern in the file:

Import the JSON schemas, created in the previous step: import getTodosV1 from './getTodos.schema.json'
In the components section, identify these schemas:

  components: {
    schemas: {
      getTodosV1: getTodosV1.definitions as OpenAPIV3_1.SchemaObject,
    },

Reference the components

  content: {
    'application/json': {
      schema: {
        $ref: '#/components/schemas/getTodosV1',
      },
    },
  },

Aside from having to add new endpoints to our API, this file does not need any maintenance. If we have new endpoints though, we might need some copy pasting for them.

// ./infrastructure/api-specs/v1/openapi.ts

import type { OpenAPIV3_1 } from "openapi-types";
import fs from "fs";
import path from "path";
import getTodosV1 from "./getTodos.schema.json";
import deleteTodoV1 from "./deleteTodo.schema.json";
import postTodoV1 from "./postTodo.schema.json";
import putTodoV1 from "./putTodo.schema.json";

export const openapi: OpenAPIV3_1.Document = {
  openapi: "3.0.1",
  info: {
    title: "aws cdk in practice specification",
    version: "1.0.0",
  },
  paths: {
    "/": {
      get: {
        responses: {
          200: {
            description: "Success",
            content: {
              "application/json": {
                schema: {
                  $ref: "#/components/schemas/getTodosV1",
                },
              },
            },
          },
        },
      },
      post: {
        requestBody: {
          required: true,
          content: {
            "application/json": {
              schema: {
                $ref: "#/components/schemas/postTodoV1",
              },
            },
          },
        },
        responses: {
          200: {
            description: "Success",
            content: {
              "application/json": {
                schema: {
                  $ref: "#/components/schemas/postTodoV1",
                },
              },
            },
          },
        },
      },
      put: {
        requestBody: {
          required: true,
          content: {
            "application/json": {
              schema: {
                $ref: "#/components/schemas/putTodoV1",
              },
            },
          },
        },
        responses: {
          200: {
            description: "Success",
            content: {
              "application/json": {
                schema: {
                  $ref: "#/components/schemas/putTodoV1",
                },
              },
            },
          },
        },
      },
    },
    "/{id}": {
      delete: {
        parameters: [
          {
            name: "id",
            in: "path",
            required: true,
            schema: {
              type: "string",
            },
          },
        ],
        responses: {
          200: {
            description: "Success",
            content: {
              "application/json": {
                schema: {
                  $ref: "#/components/schemas/deleteTodoV1",
                },
              },
            },
          },
        },
      },
    },
  },
  components: {
    schemas: {
      getTodosV1: getTodosV1.definitions as OpenAPIV3_1.SchemaObject,

      deleteTodoV1: deleteTodoV1.definitions as OpenAPIV3_1.SchemaObject,

      postTodoV1: postTodoV1.definitions as OpenAPIV3_1.SchemaObject,

      putTodoV1: putTodoV1.definitions as OpenAPIV3_1.SchemaObject,
    },
  },
};

const filePath = path.join(__dirname, "openapi.json");
fs.writeFileSync(filePath, JSON.stringify(openapi, null, 2));

├── api-specs
  ├── generate-json-schemas.ts
  └── v1
      ├── openapi.ts
      ├── deleteTodo.ts
      ├── getTodos.ts
      ├── postTodo.ts
      ├── putTodo.ts

Create a script to execute the `openapi.ts` file and generate the OpenAPI spec.

Now that we have the openapi.ts file, We need a script to find the openapi.ts file(s) and execute them, in order to generate the OpenAPI spec. There may be multiple version folders, and the script will accommodate that.

// ./infrastructure/api-specs/generate-openapi-docs.ts

import fs from "fs";
import path from "path";

// Function to recursively find all openapi.ts files
function findOpenApiFiles(dir: string, fileList: string[] = []): string[] {
  fs.readdirSync(dir, { withFileTypes: true }).forEach((dirent) => {
    const filePath = path.join(dir, dirent.name);
    if (dirent.isDirectory()) {
      fileList = findOpenApiFiles(filePath, fileList);
    } else if (dirent.isFile() && dirent.name === "openapi.ts") {
      fileList.push(filePath);
    }
  });
  return fileList;
}

// Find all openapi.ts files in src/api-specs
const openApiFiles = findOpenApiFiles(__dirname);
console.log(openApiFiles);

// Import and execute each openapi.ts file to generate openapi.json
openApiFiles.forEach((file) => {
  import(path.resolve(file))
    .then(() => console.log(`Generated OpenAPI document for ${file}`))
    .catch((err) =>
      console.error(`Error generating OpenAPI document for ${file}:`, err)
    );
});

We are including a bonus script here to reset/delete all the generated json, for demo usage.

// ./infrastructure/api-specs/delete-json-files.ts

import fs from "fs";
import path from "path";

// Function to recursively delete all .json files
function deleteJsonFiles(dir: string): void {
  fs.readdirSync(dir, { withFileTypes: true }).forEach((dirent) => {
    const fullPath = path.join(dir, dirent.name);
    if (dirent.isDirectory()) {
      // Recursively delete .json files in subdirectories
      deleteJsonFiles(fullPath);
    } else if (dirent.isFile() && dirent.name.endsWith(".json")) {
      // Delete the file if it's a .json file
      fs.unlinkSync(fullPath);
      console.log(`Deleted file: ${fullPath}`);
    }
  });
}

const apiSpecsDir = path.join(__dirname);
deleteJsonFiles(apiSpecsDir);

├── api-specs
  ### these script files can be a part of a package
  ├── generate-json-schemas.ts
  ├── generate-openapi-docs.ts
  ├── delete-json-files.ts
  └── v1
      ├── openapi.ts
      ├── deleteTodo.ts
      ├── getTodos.ts
      ├── postTodo.ts
      ├── putTodo.ts

The OpenAPI file generated is of json type, for the example purposes and not to intermix it with the http-capture approach we have referenced before which uses the yml file type.

5. Perform schema governance with Optic

Optic helps in detecting schema changes in our OpenAPI specification (openapi.json or yml). It categorizes these changes as either breaking or non-breaking; which is key for us to identify them. This is something we do not get with our own testing; we would just update the types and/or the test and would not really know if they would break future service integrations unless we are consistently very careful and knowledgeable.

We can use Optic locally, but CI is where it shines, making it obvious for us to detect such changes, or breakages in a schema, not only have analytics and history (Optic Cloud) but also have a neat representation of our OpenAPI spec in the form of online documentation.

Now that we have the OpenAPI spec, we can create a few utility scripts. You can reference the repo or the PR for the script detail, and modify them to your needs in the real world. We have differentiated the -json vs -yml scripts for reasons previously mentioned with the two approaches.

yarn update:api-docs # generates JSON schemas and OpenAPI docs
yarn optic:lint-json # lints our OpenAPI spec for validity
yarn optic:diff-json # detects breaking schema changes vs main with Optic

"reset:schemas": "npx ts-node ./api-specs/delete-json-files.ts",
"build:schemas": "npx ts-node ./api-specs/generate-json-schemas.ts",
"build:open-api": "npx ts-node ./api-specs/generate-openapi-docs.ts",
"update:api-docs": "yarn reset:schemas && yarn build:schemas && yarn build:open-api"
"optic:diff-json": "optic diff ./api-specs/v1/openapi.json --base main --check'",
"optic:lint-json": "optic lint ./api-specs/v1/openapi.json",

Note that in this repository we are already running Optic Cloud with the http-capture approach, in CI. We cannot have 2 OpenAPI specifications; one for TS approach and one for http-capture approach, and perform schema governance. However, we will propose a simple CI config, for the TS approach. Mind that in the repo you will find the http-capture approach working in the CI.

name: Optic-cloud-features
on:
  pull_request:
    types: [opened, reopened, edited, synchronize]

concurrency:
  group: ${{ github.ref }} && ${{ github.workflow }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
  optic-cloud:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: write
      pull-requests: write
    steps:
      - name: Checkout
        uses: actions/checkout@v4.1.1

      - name: Use node
        uses: actions/setup-node@v3.8.2
        with:
          node-version-file: .nvmrc
          cache: yarn

      - name: Install dependencies
        run: yarn install --frozen-lockfile

      - name: Update api docs
        run: yarn update:api-docs

      # include a preview and changelog in each PR comment
      # sync every OpenAPI spec to Optic Cloud, diff & lint the schema as well
      - name: Run Optic
        uses: opticdev/action@v1
        with:
          # Your Optic Cloud Token
          optic_token: ${{ secrets.OPTIC_TOKEN }}
          # A GitHub token with access to create comments on pull requests
          github_token: ${{ secrets.GITHUB_TOKEN }}
          # If true, standard check failures will cause this action to fail.
          # If false, standard check failures will show in PR comments and
          # in Optic Cloud but will not cause the action to fail
          standards_fail: true
          additional_args: --match **/**/openapi.json
          compare_from_pr: main
          compare_from_push: main
        env:
          CI: true

Comparison of the TS approach vs http-capture approach

TypeScript-based Approach with Optic: Use Case Scenario:

Identifying Type Changes: A breaking change is introduced to our TypeScript types, which could potentially alter the structure or behavior of our API.
Auto-Generating OpenAPI Documentation: Following these type changes, our OpenAPI documentation is automatically regenerated using the update:api-docs command. This process ensures that the OpenAPI specification reflects the latest state of our TypeScript types.
Detecting Breaking Changes with Optic: The command optic:diff-json is used to detect any breaking changes in the updated OpenAPI specification. This step is crucial for identifying discrepancies that could affect API consumers.
Decision Making and Communication:
- Rollback or Update: Based on the nature of the detected changes, we decide whether to roll back the changes to our types or proceed with updating the API documentation to reflect these changes.
- Consumer Notification and Version Update: For significant changes, especially those that are breaking, we notify service consumers of the potential impact. Additionally, we update the version of our OpenAPI specification. This step is essential to communicate the change effectively and ensure that Optic's checks recognize the update as a deliberate and managed change.

This approach emphasizes the role of TypeScript types in driving the API documentation process. By automatically generating the OpenAPI specification from TypeScript types, we ensure that our documentation is always in sync with our codebase. Moreover, it highlights the use of Optic as a tool for governing API schema changes, ensuring that updates are tracked, managed, and communicated effectively.

HTTP-based Approach with Optic: Use Case Scenario

Optic can also validate the accuracy of the OpenAPI spec by capturing traffic from E2E tests and comparing it against the OpenAPI spec (openapi.json or yml file).

# in ./infrastructure folder
yarn optic:verify # Captures E2E test traffic, detects breaking schema changes

Similar to Optic validate, Optic update allows for interactive updates to the OpenAPI spec. It's similar to optic:verify but includes prompts for additional observed changes during E2E test capture.

# in ./infrastructure folder
yarn optic:update

We generally would run optic:verify in CI to vet our http tests against our OpenAPI specification. We would run optic:update when we know there are changes in our code, not necessarily types, but could be anything. We would theoretically update the code, perhaps update the http tests, run optic:verify and record a new OpenAPI spec. This usage is very similar to Jest snapshot testing, where the snapshot is the OpenAPI spec.

Use case scenario:

Identifying Potential Breaking Changes: Any black-box breaking change is made in our service code. This change isn't necessarily related to type definitions but could impact the behavior of the API.
Verification with Optic: We use optic:verify to identify these changes.
- Real E2E Tests: This involves executing real end-to-end tests against a local server or a deployment with our code changes.
- Comparison Against OpenAPI Spec: During this process, Optic verifies the traffic captured during these tests against our existing OpenAPI specification.
Updating OpenAPI Documentation: If the actual behavior (captured traffic) does not match the current OpenAPI documentation, we are suggested by Optic to update the OpenAPI docs, so we utilize optic:update to bring the documentation in line with reality. This ensures that optic:verify can now successfully pass, providing us with an accurate coverage report of our OpenAPI documentation.
Detecting Breaking Changes: The optic:diff command is used to identify any breaking changes compared to the main/master branch. This step is crucial for understanding the impact of recent changes on the overall API.
Making Informed Decisions: Depending on the nature of the detected changes, we make a decision:
- Discarding or Updating: We either discard the recent changes or update our API documentation using optic:update.
- Communication and Versioning: For significant or breaking changes, we communicate these changes to our service consumers. Additionally, we update the version of our OpenAPI specification to reflect these changes, ensuring that Optic's checks pass.

This approach effectively leverages Optic's capabilities to manage and maintain accurate and up-to-date API documentation, especially in the context of continuous integration and delivery. It highlights the importance of aligning actual service behavior with documented API contracts, ensuring consistency and reliability for API consumers.

Conclusion

In conclusion, this exploration into generating OpenAPI documentation from TypeScript types presents a viable alternative to the HTTP-capturing method. While both approaches have their merits, the TypeScript-based method offers a cost effective and type-safe strategy, ensuring that API documentation remains closely aligned with our codebase. It is particularly beneficial for teams seeking to automate their documentation process but keep things simple and fast, albeit it does require the one time investment, and updating of the opeanapi.ts file when there are new api endpoints.

On the other hand, using real http tests to qualify our OpenAPI documentation gives us the ability to not only verify but also modify our spec. Reality is often different to the wishful perception of it, and the HTTP-capturing method is immune to that. The coverage report of our http tests versus our OpenAPI docs is a killer feature as well, giving us the proof of coverage that matters; "Are our tests covering what we publish that we feature?".

This is a much better alternative to source code coverage, akin to ui-interaction-coverage in Cypress Cloud (for UI apps).

In either approach we get the schema governing features of Optic. We get optic-diff locally or in CI, terminal output for the free version, and human readable web version with PR comments using Optic Cloud.

As a reminder, here are the Optic Cloud benefits in brief

optic diff-all (runs multiple specs at once instead of having to pick each OpenAPI spec)
PR comments
Catalogue of your API changes over time, analytics
centralized styles guides, with AI
support

Whether we use HTTP-capturing or TypeScript-based approach, we still get the same schema governance with Optic.

Looking ahead, further enhancements could include exploring ways to integrate these methods into more complex, multi-service architectures, starting with creating an internal package for scripts that will be repeated in each service repo.

Your feedback and experiences with these approaches are invaluable—feel free to share your thoughts and insights.

Addendum: Using the Types at the Lambdas as the Source of Truth (Recommended)

Previously, we created type definitions in the openapi-spec folder and then used them in the lambda. This approach ensured that the same type was being used in both the lambda and the OpenAPI spec, thereby creating an alignment.

types in getTodos -> lambda && OpenAPI spec

However, a better approach might be to define a type in the lambda and then use it directly in the openAPI spec, with the getTodos type file serving as an intermediary.

lambda -> types in getTodos -> OpenAPI spec

In this manner, the lambda becomes the source of truth.

// ./infrastructure/Lambda/delete/index.ts

// (1) define & export a type for the response body,
export type DeleteResponseBody = {
  message: string;
};

export const handler = async (event: DeleteEvent) => {
  // ...

  // (2) and use it in the Lambda Handler
  const response: DeleteResponseBody = {
    message: 'Todo deleted successfully.',
  };

  return httpResponse(200, JSON.stringify(response));
};

Reuse the exported DeleteResponseBody in API docs.

// ./infrastructure/api-specs/v1/getTodos.ts

// (3) re-use the exported type in api docs
import type {DeleteResponseBody} from '../../Lambda/delete/lambda/index';

export type ResponseBody = DeleteResponseBody;

For the POST endpoint:

// ./infrastructure/Lambda/post/index.ts

export type PostBody = {
  todo: Todo;
};

export const handler = async (event: PostEvent) => {
  // ...

  const response: PostBody = { todo };

  return httpResponse(200, JSON.stringify(response));
};

// ./infrastructure/api-specs/v1/postTodo.ts

import type {PostBody} from '../../Lambda/post/lambda/index';

export type ResponseBody = PostBody;
export type RequestBody = PostBody;

For the PUT endpoint:

// ./infrastructure/Lambda/put/index.ts

export type PutBody = {
  todo: Todo;
};

export const handler = async (event: PutEvent) => {
  // ...

  const response: PutBody = { todo: updatedTodo };

  return httpResponse(200, JSON.stringify(response));
};

// ./infrastructure/api-specs/v1/putTodo.ts

import type {PutBody} from '../../Lambda/put/lambda/index';

export type ResponseBody = PutBody;
export type RequestBody = PutBody;

The GET endpoint is particularly noteworthy as it differentiates this approach from the previous one. In the past, we struggled to make Todo[] compliant with the more generic DynamoDB expectation. Now, we can create a type that works for both our lambda and generates more precise API documentation.

// ./infrastructure/Lambda/get/index.ts

export type GetResponseBody = {
  todos: Partial<Todo>[] | undefined;
};

export const handler = async () => {
  const {Items} = await dynamoDB.scan({TableName: tableName}).promise();

  const response: GetResponseBody = {todos: Items};

  return httpResponse(200, JSON.stringify(response));
};

// ./infrastructure/api-specs/v1/getTodos.ts

import type {GetResponseBody} from '../../Lambda/get/lambda/index';

export type ResponseBody = GetResponseBody;

Here is the PR that captures the above changes.

Documenting and Testing Schemas of Serverless Stacks with Optic & Cypress

Murat K Ozcan — Thu, 17 Aug 2023 13:48:12 +0000

Serverless testing is unique. Recently I went through Yan Cui's Testing Serverless Architectures, highly recommended for all cloud engineers, learned a lot and agreed to these conclusions:

Increased Use of Managed Services: With tools like AWS Lambda, there's a notable shift towards leveraging managed services.
Focused, Single-Purpose Functions: Lambda functions typically hone in on a singular task, simplifying individual function complexity. With this structure, the main risk in serverless architectures emerges from integration points—how your Lambda functions communicate and interact with external managed services.
Granularity in Deployment: Serverless promotes deploying in smaller units. While this granularity provides precise control over access, it also introduces complexities.This increased granularity inherently demands more configurations, heightening security measures and careful management. In turn, the potential for misconfigurations is increased—be it within the application or Identity and Access Management (IAM).

Given these distinct challenges and benefits of serverless architectures, our testing strategies must adapt.

Unit Testing: Focused on individual objects or modules. These tests often don't yield a high return on investment unless the business logic is particularly intricate. Their value-to-cost ratio often leans unfavorably when compared to integration tests. They are useful for testing your business logic, particularly your branching logic, but they fall completely flat when testing your cloud infrastructure, making testing this code in a deployed environment essential.
Integration Testing: Tests interaction with external components, like AWS resources. Integration is the same cost, and more value than unit; feed an event into the lambda handler, validate the consequences. Running these tests against deployed AWS resources locally is more practical and beneficial than attempting to emulate AWS in a local environment. Emulations are often brittle and labor-intensive. Temporary stacks are mandatory in this approach, albeit there are teams that can exercise their handlers without local simulation or remote AWS resources; all the better.
End-to-End (E2E) Testing: Holistic testing of the system, including its deployment process. There are things integration tests cannot cover, such as IAM Permissions, our IaC/configuration, how the service is built and deployed. With e2e we get all that, alongside testing the functionality. While E2E tests offer the highest confidence, they're also the most resource-intensive.

Serverless architectures are increasingly prevalent, offering scalability and cost-efficiency. However, they also introduce the above unique testing challenges. Enter schema testing, a method that seems tailor-made for the serverless paradigm. Let's dive in and analyze how it can fit in to our serverless engineering & testing strategy in theory and practice.

Schema testing is a good fit with serverless applications/services/APIs
Optic stands apart
Optic vs Postman and Swagger
How can it all work in a generic serverless application?
- Install Optic to the project, and also globally.
- Generate your openapi.yml file
- Create convenience scripts in package.json
- One time setup to initialize the Optic (traffic) capture configuration
- Setup Optic cloud
- Capture the http traffic using the Optic proxy
Questions, Concerns, other comparisons
- Why both e2e and schema test?
- How about increased CI response time?
- Do we keep having to record the schema with schema:verify ?
- Optic approach vs consumer driven contract testing (Pact)
Conclusion

Schema testing is a good fit with serverless applications/services/APIs

Schema testing confirms an API's response structure, data types, and more, against a predefined schema, e.g., an OpenAPI specification. This kind of testing is useful for two main purposes; documenting the api and spot testing the implementation against potential breaking changes. It can give faster feedback in comparison to traditional e2e, as a preliminary test before heavy weight e2e tests.

Schema testing stands out as a fitting solution. Here's why:

Rapid feedback for integration integrity: Schema testing swiftly validates Lambda functions against set structures and ensures communication with external services follows expected patterns.
Cost effective: With minimal resource demands, schema tests sit in a cost effective spot, addressing the confidence gap between unit and E2E tests.
Guard against unintentional breaking changes: In the face of potential misconfigurations in serverless architectures, schema testing indirectly confirms correct integrations and guards against unintentional breaking changes in evolving applications.

Equally as important, is documenting our schema / API for the consumers. Creating the API docs and keeping them up to date is usually a chore, but there are ways we can automate the documentation upkeep as well as the testing of the schema.

Optic stands apart

We recently tried Optic and identified standout features that address the unique needs of serverless applications:

Traffic Capture and Real-time Insights: While optic offers a variety of methods to capture traffic, its ability to observe API traffic during testing through a reverse proxy mechanism streamlines API updates and verifications, akin to Jest snapshot test record and tests.
Seamless Documentation Sync: Any changes to the API automatically reflect in its documentation, ensuring alignment.
API Lifecycle Management:
- Lifecycle Awareness: Optic's comprehensive understanding of your API's lifecycle allows it to detect breaking changes and oversee appropriate API versioning.
- Forward-Only Governance: While new API endpoints comply with current standards, legacy endpoints remain intact, giving developers timely and pertinent feedback.
- API Version Control: Much like Git's approach to code, Optic introduces version control for APIs.
Enhanced Feedback and Integration:
- Visual Diff View: Optic presents intuitive visual depictions of changes in OpenAPI specifications.
- Integration with CI Pipelines: Beyond the usual CI integrations, Optic delivers changelog previews directly within pull requests, elucidating API alterations.
- Git Integration: Tightly integrated with Git, Optic allows API changes comparison across branches, aiding developers in monitoring developments across the entire project lifecycle.

By pairing Optic with e2e tests - we prefer Cypress for that because it is a great fit for api e2e testing event driven systems - you can harness Optic's traffic capture to either generate or revise your OpenAPI specification.

Optic vs Postman and OpenAPI/Swagger

Vs Postman:

Postman is really focussed on their collections format and do not have good tooling for generating OpenAPI or automatically updating it when the API no longer matches the spec.
Everything in Postman happens in their UI, whereas Optic is triggered in the local developer flows and CI. This gives developers the right feedback, in the right places.
Optic has tools for API change review that live right in the CI pipeline, there is no comparison for this on Postman side.
Optic and Postman both have API documentation viewers - every API tool does. Optic is optimized for internal teams + partner APIs and focused on showing consumers how the APIs they use change over time.

Vs Swagger

Swagger is primarily an API specification framework. Optic focuses on automated OpenAPI spec updates and change management.
Generation: Swagger tools like Swagger Editor are for manual API definition. Optic auto-captures API traffic to update specs. This is a key difference.
Integration: Swagger often demands manual updates. Optic integrates with developer flows and CI, alerting discrepancies between observed API behavior and the spec.
Documentation: Swagger UI provides interactive docs with request execution. Optic tracks API changes over time, ideal for teams to understand service evolution.
Ecosystem: Swagger boasts tools for editing, visualization, and code generation. Optic specializes in automatic change management.
Audience: Swagger is versatile for broad audiences, while Optic is tailored for environments with frequent API changes, emphasizing automated updates and change tracking.

How can it all work in a generic serverless application?

Our sample app is from Yan Cui's highly acclaimed Production-Ready Serverless workshop. Yan guides us through building an event driven serverless app in NodeJs with AWS API Gateway, Lambda, Cognito, EventBridge, DDB, and server-side rendering, which uses AWS Cognito for authentication.

Here is the link to the repo and Optic PR.
Here is a link to another repo and its Optic PR.

Optic has great documentation, and the below is a cohesive description of the setup with a focus on a serverless app.

Install Optic to the project, and also globally.

npm i -D @useoptic/optic
npm i -g @useoptic/optic

We need to create an OpenAPI specification. While there are many ways, a quick and dirty way is by using the AWS cli. It needs some tweaks, but we can work with it to start things out.

In the repository we are using serverless-export-env which allows us to extract the environment variables of our serverless stack to a .env file. The values here mostly change per the deployment.

Generate your `openapi.yml` file

Create the below script, which will be used to create an OpenApi file using
AWS cli. It uses the get-export command from AWS cli to hit the API gateway and generate the openapi.yml file. It assumes you have environment variables baseUrl and
deployment in your .env file, so that we know which api gateway we are concerned with on which
deployment. You can use any var name as long as they match between the .env
file and the script.

# .env
baseUrl=https://myApiGwId.execute-api.us-east-1.amazonaws.com/dev
deployment=dev # this could be a temp branch, or stage
# create-openapi.sh

# Load .env variables
set -a
source .env
set +a

# Extract the REST API id from the baseUrl
rest_api_id=$(echo $baseUrl | cut -d '/' -f3 | cut -d '.' -f1)

echo "Rest API ID: $rest_api_id"

# Run the aws apigateway get-export command
aws apigateway get-export --rest-api-id $rest_api_id --stage-name $deployment --export-type oas30 --accepts application/yaml ./openapi.yml

Give permissions to the file and execute:

chmod +x create-openapi.sh
./create-openapi.sh

You need the latest AWS CLI version. Here are MAC instructions, here's the AWS reference
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Like some other AWS tools, it does not have the best developer experience but we can work around it. The initial openapi.yml that gets created with aws-cli does not pass checks
at https://apitools.dev/swagger-parser/online/

Each path has options > responses, but this needs to be copied also to http-verb > options for each path.
Set openapi version to '3.1.0'.
Remove the server property.

Create convenience scripts in `package.json`

"optic:lint": "optic lint openapi.yml",
"optic:diff": "optic diff openapi.yml --base main --check'",
"optic:verify": "dotenv -e .env -- bash -c 'echo $baseUrl && optic capture openapi.yml --server-override $baseUrl'",
"optic:update": "dotenv -e .env -- bash -c 'echo $baseUrl && optic capture openapi.yml --server-override $baseUrl --update interactive'"

The optic:update script will update the OpenAPI specification (the openapi.yml file), similar to Jest snapshot update. Optic team recommends to use this locally to "record the API snapshot".

The optic:verify script works similar to a Jest snapshot test, check whether the traffic captured in the e2e matches your current openapi specification.

optic:lint and optic:diff can quickly lint the OpenAPI spec and diff it against main. These scripts are mostly for local use. However, unless the spec has been updated with optic:update, the diff will naturally not find any issues.

One time setup to initialize the Optic (traffic) capture configuration

This creates anoptic.yml file:

optic capture init openapi.yml

We need to fine tune it for a serverless stack. In the yml file, enter any placeholder for server.url. It has to exist with https prefix but does not have to be a valid url.

Remove server.command , our server is already deployed and running.

Replace requests.run.command wit the e2e test command.

requests.run.proxy_variable should be set to your api gateway url, below we are using an environment variable with the name rest_api_url.

We are using the e2e test script cy:run-fast which turns off video, screenshots and command log. This command could be any kind of e2e/http test for instance in this repository we also have mirroring Jest e2e tests using Axios. So long as they are http tests that hit the API gateway, and kind of test can be used.

# ./optic.yml

ruleset:
  # Prevent breaking changes
  - breaking-changes
capture:
  openapi.yml:
    server:
      # specified in package.json with --server-override
      url: https://api.example.com # need a placeholder

      ready_endpoint: /
      # The interval to check 'ready_endpoint', in ms.
      # Optional: default: 1000
      ready_interval: 1000
      # The length of time in ms to wait for a successful ready check to occur.
      # Optional: default: 10_000, 10 seconds
      ready_timeout: 10_000
    # At least one of 'requests.run' or 'requests.send' is required below.
    requests:
      # Run a command to generate traffic. Requests should be sent to the Optic proxy, the address of which is injected
      # into 'run.command's env as OPTIC_PROXY or the value of 'run.proxy_variable', if set.
      run:
        # The command that will generate traffic to the Optic proxy. Globbing with '*' is supported.
        # Required if specifying 'requests.run'.
        command: npm run cy:run-fast
        # The name of the environment variable injected into the env of the command that contains the address of the Optic proxy.
        # Optional: default: OPTIC_PROXY
        proxy_variable: rest_api_url

Setup Optic cloud

Optic Cloud has a few key benefits that make it compelling, and a very generous pricing model:

PR comments.
Centralized api style governance (setup by optic.yml file) for all your services, supported with AI.
An internal catalogue of all your APIs + changes so developers always know how the latest version works. This is much easier to read than openapi.yml files in the repos, and the changes between versions are visualized for easier comprehension.
Dashboards tracking OpenAPI accuracy over time, how well you follow the API standards, and breaking changes + issues over time
API Design and Collaboration: Optic facilitates an interactive feedback mechanism, allowing stakeholders to weigh in on API designs prior to deployment. This proactive approach ensures optimal design choices are made before consumers start relying on them, aligning with the API-first development mantra.
Support.

Create a token at Optic app. Save this as GitHub secret.
Enable Optic commenting on pull requests ( Repo > Settings > Actions > General and set Workflow permissions to Read and write permissions)

optic login
# you will copy over the token from the web prompt

# add the api
optic api add openapi.yml --history-depth=0

Once done, you will see your organization at Optic (updated a few seconds ago).

Capture the http traffic using the Optic proxy

Execute the script optic:update to capture the traffic and update the openapi.yml file. Again, this is similar to recording Jest snapshots.

From the docs

Your http tests are trafficked through Optic's reverse proxy to your api gateway
Optic observes the traffic, then creates or updates your open api spec.

We want to use this script in the CI to verify if the traffic captured in the e2e matches our current OpenAPI specification.

npm run optic:update

The other script optic:verify is to test your schema against your openapi spec. This is what we will use in the CI, and it works similar to executing Jest snapshot tests.

npm run optic:verify

We want to add Optic schema verification to our PRs so that the CI executes the schema test when a commit is pushed. There are many varieties of setting up our CI, and we will keep things simple and share the key highlights. For the full yml, take a look at PR.yml file at the repo.

# ./.github/workflows/PR.yml

# ...

jobs:
  build-deploy-test:
    runs-on: ubuntu-latest

    # ....

    # Permissions are only needed here if you are using OIDC
    permissions:
      id-token: write
      contents: write
      pull-requests: write # This provides write access to issues and PRs

    steps:
      # installation & deployment...

      # Jest & Cypress tests...

      # Schema verification
      - name: verify the schema with Optic
        run: npm run optic:verify

      # include a preview and changelog in each PR comment
      # sync every OpenAPI spec to Optic Cloud, diff & lint the schema as well
      - uses: opticdev/action@v1
        with:
          # Your Optic Cloud Token
          optic_token: ${{ secrets.OPTIC_TOKEN }}
          # A GitHub token with access to create comments on pull requests
          github_token: ${{ secrets.GITHUB_TOKEN }}
          # If true, standard check failures will cause this action to fail.
          # If false, standard check failures will show in PR comments and
          # in Optic Cloud but will not cause the action to fail
          standards_fail: true
          # If you have more than one spec, separate matches with commas
          # (openapi.yml,other.yml)
          additional_args: --generated --match openapi.yml
          compare_from_pr: cloud:default
          compare_from_push: cloud:default
        env:
          CI: true

Questions, Concerns, other comparisons

This approach is new and we are trying it out internally at my company Extend.
We will see how serious these concerns actually are and update when we find solutions that work for us.

Why both e2e and schema test?

Q: If we have e2e tests already, why are we also running schema tests? We need the serverless stack to be deployed for either to be running, because in schema testing the serverless stack we are recording or verifying the traffic with utilizing e2e tests too. What value add do we get from the schema test?

A: In the case of serverless applications, while both e2e and schema testing necessitate the deployment of the serverless stack, and schema tests often leverage e2e for traffic recording and verification, they offer unique perspectives and benefits.

1. Precision and Specificity: E2E tests provide a holistic system assessment, a comprehensive overview. In contrast, schema tests zero in on the API's structural integrity, ensuring its responses adhere strictly to the predefined structure. This granular approach might capture subtle inconsistencies broader tests could miss.

2. Documentation, Drift Detection, and Change Management: schema tests offer more than mere validation. They play a pivotal role in:

Dynamic Documentation Syncing: Immediate reflection of API changes in its documentation ensures continuous alignment, fostering improved developer communication and minimizing documentation drift risks.
API Lifecycle Oversight: They aid in detecting breaking changes, streamlining versioning, and managing the evolution of the API. This proactive, automated approach uniquely distinguishes schema testing from its e2e counterpart.

How about increased CI response time?

Q: How do we ensure schema testing has less impact on CI response time?

A: In a CI pipeline our setup can look like so:

install----deploy---------unit & integration
--------------------------e2e
--------------------------schema:verify

With that setup we can have a minimal impact on the duration of the pipeline.

Additionally, if we have local testing capabilities - we are working on this at my company Extend - schema testing can happen prior to deploy, or concurrently.

install----deploy-----------unit & integration
-----------schema:verify----e2e

install----schema:verify----deploy----unit & integration
--------------------------------------e2e

Do we keep having to record the schema with `schema:verify` ?

Q: In the CI examples, we are executing the schema:verify , but when do we execute schema:update ? Because, if we do not update the schema, then the tests may give false positives.

A: Usually schema update is a chore; a manual process where we update the openapi.yml file. If we have other methods of updating the schema -perhaps we update it via our code, and or types- we do not need to schema:update and in turn run the e2e suite locally.

If we do not have any practical way to update the OpenAPI specification, being able to do so with Optic is a boon. The engineers are responsible to own the schema update when they are making potential changes to the schema, so they would take advantage of automatically updating it by executing schema:update and not having to think about it.

Additionally, we can have a cron job in CI that runs nightly and updates the schema. The only gap in this approach would be is if we neglect the local schema:update execution, and we are changing the schema and releasing that day something. The cron job can reduce the human negligence error, but not entirely eliminate it.

Optic approach vs consumer driven contract testing (Pact)

Q: How does this approach relate to consumer driven contract testing (Pact)? How do they compare to just e2e?

A: First, a comparison of both with e2e:

With e2e, you deploy and release your service, and you can break consumer expectations, letting them know too late (usually never letting them know and they are surprised)
With Pact io, you would run a test and realize it is breaking the consumer’s expectation, communicate with the consumer you are going to break their expectations. They are informed prior to the deployment. Alternatively, esspecially if it is an external consumer, you do not make the breaking change.
With Optic, you realize you are changing your schema, because the schema test fails, so you realize you are probably breaking consumer expectations. A similar communication or decision making process takes place; either you let them know about the breaking change or do not make it.

Now, let's compare API governance with consumer-driven contract testing and the Optic approach:

Consumer-Driven Contract Testing with Pact:

Prevention Over Correction: Instead of finding out post-deployment that an API change has disrupted a consumer, you find out during the development phase.
Consumer-Centric: Contracts are based on the consumer's perspective. Providers ensure they meet these contracts.
Continuous Feedback: As API providers and consumers evolve, the contracts can be continually validated and updated.

Optic:

Observation Over Definition: Instead of pre-defined contracts, Optic observes actual API behavior and alerts developers to unintended changes.
Documentation & Drift Detection: As API evolves, Optic helps ensure documentation is automatically and accurately updated, reducing the chances of drift.
Automated Schema Testing: You understand changes to your schema, allowing for proactive communication with consumers about potential disruptions.

Key differences:

You have to write the consumer driven contract tests with Pact which is a lot of work compared to just maintaining automated schema test with Optic.
Pact is not going to tell you anything about API best practices, design standards, etc which is also another really important part of doing governance.

Conclusion

Schema testing, Optic, and e2e tests together create a formidable toolkit for enhancing API development in serverless architectures:

Enhanced Testing Workflow: Pairing Optic with e2e tests, especially with tools like Cypress, offers a dynamic way to test APIs in event-driven systems. Schema testing is adding another cost-effective & responsive layer to our testing toolbox.
Automated Schema Generation: Using AWS CLI and OpenAPI specs, API schemas can be initially generated.
Real-time Traffic Capturing with Optic: The OpenAPI specifications are kept accurate through real-time traffic observation, ensuring a true representation of system interactions.
Effortless API Documentation Updates: With Optic's traffic capture capabilities, not only is API behavior tracked, but documentation is also automatically updated. This means every time you run your e2e tests, you're also refreshing your API documentation, ensuring it remains relevant and up-to-date.
Simplified Integration & Collaboration: Embedding Optic commands in package.json and using Optic Cloud for team-based API review streamlines collaboration. With CI integration, it enforces an API-first development approach that remains consistent with the defined schema.
Visual Oversight: Optic Cloud's visual tools provide a clear, intuitive view of API changes, negating the need to delve into raw specification files.

In essence, the synergistic effect of schema testing, Optic, and end-to-end testing tools like Cypress offers a comprehensive solution to API development challenges in serverless environments.

Testing Email-Based Authentication Systems with Cypress, Mailosaur and cypress-data-session

Murat K Ozcan — Mon, 31 Jul 2023 16:23:36 +0000

These days our inboxes frequently buzz with verification codes, organization invites, or one-time links, each constituting a variant of authentication security. How can we ensure these email processes are robust? In this post we will cover test strategies for email authentication, using an open source application (link to the repo) as well as real life examples of how there could be variants to the same strategy. We will also touch upon how to reduce redundant consumption of such emails using cypress-data-session.

TOC:

Receiving authentication codes - AWS Cognito example
Gleb's email-authentication-code example at Mercari
User invite at Extend & passwordless-login example
Reducing redundant email consumption with cypress-data-session
Sharing the email between it blocks
Using one email per machine
Details about the data-session logic (specific to repo example)
Magic links with cypress-data-session
Wrap-up

Receiving authentication codes - AWS Cognito example

This application showcases an exemplary user registration flow, where the user we create receives an email from AWS Cognito containing a one-time code. This code then grants the user access to the application. The following sign-ins only require an email and a password. Below are the visual steps that should be familiar to all users.

Let us take a look at the automated test at sign-up-new-user.

We use a utility to randomize a user and visit the app. The example uses Chance library, but a FakerJs version is also provided. The only noteworthy item here is the randomized password. If the randomized password is not compliant, we might get 400 errors.

import {getConfirmationCode} from '../../support/e2e'
import {generateRandomUser} from '../../support/generate-random-user'

describe('sign up a new user', () => {
  it('should register the new user and log in', () => {
    const {firstName, lastName, userName, email, password} = generateRandomUser(
      Cypress.env('MAILOSAUR_SERVERID'),
    )

    cy.visit('/')

    // ... the rest of the test
  })
})

In the next section we drive the UI to fill in the form and register the user. It is important to check the backend response here, as the UI is not sophisticated to show invalid password or user already existing errors. In the real world, we would both verify the backend response and the UI, and possibly test the error/edge cases by stubbing the network via cy.intercept preferably in lower level component tests.


  cy.intercept('POST', 'https://cognito-idp*').as('cognito')
  cy.contains('Register').click()
  cy.get('#reg-dialog-form').should('be.visible')
  cy.get('#first-name').type(firstName, {delay: 0})
  cy.get('#last-name').type(lastName, {delay: 0})
  cy.get('#email').type(email, {delay: 0})
  cy.get('#username').type(userName, {delay: 0})
  cy.get('#password').type(password, {delay: 0})
  cy.contains('button', 'Create an account').click()
  cy.wait('@cognito').its('response.statusCode').should('equal', 200)

In the next section we check the email with a helper function getConfirmationCode which is using Mailosaur's Cypress commands (details later), extract the verification code from the email, and continue driving the user until the user is signed in.

getConfirmationCode(email).then(code => {
  cy.get('#verification-code').type(code, {delay: 0})
  cy.contains('button', 'Confirm registration').click()
  cy.wait('@cognito')
  cy.contains('You are now registered!').should('be.visible')
  cy.contains('button', /ok/i).click()

  cy.contains('Sign in').click()
  cy.get('#sign-in-username').type(userName, {delay: 0})
  cy.get('#sign-in-password').type(password, {delay: 0})
  cy.contains('button', 'Sign in').click()
  cy.wait('@cognito')

  cy.contains('Sign out')
})

The interesting part here is receiving the email and extracting the confirmation code from the email via regex.
Receiving the email is near-instant and effortless via Mailosaur. We accomplish that with the command from cypress-mailosaur plugin cy.mailosaurGetMessage(<yourEmailServerID>, {sentTo: <yourEmail>}).

Remember that the email is just a string with the confirmation code. We can extract the code using a regular expression.

const parseConfirmationCode = str => {
  const regex = /Your confirmation code is (\w+)/
  const match = str.match(regex)

  return match ? match[1] : null
}

export const getConfirmationCode = userEmail => {
  return cy
    .mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
      sentTo: userEmail,
    })
    .its('html.body')
    .then(parseConfirmationCode)
}

Addendum: Mailosaur makes parsing confirmation codes effortless!

We found about this later, but when looking at the html.codes property, we see that the Mailosaur already extracts the codes for us. That is so convenient because the regex may be different between authentication providers. The previous code becomes so much easier without having to worry about the regex.

export const getConfirmationCode = userEmail => {
  return cy
    .mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
      sentTo: userEmail,
    })
    .its('html.codes.0.value')
}

Gleb's email-authentication-code example at Mercari

In the AWS Cognito example we only needed the verification code so that we can enter it in the UI.

In my friend Gleb Bahmutov's blog, the case study at Mercari is a similar variant, receiving a confirmation code and performing further testing on the email. We randomize a user, fill the form, get the email, and display the email at our test runner, which is a very neat trick because it can allow us to visualize the email as we are executing the test.

beforeEach(() => {
  const userName = 'Joe Bravo'
  const serverId = Cypress.env('MAILOSAUR_SERVER_ID')
  const randomId = Cypress._.random(1e6)
  const userEmail = `user-${randomId}@${serverId}.mailosaur.net`

  // fill the form...

  // get the email
  cy.mailosaurGetMessage(serverId, {
    sentTo: userEmail,
  })
    .then(console.log)
    .its('html.body')
    // store the HTML under an alias
    .as('email)
})

beforeEach(function () {
  cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
})

User invite at Extend & passwordless-login example

At our company Extend, in one of our applications we have a user being invited by an admin to an organization. Simply, the user receives an email, then follows the link where they create an account and continue to sign on to the app & the org.

In the test, the interesting part is extracting the href and visiting it. The browser built-in DOMParser Provides the ability to parse XML or HTML source code.

/**
 * Extracts the href value from an HTML string.
 *
 * @param {string} htmlString - The HTML string to parse.
 * @returns {string|null} The href value, if it exists, otherwise null.
 */
const extractHref = (htmlString: string): string | null => {
  const parser = new DOMParser() 
  const doc = parser.parseFromString(htmlString, 'text/html')
  const link = doc.querySelector('#reset-password-link')
  return link ? (link as HTMLAnchorElement).href : null
}

The htmlString is simply the html.body we receive in the email. The function chain yields the link that we visit.

// as the admin, fill the form with the user's info

// the user receives the email
  cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
    sentTo: email,
  })
    .its('html.body')
    .then(extractHref)
    .should('exist')
    .then(cy.visit)

// the user continues creating the account

In a very similar fashion, in the case of a password-less login, we could receive a magic link in the email, extract and visit it. We fathom even the test code would be the same.

// we randomize a user, they receive a magic link

const extractHref = (htmlString: string): string | null => {
  const parser = new DOMParser() 
  const doc = parser.parseFromString(htmlString, 'text/html')
  const link = doc.querySelector('#reset-password-link')
  return link ? (link as HTMLAnchorElement).href : null
}

// we check the email, extract the link and visit it
cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
  sentTo: email,
})
  .its('html.body')
  .then(extractHref)
  .should('exist')
  .then(cy.visit)

// we are logged in to the app

Addendum: Mailosaur makes parsing links effortless!

Similar to not having to parse the regex, we later found out that we do not have to parse the links either. After all, confirmation codes and links are widespread, and Mailosaur already provides this data in the email response. Such convenience!

cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
  sentTo: email,
})
  .its('html.links.0.href')
  .should('exist')
  .then(cy.visit)

Reducing redundant email consumption with `cypress-data-session`

In the sample repo prod-ready-serverless, we are using AWS Cognito which has a soft limit of 50 emails per day. Not only there are limits that are not-easy to increase, but also there is a cost to Cognito. Similarly at Extend, we have limits with Okta. In one of the UI apps we have 80 e2e tests and ~500 it blocks, and these can fully execute at a pull request. We may have dozens of commits per day. If every test created a new user via email, even if we had a generous Mailosaur pricing band, email based authentication cannot scale. We would be blocked by our auth provider long before reaching the email limit. Gleb Bahmutov's cypress-data-session plugin is the solution.

Sharing the email between it blocks

In Gleb's example, we see data session being utilized across the it blocks in a spec, in order to prevent an email being received for each it block. The random user is created in the beginning, the form is filled, the email is received only once. The return value from the setup function is the html.body which is auto-aliased to the session name "email". The DOM is populated with the email in each test.

beforeEach(() => {
  cy.dataSession({
    name: 'email',
    setup() {
      const userName = 'Joe Bravo'
      const serverId = Cypress.env('MAILOSAUR_SERVER_ID')
      const randomId = Cypress._.random(1e6)
      const userEmail = `user-${randomId}@${serverId}.mailosaur.net`

      // fill the form...

      cy.mailosaurGetMessage(serverId, {
        sentTo: userEmail,
      })
        .its('html.body')
    },
    shareAcrossSpecs: true, // the email is reused between the tests
  })
})

beforeEach(function () {
  cy.document({ log: false }).invoke({ log: false }, 'write', this.email)
})

it('shows the code by itself', () => ...)
it('has the confirmation code link', () => ...)
it('has the working code', () => ...)

In the prod-ready-serverless application -we will share the details of the data-session configuration in the registerAndSignIn command later- we can contemplate about how a test would look if we would use one email per file. The first test in the suite would go through the complete flow of filling the form, receiving the one time code, using it to sign in. The 2nd test would only sign in. The spec would consume 1 email, similar to the previous example.

// ./cypress/e2e/front-end/place-order.cy.js
const {generateRandomUser} = require('../../support/generate-random-user')

describe('place an order', () => {
  // randomize a user
  const {fullName, userName, email, password} = generateRandomUser(
    Cypress.env('MAILOSAUR_SERVERID'),
  )

  beforeEach(() => {
    cy.visit('/')

    cy.registerAndSignIn({
      fullName,
      userName,
      email,
      password,
    })
  })

  // the first test registers, receives the email and signs in
  it('should place an order', () => {
    cy.on('window:alert', cy.stub().as('alert'))
    cy.intercept('POST', '**/orders').as('placeOrder')

    cy.get('#restaurantsUl > :nth-child(1)').click()
    cy.wait('@placeOrder').its('response.statusCode').should('eq', 200)
    cy.get('@alert').should('be.calledOnce')
  })

    // the 2nd test only signs in
  it('should do something else', () => {
    cy.log('we are logged in with the same user')
  })
})

Using one email per machine

50 emails with Cognito is not a very safe limit. In our repo we want to further reduce the email consumption, and share the random user, therefore the email, between all the tests that execute on that machine.

The way to accomplish this is to create the random user at the cypress.config file, which executes once per cy:run or cy:open . We can assign the values as environment variables, and use them as arguments when calling the registerAndSignIn command, ensuring the same values are used not only between the it blocks, but the spec files as well.

const {fullName, userName, email, password} =
  generateRandomUser(MAILOSAUR_SERVERID)

module.exports = defineConfig({
  // ...
  env: {
    fullName,
    userName,
    email,
    password,
  },
  e2e: {
    // ...
  },
})

We get the same effect at the single test file from before.

// ./cypress/e2e/front-end/place-order.cy.js

describe('place an order', () => {
  beforeEach(() => {
    cy.visit('/')

    cy.registerAndSignIn({
      fullName: Cypress.env('fullName'),
      userName: Cypress.env('userName'),
      email: Cypress.env('email'),
      password: Cypress.env('password'),
    })
  })

  it('should place an order', () => {..})

  it('should do something else', () => {..})
})

The key distinction is that other spec files utilize the same user and email as well.

// cypress/e2e/front-end/sign-up-test-user.cy.js
describe('sign in with test user', () => {
  it('should register & log in with the test user', () => {
    cy.visit('/')

    cy.registerAndSignIn({
      fullName: Cypress.env('fullName'),
      userName: Cypress.env('userName'),
      email: Cypress.env('email'),
      password: Cypress.env('password'),
    })
  })
})

The idea of sharing the test user between different spec files applies to magic-link/passwordless login scenarios as well. Mirroring the above, we would randomize the user & assign the values as environment variables upon launching Cypress, then in the specs we would use the environment variables for the command arguments.

Details about the data-session logic (specific to repo example)

One thing is for certain; with email based authentication, we cannot utilize the built-in cy.session because the email portion of the flow requires more control over the logic compared to a UI login not being repeated. cypress-data-session can be applied to login scenarios that cy.session would satisfy, but not the other way around. For a comparison of cypress-data-session vs cy.session, take a look at this repo and this video.

The way to configure the data-session logic will be different in any app. In Gleb's example, the data-session was focused on the email, receiving one email, rendering it in the DOM and performing assertions on it. In the prod-ready-serverless example we have to satisfy 2 concerns:

If it's a new user, we have to go through the registration flow (fill form, receive one time code, login with the code)
If the user already exists, only sign in

Let us start breaking apart the functions and build up to the final registerAndSignIn command.

We have a function that simply drives the UI to fill the form for a new user.

// first part of registration
const fillRegistrationForm = ({fullName, userName, email, password}) => {}

We have a function to get the confirmation code, using it one time at the UI, yielding out the confirmation code.

// second part of registration
const confirmRegistration = email =>
  getConfirmationCode(email).then(confirmationCode => {
    cy.intercept('POST', 'https://cognito-idp*').as('cognito')
    cy.get('#verification-code').type(confirmationCode, {delay: 0})
    cy.contains('button', 'Confirm registration').click()
    cy.wait('@cognito')
    cy.contains('You are now registered!').should('be.visible')
    cy.contains('button', /ok/i).click()
    return cy.wrap(confirmationCode)
  })
}

We have a function that combines the above two parts for the full registration.

// the registration flow; the above 2 parts
const register = ({fullName, userName, email, password}) => {
  fillRegistrationForm({fullName, userName, email, password})
  return confirmRegistration(email)
})

Another function that drives the UI, for sign in.

// nothing interesting, just driving the UI
const signIn = ({userName, password}) => {}

The final command makes clear why we had to separate register into two.

Initially we want the full flow of filling out the form, getting the confirmation and signing in; init() does the form fill and receives the code, and recreate() signs in.

In subsequent tests, we only want to check if the email & verification code exist, and sign in; setup() checks the email & verification code, recreate() signs in.

const registerAndSignIn = ({fullName, userName, email, password}) =>
  cy.dataSession({
    // unique name of the data session will be the email address. 
    // With any new email address, the data session will be recreated
    name: email, 
    // initially, we do the full registration flow
    init: () => register({fullName, userName, email, password}),
    // subsequent tests start from setup, just checking the email
    setup: () => confirmRegistration(email),
    // recreate runs always, either after init or setup
    recreate: () => signIn({userName, password}),
    shareAcrossSpecs: true,
  })
Cypress.Commands.add('registerAndSignIn', registerAndSignIn)

Magic links with `cypress-data-session`

How could the above example look for a magic link scenario, where we do not want to keep receiving an email?

After the setup step where we visitEmailLink, we need some logged-in state to be preserved for a of subsequent tests. This state could be local storage, session storage, and or cookies; whatever we need to be logged in after clicking the magic link.

For instance, in this example our setup step logs in an returns the localstorage. That return value is fed-in as the argument to recreate step, and the recreate step rebuilds the localstorage.

// preserving local storage example from 
// https://github.com/muratkeremozcan/appsyncmasterclass-frontend/blob/main/cypress/support/e2e.js#L65

Cypress.Commands.add('progLogin', (username, password) => {
  cy.then(() => Auth.signIn(username, password)).then(cognitoUser => {
    const idToken = cognitoUser.signInUserSession.idToken.jwtToken
    const accessToken = cognitoUser.signInUserSession.accessToken.jwtToken
    const makeKey = name =>
      `CognitoIdentityServiceProvider.${cognitoUser.pool.clientId}.${cognitoUser.username}.${name}`
    cy.setLocalStorage(makeKey('accessToken'), accessToken)
    cy.setLocalStorage(makeKey('idToken'), idToken)
    cy.setLocalStorage(
      `CognitoIdentityServiceProvider.${cognitoUser.pool.clientId}.LastAuthUser`,
      cognitoUser.username,
    )
  })
  cy.saveLocalStorage()
  // IMPORTANT: preserving the localstorage state
  // this could be session storage, or cookies 
  return cy.visit('/home').then(() => JSON.parse(JSON.stringify(localStorage)))
})

Cypress.Commands.add('dataSessionLogin', (email, password) => {
  return cy.dataSession({
    name: email,
    setup: () => cy.progLogin(email, password),
    validate: validateLocalStorage,
    // the return of the setup step is yielded as an arg to recreate
    // and we rebuild the localstorage
    recreate: ls => {
      for (const key in ls) {
        localStorage[key] = ls[key]
      }
      cy.visit('/home')
      return cy.contains('Home', {timeout: 10000})
    },
    cacheAcrossSpecs: true,
  })
})

Here's how the magic link scenario could work:

const visitEmailLink = (email) => {
  // we randomize a user, they receive a magic link
  const extractHref = (htmlString: string): string | null => {
    const parser = new DOMParser() 
    const doc = parser.parseFromString(htmlString, 'text/html')
    const link = doc.querySelector('#reset-password-link')
    return link ? (link as HTMLAnchorElement).href : null
  }

  // we check the email, extract the link and visit it
  cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVERID'), {
    sentTo: email,
  })
    .its('html.body')
    .then(extractHref)
    .should('exist')
    .then(cy.visit)

  // IMPORTANT: preserve some state here, and return that
  // could be localstorage, sessionstorage, and/or cookies
  return cy.then(() => JSON.parse(JSON.stringify(localStorage)))
}

cy.dataSession({
  // we can again make the unique email the session
  name: email,
  // something that runs the first time,
  // causes an email to be sent
  init: () => {..},
  // exract the magic link and visit it
  // in this function, make sure to preserve some state
  // localstorage, sessionstorage, and/or cookies
  setup: () => visitEmailLink(email),
  // the return value of the setup gets fed to recreate as an arg
  recreate: ls => {
    // rebuild the preserved state before visiting
    for (const key in ls) {
      localStorage[key] = ls[key]
    }  
  cy.visit('/')
  }
})

Wrap-up

We examined the use of Cypress, a powerful testing tool, and Mailosaur, a service designed specifically for testing emails, as a potent combination for testing email based authentication.

We delved into methods to extract required data from emails, either by using Regular Expressions or directly pulling href values from the email's HTML content. These techniques provide the flexibility to extract almost any information that might be needed from an email for testing purposes.

Lastly, we looked at how to use cypress-data-session to minimize redundant email consumption. This plugin not only makes tests more efficient by eliminating unnecessary email retrievals but also enables sharing of email content within the same test, or even across tests running on the same machine.

Testing is never an easy task, but with these tools and strategies, even complex systems like email-based authentication can be effectively and efficiently validated.

Cypress Component Testing vs React Testing Library - the complete comparison

Murat K Ozcan — Wed, 07 Jun 2023 16:19:25 +0000

Cypress component testing was released about a year prior to this blog post and has been a disruptor not only to the de-facto JS framework component testing solutions but also initiated a paradigm shift in developing frontend components. At the core of this is the ability to render a component in isolation in the actual browser, and to see and observe how the component behaves in isolation while time-travel debugging via Cypress. This changes the entire development experience, with testing - the actual intent - being the icing on the cake.

Rolling out component testing in a community that is so invested in de-facto component testing solutions can be a daring challenge. In this post, we will analyze the differences between React Testing Library (RTL) and Cypress Component Testing (CyCT), and provide you with resources that can help with adopting component testing in your organization.

Toc:

CyCT vs RTL examples - Tour of Heroes (ToH)
- HeaderBarBrand component
- InputDetail component
- NavBar component
Comparison of low level spies & mocks: Sinon vs Jest
- Sinon vs Jest: Spy
- Sinon vs Jest: Stub/Mock
Comparison of network spies & mocks: cy.intercept vs MSW
CyCT vs RTL examples - Epic React
Wrapping up
Addendum: Gleb Bahmutov's The Missing Comparison Part video: a comparison of the developer experience
- 1. Compare the devex making a simple breaking change to the source code: HeaderBarBrand
- 2. Compare test stability by simulating an asynchronous process: InputDetail
- 3. Compare the devex making a "complex" change: Heroes

Our experience at Extend

Our first component test at Extend was at the end of 2022, and we have been piloting it in one of our apps for about six months. Comparing the Cypress Cloud reports between then and now, we can see a preference towards component testing.

Bear in mind that we are migrating from Enzyme to CyCT, to be able to upgrade React beyond version 16. If we had RTL tests already in this app - as we do in a second UI app which had already migrated out of Enzyme before CyCT rollout - then the RTL tests and CyCT could coexist. We have about 130 Enzyme tests remaining to migrate in our pilot app. This means we will potentially have around 300 Cypress component tests once the migration is complete. This is aligned with the proportion of e2e to CT we have observed in other apps; usually it is anywhere between 1:3 to 1:5.

December 2022 (first component test commit):

June 2023:

If you are curious about how the e2e execution time became so low with more tests, check out the video Improve Cypress e2e test latency by a factor of 20!!

CyCT vs RTL examples - Tour of Heroes (ToH)

ToH is the final app built in the book CCTDD: Cypress Component Test Driven Design. It has a few dozen Cypress component tests and their RTL mirrors. We will cover a few examples to showcase the main differences.

HeaderBarBrand component

The way the component is mounted is very similar between CyCT and RTL. So are custom mounts / renders.

There are less imports in CyCT, because either these are built-in or come with the browser.

The API is the primary difference; with Cypress we have a left-to-right chain style, with RTL we have a right-to-left variable assignment style.

28 lines CyCT vs 35 lines in RTL.

HeaderBarBrand.cy.tsx

import HeaderBarBrand from "./HeaderBarBrand";
import { BrowserRouter } from "react-router-dom";

describe("HeaderBarBrand", () => {
  beforeEach(() => {
    cy.mount(
      <BrowserRouter>
        <HeaderBarBrand />
      </BrowserRouter>
    );
  });

  it("should verify external link attributes", () => {
    cy.get("a")
      .should("have.attr", "href", "https://reactjs.org/")
      .and("have.attr", "target", "_blank")
      .and("have.attr", "rel", "noopener noreferrer");
    cy.getByCy("header-bar-brand").within(() => cy.get("svg"));
  });

  it("should verify internal link spans and navigation", () => {
    cy.getByCy("navLink").within(() =>
      ["TOUR", "OF", "HEROES"].map((part: string) => cy.contains("span", part))
    );
    cy.getByCy("navLink").click();
    cy.url().should("contain", "/");
  });
});

HeaderBarBrand.test.tsx

import HeaderBarBrand from "./HeaderBarBrand";
import { render, screen, within } from "@testing-library/react";
import { BrowserRouter } from "react-router-dom";
import userEvent from "@testing-library/user-event";
import "@testing-library/jest-dom";

describe("HeaderBarBrand", () => {
  beforeEach(() => {
    render(
      <BrowserRouter>
        <HeaderBarBrand />
      </BrowserRouter>
    );
  });
  it("should verify external link attributes", async () => {
    const link = await screen.findByTestId("header-bar-brand-link");
    expect(link).toHaveAttribute("href", "https://reactjs.org/");
    expect(link).toHaveAttribute("target", "_blank");
    expect(link).toHaveAttribute("rel", "noopener noreferrer");

    // not easy to get a tag with RTL, needed to use a test id
    within(await screen.findByTestId("header-bar-brand")).getByTestId(
      "react-icon-svg"
    );
  });

  it("should verify internal link spans and navigation", async () => {
    const navLink = await screen.findByTestId("navLink");
    const withinNavLink = within(navLink);
    ["TOUR", "OF", "HEROES"].forEach((part) => withinNavLink.getByText(part));

    await userEvent.click(navLink);
    expect(window.location.pathname).toBe("/");
  });
});

InputDetail component

We see similar contrast between CyCT and RTL as before; similar mount/render, different API styles, terser syntax on Cypress side. Note that testing library has a Cypress version and can be used to make the examples even more similar.

The key difference we want to point out is stubbing the onChange property the component. cy.stub() vs its counterpart jest.fn(). Cypress comes with Sinon, and Cypress' API allows us to stub in-line.

// CyCT: we can stub the property in-line
cy.mount(
  <InputDetail
    name={name}
    value={value}
    placeholder={placeholder}
    onChange={cy.stub().as("onChange")}
  />
);

// RTL: variable assignment first
const onChange = jest.fn();

render(
  <InputDetail
    name={name}
    value={value}
    placeholder={placeholder}
    onChange={onChange}
  />
);

InputDetail.cy.tsx

import InputDetail from "./InputDetail";
import "@testing-library/cypress/add-commands";

describe("InputDetail", () => {
  const placeholder = "Aslaug";
  const name = "name";
  const value = "some value";
  const newValue = "42";

  it("should allow the input field to be modified", () => {
    cy.mount(
      <InputDetail
        name={name}
        value={value}
        placeholder={placeholder}
        onChange={cy.stub().as("onChange")}
      />
    );

    cy.contains("label", name);
    cy.findByPlaceholderText(placeholder).clear().type(newValue);
    cy.findByDisplayValue(newValue).should("be.visible");
    cy.get("@onChange").its("callCount").should("eq", newValue.length);
  });

  it("should not allow the input field to be modified", () => {
    cy.mount(
      <InputDetail
        name={name}
        value={value}
        placeholder={placeholder}
        readOnly={true}
      />
    );

    cy.contains("label", name);
    cy.findByPlaceholderText(placeholder)
      .should("have.value", value)
      .and("have.attr", "readOnly");
  });
});

InputDetail.test.tsx

import InputDetail from "./InputDetail";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";

describe("InputDetail", () => {
  const placeholder = "Aslaug";
  const name = "name";
  const value = "some value";
  const newValue = "42";

  it("should allow the input field to be modified", async () => {
    const onChange = jest.fn();
    render(
      <InputDetail
        name={name}
        value={value}
        placeholder={placeholder}
        onChange={onChange}
      />
    );

    await screen.findByText(name);
    const inputField = await screen.findByPlaceholderText(placeholder);
    await userEvent.clear(inputField);
    await userEvent.type(inputField, newValue);
    expect(inputField).toHaveDisplayValue(newValue);
    expect(onChange).toHaveBeenCalledTimes(newValue.length);
  });

  it("should not allow the input field to be modified", async () => {
    render(
      <InputDetail
        name={name}
        value={value}
        placeholder={placeholder}
        readOnly={true}
      />
    );

    await screen.findByText(name);
    const inputField = await screen.findByPlaceholderText(placeholder);
    expect(inputField).toHaveDisplayValue(value);
    expect(inputField).toHaveAttribute("readOnly");
  });
});

NavBar component

We can observe the same similarities and contrast between CyCT and RTL in this component. The most striking of them all is the API style difference; with Cypress we are able to cover a each link with forEach, to be able to do the same in RTL we have to do a little bit of more work with it.each.

NavBar.cy.tsx

import NavBar from "./NavBar";
import { BrowserRouter } from "react-router-dom";

const routes = ["heroes", "villains", "boys", "about"];

describe("NavBar", () => {
  it("should navigate to the correct routes", () => {
    cy.mount(
      <BrowserRouter>
        <NavBar />
      </BrowserRouter>
    );

    cy.contains("p", "Menu");
    cy.getByCy("menu-list").children().should("have.length", routes.length);

    routes.forEach((route: string) => {
      cy.get(`[href="/${route}"]`)
        .contains(route, { matchCase: false })
        .click()
        .should("have.class", "active-link")
        .siblings()
        .should("not.have.class", "active-link");

      cy.url().should("contain", route);
    });
  });
});

NavBar.test.tsx

import NavBar from "./NavBar";
import { render, screen, within, waitFor } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { BrowserRouter } from "react-router-dom";
import "@testing-library/jest-dom";

const routes = ["Heroes", "Villains", "Boys", "About"];

describe("NavBar", () => {
  beforeEach(() => {
    render(
      <BrowserRouter>
        <NavBar />
      </BrowserRouter>
    );
  });

  it("should verify route layout", async () => {
    expect(await screen.findByText("Menu")).toBeVisible();

    const menuList = await screen.findByTestId("menu-list");
    expect(within(menuList).queryAllByRole("link").length).toBe(routes.length);

    routes.forEach((route) => within(menuList).getByText(route));
  });

  it.each(routes)("should navigate to route %s", async (route: string) => {
    const link = async (name: string) => screen.findByRole("link", { name });
    const activeRouteLink = await link(route);
    userEvent.click(activeRouteLink);
    await waitFor(() => expect(activeRouteLink).toHaveClass("active-link"));
    expect(window.location.pathname).toEqual(`/${route.toLowerCase()}`);

    const remainingRoutes = routes.filter((r) => r !== route);
    remainingRoutes.forEach(async (inActiveRoute) => {
      expect(await link(inActiveRoute)).not.toHaveClass("active-link");
    });
  });
});

In all these examples we have not said anything about the component code, but instead shared a CyCT screen shot to communicate what the component is about. On one side we are staring at html, on the other we are looking at the component in the browser. The picture demonstrates a bug during development that we caught by just looking at the component.

Comparison of low level spies & mocks: Sinon vs Jest

We prepared a repository to cover core examples of using Sinon with Cypress, and created mirror tests in Jest.

The original Sinon examples are from Gleb Bahmutov's Cypress Examples.

Sinon vs Jest: Spy

A spy does not modify the behavior of the function - it is left perfectly intact. A spy is most useful when you are testing the contract between multiple functions and you don't care about the side effects the real function may create (if any).

spy-practice.cy.ts vs spy-practice-test.ts are linked for you to view the files in comparison; we will summarize the key differences:

Spying on methods: Both libraries allow you to create a spy that wraps around a function and records all calls to it, along with arguments and return values. Both Jest and Sinon allow you to assert if the spy was called, how many times it was called, and with what arguments. They do differ in syntax and some functionalities.

cy.spy(obj, 'method') vs jest.spyOn(obj, 'method')

Syntax: have.been.called vs toHaveBeenCalled, have.been.calledTwice vs toHaveBeenCalledTimes(2)
Matchers: sinon.match.string vs expect.any(String)
Custom matchers: easy with Sinon, not great devex with Jest. We will not share the Jest code, because it is very verbose.

// Cy + sinon
const { match } = Cypress.sinon;
const isEven = (x: number) => x % 2 === 0;
const isOdd = (x: number) => x % 2 === 1;

const spy = cy.spy(calculator, "add").as("add");
calculator.add(2, 3);

// expect the value to pass a custom predicate function
// the second argument to "match(predicate, message)"
// is shown if the predicate does not pass and assertion fails
expect(spy).to.be.calledWith(match(isEven), match(isOdd, "is odd"));

Asynchronous testing: In Jest, it is required to explicitly wait for asynchronous operations to complete before assertions are checked. This is illustrated in using the setTimeout and await new Promise combination. Cypress + Sinon does not necessitate this explicit wait. There are certain advantages to Cypress chain style and built-in retry ability. In the below example with Jest we have to gather the promises, await them with Promise.all and have to use toHaveBeenNthCalledWith to specify the order.

// Cy + Sinon
it("resolved value (promises)", () => {
  const calc = {
    async add(a: number, b: number) {
      return /* await */ Cypress.Promise.resolve(a + b).delay(100); // don't use await redundantly
    },
  };

  cy.spy(calc, "add").as("add");
  // wait for the promise to resolve then confirm its resolved value
  cy.wrap(calc.add(4, 5)).should("equal", 9);
  cy.wrap(calc.add(1, 90)).should("equal", 91);
  cy.wrap(calc.add(-5, -8)).should("equal", -13);

  // example of confirming one of the calls used add(4, 5)
  cy.get("@add").should("have.been.calledWith", 4, 5);
  cy.get("@add").should("have.been.calledWith", 1, 90);
  cy.get("@add").should("have.been.calledWith", -5, -8);

  // now let's confirm the resolved values
  // first we need to wait for all promises to resolve
  cy.get("@add")
    .its("returnValues")
    .then((ps) => Promise.all(ps))
    .should("deep.equal", [9, 91, -13]);
});

// Jest
it("resolved value (promises)", async () => {
  const calc = {
    async add(a: number, b: number) {
      return a + b;
    },
  };

  const spy = jest.spyOn(calc, "add");

  // Let's gather the promises first
  const promises = [calc.add(4, 5), calc.add(1, 90), calc.add(-5, -8)];

  // Now we wait for all the promises to resolve
  const results = await Promise.all(promises);

  // We can check if the spy was called with the correct arguments at each call
  expect(spy).toHaveBeenNthCalledWith(1, 4, 5);
  expect(spy).toHaveBeenNthCalledWith(2, 1, 90);
  expect(spy).toHaveBeenNthCalledWith(3, -5, -8);

  // Finally, we can verify the resolved values
  expect(results).toEqual([9, 91, -13]);
});

Sinon vs Jest: Stub/Mock

A stub is a way to modify a function and delegate control over its behavior to you (the programmer).

Create a standalone stub (generally for use in unit test):

cy.stub();
jest.fn();

Replace obj.method() with a stubbed function:

cy.stub(obj, "method");
jest.spyOn(obj, "foo").mockImplementation(jest.fn());

Force obj.method() to return a value:

cy.stub(obj, "method").returns("Cliff");
jest.spyOn(obj, "method").mockReturnValue("Cliff");

Force obj.method() when called with "bar" argument to return "foo":

cy.stub(obj, "method").withArgs("bar").returns("foo");

jest.spyOn(obj, "method").mockImplementation((arg) => {
  if (arg === "bar") return "foo";
});

Force obj.method() to return a promise which resolves to "foo"

cy.stub(obj, "method").resolves("foo");

jest.spyOn(obj, "method").mockImplementation(() => {
  return Promise.resolve("foo");
});

Force obj.method() to return a promise rejected with an error

cy.stub(obj, "method").rejects(new Error("foo"));

jest.spyOn(obj, "method").mockImplementation(() => {
  return Promise.reject(new Error("foo"));
});

It is interesting to note that the equivalent of cy.stub() is jest.fn() but in many of the comparisons we are using jest.spyOn(...).mockImplementation(...)

In Jest, we can use .mockImplementation() to provide a custom implementation for the mock function.
In Sinon, we can use .callsFake() or .returns() to specify custom behavior for the stub.

jest.fn() can be used more in scenarios where you're not spying on or modifying existing object methods, but rather creating standalone mock functions. For instance, when testing if a function passed as a prop or callback is called correctly in a component test or when needing to create a mock implementation for a function from a module that your function under test is calling.

Here is an example scenario where jest.fn() could be used

it("should call the callback", () => {
  const mockCallback = jest.fn();

  function doSomething(callback: (arg: string) => void) {
    callback("test argument");
  }

  doSomething(mockCallback);

  expect(mockCallback).toHaveBeenCalledTimes(1);
  expect(mockCallback).toHaveBeenCalledWith("test argument");
});

Comparing stub-practice.cy.ts vs stub-practice.test.ts, here are some of the other highlights:

Restoring the original method after stub:

const person = {
  getName() {
    return "Joe";
  },
};

/// Cy + Sinon
expect(person.getName()).to.eq("Joe");

const stub = cy.stub(person, "getName").returns("Cliff");
expect(person.getName()).to.eq("Cliff");

// restore the original method
stub.restore();
expect(person.getName()).to.eq("Joe");

/// Jest
expect(person.getName()).toBe("Joe");

const stub = jest.spyOn(person, "getName").mockReturnValue("Cliff");
expect(person.getName()).toBe("Cliff");

// restore the original method
stub.mockRestore();
expect(person.getName()).toBe("Joe");

Matchers: .callThrough(), withArgs(), match.type, match(predicate):

describe("matchers: .callThrough(), withArgs(), match.type, match(predicate)", () => {
  const { match } = Cypress.sinon;

  it("Matching stub depending on arguments", () => {
    const greeter = {
      greet(name: string | number | undefined) {
        return `Hello, ${name}!`;
      },
    };

    const stub = cy.stub(greeter, "greet");

    stub.callThrough(); // if you want non-matched calls to call the real method
    stub.withArgs(match.string).returns("Hi, Joe!");
    stub.withArgs(match.number).throws(new Error("Invalid name"));

    expect(greeter.greet("World")).to.equal("Hi, Joe!");
    expect(() => greeter.greet(42)).to.throw("Invalid name");
    expect(greeter.greet).to.have.been.calledTwice;

    // non-matched calls goes the actual method
    expect(greeter.greet()).to.equal("Hello, undefined!");
  });
});

There is no direct equivalent in Jest, but the below does the same thing

describe("matchers: .mockImplementation()", () => {
  it("Matching stub depending on arguments", () => {
    const greeter = {
      greet(name: string | number) {
        return `Hello, ${name}!`;
      },
    };

    jest
      .spyOn(greeter, "greet")
      .mockImplementation((name: string | number | undefined) => {
        if (typeof name === "string") {
          return "Hi, Joe!";
        } else if (typeof name === "number") {
          throw new Error("Invalid name");
        } else {
          return "Hello, undefined!";
        }
      });

    expect(greeter.greet("World")).toEqual("Hi, Joe!");
    expect(() => greeter.greet(42)).toThrow("Invalid name");
    expect(greeter.greet).toHaveBeenCalledTimes(2);

    expect(greeter.greet()).toEqual("Hello, undefined!");
  });
});

Calling the original method from the stub:

describe("Call the original method from the stub: callsFake(...), wrappedMethod()", () => {
  it("Sometimes you might want to call the original method from the stub and modify it", () => {
    const person = {
      getName() {
        return "Joe";
      },
    };

    cy.stub(person, "getName").callsFake(() => {
      // call the real person.getName()
      return person.getName.wrappedMethod().split("").reverse().join("");
    });

    expect(person.getName()).to.eq("eoJ");
  });
});

There is no direct equivalent in Jest, but the below does the same thing.

describe("Call the original method from the stub: mockImplementation(), originalName", () => {
  it("Sometimes you might want to call the original method from the stub and modify it", () => {
    const person = {
      getName() {
        return "Joe";
      },
    };

    const originalGetName = person.getName.bind(person);

    jest.spyOn(person, "getName").mockImplementation(() => {
      return originalGetName().split("").reverse().join("");
    });

    expect(person.getName()).toEqual("eoJ");
  });
});

Controlling time cy.clock vs jest.useFakeTimers:

When running Cypress tests, the tests themselves are outside the application's iframe. When you use cy.clock() command you change the application clock, and not the spec's clock.

describe("cy.clock", () => {
  it("control the time in the browser", () => {
    const specNow = new Date();
    const now = new Date(Date.UTC(2017, 2, 14)).getTime();

    cy.clock(now) // sets the application clock and pause time
      .then(() => {
        // spec clock keeps ticking
        const specNow2 = new Date();
        // confirm by comparing the timestamps in milliseconds
        expect(+specNow2, "spec timestamps").to.be.greaterThan(+specNow);
      });
    // but the application's time is frozen
    cy.window()
      .its("Date")
      .then((appDate) => {
        const appNow = new appDate();
        expect(+appNow, "application timestamps")
          .to.equal(+now)
          .and.to.equal(1489449600000); // the timestamp in milliseconds
      });
    // we can advance the application clock by 5 seconds
    cy.tick(5000);
    cy.window()
      .its("Date")
      .then((appDate) => {
        const appNow = new appDate();
        expect(+appNow, "timestamp after 5 synthetic seconds").to.equal(
          1489449605000
        );
      })
      // meanwhile the spec clock only advanced by probably less than 200ms
      .then(() => {
        const specNow3 = new Date();
        expect(+specNow3, "elapsed on the spec clock").to.be.lessThan(
          +specNow + 200
        );
      });
  });
});

The Jest mirror cannot be run in the browser window, and has a slightly different approach.

describe("jest.useFakeTimers", () => {
  it("control the time in the browser", () => {
    jest.useFakeTimers();
    const specNow = new Date();
    const now = new Date(Date.UTC(2017, 2, 14)).getTime();

    jest.setSystemTime(now);

    // application time is frozen
    const appNow = new Date();
    expect(appNow.getTime()).toBe(now);
    expect(appNow.getTime()).toBe(1489449600000); // the timestamp in milliseconds

    // we can advance the application clock by 5 seconds
    jest.advanceTimersByTime(5000);
    const appNow2 = new Date();
    expect(appNow2.getTime()).toBe(1489449605000);

    // spec clock only advanced by probably less than 200ms
    const specNow3 = new Date();
    expect(specNow3.getTime()).toBeLessThan(specNow.getTime() + 200);

    jest.useRealTimers();
  });
});

Comparison of network spies & mocks: `cy.intercept` vs `MSW`

If you have been through Kent C. Dodd's Epic React, you are already convinced that the farther away from our component the mocking is, the more we are testing our code and having better confidence. The farthest we can mock from our code is mocking the network. To mock the network Cypress has the intercept api, and the exact mirror on RTL side is Mock Service Worker (msw).

Let us compare RTL + MSW Heroes.test.tsx vs CyCT + cy.intercept from Tour of Heroes repo.

Here is a summary of the key points; with msw we have a bit more setup, and a required clean up, on the other side with cy.intercept the syntax is succinct and no clean up is required.

// RTL + MSW
it("should see error on initial load with GET", async () => {
  // have to define handlers, setup server and listen
  const handlers = [
    rest.get(
      `${process.env.REACT_APP_API_URL}/heroes`,
      async (_req, res, ctx) => res(ctx.status(400))
    ),
  ];
  const server = setupServer(...handlers);
  server.listen({
    onUnhandledRequest: "warn",
  });

  // test code...

  // have to clean up
  server.resetHandlers();
  server.close();
});

// CyCT + cy.intercept
it("should see error on initial load with GET", () => {
  // in comparison, this is all we have to do with cyct
  cy.intercept("GET", `${Cypress.env("API_URL")}/heroes`, {
    statusCode: 400,
    delay: 100,
  }).as("notFound");

  // test code...

  // no clean up needed
});

The difference is more clear when we want to structure our tests. RTL + msw ends up being plenty more boilerplate compared to CyCT and cy.intercept. Not only the concise syntax, but also the ability to define or change our network mock on the fly with cy.intercept makes our code easier to understand. Check out the code comments below for precise examples:

// RLT + msw
describe("200 flows", () => {
  const handlers = [
    rest.get(
      `${process.env.REACT_APP_API_URL}/heroes`,
      async (_req, res, ctx) => res(ctx.status(200), ctx.json(heroes))
    ),
    // we have to have all definitions in the handler
    // once declared, these are it;
    // we would need a new describe block to change the mock
    rest.delete(
      `${process.env.REACT_APP_API_URL}/heroes/${heroes[0].id}`, // use /.*/ for all requests
      async (_req, res, ctx) => res(ctx.status(400), ctx.json("expected error"))
    ),
  ];
  const server = setupServer(...handlers);
  beforeAll(() => {
    server.listen({
      onUnhandledRequest: "warn",
    });
  });
  afterEach(server.resetHandlers);
  afterAll(server.close);

  it("should display the hero list on render, and go through hero add & refresh flow", async () => {
    expect(await screen.findByTestId("list-header")).toBeVisible();
    expect(await screen.findByTestId("hero-list")).toBeVisible();

    await userEvent.click(await screen.findByTestId("add-button"));
    expect(window.location.pathname).toBe("/heroes/add-hero");

    await userEvent.click(await screen.findByTestId("refresh-button"));
    expect(window.location.pathname).toBe("/heroes");
  });

  const deleteButtons = async () => screen.findAllByTestId("delete-button");
  const modalYesNo = async () => screen.findByTestId("modal-yes-no");
  const maybeModalYesNo = () => screen.queryByTestId("modal-yes-no");
  const invokeHeroDelete = async () => {
    userEvent.click((await deleteButtons())[0]);
    expect(await modalYesNo()).toBeVisible();
  };

  it("should go through the modal flow, and cover error on DELETE", async () => {
    expect(screen.queryByTestId("modal-dialog")).not.toBeInTheDocument();

    await invokeHeroDelete();
    await userEvent.click(await screen.findByTestId("button-no"));
    expect(maybeModalYesNo()).not.toBeInTheDocument();

    await invokeHeroDelete();
    await userEvent.click(await screen.findByTestId("button-yes"));

    expect(maybeModalYesNo()).not.toBeInTheDocument();
    expect(await screen.findByTestId("error")).toBeVisible();
    expect(screen.queryByTestId("modal-dialog")).not.toBeInTheDocument();
  });
});

// CyCT + cy.intercept
context("200 flows", () => {
  beforeEach(() => {
    // the GET is common to both tests
    cy.intercept("GET", `${Cypress.env("API_URL")}/heroes`, {
      fixture: "heroes.json",
    }).as("getHeroes");

    cy.wrappedMount(<Heroes />);
  });

  it("should display the hero list on render, and go through hero add & refresh flow", () => {
    cy.wait("@getHeroes");

    cy.getByCy("list-header").should("be.visible");
    cy.getByCy("hero-list").should("be.visible");

    cy.getByCy("add-button").click();
    cy.location("pathname").should("eq", "/heroes/add-hero");

    cy.getByCy("refresh-button").click();
    cy.location("pathname").should("eq", "/heroes");
  });

  const invokeHeroDelete = () => {
    cy.getByCy("delete-button").first().click();
    cy.getByCy("modal-yes-no").should("be.visible");
  };

  it("should go through the modal flow, and cover error on DELETE", () => {
    cy.getByCy("modal-yes-no").should("not.exist");

    cy.log("do not delete flow");
    invokeHeroDelete();
    cy.getByCy("button-no").click();
    cy.getByCy("modal-yes-no").should("not.exist");

    cy.log("delete flow");
    invokeHeroDelete();

    // DELETE mock is unique to this test
    // we can define it or change our network mock on the fly
    // we could for example have a new GET definition here
    cy.intercept("DELETE", "*", { statusCode: 500 }).as("deleteHero");

    cy.getByCy("button-yes").click();
    cy.wait("@deleteHero");
    cy.getByCy("modal-yes-no").should("not.exist");
    cy.getByCy("error").should("be.visible");
  });
});

In our opinion, the intercept API is simpler and is more flexible compared to MSW. We can observe this in the significant difference in the amount of code we have to write, doing the same thing in RTL + msw vs CyCT + cy.intercept. You can compare them here RTL + MSW Heroes.test.tsx vs CyCT + cy.intercept.

CyCT vs RTL examples - Epic React

In the interest of brevity, we will share links to 13 Cypress component tests and their RTL mirrors. All CyCT examples are from the repo cypress-react-component-test-examples where you can find 400+ individual CyCT examples.

Simple counter: cyct vs rtl

Testing with context: cyct vs rtl

Simple redux: cyct vs rtl

A11y: cyct vs rtl

Geolocation: cyct vs rtl

Mocking http (intercept vs msw) : cyct vs rtl, another cyct vs rtl

Router-redirect: cyct vs rtl

React-router: cyct vs rtl

Modal: cyct vs rtl

Stub window fetch: cyct vs rtl

Wrapping up

We went through, in detail, 3 examples of CyCT vs RTL from the repo & book CCTDD: Cypress Component Test Driven Design where you can find a few dozen more examples of CyCT vs RTL.

We compared low level mocking in CyCT with Sinon vs mocking in RTL with Jest, with many examples and a sample cheat sheet repo.

We compared network level mocking using cy.intercept vs msw with repository links.

Finally we shared about a dozen more CyCT vs RTL examples from Kent C. Dodds' Epic React.

Equipped with these resources, you have cheat sheets at your fingertips and an information toolset to begin rolling out Cypress component testing in your organizations.

Addendum: Gleb Bahmutov's The Missing Comparison Part video: a comparison of the developer experience

In his video, Gleb covered an important part of comparing RTL and CyCT; the developer experience. "Jest (with RTL) use the terminal JSDom, while Cypress gives you the real browser with time traveling debugger, making debugging errors so so so much simpler in Cypress" We strongly suggest to see the video, and we will cover the three key points below.

1. Compare the devex making a simple breaking change to the source code: HeaderBarBrand

Pull the repo, install and start CyCT with RTL side by side; yarn cy:open-ct , yarn test. Execute the CyCT and RTL tests for HeaderBarBrand.

Make a breaking change in HeaderBarBrand component; on line 20 change the spelling of OF to ON.

<NavLink data-cy="navLink" to="/" className="navbar-item navbar-home">
  <span className="tour">TOUR</span>
  {/* Change OF to ON */}
  <span className="of">ON</span>
  <span className="heroes">HEROES</span>
</NavLink>

Compare how you would diagnose this failure in RTL and CyCT.

Here is the RTL failure:

Here is the CyCT failure:

2. Compare test stability by simulating an asynchronous process: InputDetail

Execute the CyCT and RTL tests for InputDetail.

On line 34 of InputDetail component, introduce a setTimeout to simulate an asynchronous process. These are very common in real applications, and it is a solid comparison of the stability of the two tools, especially in CI.

<input
  name={name}
  role={name}
  defaultValue={shownValue}
  placeholder={placeholder}
  // @ts-expect-error add a setTimeout to simulate an asynchronous delay
  onChange={() => setTimeout(onChange, 1000)}
  readOnly={readOnly}
  className="input"
  type="text"
></input>

CyCT executes the same, with a slight delay retrying.

We cannot make the RTL test pass as is; it is synchronous. We would have to add asynchronous assertions to the test to make it work.

3. Compare the devex making a "complex" change: Heroes

The test of concern stubs the deleting of a hero with a 500 network response (MSW with RTL, and cy.intercept with CyCT) to simulate a deletion error.

In Heroes component, comment out line 40 so that we are doing nothing upon hero deletion.

const handleDeleteFromModal = () => {
  // heroToDelete ? deleteHero(heroToDelete) : null
  setShowModal(false);
};

We are in the dark, trying to identify why the test did not work looking at RTL results (which are a few times longer than the screenshot):

Looking at the CyCT failure, we can easily tell that the 500 network call never happened:

DEV Community: Murat K Ozcan

The testing meta most teams have not caught up to yet

A QA operating system for AI testing

Fixing DX first: Playwright-Utils

The brain: TEA (Test Architect agent from BMAD)

The hands and eyes: Playwright MCPs

How it works in practice

Why this stack matters

What you get out of the box

Call to action

Turkish and Functional Programming: A Surprisingly Perfect Match

1. No Redundant Elements (Minimalism)

2. Agglutination = Function Composition

3. Harmony & Consistency

4. No Extra Noise (Declarative Style)

Final Thoughts

Page Objects vs. Functional Helpers

Page Objects vs. Functional Helpers

🚀 Why Functional Helpers > Page Objects?

❌ Why Page Objects No Longer Make Sense

1️⃣ Unnecessary Abstraction

2️⃣ Base Page Inheritance is Overkill

3️⃣ Harder Debugging

🔴 Traditional Page Object Model (POM)

✅ Functional Helper Approach (Better)

📌 Why is this better?

🔥 Final Thoughts

Addendum

Solving Cross-Execution Issues in Pact Testing with Kafka and Message Queues

Solving Cross-Execution Issues in Pact Testing with Kafka and Message Queues

The Problem: Cross-Execution of Tests Between Provider-Consumer Pairs

Root Cause Analysis

Why Was the Problem Worth Solving

How We Solved It

1. Refactoring the URL Handling Logic

Original Issues:

Refactored Solution:

2. Implementing Conditional Logic

3. Ensuring Proper Fallback

4. Updating Test Files

5. Verifying the Solution

Conclusion

Building Custom Request Filters for PactJs Verifications in Express and Non-Express Environments

The Problem:

The Solution: A Higher-Order Function

Key Takeaways:

Conclusion:

Handling Pact Breaking Changes Dynamically in CI/CD

The Problem: Breaking Changes in Pact Verification

The Solution: Managing Breaking Changes with Environment Variables

Implementation Details

Verifier Options

Building Consumer Version Selectors

Integrating with CI/CD

GitHub Actions Workflow

PR Description Checkbox

Handling Verification Failures Due to Breaking Changes

Workflow Summary

Breaking Change Flows

Consumer Flow

Provider Flow

Conclusion

Strengthening Pact Contract Testing with TypeScript and Data Abstraction

What are State Handlers?

The Problem: Type Mismatch in Provider State Handling

Introducing createProviderState: A Type-Safe Abstraction

What is JsonMap?

So, how do we start fixing things?

The abstraction we want: createProviderState

Example: Consumer Side Tests

Aligning Provider State Handlers with Strong Typing

Conclusion

My thoughts and notes about Consumer Driven Contract Testing

My thoughts and notes about Consumer Driven Contract Testing

Contract Testing

Consumer side

Provider side

The Pact Broker

Example new release scenario:

Environments:

Introducing `createProviderState`: A Type-Safe Abstraction

What is `JsonMap`?

The abstraction we want: `createProviderState`

`messageProviders` vs `stateHandlers` on the producer/provider side

3. Generate JSON schemas with the `ts-json-schema-generator` package

4. Generate the OpenAPI spec with the `openapi-types` package

Create the `openapi.ts` file

Create a script to execute the `openapi.ts` file and generate the OpenAPI spec.

Generate your `openapi.yml` file

Create convenience scripts in `package.json`

Do we keep having to record the schema with `schema:verify` ?

Reducing redundant email consumption with `cypress-data-session`

Magic links with `cypress-data-session`