DEV Community: Serhiy Pikho

Building a Transpiler to Migrate over 4,000 Tests to Playwright

Serhiy Pikho — Mon, 13 Jul 2026 08:14:21 +0000

Preface

I migrated a large existing test suite to a new framework. Some of the same ideas and techniques may be useful for migrations that seem too large to start.

Background

I joined a team with the goal of improving the QA pipeline. The team supported and maintained a chatbot. It was a complex AWS Connect setup: hundreds of flows, free text input and dynamic Lambda functionality.

After spending some time interviewing people and looking at the existing QA tooling, I came to the conclusion that the largest bottleneck on the QA side was the framework they were using.

Existing Framework

The existing framework reflected how the chatbot had originally been developed. It stored all test cases in a workbook. Each row represented a test case, with title, tags, other metadata and the test body itself. The test body used BDD-style keywords: VERIFY, SELECT, INPUT, SELECT, followed by a string that represented a snippet of what appears in the chatbot.

The runner loaded the workbook, initialised WDIO and executed each row as a test.

The approach was understandable, but it came with significant trade-offs. Most of the contact flows themselves were tracked in workbooks, so the translation from flow to test case was theoretically easy. The downsides were severe:

Git treated it as a binary file. If multiple QAs edited it at the same time, they had to stop and co-ordinate a merge process.
Tooling was non-existent. For example, if a root node of the chat journey changed, the affected string sequence had to be found and updated manually through the workbook. In TypeScript/Playwright, with the right test design, this could be a one line change.
There was no practical way to use AI to help generate test cases.
Debugging was hard. Running the tests in parallel would make that even harder.

The Migration

There were already over 4,000 tests, and the team was feeling the strain of this system. Fixing it was daunting and urgent. I gave myself two weeks to solve the problem, either by patching the existing framework or by migrating everything to something more maintainable.

I considered using an LLM for the whole migration; there is precedent for this in much more complex codebases. But that would not work for me. Cost was one problem, as my work does not allocate an unlimited token budget. I also needed determinism and traceability. Those three constraints ruled out a full LLM-based migration.

The original source of the tests was largely unstructured. While changing the representation of the tests was one obvious result, I also needed to recover the meaning of each test and categorise it; otherwise I would end up with .spec files that were several thousand lines long.

To do the actual migration I took inspiration from transpiler and compiler pipelines. The input format was well defined at a syntax level and the output format was well defined. I needed a pipeline that could transform one into the other.

I split the pipeline into small stages, with each stage solving a specific problem:

Workbook -> extraction -> content graph mining -> content sema -> journey sema -> journey mining -> emission.
// Note: sema = semantic analysis

The conversion scripts were written in TypeScript. I mainly used the standard library, with the notable exceptions of xlsx to read the file and ts-morph, which made the emission stage possible by allowing programmatic generation of TypeScript code.

A before and after

Here is one small test case before migration:

Worksheet: HomeAuthOfficeHours
Row: 87
Product: Home
Flow: Payments - Verify 'Missed payment' feature - Connect to customer specialist:No - Is Info Helped:No - Anything Else Help:No in auth journey
Auth: Y
Payment: ANNUAL
Mode: activation
MenuPath: Payments

Steps:
OPEN_HELP;
VERIFY:Hi, I’m your Virtual Assistant. I can help you with simple queries;
VERIFY:Please choose from the following menu.;
CLICK:Renewals;
VERIFY:Choose an option below, or type your question if you don’t see what you’re looking for.;
CLICK:Missed payment;
VERIFY:If you've missed a payment, you'll receive a letter and email from us confirming;
CLICK:No;
VERIFY:Was the information I provided helpful?;
CLICK:No;
VERIFY:Is there anything else I can help you with?;
CLICK:No;
VERIFY:Thank you for contacting us. Have a nice day.;
CLOSE_CHAT

After migration, the transpiler emitted a normal Playwright test that fitted the framework conventions:

test.beforeEach(async ({ brand, userPool, landingPage }) => {
  const user = await userPool.homeUser({ overrides: { brand: brand as HomeBrandNames } });
  await userPool.openLandingPageForCustomer({ customerNumber: user.customerNumber, landingPage });
});

test(
  "missed payment -> Self-service path -> Anything else: no -> Payments -> Helpful: no",
  {
    tag: [tag.e2e, tag.home, tag.payments, tag.magicLinkEntry],
    annotation: [
      { type: "code", description: "HomeAuthOfficeHours#87" },
      { type: "sourceCase", description: "HomeAuthOfficeHours#87" },
      { type: "sourceBacklink", description: "worksheet=HomeAuthOfficeHours | row=87" }
    ]
  },
  async ({ landingPage }) => {
    const chat = await landingPage.openChat();
    await chat.verify(common.introduction.greeting);
    await chat.verify(common.menu.prompts.menu);
    await chat.select(common.routing.options.renewals);
    await chat.verify(common.menu.prompts.chooseOption);
    await chat.select(payments.missedPayment.options.missedPayment);
    await chat.verify(payments.missedPayment.prompts.missedPaymentLetterAndEmail);
    await chat.flows.navigation.feedback.feedbackPrompt();
    await chat.close();
  }
);

The pipeline had to do this kind of conversion thousands of times. Raw workbook strings became shared content constants, repeated sequences became flow calls, user setup moved into fixtures, and the generated test still pointed back to the original worksheet row via annotations.

Extraction

Extraction solves the first problem: the workbook is too inconsistent to reason about safely. Before the pipeline can infer anything about the chatbot, it first needs a deterministic representation of the source material.

The pipeline pulls each row apart, expands keyword macros and normalises the parts that have drifted over time.

The normalisation step handles issues such as:

expanded macros inside the workbook
whitespace drift
casing differences
punctuation differences
smart quotes

After that, each workbook row becomes a test case object containing the source metadata together with an AST-like breakdown of the journey. In compiler terms, this is the intermediate representation (IR): a data structure that successive passes progressively enrich.

Content Graph Mining

At this point the pipeline can replay individual workbook rows, but it still has no understanding of how the content itself is reused across the suite. Here, "content" refers to the raw strings present in the workbook.

Content graph mining solves that problem by discovering relationships between content instances.

The pipeline de-duplicates chatbot prompts, options and user inputs before constructing a structural graph from the deduplicated content. Nodes represent prompts, options and inputs. Edges represent transitions between them. Paths show how those pieces of content are combined within workbook journeys. Heat maps highlight the most frequently reused areas of the graph.

The graph answers questions the raw workbook cannot:

Does this content appear at the start of journeys?
Is it a branch point?
Is it a merge point?
What usually comes before it?
What usually comes after it?
Which workbook rows and journey families use it?

Those relationships make later semantic analysis possible. A yes/no option is not meaningful by itself; its surrounding content usually determines whether it belongs to authentication, feedback, handoff acceptance or a return-to-menu question.

The graph also validates the earlier extraction work. If content appears in impossible locations, either the workbook contains inconsistent data or an earlier transformation has misunderstood it.

Content Semantic Analysis

The pipeline now understands how pieces of content relate to one another, but it still does not understand what that content represents.

Content semantic analysis solves that problem.

The pipeline converts the enriched graph into stable content tokens. In compiler terms, this is interning. Every occurrence of "Do you have an existing policy with us?" becomes a reference to the same stable content identity rather than another copy of the same string.

The pipeline derives token IDs from the canonical content key: keyword, content role and canonical value. Getting that identity right proves far more important than it first appears because every later stage depends on those references remaining stable.

The stage also classifies tokens using keyword shape, workbook metadata and graph context:

authentication prompts
common yes/no options
payment content
claims content
handoff messages
renewal content
policy information
service window messaging

The classifier attaches confidence and evidence to every decision. When confidence falls below an acceptable threshold, the pipeline marks the content for review instead of guessing.

By the end of this stage, the pipeline no longer carries workbook strings. It carries stable content identities enriched with semantic meaning.

Journey Semantic Analysis

Understanding individual pieces of content is still not enough. The pipeline now needs to understand what complete journeys represent.

Journey semantic analysis solves that problem by lifting the semantic information from individual content into complete journeys.

The pipeline classifies each journey using its token sequence together with workbook metadata and the relationships discovered during content mining. It derives product, entry point, service window, business topic, authentication route, handoff behaviour, generated tags and mutation profile.

This stage deliberately stops short of deciding how the final Playwright code should look. Instead, it produces the semantic facts that later stages depend on: product grouping, route shape, suite placement, tags and mutation behaviour.

By the end of this stage, the intermediate representation contains enough semantic information that later passes no longer need to infer behaviour from workbook data.

Journey Mining

By this point the pipeline understands what every journey represents, but the generated suite is still not optimised for long-term maintenance.

Journey mining solves that problem.

A naïve generator can emit perfectly valid Playwright tests while duplicating the same interaction sequences hundreds of times. If one of those commonly reused journeys changes, every generated copy has to change with it.

Instead, the pipeline searches the semantically classified journeys for behaviour that deserves to become a shared abstraction.

The stage answers questions such as:

Which paths genuinely repeat?
Which steps are simply common shell around the journey?
Which repeated sequences deserve helper methods?
Which repeated paths should become shared flows?
Which journeys require manual review?

The repeated corridor view highlights common routes through the classified journeys. Those corridors become candidates for shared flow methods.

The pipeline produces both the flow manifest and a triage artifact explaining every decision. Some candidates are too short. Some lack enough supporting journeys. Others are subsumed by stronger abstractions or introduce more indirection than value.

By the end of this stage, the intermediate representation no longer consists solely of token actions. Reusable behaviour has been promoted into shared flow references, leaving only journey-specific interactions inline.

Emission

The remaining problem is transforming the final intermediate representation into Playwright code that looks like an engineer wrote it rather than a generator.

Conceptually, emission solves that problem. Although the implementation emits some artifacts earlier than others, its responsibility remains the same: transform stable pipeline data into maintainable TypeScript.

So far the pipeline has only been using stable IDs. They are good for machines and terrible for humans to read. Some are also invalid TypeScript syntax.

Humanisation manifests solve that problem. They map stable content and flow identities onto readable runtime names.

The pipeline emits three outputs:

shared content constants
reusable chat flows
Playwright specifications

The content emitter generates shared TypeScript content modules. The flow emitter generates reusable chat flow classes. Finally, the journey emitter combines those abstractions with the remaining inline interactions to produce Playwright tests.

Journey semantic analysis has already solved the difficult architectural questions. The emitter simply applies those decisions. It selects the correct fixtures, startup profile, service-window preset, tags, annotations and generated users before rendering the remaining journey.

I also wanted the generated output to remain reviewable. The emitter preserves file ordering wherever possible by reading the existing generated output before writing updates.

Several passes exist solely to detect the pipeline's own mistakes: graph validation, classification drift checks, flow triage and journey quality audits. That matters because a faulty migration pipeline can generate code that looks perfectly reasonable while being fundamentally wrong.

By the end of the pipeline, workbook rows have become maintainable Playwright code consisting of shared content constants, reusable flow abstractions, generated specifications and traceability back to the original source rows.

Benefits of the new framework

The transpiler was created so the suite could use capabilities the old framework could not support cleanly.

Parallel Playwright workers. The generated suite could run across multiple Playwright workers, which mattered once it stopped being a single workbook-driven execution path.
A reusable user pool. Generated specs could ask for userPool.homeUser(...) or userPool.motorUser(...) instead of carrying policy creation details in every test. Where a journey mutated policy state, the fixture could provide a fresh user without changing the test body.
Websocket-backed chat timing and verification. The chat helpers listened to the underlying AWS Connect websocket frames, so chat.verify(...) could check transcript messages as well as the visible UI. Actions could wait for customer turns and settled chat activity instead of relying on blind sleeps.

Results and lessons learned

Before the migration, tests ran sequentially. Each one took between one and two minutes to execute. At that rate, running the full suite of 4,000 tests would have taken days. With ten Playwright workers the same suite averaged three seconds per test. Failures produced traces, so errors that had previously been hard to explain had evidence.

Adding new tests became routine instead of a manual translation exercise.

Treating the migration like a compiler problem made it much easier to debug. Once I had a source model, intermediate artifacts and an emitter, I could debug each step in isolation. When the generated code looked wrong, I rarely had to debug the Playwright test itself. I debugged the pipeline stage that produced it.

Generating TypeScript was easier than categorising ambiguous data from context. Most of the work went into working out what each piece of chatbot content meant, where journeys belonged, and which repeated paths were real abstractions rather than accidental duplication.

This migration started from a workbook, so a lot of the work was recovering structure. A more conventional framework migration should have less of that problem. For example, a Selenium suite already has a source-language AST, which means the pipeline can focus on mapping Selenium concepts into an explicit intermediate representation before emitting the intended Playwright architecture.

The Quest for Ultra-Reliable E2E UI Tests: Making API Contracts First-Class in Page Objects

Serhiy Pikho — Tue, 24 Feb 2026 10:40:46 +0000

Over the last couple of years I’ve repeatedly encountered the same frustrations relating to test stability while writing and maintaining UI tests across different frameworks, Cypress, WebdriverIO, and Playwright.

Much of the literature on reliable UI testing focuses on selector strategy, polling mechanics, and retry behaviour. These techniques are useful and often necessary. I’ve relied on them extensively. But they all operate within the same assumption: that the DOM is the primary boundary of the system under test.

Increasingly, I’ve come to the conclusion that this assumption is incomplete.

Core Idea

A UI is not a self-contained system. It is a rendering layer over a set of network contracts.

If the API calls required by a page do not complete successfully, the UI cannot function correctly. The DOM may render and elements may exist, but the underlying state is already invalid.

The practical implication is that readiness should be defined in terms of contract completion rather than DOM stability.

Instead of waiting for locators to stabilise, readiness can be defined by the successful resolution of specific, declared network dependencies.

Effects of This Change

Moving readiness to the network layer changes several properties of a test suite.

Polling logic becomes simpler. Instead of repeatedly checking element state, we track deterministic request/response pairs.
Diagnostics improve. Failures are attributed to specific contracts. Instead of a late-stage “locator not found”, the failure identifies the exact request that did not occur or returned non-OK (examples below).
Run time improves. Tests terminate earlier. If a required API fails, there is little value in continuing execution.
Parallel execution becomes more predictable. When tests are aware of backend dependencies, they partially account for system load rather than interpreting load-related failures as UI instability.

This does not eliminate flakiness entirely. It reduces ambiguity and moves failures closer to their cause.

Prior Art and Limits of Existing Signals

Network awareness is not new. Most modern frameworks expose mechanisms to wait for network conditions.

For example, Playwright allows:

await page.goto(url, { waitUntil: "networkidle" })

In practice, broad signals such as “network idle” are imprecise in modern applications. Background traffic, analytics calls, and third-party injections can keep the network active indefinitely. Waiting for the absence of traffic is not equivalent to waiting for the contracts that actually matter.

Direct use of waitForResponse inside individual tests can address specific cases, but it distributes readiness logic across the suite. Each test author decides what to wait for and how strictly to enforce it. The approach described here centralises that concern in the Page Object layer, making readiness a structural property rather than a local decision.

Similarly, network mocking tools solve a different problem, determinism of inputs, rather than enforcing correctness of contract completion in integrated environments.

The distinction is subtle: this approach targets explicitly declared dependencies rather than relying on global heuristics or per-test decisions.

Implementation Example

Large-scale UI test suites commonly model each page as a Page Object.

A typical Page Object defines:

Locators
Interaction helpers
Page-specific assertions

If we treat a Page Object as a model of a page, then the model should include more than visible elements. It should also describe the network contracts required for that page to function.

One way to structure this is for each page to declare its required and optional API dependencies. A shared base layer observes network traffic, matches it against those declarations, and enforces readiness rules.

Readiness becomes an architectural concern rather than something encoded repeatedly inside tests.

Base Page Layer

All Page Objects inherit from a shared base class. The example below is simplified but representative.

import { type Page } from "@playwright/test";
import { type DataDependency, DataDependencyTracker } from "./dataDependencyTracker";

export abstract class BasePage {
  protected readonly tracker: DataDependencyTracker;

  constructor(protected readonly page: Page) {
    this.tracker = DataDependencyTracker.getInstance(page);
  }

  protected apiContracts = {
    profile: { method: "GET", url: /\/api\/profile(?:\?|$)/ },
    basket: { method: "GET", url: /\/api\/basket(?:\?|$)/ },
    quote: { method: "POST", url: /\/api\/quote(?:\?|$)/ },
    confirm: { method: "POST", url: /\/api\/confirm(?:\?|$)/ }
  } satisfies Record<string, DataDependency>;

  protected async transition(
    action: () => Promise<unknown>,
    override?: DataDependency[]
  ) {
    const dependencies = override ?? this.dataDependencies;

    await Promise.all([
      this.tracker.waitForAll(dependencies),
      action()
    ]);
  }

  abstract get dataDependencies(): DataDependency[];
}

Several structural decisions are important:

A tracker instance is scoped to the Playwright Page (isolated per page / worker).
A central registry of API contracts.
A transition boundary that gates navigation.
A dataDependencies getter, allowing dependencies to be declared statically or computed conditionally.

The ordering inside Promise.all is intentional. Dependency tracking begins before the action resolves. If tracking were started after the click, fast requests could be missed. The boundary must observe the transition, not react to it.

Data Dependency Tracker

The tracker is a thin layer over Playwright’s network primitives.

import { type Page, type Request, type Response } from "@playwright/test";

export type DataDependency = {
  url: RegExp;
  method: string;
};

export type TimeoutOpts = {
  requestMs: number;
  responseMs: number;
  requireRequest: boolean;
  requireOk: boolean;
};

export class DataDependencyTracker {
  private static trackerRegistry = new WeakMap<Page, DataDependencyTracker>();

  private readonly cached = new Set<string>();
  private readonly notApplicable = new Set<string>();

  private readonly reducedRequestMs = 1_000;

  private readonly defaults: TimeoutOpts = {
    requestMs: 30_000,
    responseMs: 60_000,
    requireRequest: true,
    requireOk: true
  };

  private constructor(private readonly page: Page) {}

  static getInstance(page: Page): DataDependencyTracker {
    let tracker = this.trackerRegistry.get(page);

    if (!tracker) {
      tracker = new DataDependencyTracker(page);
      this.trackerRegistry.set(page, tracker);
    }

    return tracker;
  }

  async waitFor(dep: DataDependency, opts: Partial<TimeoutOpts> = {}) {
    const method = dep.method.toUpperCase();
    const key = `${method} ${dep.url.toString()}`;

    const { requestMs, responseMs, requireRequest, requireOk } = {
      ...this.defaults,
      ...opts
    };

    const effectiveRequestMs =
      this.cached.has(key) || this.notApplicable.has(key)
        ? this.reducedRequestMs
        : requestMs;

    let request: Request | undefined;

    try {
      request = await this.page.waitForRequest(
        r =>
          dep.url.test(r.url()) &&
          r.method().toUpperCase() === method,
        { timeout: effectiveRequestMs }
      );
    } catch {
      this.notApplicable.add(key);

      if (requireRequest) {
        throw new Error(`Expected request did not occur: ${method} ${dep.url}`);
      }

      return;
    }

    const response = await this.page.waitForResponse(
      r => r.request() === request,
      { timeout: responseMs }
    );

    this.notApplicable.delete(key);
    this.cached.add(key);

    if (!response.ok() && requireOk) {
      throw new Error(
        `Non-ok response: ${method} ${dep.url} (${response.status()})`
      );
    }
  }

  async waitForAll(deps: DataDependency[], opts: Partial<TimeoutOpts> = {}) {
    await Promise.all(deps.map(d => this.waitFor(d, opts)));
  }
}

Example: A Full Contract-Gated Flow

Consider a minimal checkout flow:

The landing page loads profile data.
The order page loads basket data.
The review page prepares totals.
Submitting the order triggers confirmation.
The success page depends on that confirmation succeeding.

Each page declares its required contracts by referencing the central apiContracts repository. Transitions are gated through transition().

Page Objects

export class LandingPage extends BasePage {
  get dataDependencies(): DataDependency[] {
    return [this.apiContracts.profile];
  }

  async goToCompleteOrderPage(): Promise<CompleteOrderPage> {
    const next = new CompleteOrderPage(this.page);

    await this.transition(
      () => this.page.getByRole("link", { name: "Checkout" }).click(),
      next.dataDependencies
    );

    return next;
  }
}

export class CompleteOrderPage extends BasePage {
  get dataDependencies(): DataDependency[] {
    return [this.apiContracts.basket];
  }

  async goToReviewPage(): Promise<ReviewPage> {
    const next = new ReviewPage(this.page);

    await this.transition(
      () => this.page.getByRole("button", { name: "Continue" }).click(),
      next.dataDependencies
    );

    return next;
  }
}

The remaining pages (ReviewPage, OrderSuccessPage, etc.) follow the same pattern.

The Test

test("user completes checkout", async ({ page }) => {
  const landingPage = new LandingPage(page);
  const completeOrderPage = await landingPage.goToCompleteOrderPage();
  const reviewPage = await completeOrderPage.goToReviewPage();
  await reviewPage.verifyTotals();
  const successPage = await reviewPage.submitOrder();
  await successPage.expectSuccess();
});

No explicit waits are required here.

No waitForResponse().
No expectLoaded().
No arbitrary timeouts.

If a method returns a Page Object, its declared contracts have completed. Readiness is enforced at the architectural boundary, not inside the test.

Failure Modes

If the submission request never occurs:

Expected request did not occur: POST /\/api\/confirm(?:\?|$)/

If the response is non-OK:

Non-ok response: POST /\/api\/confirm(?:\?|$)/ (500)

Failures surface at the transition boundary, not at a later DOM assertion.

Under parallel load, slow paths remain localised to their declared dependencies rather than inflating global timeouts.

Limitations

This pattern is not a replacement for DOM-level checks. A page can satisfy its network contracts and still render incorrectly.

Contract-driven readiness complements DOM assertions; it does not eliminate them.

It also introduces coupling between Page Objects and backend contracts. When APIs change, test models must change with them. In larger systems this can be desirable, contract drift becomes visible, but in smaller or rapidly evolving projects it may feel heavy.

Another practical difficulty is discovery. Modern UI applications often abstract API calls behind state management layers, Zustand, MobX, Redux, or custom data clients. From the perspective of the test author, the network dependencies of a page are not always obvious.

Declaring API contracts manually requires either familiarity with the frontend implementation or careful inspection of network traffic. This adds friction, particularly in large or fast-moving codebases.

In practice, this led to additional tooling to assist with dependency discovery rather than relying solely on manual declaration. That is likely a separate discussion.