Building a Transpiler to Migrate over 4,000 Tests to Playwright

Preface

I migrated a large existing test suite to a new framework. Some of the same ideas and techniques may be useful for migrations that seem too large to start.

Background

Recently I joined an internal team at my current place of work with the goal of improving the QA pipeline. The team supported and maintained a chatbot. It was a complex AWS Connect setup: hundreds of flows, free text input and dynamic Lambda functionality.

After spending some time interviewing people and looking at the existing QA tooling, I came to the conclusion that the largest bottleneck on the QA side was the framework they were using.

Existing Framework

The existing framework reflected how the chatbot had originally been developed. It stored all test cases in a workbook. Each row represented a test case, with title, tags, other metadata and the test body itself. The test body used BDD-style keywords: VERIFY, SELECT, INPUT, SELECT, followed by a string that represented a snippet of what appears in the chatbot.

The runner loaded the workbook, initialised WDIO and executed each row as a test.

The approach was understandable, but it came with significant trade-offs. Most of the contact flows themselves were tracked in workbooks, so the translation from flow to test case was theoretically easy. The downsides were severe:

1. Git treated it as a binary file. If multiple QAs edited it at the same time, they had to stop and co-ordinate a merge process.
2. Tooling was non-existent. For example, if a root node of the chat journey changed, the affected string sequence had to be found and updated manually through the workbook. In TypeScript/Playwright, with the right test design, this could be a one line change.
3. There was no practical way to use AI to help generate test cases.
4. Debugging was hard. Running the tests in parallel would make that even harder.

The Migration

There were already over 4,000 tests, and the team was feeling the strain of this system. Fixing it was daunting and urgent. I gave myself two weeks to solve the problem, either by patching the existing framework or by migrating everything to something more maintainable.

I considered using an LLM for the whole migration; there is precedent for this in much more complex codebases. But that would not work for me. Cost was one problem, as my work does not allocate an unlimited token budget. I also needed determinism and traceability. Those three constraints ruled out a full LLM-based migration.

The original source of the tests was largely unstructured. While changing the representation of the tests was one obvious result, I also needed to recover the meaning of each test and categorise it; otherwise I would end up with .spec files that were several thousand lines long.

To do the actual migration I took inspiration from transpiler and compiler pipelines. The input format was well defined at a syntax level and the output format was well defined. I needed a pipeline that could transform one into the other.

I split the pipeline into small stages, with each stage solving a specific problem:

Workbook -> extraction -> content graph mining -> content sema -> journey sema -> journey mining -> emission.
// Note: sema = semantic analysis

The conversion scripts were written in TypeScript. I mainly used the standard library, with the notable exceptions of xlsx to read the file and ts-morph, which made the emission stage possible by allowing programmatic generation of TypeScript code.

A before and after

Here is one small test case before migration:

Worksheet: HomeAuthOfficeHours
Row: 87
Product: Home
Flow: Payments - Verify 'Missed payment' feature - Connect to customer specialist:No - Is Info Helped:No - Anything Else Help:No in auth journey
Auth: Y
Payment: ANNUAL
Mode: activation
MenuPath: Payments

Steps:
OPEN_HELP;
VERIFY:Hi, I’m your Virtual Assistant. I can help you with simple queries;
VERIFY:Please choose from the following menu.;
CLICK:Renewals;
VERIFY:Choose an option below, or type your question if you don’t see what you’re looking for.;
CLICK:Missed payment;
VERIFY:If you've missed a payment, you'll receive a letter and email from us confirming;
CLICK:No;
VERIFY:Was the information I provided helpful?;
CLICK:No;
VERIFY:Is there anything else I can help you with?;
CLICK:No;
VERIFY:Thank you for contacting us. Have a nice day.;
CLOSE_CHAT

After migration, the transpiler emitted a normal Playwright test that fitted the framework conventions:

test.beforeEach(async ({ brand, userPool, landingPage }) => {
  const user = await userPool.homeUser({ overrides: { brand: brand as HomeBrandNames } });
  await userPool.openLandingPageForCustomer({ customerNumber: user.customerNumber, landingPage });
});

test(
  "missed payment -> Self-service path -> Anything else: no -> Payments -> Helpful: no",
  {
    tag: [tag.e2e, tag.home, tag.payments, tag.magicLinkEntry],
    annotation: [
      { type: "code", description: "HomeAuthOfficeHours#87" },
      { type: "sourceCase", description: "HomeAuthOfficeHours#87" },
      { type: "sourceBacklink", description: "worksheet=HomeAuthOfficeHours | row=87" }
    ]
  },
  async ({ landingPage }) => {
    const chat = await landingPage.openChat();
    await chat.verify(common.introduction.greeting);
    await chat.verify(common.menu.prompts.menu);
    await chat.select(common.routing.options.renewals);
    await chat.verify(common.menu.prompts.chooseOption);
    await chat.select(payments.missedPayment.options.missedPayment);
    await chat.verify(payments.missedPayment.prompts.missedPaymentLetterAndEmail);
    await chat.flows.navigation.feedback.feedbackPrompt();
    await chat.close();
  }
);

The pipeline had to do this kind of conversion thousands of times. Raw workbook strings became shared content constants, repeated sequences became flow calls, user setup moved into fixtures, and the generated test still pointed back to the original worksheet row via annotations.

Extraction

Extraction solves the first problem: the workbook is too inconsistent to reason about safely. Before the pipeline can infer anything about the chatbot, it first needs a deterministic representation of the source material.

The pipeline pulls each row apart, expands keyword macros and normalises the parts that have drifted over time.

The normalisation step handles issues such as:

1. expanded macros inside the workbook
2. whitespace drift
3. casing differences
4. punctuation differences
5. smart quotes

After that, each workbook row becomes a test case object containing the source metadata together with an AST-like breakdown of the journey. In compiler terms, this is the intermediate representation (IR): a data structure that successive passes progressively enrich.

Content Graph Mining

At this point the pipeline can replay individual workbook rows, but it still has no understanding of how the content itself is reused across the suite. Here, "content" refers to the raw strings present in the workbook.

Content graph mining solves that problem by discovering relationships between content instances.

The pipeline de-duplicates chatbot prompts, options and user inputs before constructing a structural graph from the deduplicated content. Nodes represent prompts, options and inputs. Edges represent transitions between them. Paths show how those pieces of content are combined within workbook journeys. Heat maps highlight the most frequently reused areas of the graph.

The graph answers questions the raw workbook cannot:

1. Does this content appear at the start of journeys?
2. Is it a branch point?
3. Is it a merge point?
4. What usually comes before it?
5. What usually comes after it?
6. Which workbook rows and journey families use it?

Those relationships make later semantic analysis possible. A yes/no option is not meaningful by itself; its surrounding content usually determines whether it belongs to authentication, feedback, handoff acceptance or a return-to-menu question.

The graph also validates the earlier extraction work. If content appears in impossible locations, either the workbook contains inconsistent data or an earlier transformation has misunderstood it.

Content Semantic Analysis

The pipeline now understands how pieces of content relate to one another, but it still does not understand what that content represents.

Content semantic analysis solves that problem.

The pipeline converts the enriched graph into stable content tokens. In compiler terms, this is interning. Every occurrence of "Do you have an existing policy with us?" becomes a reference to the same stable content identity rather than another copy of the same string.

The pipeline derives token IDs from the canonical content key: keyword, content role and canonical value. Getting that identity right proves far more important than it first appears because every later stage depends on those references remaining stable.

The stage also classifies tokens using keyword shape, workbook metadata and graph context:

1. authentication prompts
2. common yes/no options
3. payment content
4. claims content
5. handoff messages
6. renewal content
7. policy information
8. service window messaging

The classifier attaches confidence and evidence to every decision. When confidence falls below an acceptable threshold, the pipeline marks the content for review instead of guessing.

By the end of this stage, the pipeline no longer carries workbook strings. It carries stable content identities enriched with semantic meaning.

Journey Semantic Analysis

Understanding individual pieces of content is still not enough. The pipeline now needs to understand what complete journeys represent.

Journey semantic analysis solves that problem by lifting the semantic information from individual content into complete journeys.

The pipeline classifies each journey using its token sequence together with workbook metadata and the relationships discovered during content mining. It derives product, entry point, service window, business topic, authentication route, handoff behaviour, generated tags and mutation profile.

This stage deliberately stops short of deciding how the final Playwright code should look. Instead, it produces the semantic facts that later stages depend on: product grouping, route shape, suite placement, tags and mutation behaviour.

By the end of this stage, the intermediate representation contains enough semantic information that later passes no longer need to infer behaviour from workbook data.

Journey Mining

By this point the pipeline understands what every journey represents, but the generated suite is still not optimised for long-term maintenance.

Journey mining solves that problem.

A naïve generator can emit perfectly valid Playwright tests while duplicating the same interaction sequences hundreds of times. If one of those commonly reused journeys changes, every generated copy has to change with it.

Instead, the pipeline searches the semantically classified journeys for behaviour that deserves to become a shared abstraction.

The stage answers questions such as:

1. Which paths genuinely repeat?
2. Which steps are simply common shell around the journey?
3. Which repeated sequences deserve helper methods?
4. Which repeated paths should become shared flows?
5. Which journeys require manual review?

The repeated corridor view highlights common routes through the classified journeys. Those corridors become candidates for shared flow methods.

The pipeline produces both the flow manifest and a triage artifact explaining every decision. Some candidates are too short. Some lack enough supporting journeys. Others are subsumed by stronger abstractions or introduce more indirection than value.

By the end of this stage, the intermediate representation no longer consists solely of token actions. Reusable behaviour has been promoted into shared flow references, leaving only journey-specific interactions inline.

Emission

The remaining problem is transforming the final intermediate representation into Playwright code that looks like an engineer wrote it rather than a generator.

Conceptually, emission solves that problem. Although the implementation emits some artifacts earlier than others, its responsibility remains the same: transform stable pipeline data into maintainable TypeScript.

So far the pipeline has only been using stable IDs. They are good for machines and terrible for humans to read. Some are also invalid TypeScript syntax.

Humanisation manifests solve that problem. They map stable content and flow identities onto readable runtime names.

The pipeline emits three outputs:

1. shared content constants
2. reusable chat flows
3. Playwright specifications

The content emitter generates shared TypeScript content modules. The flow emitter generates reusable chat flow classes. Finally, the journey emitter combines those abstractions with the remaining inline interactions to produce Playwright tests.

Journey semantic analysis has already solved the difficult architectural questions. The emitter simply applies those decisions. It selects the correct fixtures, startup profile, service-window preset, tags, annotations and generated users before rendering the remaining journey.

I also wanted the generated output to remain reviewable. The emitter preserves file ordering wherever possible by reading the existing generated output before writing updates.

Several passes exist solely to detect the pipeline's own mistakes: graph validation, classification drift checks, flow triage and journey quality audits. That matters because a faulty migration pipeline can generate code that looks perfectly reasonable while being fundamentally wrong.

By the end of the pipeline, workbook rows have become maintainable Playwright code consisting of shared content constants, reusable flow abstractions, generated specifications and traceability back to the original source rows.

Benefits of the new framework

The transpiler was created so the suite could use capabilities the old framework could not support cleanly.

1. Parallel Playwright workers. The generated suite could run across multiple Playwright workers, which mattered once it stopped being a single workbook-driven execution path.
2. A reusable user pool. Generated specs could ask for userPool.homeUser(...) or userPool.motorUser(...) instead of carrying policy creation details in every test. Where a journey mutated policy state, the fixture could provide a fresh user without changing the test body.
3. Websocket-backed chat timing and verification. The chat helpers listened to the underlying AWS Connect websocket frames, so chat.verify(...) could check transcript messages as well as the visible UI. Actions could wait for customer turns and settled chat activity instead of relying on blind sleeps.

Results and lessons learned

Before the migration, tests ran sequentially. Each one took between one and two minutes to execute. At that rate, running the full suite of 4,000 tests would have taken days. With ten Playwright workers the same suite averaged three seconds per test. Failures produced traces, so errors that had previously been hard to explain had evidence.

Adding new tests became routine instead of a manual translation exercise.

Treating the migration like a compiler problem made it much easier to debug. Once I had a source model, intermediate artifacts and an emitter, I could debug each step in isolation. When the generated code looked wrong, I rarely had to debug the Playwright test itself. I debugged the pipeline stage that produced it.

Generating TypeScript was easier than categorising ambiguous data from context. Most of the work went into working out what each piece of chatbot content meant, where journeys belonged, and which repeated paths were real abstractions rather than accidental duplication.

This migration started from a workbook, so a lot of the work was recovering structure. A more conventional framework migration should have less of that problem. For example, a Selenium suite already has a source-language AST, which means the pipeline can focus on mapping Selenium concepts into an explicit intermediate representation before emitting the intended Playwright architecture.

Comments

[Frank]

How did you handle async/await conversion in the transpiler, was it a straightforward process? I'm following your work for more insights on large-scale test migrations.