I'm Nikolay Advolodkin from UltimateQA, and in this article I unpack a two-year journey I led alongside our automation engineer Oles Nikaniuk, on a massive European enterprise migration. If you watched the episode, you already know the highlights; if not, this post compresses everything—decisions, processes, pitfalls, and the exact engineering moves that produced measurable Automation Success across UI, API, and integration testing.
💌 If you're interested in more JavaScript testing tips, click here to subscribe to our newsletter.
Table of Contents
- Project overview 🧭
- Choosing the right tools 🛠️
- Building the API testing strategy 🔐
- CI/CD and parallelization ⚡
- Schema management: RAML → OpenAPI → Types 🔁
- UI testing: challenges & lessons 🖥️
- Mobile testing note 📱
- ROI and metrics 📊
- Developer experience and static code analysis 🧩
- Lessons learned & culture 🤝
- How to replicate this — a step-by-step plan 🛣️
- FAQ ❓
- Conclusion ✅
Project overview 🧭
We supported three major areas inside the customer’s ecosystem: a supply-chain/logistics application (One Network), an internal recommendations platform built on a low-code environment (Appian), and a central integration layer built on Mulesoft that ties HR, finance, legal, transportation and other core systems together. From the outset, the project was a multi-year migration from legacy internal systems to modern stacks. Achieving Automation Success in a landscape like that required a different approach for each domain.
The supply chain product had long, manual end-to-end workflows that made test setup painfully slow. A single manual reproduction of a workflow could take me nearly an hour on the first try, and even after familiarity it was still often 5–10 minutes. Multiply that by dozens of scenarios and you see why teams were frustrated and why manual testing was unsustainable. In this environment Automation Success wasn't optional—it was mandatory.
The recommendations platform added complexity because many flows involved multiple users and concurrent browser sessions. That forced us to solve test orchestration, user state, and credential management in an automated way. Finally, the integration (Mulesoft) work required a focused API testing strategy: functional, contract/schema testing, and security tests (including fuzzing, header combinations, JWT handling, and policy enforcement checks). The combination of UI and API work led us to define a robust testing strategy that prioritized quick feedback, maintainability, and developer collaboration—three fundamentals to achieving Automation Success.
Choosing the right tools 🛠️
Picking tooling was one of our most consequential early decisions. We started with an objective evaluation rather than choosing tools we happened to know. The organization was primarily a .NET shop so Selenium with C# was the expected choice. We also evaluated Playwright (TypeScript and C#), Cypress (TypeScript), and Selenium (C#) in a PC-style comparison against criteria like testing capabilities, reporting, CI compatibility, community support, and speed.
I’ll be blunt: engineers frequently pick the tool they already know and shoehorn it into every problem. That approach kills Automation Success. Instead, we scored tools against the project requirements. Playwright with TypeScript won for our environment because it handled cross-domain navigation (important for Salesforce-like flows), supported parallel workers and sharding for fast CI runs, and offered a unified test runner that worked well for both UI and API tests. Cypress had strengths, and Selenium still has a place—especially when enterprises need a “one tool to rule everything” approach that ties into Appium for mobile/desktop coverage—but Playwright fit our needs best.
Playwright C# existed, but many features are more mature in the TypeScript implementation. Choosing a language is not just personal preference—it's about community, plugins, and long-term viability. We deliberately accepted a learning curve to achieve the Automation Success we wanted.
Building the API testing strategy 🔐
Because Mulesoft acted as the central integration platform, API testing became our primary focus. We designed a testing approach that included:
- Functional tests for endpoints (parameters, responses, error paths)
- Contract/schema validation (living OpenAPI-driven tests)
- Security tests: missing/invalid headers, authentication checks, JWT forging, fuzzing
- Boundary tests and negative cases (malformed payloads, rate limits)
One early, practical win was catching integration policy mistakes. For example, one API should reject requests without client ID and secret—this policy was occasionally omitted. Automated security checks uncovered such lapses quickly, preventing potentially exploitable behavior from reaching production. That is tangible Automation Success: preventing real-world risks through automated verification.
We separated functional test runs from security test runs to keep fast feedback loops during development while still running thorough security suites as part of every build. Over time this matured into nearly 1,000 API tests that run quickly in CI.
CI/CD and parallelization ⚡
Fast, reliable CI is the engine of Automation Success. Our CI/CD lived in Azure DevOps. To get from hundreds of tests to sub-5-minute runs we used three techniques together:
- Playwright sharding (CLI shard indexes)
- Azure DevOps job matrix (multiple instances of the same job)
- Playwright worker processes inside each job
Here's a simplified example of what we did: create two CI jobs in the pipeline, each with a job matrix producing multiple instances. Each job runs Playwright with shard indexes so tests partition across instances. Inside each job we enabled multiple Playwright worker processes. In one realistic configuration we ended up with two jobs, each spinning up two workers and each worker running two shards—yielding multiple parallel consumers that execute independent subsets of tests. The end result: an 846-test suite executing in roughly 4 minutes—about 0.3 seconds per test on average for our API-focused checks.
That kind of performance unlocked real, measurable Automation Success. It meant developers received fast feedback on commits, builds either succeeded quickly or failed fast, and our testing suite became part of everyday development rather than an afterthought.
Schema management: RAML → OpenAPI → Types 🔁
Managing API schemas was one of the most time-consuming and impactful parts of the project. Mulesoft uses RAML for documentation. The native conversion from RAML to Swagger v2 existed, but Swagger v2 is outdated and many modern tools expect OpenAPI v3. Additionally, the intermediate Swagger 2 files Mulesoft produced sometimes contained custom properties that violated the spec, which broke automated conversion tools.
Our transformation pipeline looked like this:
- Convert RAML → Swagger v2 (generated by Mulesoft)
- Fix non-standard custom properties injected by Mulesoft (custom transformation code)
- Convert Swagger v2 → OpenAPI v3
- Generate JSON Schema artifacts from OpenAPI v3
- Generate TypeScript types and interfaces from JSON Schema
Generating TypeScript types allowed us to import response and request types directly into our tests and API clients. When a schema changed, TypeScript would flag mismatches in the tests at compile time—forcing an immediate, visible update. This reduced the maintenance burden and increased the accuracy of our contract validations. In short: putting schemas at the center enabled Automation Success by making tests resilient to change and keeping test code synchronized with API contracts.
We considered AI to help during the transformation. AI assisted in designing the pipeline and resolving tricky transformation edge cases, but the core transformation needed deterministic tooling and custom code to handle Mulesoft’s non-standard properties. The extra investment paid off: the living documentation approach drastically reduced schema-related bugs between integrations.
UI testing: challenges & lessons 🖥️
UI automation for supply chain and the Appian platform had unique challenges. Workflows were long and brittle, required heavy test data setup, and involved multi-role scenarios where one user’s actions needed to be visible to another user in the same scenario. Creating and maintaining end-to-end UI tests was harder and delivered less ROI than API tests.
Key issues we solved in UI automation:
- Managing multiple sessions and credentials for role-based flows
- Stabilizing selectors in a low-code environment that frequently changed DOM structures
- Speed: UI tests are slower and more fragile, so we limited the number of full end-to-end UI tests
Playwright handled cross-domain and session management well, which is why it beat alternatives like Cypress for some of our use cases. However, UI automation’s overall cost-benefit ratio was lower than API testing—so we focused UI automation on the highest value end-to-end paths (happy paths and critical flows) and relied on robust API testing for the majority of verification logic. That combination delivered balanced Automation Success.
Mobile testing note 📱
A short but important note: Playwright is not a replacement for native mobile testing. If you need deep native mobile compatibility, use the native frameworks (XCUITest for iOS and Espresso for Android). Native test frameworks are fast, well-supported by device vendors, and provide the most reliable results for mobile behavior. Appium and similar tools have their place, especially when you need cross-platform coverage, but for the best performance and value we recommend native frameworks for mobile automation.
Mixing web automation and mobile automation into one “golden hammer” rarely produces the best Automation Success. Choose the right tool for the job.
ROI and metrics 📊
Numbers tell the story better than words. Starting from zero, we implemented security and functional pipelines and grew the automation suite to the following:
- 571 security tests (from 0)
- ~850 total automated tests (at times approaching 1,000)
- 846 tests executing in about 4 minutes
- ~56 hours saved per release (assuming conservative per-test manual execution times)
- Over 100 production bugs prevented from leaking through (tracked via metrics)
We measured time saved, bug prevention, and deployment velocity improvements to quantify Automation Success. Because the API suite runs as part of each build and deployment, developers get immediate feedback. Security regressions that would previously require manual verification are now caught automatically, and schema mismatches show up before they can break downstream consumers.
Developer experience and static code analysis 🧩
Automation Success wasn't just about tests—it also involved improving the developer experience. We built custom static code analysis checks tailored to the Mulesoft environment and team best practices. These checks run in the pipeline and enforce standards (we had roughly seven sets of important rules). If a pull request violates the rules, the pipeline shows a red status and the PR cannot be merged until issues are fixed.
That enforcement had two effects:
- It improved long-term code quality and prevented anti-patterns that would create test flakiness.
- It sparked collaboration. Developers started asking questions, raising counter-proposals, and working with us to adjust the rules where appropriate. Static analysis became a conversation starter, not just a blocker.
Being the “automation person” turned from a lone gatekeeper role into a trusted technical partner for the development teams. That cultural shift is one of the hardest but most important parts of Automation Success.
Lessons learned & culture 🤝
I want to distill the practical lessons that guided our path to Automation Success. These are the behavioral and strategic takeaways that matter as much as technical choices.
Lesson 1 — Collaborate, don’t silo
“Without collaboration... you will reach nothing.” This quote became our mantra. Automation is a team sport. Tests succeed when developers, testers, product owners, and operations agree on goals and share responsibilities. Don’t try to be a solo hero.
Lesson 2 — Expand your scope responsibly
Don’t be afraid to go beyond your job description if you see a systemic issue you can fix—static analysis, schema pipelines, or CI improvements. Those investments create outsized value and make you a far more effective contributor.
Lesson 3 — Choose tools objectively
Throw away your biases. Evaluate tools against real requirements, not personal comfort. Picking the right tool may require team training, but the long-term returns are worth it.
Lesson 4 — Automate the things that matter
Prioritize API and integration tests where they provide the fastest feedback and highest value. Use UI tests for critical, high-impact flows and keep their number limited to reduce maintenance costs.
Lesson 5 — Make schemas the source of truth
Automated contract validation and generated types keep your tests synchronized with the API. Living docs and generated types reduce a category of bugs that are otherwise hard to find until integration time.
Lesson 6 — Fast feedback loops win
Optimize for speed in CI. Sharding, workers, and a smart pipeline architecture turn tests from a drag into a productive, quick feedback tool—key to Automation Success.
How to replicate this — a step-by-step plan 🛣️
If you want to reach Automation Success in a large, complex enterprise environment, follow these practical steps we used (abbreviated and actionable):
1. Inventory: Catalog systems, integrations, and critical flows. Identify the “integration core” (e.g., Mulesoft) and prioritize it.
2. Tool evaluation: Build an objective scoring matrix. Compare tools against real criteria—CI support, cross-domain support, community, performance, mobile needs, language support.
3. Start small: Implement API tests first for critical integrations. Keep security tests separate from fast functional tests initially.
4. Schema-first: Make API schemas the canonical source. Automate conversions (RAML/Swagger/OpenAPI) and generate types to catch mismatches early.
5. Pipeline design: Use CI job matrices, Playwright sharding (or equivalent), and worker processes. Aim for sub-10-minute full runs for your primary suites.
6. Static analysis: Create code checks tailored to your platform and enforce them in PRs to prevent anti-patterns.
7. Limit UI scope: Automate only the highest-value E2E flows. Rely on API checks for the bulk of validation.
8. Collaborate: Engage developers and stakeholders early. Use failing tests as conversation points, not blame.
9. Measure ROI: Track test coverage, execution time, bugs prevented, and time saved per release to quantify Automation Success.
10. Iterate: Revisit tooling and practices regularly—what's optimal now might change in two years.
FAQ ❓
Why did you choose Playwright with TypeScript?
Playwright with TypeScript offered the best combination of cross-domain support, test runner integration, sharding, and community maturity for our needs. It handled both API and UI workflows well and enabled rapid CI parallelization. Choosing a language with strong community support (TypeScript) also mattered for long-term maintainability and plugin support—key to Automation Success.
Could you have used a different stack and still reached Automation Success?
Possibly. But the point is not which tool is objectively best—it's to pick the one that fits your requirements and team and then commit. If API testing were the only concern, a purpose-built API tool might be more efficient. If mobile native tests are required, native frameworks are preferable. Automation Success comes from aligning tool choice with constraints and goals.
How did you keep UI tests stable in a changing low-code platform?
We limited the UI test surface area to high-value flows, used resilient selectors, built reusable page objects, and relied on API tests for most validation. We also collaborated closely with developers to stabilize parts of the UI that mattered for automation.
What about flaky tests—how did you control them?
Reduce flakiness by shrinking UI test scope, improving selectors, adding retries sparingly, and ensuring tests are deterministic. API tests were naturally less flaky and were the backbone of reliable CI. Fast feedback and clear logs from the Playwright test runner helped root-cause flakiness quickly.
Did you involve the development team in test ownership?
Yes. Static code analysis and living schemas made APIs and patterns clearer. Developers began collaborating—fixing failing contract tests, updating schemas, and engaging in design conversations. That shared responsibility drove Automation Success.
Any advice for teams with no TypeScript experience?
Evaluate the cost of training versus the long-term benefits. If your team is heavily invested in C#, you might pick the best-supported C# tool and invest in good practices. But if you want the Playwright feature set and community, training the team can pay off in better Automation Success later.
How important was reporting and logging?
Very important. We relied on Playwright’s built-in traceability, logs, and the test runner’s reporting. For API clients that circumvented the test runner, we lost logs and traceability—another reason to prefer a unified runner like Playwright for most of our tests.
Conclusion ✅
Automation Success is not a single technology choice or a single architectural tweak. It’s the result of a focused strategy: pick the right tools for the problem, invest in living documentation and schema-first pipelines, build fast CI, enforce quality through automated code checks, and—most importantly—collaborate with your team. Those are the pillars that helped us go from nothing to nearly a thousand automated tests, fast CI runs, over a hundred prevented production bugs, and significant time savings.
If you’re starting a similar journey, begin with an objective assessment of your needs, prioritize API-first automation, and design a CI pipeline that gives fast feedback. Invest in schema generation and type safety so your tests are maintainable and reliable. Lastly, be the person who builds bridges across teams.
Automation Success depends on people as much as on code!
Thank you! 😉
Top comments (0)