How I Cut Test Automation Runtime by 70% Using a Custom WebDriver Pool

kamini mangal — Wed, 05 Aug 2026 12:15:24 +0000

When I joined PCLnXAI as a Technical Product Manager, one of my first big builds was AutoFlow — an enterprise UI test automation platform for Oracle Fusion Cloud, covering 25+ automated end-to-end flows across modules like Accounts Payable, Accounts Receivable, General Ledger, Procurement, and Fixed Assets.

The stack: Python, Flask, and Selenium WebDriver.

Here's the problem we ran into, and how we solved it.

The problem: cold starts were killing us

Every time a test flow spun up a browser session, we paid a real cost:

Launching a fresh Edge WebDriver instance
Waiting for it to fully initialize
Only then starting the actual test

Multiply that by dozens of concurrent flows, and cold-start overhead alone was eating 2–5 seconds per test, before any real work happened.

Fix #1: A pre-warmed DriverPool

Instead of spinning up a new WebDriver instance per test, I built a custom DriverPool that keeps a set of pre-warmed Edge WebDriver instances ready to go. When a test needs a browser, it borrows one from the pool instead of paying the cold-start cost every time.

Combined with ThreadPoolExecutor-based request parallelism, this let multiple flows run concurrently without duplicating startup overhead.

Fix #2: Batched log flushing

Writing logs line-by-line during high-concurrency test runs created I/O contention. Batching log writes instead of flushing on every line reduced that overhead significantly, especially under concurrent load.

Fix #3: Real-time log streaming without polling

We wanted the UI to show live execution logs as tests ran — not logs that show up only after the test finishes. The naive approach is polling, but that's wasteful and laggy.

Instead, I built a low-latency log streaming service using WebSockets (Flask-Sock) and Gevent greenlets for non-blocking async I/O. Logs get fanned out to the UI per execution_id, in real time, without a single polling request.

Fix #4: Scheduling with APScheduler

To run flows on a schedule (nightly regression, pre-cutover validation, etc.) without a separate cron infrastructure, I used APScheduler directly inside the Flask app — simple, in-process, and easy to manage alongside the rest of the platform.

The result

Together, these changes cut per-test cold-start time by 2–5 seconds and reduced overall pipeline runtime by roughly 70% across concurrent workflows.

What I'd explore next

Moving the DriverPool to support horizontal scaling across multiple machines
Adding circuit-breaker logic for flaky flows instead of blind retries
Exporing headless execution where UI verification isn't strictly required

If you're building test automation infrastructure and have hit similar cold-start or concurrency bottlenecks, I'd love to hear how you approached it. 👇

DEV Community: kamini mangal