DEV Community: Hakan GÜL

Scaling to 100,000 RPS: Deploying Distributed Load Testing on Kubernetes with Helm and Docker

Hakan GÜL — Sun, 30 Aug 2026 07:41:47 +0000

TL;DR: Single-machine load tests inevitably hit CPU and socket descriptor limits when pushing beyond a few thousand requests per second. In LocustPilot, we packaged the entire control plane and distributed worker architecture into Kubernetes Helm charts (helm/locust), allowing engineering teams to scale to 100,000+ RPS across cloud nodes in minutes with automated resource isolation.

When a Single Machine Hits the Wall

A single modern laptop or cloud VM running Locust can comfortably generate between 2,000 and 5,000 requests per second (RPS).

However, when preparing for enterprise-scale traffic events (e.g. streaming sports events, flash sales, or banking cutovers), you need tens of thousands of concurrent users. Attempting this on one machine leads to misleading test results:

CPU Saturation: Python processes hit 100% CPU on single cores, artificially increasing response times on the client side.
Socket Exhaustion (TIME_WAIT): The operating system runs out of ephemeral ports, causing false Connection Refused errors that have nothing to do with your backend.
Bandwidth Throttling: Cloud VM network interfaces bottleneck before your target servers are even stressed.

To scale reliably, you must distribute load generation across a cluster of worker nodes.

The Distributed Master-Worker Architecture

In distributed mode, Locust separates responsibilities across two roles:

Master Node: Runs the LocustPilot web control center, coordinates worker connections, aggregates incoming metrics, and streams telemetry to ReportPortal. It generates zero HTTP traffic itself.
Worker Nodes: Connect to the master, spawn virtual users, and bombard the target backend with requests.

       ┌───────────────────────────────┐
       │   LocustPilot Master (UI)     │
       │   (Streamlit + ReportPortal)  │
       └──────────────┬────────────────┘
                      │ Port 5557 (ZMQ)
         ┌────────────┼────────────┐
         │            │            │
   ┌─────▼────┐ ┌─────▼────┐ ┌─────▼────┐
   │ Worker 1 │ │ Worker 2 │ │ Worker N │
   └─────┬────┘ └─────┬────┘ └─────┬────┘
         │            │            │
         └────────────┼────────────┘
                      ▼
         [ Target Microservices ]

Packaging for Kubernetes with Helm (`helm/locust`)

LocustPilot provides a production-ready Helm chart located in helm/locust/.

1. The Optimized Container (`k8s.Dockerfile`)

FROM python:3.10-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl build-essential && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8501 5557
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Photo by Fotis Fotopoulos on Unsplash

2. Deploying the Cluster with One Command

Using the pre-built Helm chart, deploying the entire distributed load testing cluster with ReportPortal telemetry requires a single command:

helm upgrade --install locust ./helm/locust \
  --namespace loadtest \
  --create-namespace \
  --set image.repository=my-registry.io/locust-pilot \
  --set image.tag=latest \
  --set replicaCount=10 \
  --set extraEnv[0].name=RP_TOKEN --set extraEnv[0].value=$RP_TOKEN \
  --set extraEnv[1].name=RP_ENDPOINT --set extraEnv[1].value=$RP_ENDPOINT \
  --set extraEnv[2].name=RP_PROJECT --set extraEnv[2].value=$RP_PROJECT

3. Tuning OS Limits for Massive Scale

When running high-volume workers inside Kubernetes, ensure your Pod security contexts allow high file descriptors:

# helm/locust/values.yaml
worker:
  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 1000m
      memory: 1Gi
  sysctls:
    - name: net.ipv4.tcp_tw_reuse
      value: "1"
    - name: net.core.somaxconn
      value: "65535"

What Is Next?

In the final installment (Part 5), we will connect all the pieces into your continuous delivery pipeline: Shift-Left Performance Testing with Automated Quality Gates in GitHub Actions and Bitbucket Pipelines.

👉 Explore the Helm Charts on GitHub

FAQ

How many workers are needed for 50,000 RPS?

Depending on endpoint payload size and SSL overhead, each CPU core running FastHttpUser can typically generate 1,500–3,000 RPS. A cluster of 20–25 worker pods can comfortably sustain 50,000+ RPS.

Can workers be dynamically auto-scaled?

Yes. Kubernetes Horizontal Pod Autoscaler (HPA) can scale worker pods based on CPU utilization or custom test duration parameters.

Enterprise Observability: Real-Time ReportPortal Integration with Gevent and Percentile Tracking

Hakan GÜL — Sun, 30 Aug 2026 07:41:12 +0000

TL;DR: High-throughput load tests generating thousands of requests per second can easily overwhelm reporting backends with log spam. In LocustPilot, we solved this by implementing an asynchronous gevent listener for ReportPortal that deduplicates error cascades, streams live endpoint metrics every 30 seconds, and calculates comprehensive response time percentiles (P50 to P99) with automated quality gate verdicts.

The Log Flood Dilemma in Load Testing

In standard functional UI or API tests, every failure is unique and warrants a detailed log entry.

In performance testing, the dynamic is completely inverted:

When an endpoint begins to degrade under 5,000 RPS, it does not fail once—it fails 50,000 times with the exact same 504 Gateway Timeout or Connection Refused error.
If your test framework attempts to log every single failure to a centralized telemetry platform like ReportPortal, you will DDoS your own reporting database, exhaust memory, and freeze the test execution.

To provide enterprise-grade observability without degrading load test performance, we built ReportPortalListener (app/core/rp_listener.py) in LocustPilot.

1. Background Telemetry with Gevent Greenlets

Locust relies on gevent for lightweight, non-blocking coroutines. Rather than blocking worker execution during network logging, LocustPilot spawns a background greenlet (_periodic_stats_logger) that flushes aggregated endpoint statistics every 30 seconds:

# app/core/rp_listener.py
import gevent
from reportportal_client.helpers import timestamp

def on_test_start(self, **kwargs):
    # Initialize ReportPortal Launch and Test Item
    self.launch_uuid = self.rp_client.start_launch(
        name=self.rp_launch_name,
        start_time=timestamp(),
        description=self.rp_launch_desc,
    )
    self.test_item_uuid = self.rp_client.start_test_item(
        name="Load Test Execution",
        start_time=timestamp(),
        item_type="STEP",
    )

    # Spawn background stats publisher
    self.running = True
    gevent.spawn(self._periodic_stats_logger)

def _periodic_stats_logger(self):
    """Log live endpoint stats every 30 seconds without blocking load workers"""
    while self.running:
        gevent.sleep(30)
        if self.running and self.rp_client and self.test_item_uuid:
            lines = ["📊 Live Endpoint Telemetry", "=" * 80]
            lines.append(f"{'Method':<8} {'Name':<40} {'Reqs':<8} {'Fails':<8} {'Avg(ms)':<10} {'RPS':<8}")
            lines.append("-" * 80)

            for stat in self.env.stats.entries.values():
                if stat.num_requests > 0:
                    lines.append(
                        f"{stat.method:<8} {stat.name[:40]:<40} "
                        f"{stat.num_requests:<8} {stat.num_failures:<8} "
                        f"{stat.avg_response_time:<10.0f} {stat.current_rps:<8.1f}"
                    )

            self.rp_client.log(
                time=timestamp(),
                message="\n".join(lines),
                level="INFO",
                item_id=self.test_item_uuid,
            )

This ensures team leads and developers watching the ReportPortal dashboard receive live health updates without injecting latency into the simulated users.

2. Smart Error Deduplication (Unique Error Logging)

When a microservice falters, LocustPilot captures the first instance of an error to preserve diagnostic context, while silently aggregating subsequent identical failures into a counter:

def on_request(self, request_type, name, response_time, response_length, exception=None, response=None, **kwargs):
    has_exception = exception is not None
    has_manual_failure = response is not None and getattr(response, "_manual_result", None) is False

    if has_exception or has_manual_failure:
        error_key = f"{request_type}:{name}"

        # Initialize counter on first occurrence
        if error_key not in self.last_stats:
            self.last_stats[error_key] = {"count": 0, "logged": False}

        self.last_stats[error_key]["count"] += 1

        # Only send the first failure payload to ReportPortal
        if not self.last_stats[error_key]["logged"] and self.rp_client:
            error_msg = str(exception) if has_exception else getattr(response, "_manual_result_msg", "Manual failure")
            self.rp_client.log(
                time=timestamp(),
                message=f"❌ First Failure Detected: {request_type} {name}\nDetails: {error_msg}",
                level="ERROR",
                item_id=self.test_item_uuid,
            )
            self.last_stats[error_key]["logged"] = True

Result: Zero log spam. If 100,000 requests fail with the same connection error, ReportPortal receives exactly 1 clean diagnostic entry with a summary count at the end of the test.

Photo by Mika Baumeister on Unsplash

3. Comprehensive Percentiles and Automated Quality Gates

Average response time is a dangerous vanity metric in performance engineering; a 200ms average can hide a 99th percentile spike of 12 seconds.

When a test completes (on_test_stop), LocustPilot computes full percentile distributions and enforces automated quality gates:

def on_test_stop(self, **kwargs):
    total = self.env.stats.total

    # 1. Percentile Distribution Table (50% to 100%)
    for stat in self.env.stats.entries.values():
        if stat.num_requests > 0:
            p50 = stat.get_response_time_percentile(0.50)
            p90 = stat.get_response_time_percentile(0.90)
            p95 = stat.get_response_time_percentile(0.95)
            p99 = stat.get_response_time_percentile(0.99)
            max_rt = stat.max_response_time

    # 2. Automated Quality Gate Rules
    has_issue = False
    if total.avg_response_time > 200:
        log_warning(f"⚠️ Average response time exceeded 200ms! (Actual: {total.avg_response_time:.2f} ms)")
        has_issue = True
    if total.fail_ratio > 0.05:
        log_warning(f"⚠️ Failed request ratio exceeded 5%! (Actual: {total.fail_ratio:.2%})")
        has_issue = True

    # 3. Final Build Verdict
    success_rate = (1 - total.fail_ratio) * 100
    status = "PASSED" if success_rate >= 75 and not has_issue else "FAILED"

    self.rp_client.finish_test_item(item_id=self.test_item_uuid, end_time=timestamp(), status=status)
    self.rp_client.finish_launch(end_time=timestamp())

What Is Next?

In Part 4, we move from single-instance tests to large-scale distributed load: deploying LocustPilot on Kubernetes using Helm charts and Docker worker nodes to simulate over 100,000 requests per second.

👉 Check out the ReportPortal Listener on GitHub

FAQ

Why is ReportPortal preferred over standard HTML reports?

HTML reports are static and isolated on developer machines. ReportPortal centralizes performance history across sprints, enables historical trend comparisons, and automatically flags regression anomalies.

Can custom quality gate thresholds be configured?

Yes. Quality thresholds (such as max average latency or acceptable failure ratio) can be configured via .env variables or custom BaseLocustUser validators.

Under the Hood: Building a Dynamic AST Test Runner and Subprocess Engine in Python

Hakan GÜL — Sun, 30 Aug 2026 07:40:38 +0000

TL;DR: Dynamically discovering test classes at runtime by executing importlib is dangerous because module-level code runs immediately. In LocustPilot, we solved this by building an Abstract Syntax Tree (AST) static scanner that identifies HttpUser and FastHttpUser classes without executing Python code, paired with an unbuffered subprocess.Popen engine for real-time log streaming.

The Hidden Trap of Dynamic Python Discovery

When building a web control plane for load testing, the first requirement is simple: automatically discover and list all test files in the UI dropdown.

Most developers jump straight to dynamic importing:

# ❌ The Naive Way: Dangerous runtime import
import importlib.util

spec = importlib.util.spec_from_file_location("module", file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # ⚠️ Executes all top-level code!

Why is this dangerous?

Module-level side effects: If a test file establishes database connections, reads external credentials, or executes setup logic at the module level, simply browsing the UI triggers those actions.
Crash propagation: A syntax error or missing optional dependency in one test file will crash the entire UI server.
High memory footprint: Importing dozens of heavy test modules pollutes the server process memory space.

We needed a way to inspect Python test files statically and safely before execution.

The Solution: Static Analysis via Python's AST

In LocustPilot (app/core/runner.py), we use Python's built-in ast (Abstract Syntax Tree) module. We parse the source code into a syntax tree and walk the nodes without executing a single line of code.

Here is the exact heuristic implemented in LocustPilot:

import ast
import re
from pathlib import Path

def _has_locust_user_class(path: Path) -> bool:
    """Heuristic: returns True if file defines a Locust User class.
    Detects subclasses of HttpUser/FastHttpUser via AST; falls back to regex.
    """
    try:
        src = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return False

    try:
        tree = ast.parse(src)
        for node in ast.walk(tree):
            if isinstance(node, ast.ClassDef):
                for base in node.bases:
                    # Matches class MyTest(HttpUser)
                    if isinstance(base, ast.Name) and base.id in {"HttpUser", "FastHttpUser"}:
                        return True
                    # Matches class MyTest(locust.HttpUser)
                    if isinstance(base, ast.Attribute) and base.attr in {"HttpUser", "FastHttpUser"}:
                        return True
    except Exception:
        pass

    # Fallback regex check for custom decorators
    return (
        re.search(r"class\s+\w+\(.*?(HttpUser|FastHttpUser).*?\)", src) is not None
        or "@task" in src
    )

Why This Approach Wins:

⚡ Ultra-Fast: Parsing 100 Python files takes less than 15 milliseconds.
🛡️ Zero Side-Effects: No network sockets, database pools, or environment lookups are triggered.
🔍 Smart Filtering: Automatically excludes __pycache__, __init__.py, and helper utility files unless they contain runnable user classes.

Photo by Kevin Ku on Unsplash

Extracting Target Hosts Without Hardcoding

In real-world projects, different test files target different microservices or staging environments. LocustPilot uses an intelligent fallback chain to determine the default target host:

def extract_locustfile_host(path: Path) -> Optional[str]:
    # 1. Environment variable override takes highest precedence
    env_host = os.getenv("LOCUST_TARGET_HOST")
    if env_host:
        return env_host

    # 2. Extract host attribute directly from test file
    try:
        text = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return None

    m = re.search(r"^\s*host\s*=\s*['\"]([^'\"]+)['\"]", text, flags=re.MULTILINE)
    if m:
        return m.group(1).strip()

    # 3. Fallback to centralized BaseUser target host
    if re.search(r"^\s*host\s*=\s*TARGET_HOST", text, flags=re.MULTILINE):
        base_user = BASE_DIR / "locustfiles" / "utils" / "base_user.py"
        if base_user.exists():
            base_text = base_user.read_text(encoding="utf-8", errors="ignore")
            m_target = re.search(r"TARGET_HOST\s*=\s*['\"]([^'\"]+)['\"]", base_text)
            if m_target:
                return m_target.group(1).strip()

    return None

This allows engineers to either set hosts per test file, use a shared BaseLocustUser, or dynamically override them via the Streamlit UI.

Subprocess Execution and Unbuffered Log Streaming

Running load tests inside the main web server process would freeze the UI. LocustPilot spawns an isolated subprocess.Popen running headless Locust while streaming logs in real time.

def run_locust(
    locustfile: Path,
    host: Optional[str],
    users: int,
    spawn_rate: float,
    run_time: str,
    run_dir: Path,
    csv_prefix: str = "stats",
    html_report: bool = True,
    csv_full_history: bool = True,
    stream_logs: bool = True,
) -> Tuple[subprocess.Popen, Path, Path, str, List[str]]:

    cmd = [
        "locust",
        "-f", str(locustfile),
        "--headless",
        "-u", str(users),
        "-r", str(spawn_rate),
        "--run-time", str(run_time),
        "--csv", str(run_dir / csv_prefix),
        "--logfile", str(run_dir / "locust.log"),
        "--only-summary",
    ]
    if host:
        cmd += ["--host", str(host)]
    if html_report:
        cmd += ["--html", str(run_dir / "report.html")]
    if csv_full_history:
        cmd += ["--csv-full-history"]

    # Dynamic PYTHONPATH injection so test libraries import cleanly
    env = os.environ.copy()
    py_paths = [str(LOCUSTFILES_DIR), str(LOCUSTFILES_DIR / "libs"), str(BASE_DIR)]
    env["PYTHONPATH"] = os.pathsep.join(py_paths)

    proc = subprocess.Popen(
        cmd,
        stdout=(subprocess.PIPE if stream_logs else subprocess.DEVNULL),
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1,  # Line-buffered output for instant streaming
        cwd=str(BASE_DIR),
        env=env,
    )
    return proc, logfile, html_path, start, cmd

Key Subprocess Design Choices:

bufsize=1: Enables line-buffered I/O. Standard Python output buffering waits for hundreds of bytes before flushing; bufsize=1 pushes log lines to Streamlit instantly.
PYTHONPATH Injection: Automatically adds libs/ and utils/ to Python path, allowing test scripts to use shared utilities without complex packaging.
Non-Blocking Polling: The Streamlit UI polls proc.stdout.readline() without blocking the event loop.

Architectural Summary

By decoupling test discovery (AST) from test execution (Subprocess), LocustPilot achieves:

Safety: Malicious or broken test scripts cannot crash the control plane.
Responsiveness: Heavy load generation runs in isolated processes without degrading web UI performance.
Clean Code: Test authors write standard Python without needing framework boilerplate.

What Is Next?

In Part 3, we will dive into Enterprise Observability: how LocustPilot leverages gevent greenlets to stream real-time P95/P99 latency percentiles and deduplicated error alerts to ReportPortal.

👉 Explore the LocustPilot Source Code on GitHub

FAQ

Why not use regex alone to find Locust classes?

Regex can produce false positives on comments or strings (e.g. inside docstrings). AST guarantees that we are inspecting actual Python class inheritance syntax.

Does `subprocess.Popen` limit load test throughput?

No. Headless Locust runs in its own OS process utilizing gevent coroutines. The Streamlit UI only reads lightweight summary lines, keeping CPU overhead below 1%.

Can I run tests with custom Python dependencies?

Yes. By configuring PYTHONPATH dynamically, any custom library inside locustfiles/libs/ or virtualenv is immediately accessible.

The Death of XML: Why We Built LocustPilot — A Modern Control Plane for Distributed Load Testing

Hakan GÜL — Sun, 30 Aug 2026 07:40:03 +0000

TL;DR: GUI-based load testing tools like JMeter create painful Git merge conflicts with massive XML files, while standard Locust lacks persistent run history and out-of-the-box team dashboards. LocustPilot bridges this gap by combining Python's code-as-test power with a modern Streamlit control center, dynamic test registries, ReportPortal observability, and instant Docker/Kubernetes deployment.

Why Is Traditional Performance Testing Broken?

Every test automation engineer who has prepared for a major traffic event (Black Friday, product launches, or high-volume campaigns) has faced the dreaded .jmx file conflict.

It is 6:00 PM. Two engineers update a JMeter test plan simultaneously. They commit their changes, and Git screams with a 4,000-line XML merge conflict. You cannot easily read it, you cannot safely merge it in a text editor, and one wrong closing tag corrupts the entire test suite.

For decades, performance testing was trapped in GUI-heavy, XML-driven tools. When Locust arrived, it revolutionized the industry by introducing Performance Testing as Code in pure Python.

Suddenly, your load tests were version-controlled, readable, and could leverage the entire Python ecosystem.

from locust import HttpUser, task, between

class QuickOrderUser(HttpUser):
    wait_time = between(1, 2)

    @task
    def checkout(self):
        self.client.post("/api/v1/orders", json={"item_id": 42, "quantity": 1})

What Was Missing in Vanilla Locust?

While Locust's core execution engine is lightweight and lightning-fast, scaling it across an enterprise engineering team reveals immediate operational friction:

No Out-of-the-Box Test Registry: You have to remember command-line paths (locust -f locustfiles/checkout_flow.py) or manage messy bash wrapper scripts.
Ephemeral Run History: The default Locust web UI is purely session-based. Once you stop the test or restart the container, all charts, response times, and failure logs vanish forever.
Disconnected Reporting: Sharing past test runs with backend developers requires manually downloading CSVs or hunting through local log files.
Lack of Enterprise Quality Gates: Teams need automated test thresholds (e.g., Fail if 99th percentile exceeds 250ms) and centralized observability dashboards like ReportPortal.

Photo by Glenn Carstens-Peters on Unsplash

Introducing LocustPilot: The Modern Control Plane

We built LocustPilot to turn standalone Locust scripts into a collaborative, production-grade load testing platform.

Instead of wrestling with CLI flags or losing past test metrics, LocustPilot provides an interactive Streamlit Command Center that automatically manages test discovery, execution, real-time log streaming, and persistent reporting.

Key Architectural Highlights:

🧪 Smart Test Discovery (AST Scanner): Automatically parses your test directory using Python's Abstract Syntax Tree (ast), detects all HttpUser / FastHttpUser classes, and exposes them in a clean dropdown menu.
📊 Persistent History & Plotly Charts: Every run is automatically archived with detailed latency percentiles (P50, P90, P95, P99), throughput graphs, and downloadable HTML/CSV bundles.
🔌 Native ReportPortal Integration: Streams live endpoint metrics and unique failure logs directly into ReportPortal with zero manual setup.
☁️ Cloud-Native Deployment: Ready for production with pre-configured Docker images and Kubernetes Helm charts (helm/locust).

Feature Matrix: How LocustPilot Compares

Capability	Apache JMeter	Vanilla Locust	LocustPilot
Test Definition	XML / Proprietary GUI	Pure Python Code	Pure Python Code
Version Control & Diffs	❌ Painful XML conflicts	✅ Clean Python Git diffs	✅ Clean Python Git diffs
Test Selection UI	❌ Manual file opening	❌ CLI arguments only	✅ Dropdown Test Registry
Run History	⚠️ Complex DB setups	❌ Session only (lost on stop)	✅ Persistent CSV/HTML/ZIP
Enterprise Telemetry	⚠️ Heavy plugins	❌ Basic web UI	✅ Built-in ReportPortal & Plotly
K8s & CI/CD Ready	❌ Heavy JVM overhead	⚠️ Manual worker setup	✅ Helm Chart & Docker Ready

How to Get Started in 30 Seconds

LocustPilot is completely open-source under the MIT license. You can launch the entire platform locally with a single Docker command:

# Clone the repository
git clone https://github.com/hakanngul/LocustPilot.git
cd LocustPilot

# Build and launch with Docker
docker build -t locust-pilot .
docker run -p 8501:8501 locust-pilot

Open your browser at http://localhost:8501, pick your target test from the dropdown, configure your user count, and hit Start Test.

What Is Next in This Series?

This is Part 1 of our 5-part LocustPilot deep dive. Over the next parts, we will explore:

Part 2: Under the hood of the dynamic AST Python test scanner and non-blocking log streamer.
Part 3: Enterprise observability: Streaming 99th percentile latencies to ReportPortal with Gevent.
Part 4: Scaling to 100,000 RPS on Kubernetes using Helm and distributed workers.
Part 5: Shift-Left performance testing: Enforcing automated build quality gates in CI/CD.

👉 Star and Explore LocustPilot on GitHub

FAQ

Why choose Python-based Locust over JMeter?

Python-based load testing allows developers and SDETs to treat performance tests as first-class software code. You can write reusable modules, use native Git diffs, debug directly in your IDE, and eliminate fragile XML files.

Does LocustPilot modify standard Locust scripts?

No. Any standard Locust script (HttpUser, FastHttpUser, @task) runs seamlessly in LocustPilot without requiring any code modifications or vendor lock-in.

Can I run LocustPilot in headless CI/CD pipelines?

Yes. LocustPilot supports both the interactive Streamlit UI for ad-hoc load runs and fully headless execution inside GitHub Actions, Bitbucket Pipelines, or Jenkins.

TestFly Part 8: Flakiness Quarantine Engine & Java SPI Extensibility

Hakan GÜL — Sun, 30 Aug 2026 07:39:29 +0000

Quarantine Engine & Java SPI Extensibility

The final pillar of TestFly is designed for long-term maintainability: Flakiness Quarantine and Java SPI Extensibility.

1. Flakiness Quarantine Engine (`testfly-quarantine.yml`)

Instead of deleting or commenting out flaky tests, quarantine them cleanly without blocking CI pipelines:

# testfly-quarantine.yml (Project Root)
quarantine:
  enabled: true
  tests:
    - className: io.testfly.examples.PaymentTest
      methodName: testStripeWebhook
      reason: "Waiting for third-party webhook fix (JIRA-4021)"
      owner: "@payment-team"

Quarantined tests are executed in CI, but their failures are flagged as QUARANTINED rather than failing the build.

2. Java SPI Extension Points

Extend TestFly without forking or modifying framework internals:

Extension Point	Interface	Registration
Custom Driver Provider	`NamedDriverProvider`	`DriverProviderRegistry.register()` or `META-INF/services`
Report Adapter	`ReportAdapter`	`ReportAdapterRegistry.register()` or SPI
Lifecycle Hook	`ExecutionHook`	`HookRegistry.register()` or SPI
Full Plugin	`TestFlyPlugin`	`PluginRegistry.register()` or SPI

Example: Custom Slack Report Adapter

package io.testfly.examples.plugins;

import io.testfly.reporting.ReportAdapter;
import io.testfly.metrics.ExecutionMetrics;

public class SlackReportAdapter implements ReportAdapter {

    @Override
    public String getName() {
        return "slack";
    }

    @Override
    public void onSuiteFinish(ExecutionMetrics metrics) {
        // Send Slack webhook with pass rate and failed test summaries
    }
}

Summary of the TestFly Masterclass

Across all 8 parts, we explored:

Part 1: Zero-boilerplate setup with BaseTest and testfly.yml
Part 2: Semantic locators (getByRole), BasePage, and WaitEngine
Part 3: CI/CD quality gates (BuildThresholdEnforcer) and parallel execution
Part 4: AI test generation and self-healing with testfly-mcp
Part 5: BDD with Cucumber 7, BaseCucumberSteps, and @retryable
Part 6: Built-in ApiClient, BaseApiTest, and hybrid workflows
Part 7: Bundled axe-core accessibility (accessibility()) and VisualAssert
Part 8: Flakiness quarantine engine and Java SPI extensibility

👉 Explore TestFly on GitHub and build reliable test suites today!

TestFly Part 7: Bundled axe-core Accessibility Assertions & VisualAssert Pixel Matching

Hakan GÜL — Sun, 30 Aug 2026 07:38:56 +0000

Accessibility & Visual Regression in TestFly

TestFly bundles axe-core 4.10.2 and a dedicated pixel-comparison engine VisualAssert directly into the core JAR.

1. Bundled axe-core Accessibility Scans (`accessibility()`)

In BaseTest, run a WCAG scan in a single line without external dependencies:

package io.testfly.examples.a11y;

import io.testfly.accessibility.Impact;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class AccessibilityTest extends BaseTest {

    @Test
    public void checkoutPassesWcagAA() {
        open("/checkout");

        accessibility()
            .withTags("wcag2a", "wcag21aa", "wcag22aa")
            .withLevel(Impact.SERIOUS) // fails on SERIOUS or CRITICAL
            .excluding("#third-party-widget")
            .run();
    }
}

2. Pixel-Perfect Visual Regression (`VisualAssert`)

package io.testfly.examples.visual;

import io.testfly.test.BaseTest;
import io.testfly.visual.VisualAssert;
import io.testfly.visual.VisualTolerance;
import org.openqa.selenium.By;
import org.testng.annotations.Test;

public class VisualRegressionTest extends BaseTest {

    @Test
    public void homepageVisualMatchesBaseline() {
        open("/");

        // Full-page comparison with 1% tolerance
        VisualAssert.assertScreenshot("homepage", VisualTolerance.of(1));

        // Element-scoped comparison
        VisualAssert.assertScreenshot("login-box", By.id("login-card"));
    }
}

First Run: Baseline screenshot is generated automatically.
Subsequent Runs: Compares against baseline; diffs saved to target/visual-diffs/.
Update Baselines: Run with -DupdateBaselines=true.

TestFly Part 6: Built-in ApiClient, BaseApiTest & Fast Hybrid UI + API Flows

Hakan GÜL — Sun, 30 Aug 2026 07:38:22 +0000

Native API Testing in TestFly

TestFly includes a high-performance, fluent HTTP client out of the box—no third-party dependencies required.

1. Pure API Tests with `BaseApiTest`

Extend BaseApiTest for API tests without launching a browser:

package io.testfly.examples.api;

import io.testfly.test.BaseApiTest;
import io.testfly.client.ApiClient;
import io.testfly.client.ApiResponse;
import org.testng.annotations.Test;

public class UserApiTest extends BaseApiTest {

    @Test
    public void getUserAndValidateSchema() {
        ApiResponse res = ApiClient.get("/api/users/1")
            .header("Accept", "application/json")
            .send()
            .assertStatus(200)
            .assertJson("$.name", "Hakan Gül")
            .assertSchema("schemas/user.json");
    }
}

2. Hybrid UI + API Tests via `apiClient()`

In BaseTest, use apiClient() to seed data via API before opening the browser:

package io.testfly.examples.tests;

import io.testfly.test.BaseTest;
import io.testfly.client.ApiResponse;
import org.openqa.selenium.By;
import org.testng.Assert;
import org.testng.annotations.Test;
import java.util.Map;

public class OrderCheckoutTest extends BaseTest {

    @Test
    public void fastOrderVerification() {
        // 1. Create order instantly via API (100ms)
        ApiResponse order = apiClient().post("/api/orders")
            .body(Map.of("itemId", "SKU-100", "quantity", 1))
            .send()
            .assertStatus(201);

        String orderId = order.json("$.orderId");

        // 2. Open browser directly to the order page
        open("/orders/" + orderId);
        Assert.assertEquals(getText(By.id("order-status")), "Confirmed");
    }
}

Every API request is automatically logged to the TestFly step timeline!

TestFly Part 5: BDD Testing with Cucumber 7, BaseCucumberSteps & @retryable Tags

Hakan GÜL — Sun, 30 Aug 2026 07:37:48 +0000

BDD with Cucumber 7 in TestFly

TestFly integrates with Cucumber 7 out of the box with zero boilerplate:

Automatic Driver Lifecycle: Managed via CucumberHooks in io.testfly.cucumber.
Step Timeline Logging: Streamed directly into the HTML report via CucumberStepLogger.
Automatic Screenshots: Captured and embedded on failure in both reports.
Per-Scenario Retry: Configured with @retryable or @retryable=N tags.

1. Runner Class (`BaseCucumberTest`)

package com.yourcompany.bdd;

import io.cucumber.testng.CucumberOptions;
import io.testfly.cucumber.BaseCucumberTest;

@CucumberOptions(
    features = "src/test/resources/features",
    glue     = {"com.yourcompany.bdd.steps", "io.testfly.cucumber"},
    plugin   = {"pretty", "io.testfly.cucumber.CucumberStepLogger"}
)
public class CucumberRunner extends BaseCucumberTest {}

2. Step Definitions with `BaseCucumberSteps`

package com.yourcompany.bdd.steps;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.When;
import io.cucumber.java.en.Then;
import io.testfly.cucumber.BaseCucumberSteps;
import org.openqa.selenium.By;

public class LoginSteps extends BaseCucumberSteps {

    @Given("the user is on the login page")
    public void onLoginPage() {
        open("/login");
    }

    @When("they login as {string} with password {string}")
    public void login(String username, String password) {
        find(By.id("username")).type(username);
        find(By.id("password")).type(password);
        find(By.id("submit")).click();
    }

    @Then("the dashboard is visible")
    public void dashboardVisible() {
        assertThat(By.id("dashboard")).isVisible(); // auto-retrying assertion
    }
}

3. Per-Scenario Retry Tag

@retryable=2
Scenario: Flaky payment gateway widget
  Given the user is on the payment screen
  When they submit payment details
  Then the transaction should be confirmed

Shift-Left Performance: Automated Load Testing Quality Gates in GitHub Actions and Bitbucket CI

Hakan GÜL — Sun, 30 Aug 2026 07:37:15 +0000

TL;DR: Performance testing should never be a once-a-quarter panic event before major production releases. In the final installment of the LocustPilot series, we demonstrate how to shift performance testing left by embedding automated load tests into GitHub Actions, Bitbucket Pipelines, and Jenkins with strict P99 latency quality gates that break builds on performance regressions.

Why Performance Testing Must Shift Left

In traditional software organizations, performance testing happens too late:

A team develops features for three months.
One week before production release, a dedicated performance team runs a massive load test.
Disastrous response times are discovered, requiring architectural redesigns, rollbacks, and delayed launches.

The solution is Shift-Left Performance Testing: running smaller, automated load tests on every pull request or nightly build. If a newly introduced database query increases latency by 200ms, the CI pipeline fails immediately—before the code ever reaches staging.

1. Automated Quality Gates in GitHub Actions

LocustPilot supports fully headless execution designed specifically for CI/CD runners. Here is the recommended GitHub Actions workflow (.github/workflows/performance.yml):

name: Performance Quality Gate

on:
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * *' # Nightly run at 2 AM

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Set up Python 3.10
        uses: actions/setup-python@v5
        with:
          python-version: '3.10'
          cache: 'pip'

      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Execute Headless Load Test
        env:
          LOCUST_TARGET_HOST: ${{ secrets.PERF_TEST_HOST }}
          RP_TOKEN: ${{ secrets.RP_TOKEN }}
          RP_ENDPOINT: ${{ secrets.RP_ENDPOINT }}
          RP_PROJECT: ${{ secrets.RP_PROJECT }}
        run: |
          locust -f locustfiles/files/checkout_flow.py \
            --headless \
            -u 50 \
            -r 5 \
            --run-time 2m \
            --csv stats \
            --html report.html \
            --only-summary \
            --exit-code-on-error 1

      - name: Archive Performance Reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: performance-artifacts
          path: |
            stats_*.csv
            report.html
            locust.log

Photo by Austin Distel on Unsplash

2. Bitbucket Pipelines Integration

For teams using Bitbucket Pipelines with Kubernetes deployments:

# bitbucket-pipelines.yml
image: python:3.10

pipelines:
  custom:
    performance-regression:
      - step:
          name: Run Staging Load Suite
          caches:
            - pip
          script:
            - pip install -r requirements.txt
            - locust -f locustfiles/libs/api_suite.py --headless -u 100 -r 10 --run-time 3m --csv report_stats
          artifacts:
            - report_stats*.csv

3. The 3 Golden Rules of CI/CD Load Testing

Test Focused Scenarios in PRs: Do not run 2-hour stress tests in pull requests. Run 2-to-5 minute targeted benchmark tests with 50–100 users to catch regressions fast.
Enforce Strict Thresholds: Fail the build if the 95th percentile latency exceeds your SLA budget (e.g. P95 > 250ms) or if the error rate exceeds 0.5%.
Compare Against Baselines: Track test trends over time in ReportPortal to detect slow memory leaks and gradual performance degradation across releases.

Series Conclusion: The Complete LocustPilot Journey

Over this 5-part masterclass, we transformed load testing from a manual chore into an enterprise-grade platform:

Part 1: Replaced brittle XML tools with Python's Test-as-Code and the LocustPilot Control Center.
Part 2: Built a safe static AST test discovery scanner and unbuffered subprocess engine.
Part 3: Streamed real-time P99 telemetry and deduplicated errors to ReportPortal using Gevent.
Part 4: Scaled to 100,000+ RPS across Kubernetes nodes using Helm charts.
Part 5: Embedded automated performance quality gates into CI/CD pipelines.

Performance engineering is no longer an afterthought—it is a continuous, automated quality pillar.

👉 Get Started with LocustPilot on GitHub

FAQ

How do we prevent load tests from overloading test environments in CI?

Use dedicated staging or isolated containerized environments, and set conservative user caps (-u 50 -r 5) during PR verification builds.

Where are historical CI test reports stored?

All test artifacts (HTML summaries, CSV latency distributions, and logs) are automatically uploaded to both CI build artifacts (GitHub/Bitbucket) and your centralized ReportPortal server.

TestFly Part 4: AI-Assisted Test Generation with testfly-mcp & Self-Healing Locators

Hakan GÜL — Sun, 30 Aug 2026 07:36:44 +0000

The Next Frontier: Model Context Protocol & TestFly

TestFly bridges modern AI coding assistants (Claude Desktop, Cursor, Copilot, Antigravity) through the testfly-mcp companion server.

1. Connecting `testfly-mcp`

Add to your mcp_config.json:

{
  "mcpServers": {
    "testfly-mcp": {
      "command": "npx",
      "args": ["-y", "testfly-mcp@latest"]
    }
  }
}

2. Generating TestFly Code from User Journeys

The AI launches a browser via MCP, records interactions, and writes clean TestFly code utilizing accessibility-first locators:

package io.testfly.examples;

import io.testfly.test.BaseTest;
import io.testfly.locator.Role;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;

public class CheckoutFlowTest extends BaseTest {

    @Test
    public void userCompletesPurchase() {
        open("/products");
        getByRole(Role.BUTTON).withName("Add to Cart").click();
        getByRole(Role.LINK, "Checkout").click();
        getByLabel("Full Name").type("Hakan Gül");
        getByRole(Role.BUTTON).withName("Place Order").click();

        assertEquals(getByTestId("order-status").getText(), "CONFIRMED");
    }
}

3. Self-Healing Locators (`locators.selfHealing`)

Enable locator self-healing in testfly.yml:

locators:
  selfHealing: true

When an element is not found by its primary selector, TestFly's healing engine analyzes surrounding DOM attributes (accessible name, ARIA role, nearby text) to locate the element and keep your CI pipeline green.

TestFly Part 3: CI/CD Quality Gates, Parallel Execution & CI Metadata

Hakan GÜL — Sun, 30 Aug 2026 07:36:41 +0000

Enterprise CI/CD with TestFly

In modern continuous delivery pipelines, flaky tests and degraded pass rates must not block deployments silently. TestFly features dedicated CI subsystems:

CiEnvironmentDetector: Auto-detects GitHub Actions, Jenkins, GitLab CI, CircleCI, TeamCity, and Bitbucket pipelines.
BuildThresholdEnforcer: Enforces strict quality gates on build completion.
JUnitXmlReporter: Generates machine-readable TEST-TestFly.xml for CI dashboard integration.

1. Configuring CI Quality Gates in `testfly.yml`

execution:
  mode: local
  parallel: methods
  threadCount: 4
  maxActiveSessions: 4

ci:
  failOnPassRateBelow: 95.0 # Fails the build if pass rate < 95%
  maxFlakyTests: 2          # Fails if more than 2 tests are flaky

If quality thresholds are breached, TestFly throws a BuildQualityGateException and exits with a non-zero code.

2. GitHub Actions Multi-Browser Matrix Workflow

name: TestFly E2E Regression

on:
  push:
    branches: [ master, main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        browser: [ chrome, firefox ]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'
          cache: 'maven'

      - name: Run TestFly Tests
        run: mvn clean test -Dbrowser.name=${{ matrix.browser }} -Dbrowser.headless=true

      - name: Upload TestFly HTML Report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: testfly-report-${{ matrix.browser }}
          path: target/testfly-report.html

TestFly Part 2: Accessibility-First Locators, BasePage & WaitEngine

Hakan GÜL — Sun, 30 Aug 2026 07:36:02 +0000

Accessibility-First Semantic Locators

Traditional Selenium tests rely heavily on brittle CSS selectors or complex XPaths that break whenever the DOM hierarchy changes:

// Brittle — breaks on CSS redesigns:
find(By.cssSelector("div.modal > form button.btn-primary")).click();

TestFly introduces Playwright-style Semantic Locators that target the accessibility tree—interacting with the page exactly as a real human perceives it:

getByRole(Role.BUTTON).withName("Submit").click();
getByLabel("Email address").type("user@testfly.io");
getByPlaceholder("Search products…").type("laptop");
getByText("Forgot password?").click();
getByTestId("checkout-cta").click();

Every semantic locator returns an auto-waiting, chainable Locator instance with zero Thread.sleep().

Clean Page Objects with `BasePage`

BasePage provides wait-backed helper methods (type, click, getText, getAttribute, isDisplayed):

package io.testfly.examples.pages;

import io.testfly.test.BasePage;
import io.testfly.locator.Role;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class LoginPage extends BasePage {

    private static final By USERNAME = By.id("username");
    private static final By PASSWORD = By.id("password");

    public LoginPage(WebDriver driver) {
        super(driver);
    }

    public void login(String user, String pass) {
        type(USERNAME, user);
        type(PASSWORD, pass);
        getByRole(Role.BUTTON).withName("Sign In").click();
    }
}

Advanced Synchronization with `WaitEngine`

When writing custom wait conditions, WaitEngine centralizes explicit waits using timeouts.explicit from testfly.yml:

import io.testfly.wait.WaitEngine;

// Centralized explicit wait conditions:
WaitEngine.waitForVisible(By.id("dashboard-banner"));
WaitEngine.waitForClickable(By.id("pay-button"));
WaitEngine.waitForInvisible(By.cssSelector(".loading-spinner"));

DEV Community: Hakan GÜL

Scaling to 100,000 RPS: Deploying Distributed Load Testing on Kubernetes with Helm and Docker

When a Single Machine Hits the Wall

The Distributed Master-Worker Architecture

Packaging for Kubernetes with Helm (helm/locust)

1. The Optimized Container (k8s.Dockerfile)

2. Deploying the Cluster with One Command

3. Tuning OS Limits for Massive Scale

What Is Next?

FAQ

How many workers are needed for 50,000 RPS?

Can workers be dynamically auto-scaled?

Enterprise Observability: Real-Time ReportPortal Integration with Gevent and Percentile Tracking

The Log Flood Dilemma in Load Testing

1. Background Telemetry with Gevent Greenlets

2. Smart Error Deduplication (Unique Error Logging)

3. Comprehensive Percentiles and Automated Quality Gates

What Is Next?

FAQ

Why is ReportPortal preferred over standard HTML reports?

Can custom quality gate thresholds be configured?

Under the Hood: Building a Dynamic AST Test Runner and Subprocess Engine in Python

The Hidden Trap of Dynamic Python Discovery

The Solution: Static Analysis via Python's AST

Why This Approach Wins:

Extracting Target Hosts Without Hardcoding

Subprocess Execution and Unbuffered Log Streaming

Key Subprocess Design Choices:

Architectural Summary

What Is Next?

FAQ

Why not use regex alone to find Locust classes?

Does subprocess.Popen limit load test throughput?

Can I run tests with custom Python dependencies?

The Death of XML: Why We Built LocustPilot — A Modern Control Plane for Distributed Load Testing

Why Is Traditional Performance Testing Broken?

What Was Missing in Vanilla Locust?

Introducing LocustPilot: The Modern Control Plane

Key Architectural Highlights:

Feature Matrix: How LocustPilot Compares

How to Get Started in 30 Seconds

What Is Next in This Series?

FAQ

Why choose Python-based Locust over JMeter?

Does LocustPilot modify standard Locust scripts?

Can I run LocustPilot in headless CI/CD pipelines?

TestFly Part 8: Flakiness Quarantine Engine & Java SPI Extensibility

Quarantine Engine & Java SPI Extensibility

1. Flakiness Quarantine Engine (testfly-quarantine.yml)

2. Java SPI Extension Points

Example: Custom Slack Report Adapter

Summary of the TestFly Masterclass

TestFly Part 7: Bundled axe-core Accessibility Assertions & VisualAssert Pixel Matching

Accessibility & Visual Regression in TestFly

1. Bundled axe-core Accessibility Scans (accessibility())

2. Pixel-Perfect Visual Regression (VisualAssert)

TestFly Part 6: Built-in ApiClient, BaseApiTest & Fast Hybrid UI + API Flows

Native API Testing in TestFly

1. Pure API Tests with BaseApiTest

2. Hybrid UI + API Tests via apiClient()

TestFly Part 5: BDD Testing with Cucumber 7, BaseCucumberSteps & @retryable Tags

BDD with Cucumber 7 in TestFly

1. Runner Class (BaseCucumberTest)

2. Step Definitions with BaseCucumberSteps

3. Per-Scenario Retry Tag

Shift-Left Performance: Automated Load Testing Quality Gates in GitHub Actions and Bitbucket CI

Why Performance Testing Must Shift Left

1. Automated Quality Gates in GitHub Actions

2. Bitbucket Pipelines Integration

3. The 3 Golden Rules of CI/CD Load Testing

Series Conclusion: The Complete LocustPilot Journey

FAQ

How do we prevent load tests from overloading test environments in CI?

Where are historical CI test reports stored?

TestFly Part 4: AI-Assisted Test Generation with testfly-mcp & Self-Healing Locators

The Next Frontier: Model Context Protocol & TestFly

1. Connecting testfly-mcp

2. Generating TestFly Code from User Journeys

3. Self-Healing Locators (locators.selfHealing)

TestFly Part 3: CI/CD Quality Gates, Parallel Execution & CI Metadata

Packaging for Kubernetes with Helm (`helm/locust`)

1. The Optimized Container (`k8s.Dockerfile`)

Does `subprocess.Popen` limit load test throughput?

1. Flakiness Quarantine Engine (`testfly-quarantine.yml`)

1. Bundled axe-core Accessibility Scans (`accessibility()`)

2. Pixel-Perfect Visual Regression (`VisualAssert`)

1. Pure API Tests with `BaseApiTest`

2. Hybrid UI + API Tests via `apiClient()`

1. Runner Class (`BaseCucumberTest`)

2. Step Definitions with `BaseCucumberSteps`

1. Connecting `testfly-mcp`

3. Self-Healing Locators (`locators.selfHealing`)

1. Configuring CI Quality Gates in `testfly.yml`

Clean Page Objects with `BasePage`

Advanced Synchronization with `WaitEngine`