DEV Community: Jeremy Herzog

Selenium Load Tests With Cicada Distributed

Jeremy Herzog — Tue, 17 Aug 2021 13:33:41 +0000

Cicada Distributed is unique because it allows end-to-end browser tests and load tests with many users to be run with the same framework. This is particularly useful with server side rendered websites, where the site’s performance is more limited. In this tutorial, you’ll learn how to modify the Cicada Dockerfile with a Selenium installation, and run Selenium tests through CicadaD.

Source code available at https://github.com/cicadatesting/cicada-distributed-demos/tree/main/selenium-demo

Installing Selenium

First, generate the base Dockerfile and test.py:

cicada-distributed init

Next, modify the Dockerfile in order to install Chrome, Webdriver, and Selenium for Python.

FROM cicadatesting/cicada-distributed-base-image:1.3.0

RUN apt-get update
RUN apt-get install -y bash unzip curl xvfb libxi6 libgconf-2-4 gnupg

# Install chrome
RUN curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add
RUN echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
RUN apt-get -y update
RUN apt-get -y install google-chrome-stable

# Install ChromeDriver.
# NOTE: may have to change this to appropriate version of google-chrome-stable listed here: https://www.ubuntuupdates.org/pm/google-chrome-stable
RUN wget -N https://chromedriver.storage.googleapis.com/92.0.4515.107/chromedriver_linux64.zip -P ~/
RUN unzip ~/chromedriver_linux64.zip -d ~/
RUN rm ~/chromedriver_linux64.zip
RUN mv -f ~/chromedriver /usr/local/bin/chromedriver
RUN chown root:root /usr/local/bin/chromedriver
RUN chmod 0755 /usr/local/bin/chromedriver

# Install selenium
RUN pip install selenium

# Install tests
COPY . .

ENTRYPOINT ["python", "-u", "test.py"]

Adding Selenium to the Test

In this example, we’ll add the Selenium driver to a custom user loop. You don’t have to do it this way, but it is a nice way of separating the driver setup code from the test code in Cicada.

First, create a custom user loop in test.py to initialize the driver:

from datetime import datetime
from typing import Any

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from cicadad.core.scenario import UserCommands
from cicadad.core.decorators import scenario, user_loop, users_per_container
from cicadad.core.engine import Engine

CHROMEDRIVER_PATH = "/usr/local/bin/chromedriver"
WINDOW_SIZE = "1920,1080"

engine = Engine()


def chromedriver_user_loop(user_commands: UserCommands, context: dict):
    chrome_options = Options()

    chrome_options.add_argument("--headless")
    chrome_options.add_argument(f"--window-size={WINDOW_SIZE}")
    chrome_options.add_argument("--no-sandbox")

    driver = webdriver.Chrome(
        executable_path=CHROMEDRIVER_PATH, chrome_options=chrome_options
    )

    start = datetime.now()

    output, exception, logs = user_commands.run(
        context=context,
        driver=driver,
    )

    end = datetime.now()

    user_commands.report_result(
        output,
        exception,
        logs,
        time_taken=(end - start).seconds,
    )

    driver.close()

Next, pass it to the test function:

@scenario(engine)
@user_loop(chromedriver_user_loop)
def get_tutorial(context, driver: Any):
    print(driver)


if __name__ == "__main__":
    engine.start()

Creating a Selenium Test

The last step is to add test code to verify functionality of your site. Let’s add a simple test to the Cicada Docsite which verifies clicking the Tutorial navigation button will load the installation page:

@scenario(engine)
@user_loop(chromedriver_user_loop)
def get_tutorial(context, driver: Any):
    driver.get("https://cicadatesting.github.io/cicada-distributed-docs/")

    em = driver.find_element_by_link_text("Tutorial")

    assert em is not None
    em.click()

    url = driver.current_url

    assert "installation" in url, f"Did not navigate to tutorial; current url is {url}"
    return url

In addition, limit the users_per_container parameter, introduced in version 1.3.0, to 1 in order to prevent resource contention in the user containers:

Finally, run the test:

cicada-distributed run

The test should hit the docsite and return the URL of the installation page.

Congratulations! You’ve learned how to run Selenium through Cicada Distributed!

Running Cicada Distributed Tests in Kubernetes

Jeremy Herzog — Wed, 14 Jul 2021 14:55:14 +0000

Cicada Distributed’s scalability and flexibility comes from being able to run a containerized testing workload. In the initial version of Cicada Distributed, this was only available in Docker. Cicada Distributed 1.2.0 introduces Kubernetes support to run those same workloads in a K8s cluster. In this tutorial, you’ll set up a local K8s cluster using k3d, (a lightweight version of Kubernetes that runs in Docker containers), modify the Cicada Distributed cluster for local usage with Kustomize, and use that to host the tests.

This guide is also available on the Cicada Distributed docsite.

Setting up the cluster

Begin by installing k3d and ensuring that Kustomize is installed and available to use with kubectl -k. Once this is installed, start the k3d cluster:

k3d cluster create -p "8283:30083@server[0]" -p "8284:30084@server[0]"

This will create a cluster with two node ports exposed on localhost:8283 and localhost:8284. Because these ports map to 30083 and 30084 respectively in the cluster, we’ll also have to modify the install of Cicada using Kustomize.

First, create a directory for the overlay and get the installation YAML into a file:

mkdir cicada-distributed-overlay
cicada-distributed start-cluster --mode=KUBE > cicada-distributed-overlay/cicada.yaml

Next, create a file called cicada-distributed-overlay/patch.yaml and add this to the contents:

apiVersion: v1
kind: Service
metadata:
  name: cicada-distributed-datastore-client
spec:
  ports:
  - port: 8283
    protocol: TCP
    targetPort: 8283
    nodePort: 30083
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: cicada-distributed-container-service
spec:
  ports:
  - port: 8284
    protocol: TCP
    targetPort: 8284
    nodePort: 30084
  type: NodePort

This will override the datastore-client and container-service services to use a NodePort bound to 30083 and 30084 in the cluster, so we can access them locally.

Next, you’ll need to merge the files using Kustomize. To do this, add a file called cicada-distributed-overlay/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - cicada.yaml
patchesStrategicMerge:
  - patch.yaml

Next, apply the directory by using the -k flag in kubectl:

kubectl apply -k cicada-distributed-overlay

This will create all the resources and modify the services for usage in k3d.

Getting an API into K8s

In a previous article, a simple REST API was developed to demonstrate Cicada Distributed. The source code for that API is available here. For this example, you’ll need to add the API image to the k3d cluster and start it using the provided Kube YAML.

The demo app can be cloned from cicadatesting/cicada-distributed-demos:

git clone https://github.com/cicadatesting/cicada-distributed-demos.git

First, build the API and database and add the images to the cluster:

cicada-distributed-demos/rest-api/app :
docker build -t cicadatesting/demo-api-app:local .
docker build -t cicadatesting/demo-api-flyway:local -f flyway.dockerfile .
k3d image import cicadatesting/demo-api-app:local
k3d image import cicadatesting/demo-api-flyway:local

Next, install the app with the code in kube-app.yaml:

kubectl apply -f kube-app.yaml

This should start the API, database, and a job to install the database schema.

Running the tests

Once an example app is running, we can run Cicada tests against it. Navigate to the cicada-distributed-demos/rest-api/integration-tests. Since it is running in k3d, the image needs to be imported into the cluster. To build, run:

docker build -t cicadatesting/cicada-distributed-demo-integration-test:local .

Next, import the image with:

k3d image import cicadatesting/cicada-distributed-demo-integration-test:local

Finally, start the test by running:

cicada-distributed run --mode=KUBE --image=cicadatesting/cicada-distributed-demo-integration-test:local

You should see the test spin up and execute the 4 test scenarios.

(Originally posted on Medium)

Writing the Cicada Distributed Testing Framework

Jeremy Herzog — Tue, 25 May 2021 03:55:37 +0000

Last year, I wrote Cicada-2, a low-code testing framework. Since its release, I’ve considered ways to improve upon it, particularly for running load tests. This is the story of how I created Cicada Distributed, a Python based load testing framework and why I believe it should be the go-to tool for testing your services.

Why it was time for a rewrite

Cicada-2 was based on lessons I learned trying to test complex applications. While it was great for integration tests, it did not have the features I needed to write effective load tests. The hard-coded load model of Cicada-2 was limiting when it came to writing a test beyond something like “hit my API a bunch of times". I wanted a tool that could hit a service hard enough to know what it’s true limits were.

With that in mind, I started adding more programmatic testing features into Cicada-2. However, I quickly realized that I would rather be able to write tests in Python than with some awkward recursive YAML mixed with Jinja2. So I started over, and began writing Cicada from scratch.

Rewriting Cicada again… and again

The core feature I wanted the new Cicada to have was to allow users to have complete control over the load model of the test. A test should be able to not only call a service a certain number of times, but deliver ramping load, scaling to a threshold, and a bunch of other situations I hadn’t considered.

To do this, Cicada uses a virtual user model. Essentially, the code to simulate the actions of a user is run in parallel to create load. I wrote the initial version of this to run the virtual users inside of threads. Unfortunately, this turned out not to be a great approach. Print statements would break the rest of the test. I’d get weird bugs about what could and could not be pickled to run inside a thread. Code written outside of a test wouldn’t always work. In addition, the process managing the user threads became a bottleneck. It was hard to control which users could start and stop without significantly affecting performance.

How the new Cicada works

After several revisions, I settled on a distributed user model that was loosely coupled to the scenario via an event broker (Kafka as of right now). Instead of running on a managing instance, virtual users run inside of containers. This greatly simplifies the virtual user code because it allows Cicada to take advantage of a container orchestrator in managing the user pool, instead of managing individual threads across a machine or multiple machines. In addition, the event model allows users to receive commands and send back results at their own pace, making the test less prone to performance bottlenecks.

Another major improvement is in how much more configurable Cicada Distributed’s load model is than that of Cicada-2’s. It allows you to write it in plain Python and control the scenario via an API. This means you can scale users up and down programmatically, as well as divvy up load amongst the user pool. Finally, you have complete control over how results are gathered and analyzed via user definable aggregation and error filtering functions.

A quick example

To demonstrate the improved load testing features of Cicada Distributed, we’ll walk through an example of a simple test. For this example, I’ve created an API with an endpoint for creating a user and storing it in a database:

For a basic load test, we can hit this endpoint with a limited number of users for a certain time. First, we’ll need to install Docker and Cicada Distributed and create a blank project:

pip install cicadad

mkdir load-test
cd load-test

cicada-distributed init .

In the load-test directory, you’ll see a couple of files:

Dockerfile
test.py

Because Cicada uses Docker to package the tests, you can add any dependencies to the image to use in a user or scenario. Add the requests package to the Dockerfile:

FROM cicadatesting/cicada-distributed-base-image:latest

RUN pip install requests

COPY . .

ENTRYPOINT ["python", "-u", "test.py"]

Next, update test.py with a basic load test:

...
from cicadad.core.decorators import scenario, load_model, user_loop
from cicadad.core.scenario import n_seconds, iterations_per_second_limited
...

@scenario(engine)
@load_model(n_seconds(180, 30))
@user_loop(iterations_per_second_limited(4))
def post_user(context):
    requests.post(
        url="http://172.17.0.1:8080/users",
        json={
            "name": "jeremy",
            "age": 23,
            "email": f"{str(uuid.uuid4())[:8]}@gmail.com",
        },
    )

In this example, Cicada will perform the post_user test to create a user for 180 seconds with 30 users. Additionally, each user is limited to 4 requests per second. To execute the test, you’ll need to start the cluster (an event broker and a service to create containers) and run the test:

cicada-distributed start-cluster
cicada-distributed run

When this runs, we’ll end up with a load curve that looks like this (I used Prometheus + Grafana to monitor the API):

What if we wanted to see how much load the API could take in 3 minutes? Remove the line @user_loop(iterations_per_second_limited(4)) and replace it with @user_loop(while_alive()) (import it with from cicadad.core.scenario import while_alive). This will remove the constraint on requests per second so the virtual users can make as many requests as possible. On my machine, I was able to process approximately 200 requests per second (although Cicada’s virtual users are capable of putting out a much higher RPS against a more capable host). Let me know in the comments how much load your system was able to handle.

Conclusion

In conclusion, I’m much more satisfied with Cicada Distributed’s flexibility over Cicada-2. Please feel free to try Cicada Distributed out and let me know what you think!