DEV Community: Juan Ibarra

Parallel Testing: Best Practice for Load Testing & Functional Testing

Juan Ibarra — Tue, 12 Nov 2024 14:07:44 +0000

Introduction

Testing becomes crucial as the complexity and the adoption of a solution grows. In Kubecon NA 2020, Jian from Airbnb talked about 10 more weird ways to blow up your Kubernetes. The first key takeaway from the talk was to test the new features on test clusters. But how does one ensure adequate testing without adding significant time to build or deploy pipelines?
To do so, here are two preliminary tests that have to be performed by a team before moving their solution to a production environment:

The solution must be running on such systems that can support peak usage. This would require simulating many users to test the peak performance.
Each feature of the solution has the expected behavior and is seamless to use. For this, testing has to be done across multiple environments, different browsers, devices, operating systems, and inputs.

Testing for both of these scenarios ultimately requires time and resources, which is where parallel testing is a powerful approach to ensuring your system is tested at scale before it goes into production. In this post, we will understand the concept of test parallelisation and some of the common approaches like load and functional parallelisation. We will discuss in depth some of the challenges associated with those approaches, and explore a Kubernetes native testing framework that can help us address those challenges.

Parallel Testing Overview

Parallel Testing is the process of performing parallel simulations and tests to validate the functionality and performance of the application under test. Parallelisation is done to both scale the performance under load, and perform functional analysis of the system under test to increase coverage and decrease overall test execution times. Let us understand load and functional parallelisation in detail.

Parallel Load Testing

To test a solution for peak-hour performance, creating a load by sending multiple requests from a few anonymous users is not enough. We are required to simulate a real-time environment by generating a massive load from named users from multiple geographies, possibly across multiple browsers or devices.
Parallel load testing is the process of using parallel instances of load-testing tools to simulate a massive number of users from multiple nodes, possibly distributed across geographic locations. Solutions that face large volumes of concurrent users, such as social media platforms, banking applications, e-commerce websites, online gaming applications, etc. can all benefit from this approach to testing to ensure uninterrupted services for their users even under peak usage.

Enabling load parallelisation has some challenges, let us understand these.

Challenges with load parallelisation

Generate Heavy Load: Load parallelisation requires sending huge traffic to the software or application possibly from various geographical locations to test peak-hour performance and network latency. For example, for an online gaming website, we need to test if players from across geographies are playing in a multiplayer mode, they do not experience network lagging.
Parameterized Virtual Users: To simulate specific user behavior by specifying individual characteristics, parameterized virtual users are required. Generating anonymous virtual user requests will not help if we need to customize test cases. Suppose in an online gaming website, players are chatting using the chat message feature in the game, it needs to be tested that their profile details are correct.
Visualizing Results: We need real-time visualization of the test to simply understand the behavior of the application or the scenario that is causing the issue. It may also require aggregating such results to observe the behavior. For example, if the test shows large network latency when a massive load is created from multiple geographies, we need detailed results to visualize the issue and debug the underlying network infrastructure.
Monitoring Test System: For the target system under test, monitoring can help us identify the resource usage such as CPU or memory utilization, and scalability behavior and anticipate performance issues when a massive load is introduced to the solution. For example, if the test shows a spike in CPU usage during peak hours then this analysis can be used to optimize the resource allocation.

Parallel Functional Testing

Parallel functional testing is the process of performing parallel functional tests to increase test coverage and decrease overall test execution time. It is commonly paired with parameterised test execution by giving a specific input and checking to get the expected output. For example, using the browser version as input, we can test a feature across multiple browsers or operating systems in parallel and at the same time. This would not only help us with time to market but would also allow efficient testing and optimized usage of resources. In the coming section, we are going to talk about the need for functional parallelisation with the help of an example.

Why is functional parallelisation needed?

For example, we have a banking solution for which we want to make two-factor authentication mandatory for all users at the time of sign-in. Now testing this functionality across multiple browsers sequentially would not only take a lot of time but also resources. Here the input would be a parameterized user for which the two-factor authentication is enabled and happens successfully in a minimal amount of time.

With functional parallelisation, we can test this functionality with various inputs across multiple browsers at the same time. Suppose the test fails for any of the browsers, we would get to know in a lot less time and work on identifying the root cause along with its solution.

Enabling functional parallelisation also has some challenges, let us understand these.

Challenges with functional parallelisation

Test Isolation: Each test may have its own set of dependencies, configurations, and compatibility considerations. Setting up and configuring multiple test environments for different tests can be time-consuming and complex, especially when ensuring consistency across environments.
Recording Results: In the previous example, we would require a system that can record different input cases and the combination browser and the actual results. These recorded results would then be used to compare with the expected result to determine the performance of functionality.
Resource Allocation: Identifying which functionality test would require more resources and then distributing the resources accordingly could be a challenging task. Suppose for a banking solution, the sign-in and transaction functionalities have to be tested. Now these tests take time even though run in parallel so we need to determine the resource utilization limit for each.

Seeing the challenges associated with load and functional parallelisation, we need a testing solution that automates testing with parameterized virtual users or other inputs across multiple browsers or operating systems simultaneously. Also, the solution should allow dynamic resource allocation based on test requirements and optimize resource utilization. For systems under test, the testing solution should provide real-time visualization of tests and aggregated results to perform result analysis.

Parallel Testing with Testkube

Testkube recently introduced Test Workflows, which leverages an execution and orchestration engine specifically built for executing any testing tools and scripts at scale, and since Testkube leverages Kubernetes as its runtime environment for test execution, it can scale and allocate resources for test execution in line with corresponding functionality provided by Kubernetes itself.

Test Workflows are defined using a dedicated YAML vocabulary, and can be created using both extensive Wizards and samples for different scenarios which can then be further enhanced in any way required to fulfill your testing requirements using the Testkube Workflow Editor. Workflows also give fine-grained control over resource-usage and allocation for your tests, helping you maximize the utilization of your infrastructure for test execution.

Parallel testing is one of the many functionalities available via Workflows:

A parallel keyword makes it possible to run any testing tool across multiple nodes
A shard keyword makes it possible to shard tests across multiple nodes to ensure each node is running the right set of tests
A matrix keyword makes it possible to parameterise tests running both in sequence and in parallel
A service keyword makes it possible to manage depending services to run distributed tests (for example with JMeter or Selenium)
An execute keyword makes it possible to orchestrate multiple tests to run both in sequence and/or in parallel, allowing you to simulate more advanced usage scenarios - see our article on System Testing.

Workflows for Load Parallelisation

As described above, running a load test from multiple nodes in parallel is often required to simulate large amounts of user traffic. The parallel keyword in Test Workflows allows you to parallelize any load-testing tool or script for this purpose, for example:

K6 - parallel can be used to distribute and parameterise k6 scripts far beyond what can be achieved with the k6-operator - Read More.
*JMeter *- service can in combination with service.count be used to spawn any desired number of distributed worker nodes which are then invoked from a master node for running the test.
Artillery, Gatling - similar to K6, distributing and parameterising Artillery and Gatling tests is easily achievable using the parallel keyword

Examples for distributed load-testing with several of these tools are available in the Testkube Documentation.

Workflows for Functional/End-to-End Parallelisation

Parallelising functional and end-to-end (E2E) tests for the purpose of increased test coverage and reduced overall test execution times is equally well supported by the parallel keyword. Depending on the testing tool you are distributing, you can also shard test files or input parameters across nodes, which is supported by the shard keyword, for example

Playwright tests can be parallelised and sharded across multiple nodes using Playwrights built-in sharding functionality
Cypress tests can be parallelised and the Workflow Express Language can be used to shard test files accordingly
API Testing tools like Postman and SoapUI can be parallelised and parameterised similarly to increase test coverage for APIs
Acceptance testing tools like Selenium and Robot-Framework can be parallelised and parameterised for different browsers and user input, increasing acceptance-test coverage before releasing applications into production.
Any other testing tool or script can be distributed across multiple nodes, with parameterisation as applicable for each tool.

As above, examples for distributed functional testing with these tools are available in the Testkube Documentation.

Composite Workflows for Test Parallelisation

The approaches described above with the parallel and shard keywords are meant to be used within a single workflow. If you on the other hand want to combine multiple workflows to run either in sequence or in parallel you can use the execute keyword to orchestrate the execution of any other workflow in any combination required. For example you can:

Run multiple load-tests in parallel to see how the traffic they simulate affect each other
Run an E2E or API test in parallel with a load-test to ensure that functionality is maintained under load
Run a security test in parallel with a distributed load-test to ensure that security is maintained during high load on your applications.

Read more about Composite Testing in Introducing System Testing with Testkube and about the execute keyword in the documentation.

Troubleshooting, Artifacts and Reporting

Testkube automatically captures the log output of any testing tool it runs on all nodes the tools are running on, to help you ensure that tests were executed as desired and debug issues with your executions. Furthermore, Testkube can be configured to capture any artifacts produced by your tests, including aggregate reports, videos, JUnit-reports, etc. When captured, Testkube will make these available via the Dashboard.

Conclusion

Testing your applications at scale often requires a parallelised execution approach to your testing efforts, be it the parallelisation of load-tests to generate massive load, or the parallelisation of functionali/E2E tests to increase test coverage and improve test execution times.
The recently introduced Test Workflows engine in Testkube allows for parallelisation of any testing tool, both load and functional, with additional support for sharding, parameterisation and management of dependent services to execute your tests.

Head over to testkube.io/get-started to learn more and give Testkube a try using either our demo environment or your existing tests running in your own infrastructure.

Distributed Load Testing with JMeter in Testkube

Juan Ibarra — Wed, 06 Nov 2024 22:08:12 +0000

Distributed Systems face various challenges due to the system’s complexity, such as partial system failures, data inconsistency, deadlocks, high latency, and packet loss. These challenges can often be proactively addressed by proper functional and non-functional testing, including distributed load-testing to ensure your system can handle a high number of users at any given time. Using tools like JMeter, you can simulate real-world conditions and failures to ensure robustness and reliability.

However, simulating production-like conditions often requires running load tests in a distributed setup, generating load from multiple sources, which is resource-intensive and requires careful orchestration of tests and resources. Thus, you need specialized tools to make that process simpler.

In this blog, we will show how Testkube can be leveraged together with JMeter to simplify the distributed load-testing process, resulting in efficient resource allocation, horizontal scalability, and simplified orchestration of tests.

Distributed Testing with JMeter

Distributed testing is executing tests concurrently from multiple machines or environments. The load is distributed across multiple nodes to more closely simulate real-life usage scenarios. This method is beneficial for testing complex systems, cloud-native, and large-scale applications like banking or e-commerce websites.
In this section, we will discuss JMeter, a distributed load-testing tool, dig deeper into its distributed systems testing architecture, and understand the associated challenges.

JMeter

Apache JMeter is an open source software for distributed, performance, and load testing of applications. JMeter supports protocols like HTTP/HTTPS, FTP, SOAP, JDBC, TCP, UDP, and much more, allowing it to support the testing of different applications. It enables distributed testing with master-slave configuration, spreading the load across multiple nodes. This enables an extensive test of an application’s behavior under heavy traffic. Let us understand the master-slave architecture supported by JMeter for distributed testing.

JMeter Master-Slave Architecture

In the master-slave architecture of JMeter, multiple users are simulated across various machines. This type of setup helps test such applications where you need to generate significant loads but have hardware limitations.

Source: JMeter Distributed Testing

Here is an overview of each component in this architecture:

Target: An application, service, or server under test.
Master [Controller Node]: A controller that manages the execution of the test. It initiates the test execution, distributes the load, and collects the results from the slave.
Slave [Worker Nodes]: A load generator that takes requests from the master, generates the load, and executes the test on the target.

How does JMeter execute a test on a target?

There is one master that handles the execution of tests using multiple remote slaves. For proper communication, the master, slaves, and target must be on the same network. The master communicates with the slaves using Java RMI(Remote Method Invocation). Each of these slaves generates a load and executes tests. The test execution by all the slaves on the target starts at the same time. The execution details are sent back by the slave to the master, who performs the result aggregation.

Challenges with distributed testing with JMeter

While this setup seems easy, there are some challenges associated with it:

Resource allocation overhead: Even though the master-slave architecture distributes the load, which reduces the chances of over-utilizing a single slave, it is quite challenging to allocate resources to the slaves for optimum usage.
**Complex master-slave configuration: **Ensuring the same version of JMeter on all the nodes, configuring the network to allow RMI traffic, and synchronizing time between nodes is complicated.
Monitoring and debugging issues: JMeter requires configuring external plugins or tools for real-time analytics or monitoring. With a large-scale system setup and complicated configurations, this becomes more of an overhead for the team.

Thus, for performing distributed load testing on complex systems in a cloud-native environment, you require a tool to help with optimum resource allocation, easy configuration of master and slaves, scalability, and proper observability. The good news is that Testkube can do it all and gives you both a Dashboard and command line utility for test orchestration.

In the following section, we will take a detailed look at the features of Testkube that can help achieve better test orchestration for JMeter distributed testing.

Automate JMeter Distributed Testing with Testkube

Testkube is a test orchestration and execution platform that leverages the power of Kubernetes for testing cloud-native applications. It allows you to automate test execution irrespective of your testing framework, tool, or script using a powerful Test Workflows engine, and leverages a unified dashboard, for centralized test creation, execution, and result aggregation, helping you manage tests better and gain observability in overall testing.

How does Testkube automate JMeter distributed testing?

Testkube supports all popular testing frameworks, including JMeter, for distributed load testing. Testkube runs the test directly in your Kubernetes cluster, ensuring secure execution of your system or new features. Here are some of the benefits that Testkube offers when it comes to running distributed JMeter tests:

Easy RMI configuration: Testkube simplifies the major challenge of configuring RMI in JMeter’s master-slave architecture by automating the network configuration needed, such as firewall and port setup. It also automates the installation of the same JMeter version on all nodes, saving time and reducing the risk of error. Testkube provides a sample configuration template that you can use to get started easily.
Scalable system design: With Testkube, you can leverage Kubernetes native scaling capabilities and dynamically scale the number of slaves without changing your test scripts. Testkube utilizes the Kubernetes’ resource management to distribute load. This way, you can easily simulate a large amount of traffic to test your system or application without worrying about resource allocation.
Centralized monitoring dashboard: While testing, it is important to have the ability to view the details of previous test executions and current ones for comparison. In the case of JMeter, you also need test execution details, errors, and performance metrics per slave. The Testkube Dashboard aggregates all the test executions in real-time and helps you track them for easy monitoring and debugging.

Testkube handles all the complexities of performing distributed tests in JMeter. This lets you focus on developing the application and testing rather than figuring out the infrastructure. Let us see in the coming section the JMeter distributed testing in Testkube.

How do you execute JMeter distributed testing in Testkube?

Using Testkube Test Workflows, we are going to execute the test with JMeter. We have created a JMeter test that runs on our Testkube website and performs the distributed test. Let us get started with the prerequisites and create a Test Workflow to execute the test.

Prerequisites

The basic requirements while configuring the master-slave architecture in JMeter with Testkube are as follows:

A Kubernetes Cluster(We have used Minikube here.)
A Testkube Pro account (free plan is fine)
The Testkube Agent is installed on the cluster.
A Testkube API token with Admin Access rights. ‍

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent.

Creating a Test Workflow

In the Testkube Dashboard, we are going to create a Test Workflow.

Login to Testkube and select Workflows from the left menu bar.
Click on “Add a new Test Workflow” and select “Start from an example”.

Scroll to the right and select “Distributed JMeter”. Testkube loads an example YAML.

We are going to update this YAML to also process the JMeter artifacts. Provide the YAML given below:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: distributed-jmeter-example-config-artifacts
  namespace: testkube
spec:
  config:
    slavecount:
      type: integer
      default: 3
  content:
    git:
      uri: https://github.com/kubeshop/testkube
      revision: main
      paths:
      - test/jmeter/executor-tests/jmeter-executor-smoke.jmx
  container:
    workingDir: /data/repo/test/jmeter/executor-tests
  services:
    slave:
      use:
      - name: distribute/evenly
      count: config.slavecount
      logs: always
      timeout: 30s
      image: anasoid/jmeter:5.6-plugins-21-jre
      command:
      - jmeter-server
      - -Dserver.rmi.localport=60000
      - -Dserver_port=1099
      - -Jserver.rmi.ssl.disable=true
      readinessProbe:
        tcpSocket:
          port: 1099
        periodSeconds: 1
  steps:
  - name: Run tests
    run:
      image: anasoid/jmeter:5.6-plugins-21-jre
      shell: |
        jmeter -n \
          -X -Jserver.rmi.ssl.disable=true -Jclient.rmi.localport=7000 \
          -R {{ services.slave.*.ip }} \
          -t jmeter-executor-smoke.jmx \
          -j /data/artifacts/jmeter.log \
          -o /data/artifacts/report \
          -l /data/artifacts/jtl-report.jtl -e
    artifacts:
      paths:
      - /data/artifacts/**/*
status: {}

You can view the same running in the Testkube Dashboard as shown below.

‍

Let us split the above file into parts to understand the changes specific to JMeter.

Defining the slaves to have a dynamic value: Here we have used config to define the slave count. The name of the field is set as slavecount of type integer. We have set the default value as 3. This will be updated with the value you provide at the time of execution. Testkube allows easy scalability without touching the test script.

spec:
  config:
    slavecount:
      type: integer
      default: 3

Path to the test that will run on target: Using the content, we have defined the Github repository path that has the test that will be executed on the target. In this test, the target is the Testkube website: testkube.kubeshop.io. With this, Testkube helps us version control the tests.

content:
  git:
    uri: https://github.com/kubeshop/testkube
    revision: main
    paths:
    - test/jmeter/executor-tests/jmeter-executor-smoke.jmx
container:
  workingDir: /data/repo/test/jmeter/executor-tests

‍

Configuring the slave: Testkube creates JMeter slaves as Kubernetes pods. The number of slaves helps decide the number of pods to spin. Here, we have passed the JMeter image to be used while creating the containers in the pod. Also, we have defined the command that will run when each container spins up. ‍

services:
  slave:
    use:
    - name: distribute/evenly
    count: config.slavecount
    logs: always
    timeout: 30s
    image: anasoid/jmeter:5.6-plugins-21-jre
    command:
    - jmeter-server
    - -Dserver.rmi.localport=60000
    - -Dserver_port=1099
    - -Jserver.rmi.ssl.disable=true
    readinessProbe:
      tcpSocket:
        port: 1099
      periodSeconds: 1

You can see here the minimal configuration that needs to be done which is already part of the example that loads in the Testkube Dashboard when you select to get started with an example. Testkube internally does the firewall configuration to ensure secure and seamless communication between slave and master.

‍

Running test and processing the artifacts: The final step is to define the test. In the master-slave architecture, the master initiates the test execution on the slaves using the jmeter command with arguments that allow it to connect to the created slaves We also add arguments for generating a report for our execution, which we then capture using the artifacts configuration.

steps:
  - name: Run tests
    run:
      image: anasoid/jmeter:5.6-plugins-21-jre
      shell: |
        jmeter -n \
          -X -Jserver.rmi.ssl.disable=true -Jclient.rmi.localport=7000 \
          -R {{ services.slave.*.ip }} \
          -t jmeter-executor-smoke.jmx \
          -j /data/artifacts/jmeter.log \
          -o /data/artifacts/report \
          -l /data/artifacts/jtl-report.jtl -e
    artifacts:
      paths:
      - /data/artifacts/**/*

‍
We used here the sample provided by Testkube to run distributed testing in JMeter and tweaked it to process the artifacts.

From the dropdown, select “Create” to create the test workflow.

We have set the default slave count as 3. On the prompt, enter the number of slaves you want to be created and click on “Run”. We have entered the value as 6 here.

Testkube does the automatic resource allocation and creates 6 slave nodes along with performing the test execution as shown below.

After the test execution is completed successfully, you can view the execution logs of each slave by selecting slave in Testkube Dashboard. In the image below, we have shown the execution logs of slave #1.

Viewing the artifacts

Artifacts help with the analysis of an application under heavy load. Testkube integrates with JMeter and loads the UI which gives a detailed view of the application test. In the Testkube Dashboard, click on “Artifacts”.

From the artifactsdropdown, select reportand click on index.html.

Testkube loads the JMeter Dashboard as shown below. This has a detailed view of the Application Performance Index, Requests Summary, Statistics, etc which you did not have to set up explicitly. Testkube configures all this for you, making it easier for you to gather results.

Testkube provides utilities to make testing easier for you so that you can focus on the application development better.

Executing the Test Workflow using Testkube CLI

Testkube provides the ability to connect to the Testkube account using the command line. In the previous section, we created and executed the Test Workflow from the Testkube Dashboard. We will learn here how to manage the Test Workflows using Testkube CLI. Install the Testkube CLI on your machine and configure the API Token. Once the context is set, you can view your Test Workflows and run them using the following command:

testkube run tw  --config

For our use case, here is how the execution from the CLI looks like:

$ testkube run tw distributed-jmeter-example-artifacts --config slavecount=4
Context: cloud (1.17.54)   Namespace: testkube   Org: SONALI SRIVASTAVA-personal-org   Env: SONALI SRIVASTAVA-personal-env
--------------------------------------------------------------------------------------------------------------------------
Test Workflow Execution:
Name:                 distributed-jmeter-example-artifacts
Execution ID:         66a2315f7fc8371cd690a9d3
Execution name:       distributed-jmeter-example-config-artifacts-2
Execution namespace:  testkube
Execution number:     2
Requested at:         2024-08-13 21:05:03.083826997 +0000 UTC
Status:               queued

You can view the same running in the Testkube Dashboard as shown below.

This level of automation, from creating to executing the Test Workflow, allows you to work on the test and leave the execution to Testkube. With Testkube, we could run the JMeter test with minimal configuration and view the application performance-related metrics.

Conclusion

JMeter is a commonly used tool for distributed load testing. Testkube abstracts the complexities associated with it and handles the network configuration, resource allocation, and processing of the artifacts. In this blog, we have seen, using an example available in the Testkube Dashboard, how we can easily set up the master-slave architecture for distributed testing of an application with JMeter.
Using the Testkube Dashboard, we were able to configure and execute distributed JMeter tests with so much ease, and Testkube helped us view the execution logs of each slave and an aggregated results report which helps debug issues quickly.

By leveraging the Kubernetes features, Testkube simplifies the process of configuring slaves and gives the power to dynamically set the number of slaves. Testkube also supports k6 for distributed testing. So, to standardize testing for you, we invite you to try Testkube today. Witness firsthand how Testkube simplifies and empowers your testing process with its Kubernetes-native test execution capabilities. Join our active Slack community for guidance and support.

Test Execution: A 5-Step Framework for Success

Juan Ibarra — Wed, 06 Nov 2024 21:56:09 +0000

In our previous article, we made the point that coupling test execution to CI/CD pipelines has several drawbacks that become apparent as the complexity and scale of your application or deployment infrastructure increases. Let’s take a step back now and look at the initial need solved by CI/CD in this context: running your tests, which is also known as test execution. As with many things, giving test execution some extra thought and love as you build out your infrastructure can reward you in multiples. Let’s break it down.

Test Execution in the STLC

The software testing life cycle (STLC) is a well-established step-by-step breakdown of testing activities in the software development life cycle (SDLC). At a high level, the STLC consists of the
following steps:

Requirements analysis: Understand what needs to be tested.
Test planning: Plan how the requirements will be tested.
Test case development: Write actual test cases. -** Test environment setup**: Prepare your test environments.
Test execution: Execute your tests in your test environment.
Test cycle closure: Ensure that all testing activities are completed.

Source: https://www.boardinfinity.com/blog/introduction-to-stlc-software-testing-life-cycle/

As you can see, test execution is a specific step in this life cycle, and it in itself is a rabbit hole to delve into. Let’s do just that.

A 5-Step Framework for Test Execution

Executing tests and consequently managing execution results in a scalable and efficient manner turns out to be a complex undertaking as the number of testing tools, CI/CD systems, engineers and applications grows in your organization. Let’s start by breaking down test execution into five steps to help decide how to execute tests in a way that can grow correspondingly.

Define: How will you define the execution of your tests?
Trigger: How will you trigger your test executions?
Scale: What scalability needs or constraints do you have for test execution?
Troubleshoot: How can you effectively troubleshoot your (failed) test executions?
Report: What reporting do you need to plan your (future) testing activities?

Let’s dig into each of these steps in a little more detail to help you understand what questions you might need to answer within your team.

‍

Define – How are you going to run your tests in a consistent way, considering:

From CI/CD tooling as part of your build and deploy processes?
Your existing (and future?) testing tools and versions
Input data for data-driven testing
Test orchestration: for instance, execution of multiple tests in a coordinated way, possibly across multiple/remote environments

Trigger – How will you trigger the execution of your tests?

From CI/CD tooling as part of your build and deploy processes?
Scheduled execution at regular intervals? (For example, “Run our security tests on a daily basis.”)
Based on external/internal asynchronous event triggers or webhooks? (“Re-run end-to-end tests whenever these components are updated in our infrastructure.”)
Ad hoc or manually?
Custom integrations via APIs/CLIs?

Scale – As you ramp up your testing activities, make sure you’ve assessed:

How many tests do you anticipate to be running at “peak testing time”?
Do you have shared/stateful infrastructure that is shared across tests? Do you need to constrain test execution accordingly?
Do you have very-long-running tests that either need to be: Parallelized to cut down on execution time? Scheduled asynchronously instead of run for every build?
Should tests be running inside and/or outside your infrastructure (or both)?
For load-testing specifically:
How much load do you need to simulate?
Can you use your existing/internal infrastructure?
How can you coordinate with other (testing) activities?

Troubleshoot – Troubleshooting failed tests can be a pain in a complex application infrastructure:

Are the logs and artifacts from your testing tools sufficient, or do you also need logs and metrics from the application that is under test?
Do the right people have access to logs/infrastructure to troubleshoot?
Can all troubleshooting be done in one place or are there multiple points of access?
For how long do you need to keep results around?
Do logs or artifacts contain sensitive information? Do they need to be stored securely?

Report – Ask yourself:

What metrics do you need to track over time, and at what granularity? For example pass/fail ratios, total number of tests, etc.
Could or should you aggregate results from different test executions and testing tools into common reports?
Access control: Do the right people have access to reports?
Can reports/metrics be analyzed by required dimensions, such as team/application, etc.?
Do test execution results need to be pushed to external systems? For example: reporting, incident management, issue tracking
How should reports be distributed internally and be accessed over time — ephemeral/long-lived URLs? PDFs? etc.

Test Execution Assessment Criteria

Apart from the somewhat tactical approach to test execution outlined above, we can define a number of criteria that need to be assessed and planned for to scale accordingly with the needs of your team and your application.

Consistency – Getting consistent test results is key to building trust in quality metrics and downstream activities, and to that end, your test execution environments should be as homogenous as possible, given the context of your applications.
Decoupling – Test execution should not be tightly coupled to any other specific framework or pipeline in your infrastructure. The need to run tests will shift both strategically and tactically over time, and your tests should be available for execution whenever needed.
Centralization – While your tests might execute in multiple places in your infrastructure, managing these executions and their results in one place gives you a holistic view of your testing activities, making it possible to assess, analyze and control test execution consistently as your testing scales with your applications and infrastructure.
Integration – Test execution commonly needs to be integrated — but not tightly coupled! — with your existing workflows and pipelines.

The execution of tests needs to be triggerable from a variety of sources.
Notifications of test executions or failures needs to be integrated into collaboration platforms and incident/issue tracking.
Test results or metrics might need to be captured by external monitoring or reporting tools.

Scalability – Running tests at scale is one of the most common challenges for teams embracing a proactive approach to test execution.

The need to scale individual tests horizontally to improve execution times or cover more test scenarios
The need for multiple teams to run their tests using a constrained resource (infrastructure, shared database, etc.)
The need to scale load tests to generate the required load to ensure the performance and stability of your applications and infrastructure

Security and Access Control – This has several aspects:

Who should be able to run tests, see results, etc.?
If your infrastructure needs to be configured specifically for test execution, does that have any security implications?

Charting the Course for Test Execution

Neither of the above sections is meant to be exhaustive or conclusive in their respective approach. Each application infrastructure is unique, and so will your team’s needs be on how to run tests. The main point is to make you think about test execution further than “run my playwright test in Jenkins” — as that will surely hit a dead end and stop you from scaling your testing activities in line with the evolution of your applications.
A hands-on approach could be:

Break down your testing activities into the different steps of the STLC. How are you performing each of these steps? Who is responsible? What needs do you have?
Break down test execution into the five steps above and ask yourself again: What are your needs, who is responsible, etc..
Factor in the test execution assessment criteria outlined above into your test execution strategy. Make sure you have at least discussed them all, even if your course of action is “ignore.”
Make sure the right people are involved in all of these discussions (in no specific order):
QA leads/managers
DevOps/platform engineering
System architecture (if needed/applicable)
Product ownership (if needed/applicable)

Testkube for Test Execution

Perhaps not surprisingly, I’m writing this article not only to share insights into test execution, but also to show you how Testkube can help.

Put simply, Testkube is an orchestration platform for test execution in line with many (but not all) points discussed above. The five steps outlined for test execution above are cornerstones for how to work with Testkube:

Define your test execution using a powerful Test Workflow syntax that supports any testing tool or script you might be using.
Trigger your tests however you might need to; CI/CD, events/webhooks, CLI, API, etc..
Scale any testing tool horizontally or vertically to ensure your applications are tested consistently and at scale.
Troubleshoot test results using Testkube results and log analysis functionality.
Report on test results over time to guide you in your testing efforts and activities.

And although Testkube can’t solve for every issue discussed above, it provides a grounded starting point. Try it out at testkube.io/get-started. There are both open source and cloud versions available.

Leveraging Testkube for Complex System Testing

Juan Ibarra — Tue, 29 Oct 2024 16:35:00 +0000

Applications today span multiple servers and services which requires a multifaceted approach to ensure reliability and performance. Testing such distributed applications has its own challenges due to their inherent complexity.

To perform comprehensive testing of such applications, you must run various functional and non-functional tests. Moreover, different load, API, and UI tests should preferably be executed simultaneously to ensure consistent system behavior under complex usage scenarios and provide a thorough validation of your system before it goes into production.

However, without proper tools, managing multiple tests simultaneously can be difficult. Coordinating with different types of tests, analyzing their results, and maintaining consistency is difficult.

This learn article will show how Testkube can help you create custom Test Workflows combining multiple tests for seamless system testing.

Challenges With System Testing

Performing a system test for your application is crucial to ensure it always works as expected. This often involves running different tests, including unit, functional, and non-functional tests. Managing these tests and integrating multiple tools to replicate a real-world scenario is challenging. Let us look at some other challenges with system testing.

Complexity: As the number of tests and their types increases, managing and analyzing them becomes complex. Furthermore, when each test is to be performed by a different tool, it is even more complex and demands advanced tools.
Resource Optimization: Running multiple tests simultaneously means increased resource usage and requires careful orchestration of tests and allocation of resources.
Logs & Analysis: With so many tests running simultaneously, getting a complete picture of the test outcome can be difficult. Teams struggle to collate results from different tests and environments without proper tools. This inefficiency can lead to missed bugs and affect the application quality.

You need a versatile tool to orchestrate and manage multiple tests to overcome the challenges of performing comprehensive system testing with different tests. Enter Testkube.

System Testing with Testkube

Testkube is a Kubernetes-native testing framework that makes end-to-end testing in Kubernetes a breeze. Using Testkube, you can orchestrate and automate complex testing workflows using different testing tools, all from a single intuitive UI.

Benefits of Using Testkube For System Testing

Testkube enhances system testing by integrating seamlessly with Kubernetes, allowing teams to leverage its full potential for running your tests:

Orchestrate Complex Test Workflow: Test Workflows allow you to define complex workflows that enable sequential and parallel test executions to mimic real-world scenarios. Testkube does so without complex scripting and facilitates the creation of detailed test workflows, allowing for better control and customization of test executions.
In-Cluster Test Execution: Unlike other testing frameworks and tools, Testkube executes your tests within the Kubernetes clusters, ensuring a secure and production-like environment and thus improving the reliability of your test outcomes.
Leverage your own infrastructure: You can run Testkube on your existing infrastructure, which helps maintain consistency across test and production environments and decreases infrastructure costs.
Integration with Testing Tools: Testkube integrates with all popular testing tools, including k6, Cypress, and Postman, to name a few. Furthermore, Testkube allows you to combine tests for any of these tools in any way required to perform your system tests.

With these out-of-the-box benefits, Testkube simplifies the orchestration of complex system testing scenarios with a drastically shorter implementation time than other approaches like CI/CD-based solutions or DIY frameworks.

Building A System Test Workflow Using Testkube

With Testkube, you can not only create standalone test workflows but also combine different test workflows that use different tools to run sequentially or in parallel. To put it simply, you can run a system test, load test, and API test all at the same time.
Let’s have a look at how this can be done with Testkube. We’ll create a similar scenario where we’ll have a cURL test to login a user as the first step. The test fetches the access token and passes it on to the next step. Next, we configure Cypress, k6, and Postman tests to run parallelly. All these tests will use the token to perform the tests.

To summarize, we have created the following test workflows:

cURL - to authenticate a user, fetch the access token, and save it in a file.
Cypress - to perform end-to-end testing of the application.
Postman - to perform API testing for the application.

Pre-requisites

Get a Testkube account.
Kubernetes cluster - we’re using a local Minikube cluster.
Testkube Environment with Testkube Agent configured on the cluster.
Some Test Workflows used to test all aspects of your application - we’re using Cypress, Postman, and distributed k6 test workflows. Check out our documentation to learn how to create a test workflow.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured and some Workflows ready for execution.

This video below provides a visual guide to the concepts we'll be exploring in the following sections.

Creating a System Test Workflow

Login to Testkube, navigate to the Test Workflows tab for your local environment, and click the “Add a new test workflow” button.

This will provide you with four options:

Create from Wizard - use the wizard to create a Test Workflow.
Start from an example - use existing k6, cypress, and playwright examples.
Combine existing workflows - use with existing workflows.
Import from yaml - import your own Test Workflow.

We’ll choose the “Combine existing workflows” option to create this custom workflow and choose the existing test workflows we’ve created.

In the new tab, provide the name for the test workflow and click on the “Add the first workflow” button. Testkube provides an easy-to-use and intuitive interface for creating test workflows. This will give you a list of test workflows that you have already created, and you can choose one from the list. We’ll choose a cURL test workflow that we created, which tests login.

After adding your first test workflow, Testkube will allow you to add more test workflows to execute in sequence or parallel. You can click the “+” buttons on either side of the current test workflow to add a new test workflow. Let us add a “cypress-workflow.”

Similarly, let’s add “distributed-k6” and “postman-example” test workflow, but parallelly so that we have cypress, k6, and postman test workflows execute in parallel.

Finally, you’ll have something like this - a cURL test followed by cypress, k6, and postman that will run parallelly. Click on the “Next” button to view the spec file it generates.

Passing Parameters Between Tests

A common need in System testing is to reuse output from one test as input to other tests. In this case, our initial test authenticates a user. The resulting authentication token is then passed to all subsequent tests to ensure those are all running as the same user account.

Let’s do just that in our generated Workflow: we will modify the spec generated and add an initial cURL test step that adds “token” as the config variable that will be passed by the cURL test to the other tests. Below is the updated spec file.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: end-to-end-test-workflow
 namespace: testkube
spec:
 container:
   env:
   - name: USERNAME
     value: emilys
   - name: PASSWORD
     value: emilyspass
 steps:
 - name: curl-emilys
   run:
     image: curlimages/curl:8.7.1
     shell: |
       curl -s -X POST https://dummyjson.com/auth/login \
        -H "Content-Type: application/json" \
        -d '{
          "username":  "'"$USERNAME"'",
          "password":  "'"$PASSWORD"'",
          "expiresInMins": 30
        }' | grep -o '"token":"[^"]*"' | sed 's/"token":"\([^"]*\)"/\1/' > /data/http_output.txt
 - execute:
     workflows:
     - name: cypress-example
       config:
         token: '{{ file("/data/http_output.txt") }}'
     - name: postman-testkube
       config:
         token: '{{ file("/data/http_output.txt") }}'
     - name: distributed-k6
       config:
         token: '{{ file("/data/http_output.txt") }}'
status: {}

Let us understand what the above spec file does:

It configures the “USERNAME” and “PASSWORD” as environment variables for the user to perform authentication.
The cURL test defines the end point for authentication and stores the received token in a file named http_output.txt
Under workflows, we configure `token: '{{ file("/data/http_output.txt") }}' that takes the token from http_output and passes it as a variable to the other test workflows.

Executing a System Test Workflow

The spec file lists down all the test workflows that we chose from the UI and the order of their execution. Plus it now has the config parameters that we added. Click on the “Create and Run” button to create and execute the test workflow. You’ll see that your custom workflow has started executing.

You can click on an individual workflow to see its status. For instance, we can check our cypress-example’s execution and see that it has fetched the token from the cURL test and executed the steps successfully.

Similarly, we can check the status and logs for each of the test workflows. Once the entire test workflow has finished executing, you’ll see the status in the UI. In this case, we had a failed Postman test workflow.

That’s how you can create end-to-end system test workflows by combining multiple test workflows to reproduce a realistic, close-to-product scenario to test your application.

Summary

One of the most difficult things in testing is to test in production-like environments. It’s difficult to create complex test scenarios that replicate a real-world scenario. Furthermore, if you use different testing tools, it’s nearly impossible to configure all of them to work in tandem and perform comprehensive system tests on your application.
That’s where Testkube shines by allowing you to create your own test workflows through an intuitive UI and execute those workflows from inside your Kubernetes cluster(s). Irrespective of what testing tools your test workflows use, you can combine all of them to run sequentially or parallelly to perform system testing to test your application.

We would love to hear all about the custom test workflows that you created using Testkube. If you face any issues, remember that the entire Testkube team, plus a vibrant community of fellow Kubernetes testers, are on Slack. We’re just getting started in building the most comprehensive (and friendliest!) cloud-native testing framework for Kubernetes so feel free to follow us on Twitter @testkube_io.

A Guide to Scalable and Heavy Load Testing with k6 + Testkube

Juan Ibarra — Mon, 28 Oct 2024 13:37:06 +0000

Every team aspires for their application to be flexible, scalable, and capable of handling user requests efficiently. While functional testing validates the application’s features, load testing assesses how the application performs under stress.

Modern tools have simplified functional testing, but conducting distributed load tests to evaluate an application’s resiliency remains complex. It requires the ability to adjust tests to simulate production-like scenarios dynamically.

In our previous post, we discussed the benefits of using Testkube over the k6 operator for scaling load testing. In this post, we’ll cover step-by-step how to scale your k6 load tests using Testkube to ensure your application can handle heavy loads seamlessly.

Distributed K6 Testing with Testkube

We discussed distributed load testing, k6, and how the k6 operator stacks up against Testkube in this blog post. Let us apply what we have learned and create a distributed k6 test using Testkube. We’ll create a Test workflow for running distributed k6 load tests and configure it to run in parallel.

Pre-requisites

Get a Testkube account.
Kubernetes cluster - we’re using a local Minikube cluster.
Testkube Agent configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

We've created a video tutorial that walks you through the process. This video provides a visual guide to creating and running your k6 distributed test workflow using Testkube. You can watch it for a quick overview or follow along as you implement the steps yourself.

Creating a Test Workflow

Navigate to the Test Workflows tab and click on “Add a new test workflow”

This will provide you with three options:

Create from Wizard - use the wizard to create a Test Workflow.
Start from an example - use existing k6, cypress, and playwright examples
Combine existing workflows - use with existing workflows.
Import from yaml - import your own Test Workflow.

We’ll choose the “Start from an example” option to create this workflow. In the examples tab, choose “Parallel Execution” and then select “Distributed k6”. This will show a sample yaml code that will create a distributed k6 test. We’ll look at the yaml file at the end of this document.

Click on Create to create the Test Workflow.

Executing Test Workflow

Now that the Test Workflow is ready, click “Run Now” to execute the workflow. It will prompt you to provide values for parameters like the duration, number of virtual users, and workers. Provide the values as per your requirement and click the Run button.

The Test Workflow will start executing based on the parameter values that you’ve provided.

Based on the number of workers provided, you can see that k6 load tests are running in parallel. You can click on any of those to view the detailed steps.

Navigate to the “Artifacts” tab to check the logs and reports generated from the test. You’ll see that logs and reports are generated for each worker. Click on any of the reports to view them in a new tab.

Below is what a typical k6 distributed test report looks like.

This was a simple demo of creating and executing distributed k6 load tests using Testkube. To take advantage of test workflows more, you can create custom workflows and import them to Testkube.

Let us now take a look at the test definition that was generated for the Test Workflow

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: distributed-k6
 namespace: testkube
 labels:
   docs: example
spec:
 config:
   duration:
     type: string
     default: 5s
   vus:
     type: integer
     default: 10
   workers:
     type: integer
     default: 3
 content:
   git:
     uri: https://github.com/kubeshop/testkube
     paths:
     - test/k6/executor-tests/k6-smoke-test.js
 steps:
 - name: Run test
   parallel:
     count: config.workers
     transfer:
     - from: /data/repo
     use:
     - name: distribute/evenly
     container:
       workingDir: /data/repo/test/k6/executor-tests
       env:
       - name: K6_SYSTEM_ENV
         value: K6_SYSTEM_ENV_value
       - name: K6_WEB_DASHBOARD
         value: "true"
       - name: K6_WEB_DASHBOARD_EXPORT
         value: /data/k6-test-report.html
       resources:
         requests:
           cpu: 128m
           memory: 128Mi
     paused: true
     run:
       image: grafana/k6:0.49.0
       shell: |
         k6 run k6-smoke-test.js \
           -e K6_ENV_FROM_PARAM=K6_ENV_FROM_PARAM_value \
           --vus {{ config.vus }} \
           --duration {{ shellquote(config.duration) }} \
           --execution-segment {{ index }}/{{ count }}:{{ index + 1 }}/{{ count }}
     artifacts:
       workingDir: /data
       paths:
       - '*.html'
status: {}

In the above file:

The configuration specs define the number of workers, virtual users, and duration of the test. The document also specifies the environment variables for k6.
The test source is a file in a GitHub repository. Refer to the k6 test referred to in this definition.
Resource limits are provided for CPU and memory, along with the image details.
Test execution command is provided along with artifacts configuration to collect the logs.

Summary

There’s no second thought that load testing is critical to test the resilience of your application. Distributed load testing provides you with a more realistic and production-like environment. Tools like k6 make load testing simpler.
Regarding Kubernetes, Leveraging k6 and Testkube can significantly enhance distributed load testing. While the k6 Operator offers robust automation, it requires deep Kubernetes expertise. Testkube simplifies the process with flexible test triggering, Git integration, distributed parameterization, and support for provisioning dependent services.

Get started with Testkube today, or visit the Testkube documentation to learn more about running distributed tests in Testkube using other testing tools. Feel free to post a note in our active Slack community if you struggle with anything.

Comparing the K6 Operator vs Testkube for Load Testing

Juan Ibarra — Mon, 28 Oct 2024 13:32:35 +0000

Load testing is crucial to understanding how an application performs under stress and ensuring it can handle high-traffic loads. Traditional load testing methods fall short regarding scalability and simulating a production-like setup.

This is where distributed testing comes in. It involves mimicking simultaneous users by spreading tests across multiple machines and creating a more realistic setup. This helps to make your application resilient by identifying points of failure.

Running distributed load tests also has its own limitations:

Resource limitation: Load testing is constrained by the hardware limitations of the testing infrastructure, making it difficult to simulate concurrent users.
Static Test Configuration: The lack of flexibility with test scripts makes adapting to different production-like environments difficult.
Result aggregation: When running distributed load tests, combining results from multiple machines can be complex and error-prone.

These challenges, limited scalability, and distributed parametrization make the process more complex. That’s where tools like k6 streamline the load-testing process. With the k6 operator, distributed load testing with Kubernetes is easier and more efficient.

In this blog post, we’ll examine k6 and the k6 operator and analyze k6 vs Testkube for distributed load testing.

k6 & k6 Operator

k6 is an open-source load testing tool by Grafana. It offers developer-friendly APIs, scripting support using JavaScript, and various other configurations that allow users to perform automation-friendly load testing.
What sets k6 apart from other load-testing tools is its ability to efficiently generate massive loads from a single machine using all CPU cores. Their documentation suggests that if you need less than 300,000 requests per second, you should be good with running k6 on a single machine.

However, there are scenarios where you want to emulate multiple machines running a single test. You want to test your application’s load by generating traffic from different IP addresses, or your single instance cannot create the needed load. If you’re using Kubernetes in your organization, you can use the k6 operator to run distributed load tests.

The k6 Operator is designed to run on a Kubernetes cluster. It leverages Kubernetes' orchestration capabilities to scale and manage load tests. It further automates the deployment, execution, and scaling of k6 tests, reducing manual intervention. However, it is difficult to use in certain scenarios, let us look at some of the challenges.

Requires Kubernetes Expertise: k6 is primarily designed for DevOps teams. Thus, a good understanding of Kubernetes is needed. This is a challenge for teams that lack Kubernetes expertise, which makes it difficult to manage tests.
Direct Cluster Access Needed: You need direct access to the cluster to use the k6 operator, which can pose security risks and operational challenges.
Triggered by Custom Resources: Your load tests in k6 can only be triggered using custom resources within the clusters. This further complicates the testing process, requiring additional Kubernetes-specific knowledge to define and manage tests.
No Git Support: Out of the box, there’s no support for Git, meaning you cannot manage your k6 tests within Git repositories.
No UI: The absence of a UI makes it harder to visualize and manage tests. It also hinders the troubleshooting process, as users must learn CLI commands and review logs and configuration files, which can be tedious and error-prone.

By addressing many of these challenges, Testkube makes distributed load testing using K6 much easier, more accessible, and more efficient.

Distributed Testing Using Testkube

Testkube is a Kubernetes-native testing framework that automates and manages end-to-end test execution within your Kubernetes clusters. It allows you to bring in your testing tool, including k6, enabling you to perform efficient testing seamlessly. Using Testkube, you can orchestrate complex test scenarios using Test Workflows and manage test configurations and resource utilization, all from an intuitive UI. Read more about Testkube.

Benefits of Using Testkube For Distributed Load Testing

Testkube enhances distributed load testing by integrating seamlessly with Kubernetes. It offers several benefits:

Kubernetes Job Scheduler: Testkube leverages the Kubernetes job scheduler to manage parallel test executions. This ensures efficient resource allocation and, thus, optimal test performance. It also helps simulate a high number of concurrent users and makes it scalable.
Test Workflow: Test Workflows allow you to define complex test workflows that enable load generation and parallel test executions. You can configure diverse user behavior and conditions to stress-test your application.
In-Cluster Testing: Unlike other testing frameworks and tools, Testkube executes your tests within the Kubernetes clusters, ensuring a secure and production-like environment and thus improving the reliability of your test outcomes.
Leverage your own infrastructure: You can run Testkube on your existing infrastructure, thus eliminating the need for external testing environments. This helps maintain consistency across test and production environments and decreases infrastructure costs.

k6 Operator vs Testkube

Let's examine the K6 operator and Testkube closer to understand their differences and why Testkube is a better option for running distributed K6 tests at scale.

Tests can be triggered in multiple ways: Unlike the k6 operator, which relies solely on custom resources to initiate tests, Testkube allows tests to be triggered manually or through API calls, CLI commands, Cronjobs, and CI/CD pipelines, providing more flexibility.
Git Integration: Testkube integrates with Git, enabling version control and collaborative management of test scripts.
Parametrization: Testkube allows you to adjust test parameters across multiple nodes dynamically. This flexibility enables more comprehensive testing for different scenarios.
Provisioning dependent services: Testkube allows you to provision dependent services required for your tests within the Kubernetes cluster. This ensures that all the necessary components are available and correctly configured.
Combine with other tests: You can combine your distributed load tests with functional and integration tests within the same workflow. This holistic approach provides a more thorough testing of your application.
Intuitive UI: Testkube provides an intuitive UI that everyone on the team can use. It allows teams to define, execute, and manage tests, view logs, and test artifacts in a single pane.

Here’s a comprehensive list of differences between k6 and Testkube

Feature	Testkube	k6 Operator
Test Triggering	Multiple ways (API calls, CLI commands, Cronjobs, and CI/CD pipelines)	Primarily through custom resources
Git Integration	Yes	Not specified
Parametrization	Easy, dynamic configuration of test parameters	Not specified
Provisioning Dependent Services	Supported within Kubernetes cluster	Not specified
Test Combination	Can combine distributed load tests with functional and integration tests	Primarily focused on load testing
User Interface	Intuitive UI for team collaboration	CLI
Test Management	Single pane for defining, executing, and managing tests	Limited to Kubernetes custom resources
Logging, Artifacts & Reporting	Centralized view of logs and test artifacts, along with comprehensive reporting through UI	Basic logging and reporting through Kubernetes may require additional tools.
Flexibility	Higher flexibility for various testing scenarios	More focused on load testing scenarios
Learning Curve	Easier due to UI and integration features	Requires Kubernetes expertise
Test Tool Support	It supports K6 and any other testing tool such as Jmeter, Artillery, Playwright, Postman, etc.	Supports only k6 tests
Kubernetes Native	Yes	Yes
Community Support	Yes - Slack	Yes - Slack

By providing advanced features and leveraging Kubernetes’ capabilities, Testkube offers a more versatile and comprehensive approach to distributed load testing than the k6 operator.

Summary

In this blog post, we examined the K6 Operator and Testkube for running distributed K6 tests in Kubernetes environments. Leveraging k6 and Testkube can significantly enhance distributed load testing. While the k6 Operator offers robust automation, it requires deep Kubernetes expertise. Testkube simplifies the process with flexible test triggering, Git integration, distributed parameterization, and support for provisioning dependent services.
Get started with Testkube today at www.testkube.io/get-started, or follow our step-by-step tutorial for scaling your load testing with k6 and Testkube. Please visit our documentation for detailed guidance on using Testkube for distributed testing with k6 and more information on the features. Should you have any questions or need assistance, do not hesitate to contact us in Slack or email me at bruno@kubeshop.io.

Scaling Cypress Tests: Parallelise your End-to-End Tests with Testkube

Juan Ibarra — Mon, 28 Oct 2024 13:20:56 +0000

Users today want applications that are snappier and have an intuitive interface. Building and shipping such applications require thorough testing features to ensure they work as expected.

One of the most popular end-to-end testing tools for this is Cypress. Its rich feature set and developer-friendly APIs make testing your entire application easy. However, running tests in sequence is time-consuming for large, complex applications.

Hence, teams turn to test parallelization, which allows them to run multiple tests simultaneously. In this blog post, we’ll look at how test parallelization works in Cypress. We’ll also explain why parallelization is critical for end-to-end testing and how Testkube helps with test parallelization for Cypress, providing an example.

The Need For Parallelization In Testing

Test parallelization runs multiple tests simultaneously across different environments. Instead of executing tests sequentially, one after another, parallel testing divides the test suite into smaller groups that can be run concurrently.
Here’s why parallelization in testing is necessary:

Faster Results: Test parallelization reduces the time taken to execute all the tests, thus reducing the overall execution time and allowing teams to get faster feedback.
Resource Efficiency: By running tests in parallel across different environments, you can optimize your resource utilization and better use the available resources.

Test Parallelization in Cypress

As mentioned earlier, Cypress has become a popular choice for end-to-end testing. To enhance its capabilities, Cypress provides Cypress Cloud with an intuitive dashboard with additional functionalities like test parallelization and result analytics. These features allow developers to speed up test execution.
Below are some salient features of running tests in parallel in Cypress

Multi-browser Testing: Cypress allows tests to be run concurrently across different browsers, enhancing cross-browser coverage.
CI Integration: Cypress easily integrates with CI tools and allows you to execute tests on different environments in an automated manner.
Dashboard Service: Cypress provides a dashboard that helps you manage and visualize parallel test runs.

Having said that, running tests in Parallel in Cypress can be complex. Managing test data, environment setup, and teardown across parallel instances can be difficult. Further, in their free tier, the limit on the number of tests you can execute in parallel and the results you can collect is relatively low.

Cypress With Testkube

Testkube is a Kubernetes-native testing framework that allows you to create testing workflows in a declarative, version-controlled manner. It allows you to plug in any testing tool and leverage the power of Kubernetes.
Key benefits

Simplified Test Workflow Creation: Without complex scripting, Testkube facilitates the creation of detailed Test workflows that allows for better control and customization of your Test Executions. Refer to our Test Workflows documentation to learn more.
Scales your Testing Tools by leveraging Kubernetes: Testkube integrates with any testing tool, including Cypress and Playwright, and allows you to leverage your own infrastructure to run tests at scale!
Single Pane of Glass: Testkube gives you a simple dashboard that allows you to observe and troubleshoot all of your tests.

Further, Testkube integrates with existing CI/CD pipelines, enhancing end-to-end testing capabilities. Furthermore, it provides a straightforward process for incorporating custom testing tools, enabling native execution on Kubernetes with minimal setup. Read more about Testkube.

Key advantages of Testkube for Cypress test parallelization:

No dependency on Cypress libraries: It doesn't rely on any Cypress-specific libraries that could be subject to blocking or restrictions.
Flexibility: Testkube allows users to run tests without being tied to specific versions of Cypress or worrying about compatibility issues.
Container-based execution: Testkube runs tests inside containers, providing isolation and consistency.

Thus, Testkube provides stable and reliable options for teams looking to parallelize their Cypress tests.

Cypress Test Workflow using Testkube

Let's see how we can run Cypress tests in parallel using Testkube. We’ll create a Test workflow for Cypress tests and configure it to run in parallel.

Pre-requisites

A Testkube account, either on prem or in the cloud.
Kubernetes cluster - we’re using a local Minikube cluster.
Testkube Agent configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Creating a Test Workflow

Navigate to the Test Workflows tab and click on “Add a new test workflow”

This will provide you with three options:

Create from scratch - use the wizard to create a Test Workflow.
Start from an example - use existing k6, cypress, and playwright examples
Import from yaml - import your own Test Workflow.

We’ll choose the “Import from yaml” option to create this workflow. Below is the yaml file used to create the Test workflow for running Cypress tests in parallel.

apiVersion: testworkflows.testkube.io/v1

kind: TestWorkflow

metadata:

  name: cypress

  namespace: testkube

spec:

  content:

    git:

      uri: https://github.com/kubeshop/testkube

      revision: main

      paths:

      - test/cypress/executor-tests/cypress-13

  container:

    image: cypress/included:13.6.4

    workingDir: /data/repo/test/cypress/executor-tests/cypress-13

  steps:

  - name: Run tests

    parallel:

      maxCount: 3

      shards:

        testFiles: 'glob("cypress/e2e/**/*.js")'

      description: '{{ join(map(shard.testFiles, "relpath(_.value, \"cypress/e2e\")"), ", ") }}'

      transfer:

      - from: /data/repo

      fetch:

      - from: /data/artifacts

      container:

        resources:

          requests:

            cpu: 1

            memory: 1Gi

      run:

        args:

        - --env

        - NON_CYPRESS_ENV=NON_CYPRESS_ENV_value

        - --config

        - '{"video":true,"screenshotsFolder":"/data/artifacts/screenshots","videosFolder":"/data/artifacts/videos"}'

        - --spec

        - '{{ join(shard.testFiles, ",") }}'

        env:

        - name: CYPRESS_CUSTOM_ENV

          value: CYPRESS_CUSTOM_ENV_value

    artifacts:

      workingDir: /data/artifacts

      paths:

      - '**/*'

The above file creates a Cypress Test workflow and configures it to run 3 tests in parallel, specifying other details like resource requirements and artifacts folders.

Paste the yaml file's contents and click Create & Run to create and run the test workflow. This will trigger the Test workflow, which you’ll see on the dashboard.

After the Test workflow has finished, it will update the status and give you a link to view the artifacts, such as logs, screenshots, and videos

In this case, it has generated logs and videos. Clicking on any artifacts will open a new tab/window to view them.

This was a simple demo of creating and running Cypress tests in parallel using Testkube. To take advantage of test workflows more, you can create custom workflows and import them to Testkube.

Cypress & Its Alternatives

Cypress offers features like test parallelization and analytics that help teams quicken their testing process. While initially expensive, it now has a free tier with limitations on the number of parallel tests you can run.
To overcome this barrier, open-source alternatives like SorryCypress and managed solutions like Currents.dev emerged, offering unlimited parallelization and features previously exclusive to Cypress's enterprise plans.

However, starting with version 12, Cypress blocked projects using the 'cypress-cloud' module, impacting these third-party services. SorryCypress now only works with older Cypress versions, while Currents.dev has ended its official support for Cypress. This change has disrupted many teams' testing workflows.

However, with Testkube, you can still run your Cypress tests in parallel without worrying about Cypress blocking Testkube.

Summary

To summarize, Cypress is a versatile and feature-rich tool for end-to-end testing. With Testkube, you can run Cypress tests in parallel and leverage its advanced test orchestration capabilities to perform end-to-end testing.
Further, third-party services like SorryCypress and Currents.dev are no longer helpful as they don’t work with the latest version of Cypress or are forcing their users to use Playwright. But with Testkube, you’re guaranteed an uninterrupted testing experience.

Check out the Testkube website to learn more about Cypress or other testing tools you can integrate and get started with Testkube today. Feel free to post a note in our active Slack community if you struggle with anything.

‍

Testing in KinD: Using Testkube with Kubernetes in Docker

Juan Ibarra — Mon, 21 Oct 2024 19:58:59 +0000

Docker is one of the greatest advancements in application deployment. It revolutionized containerization, enabling developers to package their applications and dependencies into mobile units. As the number of containers grew, orchestrating and managing them became difficult.

Enter Kubernetes, which helps you manage containers at scale and comes with scaling, healing, and fault tolerance capabilities.

Many teams use docker-in-docker to extract most of the tool. However, as discussed in our previous post, that approach has limitations and complexities. Instead, using Kubernetes in Docker is a better alternative and changes how developers interact with Kubernetes.

In this post, we’ll examine KinD, its features, real-world use cases, and how you can use Testkube in KinD for efficient and effective testing.

Kubernetes in Docker - KinD

As the name suggests, Kubernetes in Docker, KinD allows you to run Kubernetes clusters locally using Docker. Each Kubernetes node is represented by a Docker container, which uses Docker’s underlying networking and storage capabilities to simulate a realistic Kubernetes setup.

Salient features of KinD:

Run Kubernetes clusters locally using Docker containers.
Each Kubernetes node is a Docker container, ensuring it’s lightweight and portable.
Supports custom cluster configurations that allow for tailored setups.
Suitable for ephemeral setups, ensuring isolation and reproducibility.

Using Kubernetes in Docker brings many benefits while streamlining the development and deployment process. Some of the benefits are:

Cost Effective: KinD helps reduce costs by eliminating the need for resource-intensive infrastructure.
Development Consistency: Developers can ensure consistency across their development setups and replicas of production setups.
Simplified CI/CD Integration: KinD seamlessly integrates with CI/CD pipelines, providing ephemeral clusters.
Simplified Testing: Testing is more scalable and efficient with KinD, allowing developers to spin up isolated Kubernetes clusters easily.

KinD Use Cases

It's important to understand the specific use cases where KinD is helpful before you adopt it for your development and deployment needs. Below, we explore real-world use cases where KinD shines and helps improve your overall workflow efficiency.

Local Development Environment

Developers can use KinD to create local Kubernetes clusters that mimic the production setup. This helps them with easy testing and debugging. For instance, developers working on a microservices application can use KinD to create a local setup where each microservice can be tested in isolation, ensuring that changes in one service don’t affect the other.

CI/CD - Provision Ephemeral Environments

When working with CI/CD pipelines, you can leverage KinD to provision ephemeral clusters for pipeline runs. Since these clusters can be created and destroyed dynamically, they are suitable for use with CI/CD pipelines. Further, these clusters can be configured to mirror your production setup, making them a good candidate for automated integration tests.

This ability of KinD allows your development workflow to be more reliable and efficient while ensuring that you’re testing in a production-like setup, which leads to a robust and stable application.

Running Testkube in KinD

Testkube is designed to make Kubernetes testing efficient. It’s a framework that allows you to bring any testing tool and make it ‘Kubernetes-native’ to take full advantage of Kubernetes. It allows you to treat your tests as Kubernetes resources, making it more straightforward to manage them.
Running Testkube in KinD ensures consistency and reproducibility. Integration with CI/CD pipelines further streamlines the testing process, allowing for continuous testing and validation of code changes before deployment. It can easily integrate with CI/CD tools like GitHub Actions, Jenkins, and Azure DevOps, to name a few, allowing you to create automated workflows.

Using Testkube in KinD has a lot of benefits:

Testkube automates the entire testing process and end-to-end saving efforts from developers.
Testkube ensures consistency and reproducibility with KinD clusters, yielding more reliable test results.
Integrating with CI/CD tools, Testkube easily integrates with your existing workflows for creating ephemeral environments.

Further, Testkube can be used effectively in different scenarios:

Self-Hosted Control Plane

Testkube can be deployed fully on-prem within a KinD cluster. You can install Testkube agents in your local environments to talk with your on-prem Tetskube control plane. This cost-effective solution allows developers to develop and manage their tests locally. This setup ensures that your testing environment is secured and within the local infrastructure.

Testkube Hosted Control Plane

When working with CI/CD pipelines, Testkube can be deployed as agents within a Kind cluster. To orchestrate and manage tests, these agents communicate with Testkube’s hosted control plane. Such a setup allows for automated integration and deployment testing as Testkube agents ensure seamless integration into the CI/CD workflow.

More details about Testkube offerings can be found here.

Test Workflows Using Testkube in KinD

Test Workflows provide a comprehensive, purpose-built solution for managing the full lifecycle of running tests against your applications and their components. These are stored as custom resources in your cluster, making them easy to manage using existing Kubernetes tools and GitOps pipelines. Hence, when running Testkube in KinD, you’ll benefit from creating Test Workflows.
After setting up Testkube in your KinD cluster, you can configure a Testkube agent to talk to the dashboard. Once this is done, you can start creating Test Worfklows.

As mentioned earlier, there are two ways to use Testkube in KinD: one using the Testkube dashboard, which is suitable for self-hosted control planes, and the other using the Testkube CLI, which is suitable for your CI/CD pipeline.

Using the Testkube Dashboard

Using an existing example is the easiest way to create a Test Workflow using the Testkube dashboard and understand how it works. Testkube provides some prebuilt Test Workflows for k6, PlayWright, and Cypress.

Choose any one of them from the wizard, provide a name, and you’re ready to execute your Test Workflow.

Testkube configures everything else for you, from resource allocation to artifacts and log collection. All you need to do is trigger your Test Workflow.

Once your Test Workflow is executed, you can examine the artifacts collected by Testkube that provide more details about the test. In this case, the playwright test report is generated and captured by Testkube.

Using the Testkube CLI

If you want to run Test Workflows in KinD for your CI/CD pipeline, then using Testkube CLI is suitable. After configuring the environment on Testkube, you can install Testkube CLI to perform operations using CLI.

Once you have configured Testkube CLI, you create a Test Workflow. We will create a playwright Test Workflow using the following command:

testkube create testworkflow --name playwright --file playwright.yaml

We use a playwright.yaml file to create the Test Workflow.

Run the Test Workflow after it is created.

testkube run testworkflow playwright -f

After executing the Test Workflow, you can get the results using the execution execution id from the previous command's output.

kubectl testkube get twe 6683b881923ebd28cd944418

Similarly, you can create advanced Test Workflows from scratch for different testing tools and scenarios. We have detailed guides on creating Test Workflows for Cucumber, Rest Assured, and more in our Testing in Kubernetes Handbook.

Summary

Leveraging KinD for local Kubernetes development and testing offers multiple benefits, including cost-effectiveness, development consistency, and simplified CI/CD integration. These advantages streamline your testing and development efforts and enhance the overall workflow.
By using Testkube in KinD, you can further enhance your testing process with flexible test orchestration and leverage the full benefits of running Kubernetes in Docker. Try out Testkube today and join our Slack community to connect with fellow developers, share insights, and receive support.

Using Testkube for Test Orchestration in Tekton Pipelines

Juan Ibarra — Mon, 30 Sep 2024 20:00:27 +0000

Introduction

Integrating efficient and robust testing frameworks within CI/CD pipelines is essential in today's fast-paced DevOps environment. Tekton, a Kubernetes-native open-source framework, excels in defining and running continuous integration and continuous delivery (CI/CD) pipelines. When paired with Testkube, a cloud-native test orchestration platform, Tekton offers a seamless and automated foundation for efficient software delivery.
By incorporating Testkube into Tekton Pipelines, development teams can achieve more reliable and efficient testing workflows across their entire SDLC. Testkube supports all the popular testing frameworks and you can even Bring Your Own Tool(BYOT). Developers can trigger and manage any test such as functional, acceptance, load, etc., streamline their testing processes from the pipeline themselves or from a variety of other trigger sources, and ultimately deliver high-quality software faster.

Automated testing throughout the development cycle ensures reliability and agility, preventing testing from becoming a bottleneck. This blog post explores how to effectively use Testkube for test orchestration within Tekton Pipelines, enhancing your CI/CD practices.

What is Tekton?

Tekton, a graduated Continuous Delivery Foundation (CDF) project, offers a flexible, cloud-native solution for automating application’s build, test, and deployment. It is highly extensible and can be installed on any Kubernetes cluster.
Let us understand some of the terms related to Tekton:

Task: is a fundamental unit in Tekton. In a Task, we define the steps that will be executed sequentially within a Kubernetes pod using the specified image. Each step runs in its own container within the pod and creates an output that can be used as input to the next step.
Pipeline: is a series of ordered Task. A Task in a Pipeline can use the previous Task output as input.
TaskRun: While Task has the definition of what to do, TaskRun instantiates the defined Task for the execution.
PipelineRun: Similar to TaskRun, a PipelineRun instantiates the defined Pipeline for execution.

A TaskRun is useful when we have to run only one task. To run multiple tasks in some orderly manner or to handle complex workloads such as static analysis, testing, or building complex tasks, we need to create a Pipeline.

Tekton works using its fundamental unit, Task. A Task or a Pipeline created on a cluster is just a definition. It gets executed by Tekton on creating TaskRun(in case of a single Task) or PipelineRun. Once the PipelineRun is created, Tekton schedules the Tasks and Steps to run in the specified order.

Each Task runs in its own pod, and Tekton manages the execution, monitoring, and logging of each step. Developers can monitor the status of the PipelineRun and TaskRun using the Tekton CLI.

Here are some more things that developers can achieve with Tekton:

Define custom workflows as code tailored to their specific needs, enabling precise control over their CI/CD processes.
Isolate each task execution and minimize the risk of conflicts between tasks.
Easily scale pipelines and handle increased workloads efficiently.
Utilize Kubernetes features such as namespaces, RBAC (Role-Based Access Control), and custom resource definitions (CRDs) seamlessly within their CI/CD pipelines.
Integrate with a wide range of tools and services commonly used in DevOps workflows.

In the coming section, let us explore Testkube and the benefits of integration.

What is Testkube?

Testkube is a cloud-native test orchestration platform for Kubernetes environments. It simplifies managing and running any kind of test by providing a unified platform that integrates with various testing tools and frameworks.
This makes it easier for developers to ensure their applications work correctly without needing to handle the complexity of different testing setups. Here are some of the benefits of using Testkube to run tests in a CI/CD pipeline:

Seamless Integration with any testing framework: Whether we are using unit tests, integration tests, or end-to-end tests, Testkube can integrate with our preferred tools, providing a consistent and streamlined testing process.
Automate Parallel Testing: With Testkube, we can run multiple tests simultaneously (in parallel), which speeds up the testing process. This is particularly useful when there are many tests, as it reduces the overall time needed to verify code.
Centralized Test Execution Management: Testkube offers a centralized platform to manage all the test executions. This means we can start, stop, monitor and troubleshoot tests from one place, making tracking our testing activities and results easier.
Scalability: With the increased workload, Testkube can easily scale up to handle executions efficiently. Each test runs in isolation, ensuring that our testing processes remain robust and reliable.
Trigger tests from outside CI/CD: Testkube allows us to trigger tests from outside the CI/CD pipeline on the cluster through various mechanisms, including Kubernetes events, scheduled executions, the Testkube CLI, etc.. This flexibility ensures we can run tests whenever needed, not just during automated build and deployment processes.

In the next section, we will integrate Testkube with Tekton using the Testkube CLI.

Integrating Testkube with Tekton

We will create a Tekton Task that uses the Testkube CLI Docker image, performs authentication to connect to your Testkube account, and executes the Testkube command we want to run.

Pre-requisites

To follow along, you will need to set up the following:

A Testkube Pro account (free plan is fine)
A Kubernetes cluster - we’re using a local Minikube cluster.
The Testkube Agent is configured on the cluster.
Tekton Pipeline is installed on the cluster.
A Testkube API token with Admin Access rights.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent and Tekton configured.

$ kubectl get pods -A
NAMESPACE                    NAME                                                   READY   STATUS    RESTARTS        AGE
kube-system                  coredns-5dd5756b68-5hlw4                               1/1     Running   0               3d23h
…
tekton-pipelines-resolvers   tekton-pipelines-remote-resolvers-9c4dfcb5c-9dnj8      1/1     Running   0               48s
tekton-pipelines             tekton-events-controller-7b68785c4b-q4df7              1/1     Running   0               48s
tekton-pipelines             tekton-pipelines-controller-7d9f595ff9-wtfp8           1/1     Running   0               48s
tekton-pipelines             tekton-pipelines-webhook-544849794c-ldxxz              1/1     Running   0               48s
testkube                     testkube-api-server-5987d56766-wl85n                   1/1     Running   3 (3d23h ago)   3d23h
testkube                     testkube-nats-0                                        2/2     Running   0               3d23h
testkube                     testkube-operator-controller-manager-f4bb85c97-mxg72   2/2     Running   0               3d23h

Create Tekton Task

To integrate Testkube with Tekton, we need to configure the CLI to connect to our Testkube environment. Testkube provides set context CLI command that will set the CLI Context to the expected environment. Once the context is set, we can run any testkube command from the Tekton Pipeline. Here is Tekton Task with Testkube integrated:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: testkube
spec:
  description: >-
    Run testkube cli commands with tekton
  params:
  - name: tkc-env
    description: testkube cloud environment name
    type: string
  - name: tkc-org
    description: testkube cloud org name
    type: string
  - name: api-key
    description: api key for testkube account
    type: string
  - name: tkc-command
    description: Command to run with testkube
    type: string
  steps:
  - name: run-testkube-cli
    image: "kubeshop/testkube-cli:latest"
    script: |
      #!/bin/sh
      testkube set context -c cloud --env-id $(params.tkc-env) --org-id $(params.tkc-org) -k $(params.api-key)
      testkube $(params.tkc-command)

Let us understand each of the fields we have set in the above Task:

kind: Task: Declares that this YAML defines a Tekton Task
params: Parameter that the Task expects. In the script, we have added the testkube set context command that expects the env-id, org-id, and API key to connect.some text

tkc-env: This is the environment ID. You can find the value in Testkube Dashboard -> Settings.
tkc-org: This is the org ID. You can find the value in Testkube Dashboard -> Settings.
api-key: This is the dedicated API key you created during prerequisites.
tkc-command: In the script, we run the testkube command, which allows you to pass any testkube CLI command or flags.

steps: Set the name of the step along with image to be used to create the pod to run the script. Here we have created a bash script that sets the Testkube CLI Context and runs the Testkube command.

This Task is reusable. You can run any Testkube command using it in a Testkube environment with Admin access. Let us see, with the help of a use case, how we can use this Tekton Task with Testkube integrated to run a Test Workflow.

Test Orchestration with Testkube in Tekton

In our previous blog, we automated acceptance testing in the Testkube Dashboard using the Test Workflows. We will execute the same TestWorkflow from the Tekton CI/CD. This automation will help you understand how seamlessly Testkube integrates the execution of any testing tool in the Tekton Pipeline.

Create a Test Workflow in the Testkube Dashboard

Testkube offers a comprehensive Dashboard where you can manage your Test Workflows and their execution. We have already created the Test Workflow and executed it in the dashboard:

The Robot Framework test for Restful Booker has been executed successfully.

Create a Tekton Task on the cluster

Create the Tekton Task which integrates Testkube on the cluster. We have kept Task in a file tekton-testkube.yaml and deployed it on the cluster.

$ kubectl apply -f tekton-testkube.yaml
task.tekton.dev/testkube created

$ kubectl get tasks
NAME       AGE
testkube   5s

The Tekton Task testkube is successfully created on the cluster.

Create a Tekton Pipeline

A Tekton Pipeline contains the task and the required parameters. Here is the Pipeline we created that includes the testkube Task.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: testkube-pipeline  # Name of the Pipeline
spec:
  params:                  # Parameters that the Pipeline expects
  - name: tkc-env
    description: testkube cloud environment name
    type: string
  - name: tkc-org
    description: testkube cloud org name
    type: string
  - name: api-key
    description: "The API key for Testkube"
    type: string
  - name: tkc-command
    description: Command to run with testkube
    type: string
  tasks:                   # Tasks to be executed as part of this Pipeline
    - name: testkube-cli
      taskRef:             # Reference to a previously defined Task
        name: testkube     # Name of the Task being referenced is 'testkube'
      params:
      - name: tkc-env
        value: $(params.tkc-env)
      - name: tkc-org
        value: $(params.tkc-org)
      - name: api-key
        value: $(params.api-key)
      - name: tkc-command
        value: $(params.tkc-command)

In the Pipeline, the parameters are defined that are passed in the testkube Task. Save this in a file pipeline-tekton-testkube.yaml and deploy it on the cluster.

$ kubectl apply -f pipeline-tekton-testkube.yaml
pipeline.tekton.dev/testkube-pipeline created

$ kubectl get pipeline
NAME                AGE
testkube-pipeline   9s

The Tekton Pipeline for the testkube Task is successfully created. To instantiate this Pipeline, we create a PipelineRun.

Create a Tekton PipelineRun

A Tekton PipelineRun contains the Pipeline details and the values of the parameters required by the Pipeline. Here is the PipelineRun we have created for execution.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: testkube-pipeline-run
spec:
  pipelineRef:
    name: testkube-pipeline
  params:
  - name: tkc-env
    value: "tkcenv_34xxxxxxxxxxxxx"
  - name: tkc-org
    value: "tkcorg_b8xxxxxxxxxxxxxx"
  - name: api-key
    value: "tkcapi_58xxxxxxxxxxxxxxx"
  - name: tkc-command
    value: "run testworkflow basic-acceptance-test --watch"

Here, we have added the value to the parameters. You can also make use of Kubernetes Secrets to pass encrypted values. Parameter tkc-command contains the testkube CLI command to be executed on the cluster.

Here we have passed the command to run the testworkflow basic-acceptance-test. This is the testworkflow for the acceptance testing with the Robot Framework created in the Testkube Dashboard. Save the configuration in a file pipelinerun-robottest-tekton-testkube.yaml and deploy it on the cluster.

$ kubectl apply -f pipelinerun-robottest-tekton-testkube.yaml
pipelinerun.tekton.dev/testkube-robotframework-pipeline-run created

$ kubectl get pipelinerun
NAME                                   SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
testkube-robotframework-pipeline-run   True        Succeeded   2m36s       2m24s

With this, the PipelineRun is successfully created and the Task is executed. In the coming section, we will see how to verify and get more details related to the test execution.

Verify

Tekton provides Tekton CLI which can also be utilized to see the execution of the Task. However this will not show us the complete test execution logs and artifacts-related details. Testkube provides a Dashboard that we can use to verify the execution of the testworkflow from the Task.

Tekton

We installed Tekton CLI on our local machine. Here are the logs related to the Pipeline.

$ tkn pipelinerun logs testkube-robotframework-pipeline-run
task testkube-cli has failed: "step-run-testkube-cli" exited with code 2
[testkube-cli : run-testkube-cli] Your config was updated with new values 🥇
[testkube-cli : run-testkube-cli]
[testkube-cli : run-testkube-cli] Your current context is set to cloud
[testkube-cli : run-testkube-cli]
[testkube-cli : run-testkube-cli]   API Key        : tk*********************************37
[testkube-cli : run-testkube-cli]   API URI        : https://api.testkube.io
[testkube-cli : run-testkube-cli]   Namespace      :
[testkube-cli : run-testkube-cli]   Organization   : SONALI SRIVASTAVA-personal-org (tkcorg_xx)
[testkube-cli : run-testkube-cli]   Environment    : SONALI SRIVASTAVA-personal-env (tkcenv_xx)
[testkube-cli : run-testkube-cli]
[testkube-cli : run-testkube-cli]
[testkube-cli : run-testkube-cli] Context: cloud (1.17.63)   Namespace:    Org: SONALI SRIVASTAVA-personal-org   Env: SONALI SRIVASTAVA-personal-env
[testkube-cli : run-testkube-cli] ------------------------------------------------------------------------------------------------------------------
[testkube-cli : run-testkube-cli] Test Workflow Execution:
[testkube-cli : run-testkube-cli] Name:                 basic-acceptance-test
[testkube-cli : run-testkube-cli] Execution ID:         666c4fa7c3e60220a410f063
[testkube-cli : run-testkube-cli] Execution name:       basic-acceptance-test-8
[testkube-cli : run-testkube-cli] Execution namespace:  testkube
[testkube-cli : run-testkube-cli] Execution number:     8
[testkube-cli : run-testkube-cli] Requested at:         2024-06-14 14:11:51.349926545 +0000 UTC
[testkube-cli : run-testkube-cli] Status:               queued

In the above output, we can see Task name testkube-cli and the step name run-testkube-cli. The Task is running and the execution of the testworkflow is queued. Let us head to the Testkube Dashboard for more details.

Follow a Test Execution in the Testkube Dashboard

In the Testkube Dashboard, we can see the test running as shown below.

Click on Open Details & Logs.

The above screenshot shows that all the tests have successfully passed, and Testkube has uploaded the artifacts.

With this approach, we orchestrated the Robot Framework test from the Tekton Pipeline using Testkube. Not only did Testkube make the integration seamless, but it also offered us a scalable and efficient solution for automating testing processes within your Kubernetes environment.

Conclusion

Integrating Testkube with Tekton Pipelines provides a robust, secure, and scalable solution for test orchestration and execution. In this blog, we created a reusable Tekton Task that integrates with Testkube. Using this Task, we deployed a CI/CD pipeline that executed Robot Framework tests in a Kubernetes native environment. Similarly, you can Bring Your Own Tool(BYOT) and orchestrate tests with Testkube in CI/CD pipelines.
To experience the full potential of testing on Testkube, we invite you to try Testkube today. Witness firsthand how Testkube simplifies and empowers your testing process with its Kubernetes-native test execution capabilities. Join our active Slack community for guidance and support.

Simplifying API Test Automation with Postman and Testkube

Juan Ibarra — Mon, 30 Sep 2024 19:51:32 +0000

As an industry, we are moving away from monolithic architectures for many applications and adopting API-first mode. APIs have become the building blocks of applications, making our applications modular and scalable. This approach focuses on building robust APIs from the start.

Kubernetes simplifies deploying and managing such API-based applications at scale. However, API reliability and performance require comprehensive testing as the applications grow and become more distributed. Issues like ephemeral resources, load balancing complexities, service discovery mechanisms, and the distributed nature of Kubernetes clusters make it challenging to execute API tests reliably.

Tools like Postman help you with testing your APIs. However, using Postman with Testkube allows developers to streamline API testing workflows within Kubernetes clusters. In this post, we’ll look at creating a Test Workflow using Postman in Testkube.

API Testing Using Postman

Postman is one of the most widely adopted API testing tools. It provides a suite of features for designing, testing, and monitoring APIs. Its intuitive dashboard, rich feature set, and user experience make it a go-to choice for developers and testers alike. Some of Postman’s features include:

User-friendly UI: Postman's UI is intuitive, seamless, and user-friendly, allowing you to build and test APIs easily.
Automated testing: Postman allows you to write automated tests using its built-in capabilities to create and build comprehensive test suites.
Environment management: Postman simplifies environment management by allowing you to create and seamlessly switch between environments.

You can read more about Postman here.

Creating API Tests In Postman

One of the main reasons Postman is a popular choice for API testing is the ease with which it allows developers to create and manage APIs. Once you have the API endpoints you want to test, creating a request and an associated test is straightforward.

In this case, we’ll use the https://restful-api.dev/ endpoint, a free-to-use API for testing. To use it, you create a collection and add a request to it. By tweaking the endpoints, we added three requests: GetAllObjects, GetObjectsById, and GetSingleObject.

We can also add tests to each request based on your requirements. The tests are written in JavaScript and are used to validate the responses from the endpoint.

You can then export the collection as a JSON file, which we’ll use to create the Test Workflow. To do that, Click on the three dots next to the Collection Name and click Export.

Challenges With Postman in Kubernetes

While Postman works great for API testing in traditional environments, managing them in Kubernetes using Postman is challenging and complex. Some of these challenges are:

Service discovery: The Kubernetes service discovery mechanism allows services to change their IP addresses and ports frequently, complicating the configuration of Postman tests.
Test Orchestration: Orchestrating and managing Postman tests across nodes and pods can be challenging, especially when dealing with dependencies and parallel tests.
Test data management: Without proper tooling, managing and sharing test data across different clusters and nodes can become complex.

To overcome these issues, you can turn to specialized tools like Testkube, which is explicitly designed for running tests in Kubernetes.

Using Postman With Testkube

Testkube is a Kubernetes-native testing framework designed to streamline the orchestration of tests on Kubernetes. It adopts a declarative and version-controlled approach that enables developers to manage tests as Kubernetes resources. Below are some benefits of using Testkube:

Integration With Testing Tools: Testkube integrates with many testing tools like Cucumber, k6, Cypress, etc., enabling you to leverage the power of Kubernetes for your tests.
Simplified Workflow Creation: Without complex scripting, Testkube facilitates the creation of detailed test workflows, allowing for better control and customization of test executions. Refer to our Test Workflows documentation to learn more.
Kubernetes Native: By storing tests as Kubernetes Custom Resource Definitions (CRDs), Testkube ensures compatibility and scalability.
Integration with CI/CD Tools: Testkube seamlessly integrates with existing CI/CD pipelines, enhancing end-to-end testing capabilities.

Let’s see how to create a Test Workflow for Postman using Testkube. All the files used for this demo can be found in the Testkube Examples GitHub repo.

Pre-requisites

Get a Testkube account.
Kubernetes cluster - we’re using a local Minikube cluster.
Testkube Agent configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Creating a Test Workflow

Navigate to the Test Workflows tab and click on "Add a new test workflow".

This will provide you with three options:

Create from scratch: Use the wizard to create a Test Workflow.
Start from an example: Use existing k6, cypress, and playwright examples.
Import from YAML: Import your own Test Workflow.

We’ll choose the "create from scratch" option to create this workflow.

Provide a name for the workflow and choose the type as Postman.
Provide a Postman version, we’ll use 6-alpine.
Provide the run command. In this case, we’ll provide newman run .json.

On the next screen, provide the source for the test file. This can either be a Git Repo, String, or a file. In this case, we’ll use a Git repo. Further, we’ll provide the path as “postman-workflow”.

On the next screen, it will generate the YAML spec file and display the output.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: postman-testkube
  namespace: testkube
  labels:
    test-workflow-templates: "yes"
spec:
  use:
  - name: official--postman--beta
    config:
      run: newman run RestAPITestParams.postman_collection.json
      version: 6-alpine
  content:
    git:
      uri: https://github.com/kubeshop/testkube-examples.git
      revision: main
      paths:
      - postman-workflow
  container:
    workingDir: /data/repo/postman-workflow

The yaml file is self-explanatory as it lists down the details you’ve provided in the yaml.

…
"item": [
        {
            "name": "GetAllObjects",
            "event": [
                {
                    "listen": "test",
                    "script": {
                        "exec": [
                            "// Check the HTTP status code",
                            "pm.test(\"Status code is 200\", function () {",
                            "   pm.response.to.have.status(200);",
                            "});",
                            "",
                            "// Check the number of items returned",
                            "pm.test(\"Item count is 13\", function () {",
                            "   var jsonData = pm.response.json();",
                            "   pm.expect(jsonData.length).to.eql(13);",
                            "});"
                        ],
                        "type": "text/javascript"
                    }
                }
            ],
…

Above is a snippet from the json collection which has three APIs and a few tests associated with each. All APIs make a GET request to an endpoint, some with parameters and some without. The tests here validate the API response.

Click on “Create” to create the test workflow.

Setting Up Variables

The last step before executing the Test Workflow is to set up variables. The Postman_collection currently requires the following two variables to execute:

url: This is the base URL of the API endpoint.
id: This is the object_id parameter that will be passed in the API call.

Select the newly created Test Workflow to configure these variables and navigate to Settings -> Variables & Secrets. Here, add these two variables and click Save. This will configure the Test Wofklow to use the variables, which will also reflect in the yaml spec

After this, you need to configure the Test Workflow’s yaml spec to pass these variables with the execution command. Navigate to Definition tab and edit the run command as shown below:

<span>newman run RestAPITestParams.postman_collection.json --env-var "id=${id}" --env-var
</span>"url=${url}"

With this, you have configured the Test Workflow to use the value of “2” for id and “api.restful-api.dev” for url, respectively. You can tweak these variables based on your requirements.

Executing the Test Workflow

Once the workflow is ready, you’ll see the newly created test workflow on the screen. Click on it and click “Run Now” to start the workflow.

You’ll see the workflow executing along with the real-time logs of every step.

You’ll see the test result based on the test execution. In this case, you’ll see that all the tests have passed. If you want to fail the Test Workflow, you must change the “id” variable to some other number between 1-12 and see how the output changes.

This was a simple demo of creating a Postman Test Workflow using Testkube for API testing in Kubernetes. To take advantage of test workflows more, you can create custom workflows and import them to Testkube.

Summary

In this post we looked at how API first design approaches are getting popular. Many Java applications are also adopting this design principle, and tools like RestAssured are gaining popularity in testing APIs. We also looked at RestAssured and the complexities of running it on Kubernetes.
We then saw how using RestAssured with Testkube, you can create an end-to-end workflow for API testing leveraging the power of Kubernetes.

Visit the Testkube website to learn more about the other testing tools you can integrate with. If you struggle with anything, feel free to post a note in our active Slack community.

Stop Running Tests With Your CI/CD Tool

Juan Ibarra — Tue, 10 Sep 2024 21:20:13 +0000

6 reasons to decouple test execution from your CI/CD pipeline

Implementing a consistent testing infrastructure and workflow for cloud native applications has its challenges. Multiple stakeholders have different needs in regard to testing/QA, the testing tool stack is constantly evolving in line with new technologies and requirements, CI/CD/GitOps pipelines are transforming the way we deliver software, and the testing of both legacy and cutting-edge components needs to be maintained to ensure the delivery of high-quality applications to end users.

With the advent of CI/CD tooling and workflows it felt natural to use CI/CD for running tests also. Testing is part of the software delivery life cycle after all, and automating test executions as part of builds and deployments makes sense at a conceptual level. Unfortunately though, many CI/CD tools put very little emphasis on the specific needs of testing and QA. To them testing is just another task to run in the pipeline, which often makes additional testing support in CI/CD tools feel more like an afterthought than a primary objective.

Add in the common scenario where multiple CI/CD tools are used within the same organisation: Jenkins for building your Java microservices backend, GitHub actions for building (and deploying?) your frontend applications and maybe even something like Argo for adopting a GitOps approach to deploying your applications to Kubernetes. Not only is testing often an afterthought, that afterthought is now spread across multiple tools! What can go wrong?

Let's drill down into six specific needs of a successful test-automation strategy and how relying on CI/CD tooling will often send you into the testing-swamp-of-no-return (TSONR — this is where you read it first!).

1. Consistent Test-tooling Support

No matter how you set up test runs in your CI/CD pipelines and tools, maintaining consistent support for legacy tools, modern tooling, version changes and legacy tests is a challenge.

One of the last things you want to hear at the end of the day is "Our CI/CD tool doesn't support your testing framework" or "We can't run multiple versions of in our pipelines. You'll have to upgrade all your scripts to work with version X."

Many CI/CD tools rely on plugins to support a specific testing tool / version — not a guarantee for consistency. Their fallback is usually some kind of scripting environment, which might do the job but adds complexity and maintenance overhead, making it hard to scale and diversify testing efforts.

2. Consistent Test-Execution Environment

"Works on my machine." You've surely both heard and uttered those words with disbelief when a test you've meticulously crafted in one environment doesn't give the desired results when run in a different (more important) environment.

Running the same set of tests should give consistent results, obviously. Unfortunately though, running tests in a multi-CI/CD tooling environment often results in results that vary depending on where (and how) you run them. Different CI/CD tools have different runtimes, environments and infrastructure, making it hard to predict consistency of your testing efforts, especially when it comes to non-functional tests like performance, security and compliance testing. Add to this that tests run locally during development are often run "manually" directly with the corresponding testing tool, which is usually far from a testing or production environment.

3. Run Tests Whenever Needed

Running automated tests as part of your CI/CD pipeline is common practice but running those tests outside your pipeline is hard, and you don't want to rerun an entire build just to rerun some updated tests against a development environment.

The possibility to run tests outside your CI/CD pipelines, both manually (for example load-tests) or in response to other system events (such as a Kubernetes event) is a must in a distributed and diversified infrastructure to ensure that both DevOps and QA teams can (re)run tests whenever needed.

4. Run Tests at Scale

Running automated tests at scale entails two vectors:

Scaling load tests to generate massive load for simulating peak usage scenarios for your applications or APIs
Scaling end-to-end (E2E)/functional tests to cover a matrix of execution scenarios, including different browsers, operating systems, users, etc.

CI/CD tooling will rarely have dedicated functionality that caters to either of these needs in the context of test execution. They might allow you to launch different "workers," but all logic beyond that in regard to your testing tool at hand will have to be managed by custom scripts and/or third-party solutions.

5. Single Pane of Glass for Test Results

Getting access to consistent test-results and artifacts across all testing tools used in your CI/CD pipelines is key to both tactical troubleshooting of failed tests and strategic understanding of overall testing efforts.

Unfortunately, most CI/CD tools have little inherent knowledge about test results at a higher level. They might make it easy to see the log/artifact output of each individual test, but aggregating quality metrics such as pass/fail ratios and execution numbers across all your testing tools is not their concern, and providing you with an easy way to access specific test-execution results and artifacts for in-depth troubleshooting of failed tests will often require you to do a fair amount of scripting yourself or export these to external tooling for deeper analysis.

6. Giving Control to QA

Need to update some arguments or the version of your testing tool running in your pipelines? File a ticket with the DevOps team and hope they have time later today, or this week, or month..

Once test automation is handed over to the team(s) managing CI/CD pipelines, QA often has little control or insight into that automation, which can considerably slow down the evolution of testing in your CI/CD pipelines.

CI/CD tooling rarely has the role-based access control granularity required to give testers access to just the testing aspects of build pipelines, so QA-initiated improvements/changes related to test execution often need to go through a tedious process before they get implemented, causing everything from frustration within teams to lacking test coverage.

Alternatively QA is given access to areas of the build infrastructure they should not have, which could introduce security concerns in a more regulated organization.

OK, Now What?

OK, you've heard the arguments and hopefully you'll think twice before you ask your DevOps team to automate your Playwright scripts or Postman collections in your pipelines going forward.

But how do address all these challenges and decouple test execution from your CI/CD pipelines without sacrificing the value of testing in CI/CD itself?

Solution 1: A Test Orchestration Platform

Testkube is a test-orchestration platform for CI/CD specifically built to solve the above problems (and more):

It supports the execution of tests for any testing tool/version without the need for extensive scripting.
It uses Kubernetes to run all your tests, which brings a consistent and scalable execution environment for all your tests.
It allows you to run your tests whenever needed — as part of CI/CD, manually from a CLI, dashboard or API, through external triggers, etc.
It has built-in support for scaling any testing tool - either horizontally for load-generation or vertically for multi-scenario E2E/functional tests.
It provides a single dashboard for all your test results and artifacts, ensuring you with a consistent way to troubleshoot any test and gather both operational and quality insights for your testing activities over time.
It separates test automation orchestration/authoring/configuing from your CI/CD tooling, putting QA back in control of all test-execution activities in your environment.

Testkube always runs tests in your own infrastructure, helping you manage both costs and security aspects of test executions. The Testkube Dashboard can either be hosted in the cloud or run on-prem (air-gapped if needed), giving you both easy-to-start and security-compliant alternatives going from an evaluation to production setup.

Option 2: Manual Scripting and Duct Taping

If Testkube isn't for you, what are your options to circumvent some of the above challenges?

If you're using at least one CI/CD tool in your organization, you could look into creating micro-pipelines specific to testing, and call/reuse those from your existing build pipelines. This will possibly help you with points 3, 5 and 6 above

These pipelines can be run whenever needed (although individual tests within them cannot).
All test.results will be found in the output of these pipelines, although if you use multiple testing tools they will still be disconnected.
You might be able to ensure the testers/QA have access to managing them without having to touch the rest of your build configurations.

Unfortunately the level of support for the other points will vary greatly depending on the CI/CD tool you are using, and how much effort/time you are willing to put into custom script authoring/maintenance.

Summary

Automated test execution is a mandatory practice in large-scale CI/CD pipelines but comes with many challenges not addressed by CI/CD tooling. The shortcomings of CI/CD tools in this regard hamper a successful testing strategy that can scale across teams, projects and testing tools. Testkube provides a holistic solution to these challenges while maintaining compatibility with any testing tool, workflow or pipeline already deployed in your organization. An open source version is available. Give it a try at testkube.io/get-started to take your automated test execution to the next level

Functional Testing with Selenium in Testkube

Juan Ibarra — Tue, 10 Sep 2024 21:16:53 +0000

Introduction

Functional testing tools like Selenium are essential for verifying that the different components of an application work together as intended. Integrating Selenium into a CI/CD pipeline can demand significant rework and high resource utilization, and running Selenium tests in a Kubernetes environment can be complex and resource-intensive. Running the pipeline multiple times for functional testing can quickly become inefficient and costly.

To address these challenges, we explored Testkube’s innovative Test Workflows for executing Selenium tests. Test Workflows provides a streamlined approach to defining, running, and automating tests in Kubernetes, allowing you to define your tests as code, manage test executions, and generate artifacts without disrupting your existing setup.

In this blog, we will delve into how Testkube’s capabilities simplify the functional testing process with Selenium. We will explore how to set up and automate your Selenium tests within a Kubernetes environment using Testkube, overcoming the common challenges of integration and resource management.

What is Functional Testing?

Functional testing is the process of verifying whether the functionality of a product is built right. It involves systematically testing each feature to ensure they function correctly according to specifications. This type of testing validates input, output, and user interactions to confirm that the product behaves as expected.

With functional testing, we can ensure that tests execute without errors, including handling exceptions and HTTP 404 responses. It also involves verifying correct redirections, ensuring usability, and assessing accessibility according to specifications. For web applications, functional testing can be automated using Selenium, which tests for expected outcomes in your web applications.

Functional Testing with Selenium

Selenium is an open source project that comprises many tools and libraries for browser automation. In Selenium, we can write test scripts in different languages like Python, Ruby, Java, Perl, etc. It supports different browsers such as Firefox, Chrome, Edge, Safari, etc. Selenium allows various types of testing such as acceptance, functional, regression, performance, load, etc.

There are the main Selenium components that allow testing:

Selenium IDE: For developing Selenium test cases, Selenium provides a Chrome and Firefox extension. It records user actions in the browser using Selenium commands, making it efficient and educational for learning Selenium script syntax.
Selenium WebDriver: WebDriver provides an abstraction layer over browser engines, exposing APIs that command the browser to run tests, simulating real user interactions. WebDriver is non-intrusive, allowing you to test the same application you deploy live.
Selenium Grid: Once WebDriver tests are ready, we may require testing on multiple browsers, different operating systems, and their combinations. Selenium Grid allows parallel execution of test cases across multiple machines and also enables running tests remotely.

These Selenium components help create automated test scripts, cross-browser/platform testing, data-driven testing, and parallel testing with different language support across multiple browsers or operating systems. Let us understand in the coming section the working of Selenium and where these components fit.

How Selenium Test Works

Web browsers, despite appearing similar, operate differently beneath the surface. Selenium abstracts these differences, enabling code to execute seamlessly across browsers like Firefox, Microsoft Edge, and Chrome. This simplifies coding, allowing complex workflows to be written in just a few lines while ensuring consistent functionality across supported browsers.

To build a test suite in Selenium using WebDriver, we need to understand some of the commonly used terms:

Driver: A driver controls the actual browser. They are also known as proxies. These are created and provided by the browser vendors in most browsers.
WebDriver: The task of WebDriver is to command the browser but to do so, it needs the driver. WebDriver talks to the browser through the driver and receives information back via the same route.
Framework: The task of the framework is to run and execute the WebDriver. These could test frameworks like JUnit for Java, RSpec for Ruby, NUnit for .Net, etc.

How does a WebDriver communicate with a browser?

In the execution of the Selenium test, the first step is to understand how the communication is happening between the WebDriver and the browser. Here are the two methods:

Direct: In the direct communication setup, the WebDriver, Driver, and the browser are on the same system.

Remote: In the case of remote communication setup, the communication between the WebDriver and the browser happens through a Selenium Server/Grid or RemoteWebDriver. The Selenium Server or Grid runs on a different system while the RemoteWebDriver can run on the same system as the driver and browser.

Why do we need a Framework?

The WebDriver communicates with the browser but is unaware of the testing. It has no information on asserting a test fail or pass, or reporting an issue. This is where the frameworks help. A test framework allows the execution of WebDriver that runs the steps mentioned in the test.

The above image shows the working of a Selenium test with remote communication. Let us understand with the help of an example. Let us suppose, you have to create a Selenium test in Java. To execute this test, you will need a test framework like JUnit that will call the WebDriver which runs the test. Once the WebDriver is invoked, it communicates(remote) with the browser via the driver. Thus a Selenium test is run to verify the functionality of a browser.

Now we understand here that multiple components need to be set up in a Kubernetes native environment to perform tests. This comes with some challenges. One major challenge is scaling Selenium tests to run in parallel. Kubernetes can handle scaling, but configuring Selenium to run multiple browser instances effectively within Kubernetes can be complex. You need to figure out browser-specific dependencies, ensure that Kubernetes nodes have optimum resources, and that the test environment closely mimics the production environment.

Addressing these challenges requires a well-planned approach, leveraging Kubernetes features effectively, and utilizing tools like Testkube to simplify and streamline the deployment and management of Selenium tests.

Functional Testing with Testkube

Testkube offers a comprehensive solution to address the challenges of deploying Selenium tests on Kubernetes. With Testkube, teams can efficiently manage resource consumption by leveraging Kubernetes' scalable infrastructure, ensuring optimal use of CPU and memory resources. Its simplified setup process eliminates the complexities of configuring Selenium within Kubernetes, facilitating seamless communication between components while managing browser dependencies effortlessly.

As a Kubernetes-native test orchestration framework, Testkube simplifies the creation and execution of complex test workflows across various environments without the need for extensive scripting. Test Workflows provide granular control over test execution parameters, resource usage, and setup/teardown procedures. Also, Test Workflows are stored as Kubernetes Custom Resources (CRs), ensuring compatibility, scalability, and portability.

Let’s explore how to use Selenium with Testkube for functional testing in the coming section.

Pre-requisites

To follow along, you will need to set up the following:

A Testkube Pro account (free plan is fine)
A Kubernetes cluster - we’re using a local Minikube cluster (https://minikube.sigs.k8s.io)
The Testkube Agent is configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Use Case: Single Browser

We have written a test in Java that verifies a specific button on the website exists and is clickable. This test has been designed to check button functionality on Chrome, Firefox, and Edge. To build the dependencies and run the test, we are using Maven. You can find the Project Object Model (POM) configuration for Maven (pom.xml) and the Java Test (SeleniumTest.java) in this repository.

We will now create a Test Workflow in Testkube using Maven and integrate Selenium tests into it. In this first use case, we will run the test only for the Chrome browser and store corresponding artifacts to get an understanding of the process.

Creating a Test Workflow

Here are the steps to create a test workflow from the Testkube Dashboard:

Navigate to the Test Workflows tab on Testkube and click on “Add a new test workflow”.

This will provide you with three options:

Create from scratch - use the wizard to create a Test Workflow.
Start from an example - use existing k6, cypress, and playwright examples.
Import from yaml - import your own Test Workflow.

Select the “Import from yaml” option to create this workflow and provide the configuration as shown below.

Click on Create and a Test Workflow will be created. Here you can trigger the Test Workflow and execute the tests.

Before we execute the Test Workflow, let us glance at the Test Workflow and the Java test.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: selenium-single-browser-test #Name of the workflow
 namespace: testkube #Namespace where TestWorkflow run
spec:
 content:
   git:
     uri: https://github.com/cerebro1/selenium-testkube.git
     paths:
     - selenium-java #git repo and path that contains Selenium Java Test
 container:
   workingDir: /data/repo/selenium-java #Mount point for the repo
   image: maven:3.9.6-eclipse-temurin-22-alpine #Maven image for container
 services:
   chrome:
     image: selenium/standalone-chrome:4.21.0-20240517
     readinessProbe:
       httpGet:
         path: /wd/hub/status
         port: 4444
       periodSeconds: 1
 steps:
 - name: Run Chrome Test
   run:
     env:
     - name: SELENIUM_HOST
       value: '{{ services.chrome.0.ip }}:4444' # get hostip from service
     shell: |
       mvn test #execute the test
       mvn surefire-report:report #generate reports
   artifacts:
     paths:
      paths:
      - target/site/**/*
      - target/surefire-reports/**/*

In this Test Workflow, we have used the Selenium Standalone Chrome image that allows WebDriver tests to be run remotely. The WebDriver will use the chromedriver to communicate with the Google Chrome browser. We have updated the path of the directory in the Workflow that contains the functionality test case in Java for Selenium. In the code, we have added comments to help you understand each step.

Here is the test that we are going to run. In this test, we sent a GET request to a website, searched for the button using the text on the button, and made a click. This test will check if the click was successful and return the outcome based on that.

Maven refers to the pom.xml file and installs the dependencies such as selenium-java and junit-jupiter. We are using JUnit as the test framework and enabled the maven-surefire-report-plugin for gathering the artifacts.

Now that we have our Selenium test, POM, and the Test Workflow ready, let us go ahead and execute it.

Executing the Test Workflow

Here are the steps to execute the workflow and view the artifacts:

Click on “Run this Test Workflow now” to trigger the first run. As the execution starts, you will see the workflow running as shown below.

Click on the running workflow to view the log output.

The above image shows that the Chrome Test is running. The next step here is uploading the artifacts once the shell commands are executed successfully.

Once the test execution is complete, click on Artifacts.

The above image shows the list of artifacts generated and stored at the path target/chrome-artifacts.

In the site directory, click on surefire-report.html. Testkube loads the Surefire report in the browser as shown below.

Thus, with the help of Testkube, we were able to automate the execution of a Selenium Java test on a Kubernetes environment using Maven.

In this use case, we ran the test only for one browser—Chrome. In the real world, for testing an application development, you would want to perform the functional test across multiple browsers to ensure your application works for the majority of users. Here you have two options: you can either run a test on each browser serially or invoke parallel testing. Let us see with the help of the following use case how parallelization is better than serial test execution and can help improve the overall testing process.

Use Case: Sequential Test Execution with Multiple Browser Engines

We utilized the previous use case to create a serial Test Workflow for the test execution on multiple browsers. You can create and run this workflow as described for the basic use-case above—the execution will look as follows:

(clicking on the individual items will reveal their logs as applicable)

The workflow is configured to collect artifacts from each execution into a folder named after the browser:

As you can see, we now have three browser instances (Chrome, Firefox, Edge) and three corresponding test executions, one after the other. Although this works as expected, there are some points we noticed:

Tests run one after another will make the entire testing process time-consuming, particularly for large test suites.
If a test fails, it can halt the entire testing process until the issue is resolved, causing delays.
Serial testing is not an optimal use of available computing resources.

In the coming section, we will see with the help of a use case how we can run the same Selenium test across different browsers using parallelization in Testkube.

Use Case: Parallel Execution with Multiple Browser Engines

With parallelization, not only could we improve the use of infrastructure resources for test execution, but also execute more tests in the same amount of time. This would lead to earlier detection of issues and an increase in the overall efficiency of the testing process.

Let’s modify our Test Workflow to make use of two powerful Testkube features: matrix parameters and parallel execution. To achieve this, in the Test Workflow we have defined the IP and name of each browser service in parallel.matrix.browser. The execution of shell commands and storing of artifacts will happen in parallel for all browsers using the parallel execution object (the full workflow is here):

...
 steps:
  - name: Run Cross-browser tests
    parallel:
      matrix: # define matrix of browsers to test against
        browser:
        - ip: '{{ services.chrome.0.ip }}'
          name: chrome
        - ip: '{{ services.firefox.0.ip }}'
          name: firefox
        - ip: '{{ services.edge.0.ip }}'
          name: edge
      transfer: # make sure each parallel node has the test
      - from: /data/repo/selenium-java
      fetch: # retrieve generated artifacts from node
      - from: /data/repo/selenium-java/target
        to: /data/artifacts/{{ matrix.browser.name }}
      container:
        workingDir: /data/repo/selenium-java
        image: maven:3.9.6-eclipse-temurin-22-alpine
      steps:
      - run:
          env:
          - name: SELENIUM_BROWSER
            value: '{{ matrix.browser.name }}'
          - name: SELENIUM_HOST
            value: '{{ matrix.browser.ip }}:4444'
          shell: |
            mvn test
            mvn surefire-report:report
  - condition: always
    workingDir: /data/artifacts
    artifacts:
      paths:
      - '**/*'

Create and Execute the Workflow as we have done previously:

In the above image, we can see that three workers are running at the same time. Testkube not only has improved the process of test execution, but also made optimal use of available computing resources. Since these workers are independent of each other, the failure of any one of them will not impact the execution of the others, providing test isolation.

Performance Comparison: Parallel vs. Sequential Test Execution

On the completion of the above tests, we compared the execution times of five serial and parallel test executions. (Execution times are all in seconds)

While the execution of the entire serial test workflow took roughly 73 seconds on average, the parallel execution workflow completed in 48 seconds on average, an improvement of roughly 34%. Drilling down into the actual times spent on running tests within each workflow (excluding setup/teardown/etc.), the difference was 57 seconds on average in serial flows vs. 30 seconds on average for parallel executions—an improvement of almost 50%.

Conclusion

Selenium is a widely used suite of tools for browser automation. Leveraging its power in Testkube without the complexities of resource management, environment configuration, and setting up dependencies streamlines the testing process for Kubernetes applications.

In this blog post, we have seen how functional testing with Selenium could be done using Testkube. We started with setting up a Testkube workflow for a single browser and explored parallelization to optimize multiple browser testing. We were able to run a Selenium test in Testkube effortlessly for single and multiple browsers and explored the benefits of parallel execution in the case of multiple browsers. Testkube abstracts the complexities and allows optimized test execution in a containerized environment.

Using Testkube, you can ensure compatibility across Selenium, browser drivers, and Kubernetes versions, reducing maintenance overhead. Overall, it offers a strategic and efficient approach to deploying Selenium tests on Kubernetes, enabling teams to focus on delivering high-quality software with confidence.

We invite you to try Test Workflows. Visit the Testkube website to get started. If you find yourself struggling with anything, feel free to drop a note in our active Slack community, and someone will help you out.