Event-driven Kubernetes testing with Testkube and Tracetest

We are pleased to announce that Tracetest now works with Testkube, the Kubernetes-native testing framework. By using a Testkube executor to build an integration with Tracetest, you can now run event-driven trace-based tests in your Kubernetes cluster!

Note: Check out this hands-on Demo example of how Tracetest works with Testkube!

Configuring CI pipelines for running trace-based tests in Kubernetes is tedious work. Especially if you need to trigger tests based on Kubernetes events. Look no further!

Once you’re done reading, you’ll learn how to set up event-driven, trace-based testing in Kubernetes!

What is Testkube?

Testkube is an open-source project, part of the CNCF landscape, and testing framework designed for testers and developers who use Kubernetes. It integrates test orchestration and execution into Kubernetes and your CI/CD/GitOps pipeline. You can automate the execution of your tests, regardless of the testing framework, by using Testkube's executors or creating your own.

By adopting Kubernetes constructs and GitOps, you can perform K8s-native testing. You can use Kubernetes CRDs to manage and store tests, allowing you to validate your applications by executing tests from inside your cluster.

Use any CI/CD framework for any testing scenario. By decoupling them from your CI/CD, you will spend less time integrating different testing tools.

Analyze all your test results in a centralized place. After running your tests, you can view the results in an intuitive UI, regardless of which testing framework you used.

Debug test failures with ease. With Testkube, it's easy to see all the results, logs, and artifacts of your tests in one place.

In addition, Testkube allows you to easily store and download files generated by your tests from your Kubernetes cluster. All files generated from your tests are saved.

What is Tracetest?

Tracetest is an open-source project, part of the CNCF landscape. It allows you to quickly build integration and end-to-end tests, powered by your distributed traces.

Tracetest uses your existing distributed traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Tracetest makes it possible to:

• Define tests and assertions against every single microservice that a trace goes through.
• Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.
• Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
• Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.
• Save and run the tests manually or via CI build jobs with the Tracetest CLI.

Tracetest Now Works with Testkube!

Tracetest now works with Testkube, allowing you to unlock Testkube's capacity with Tracetest, and leverage OpenTelemetry instrumentation in your services to run end-to-end and integration testing. It works thanks to the Testkube Tracetest Executor — a test executor to run Tracetest tests with Testkube.

Why is the Tracetest integration with Testkube important?

By integrating with Testkube you can now add Tracetest to the native CI/CD/GitOps pipeline in your Kubernetes cluster. It allows you to run scheduled test runs on set intervals, as well as asynchronous tests triggered by Kubernetes events. All while following the trace-based testing principle and enabling full in-depth assertions against trace data, not just the response.

Combining the ability to create tests with Tracetest with a Kubernetes-native test runner like Testkube enables you to use the native events from the environment of your Kubernetes cluster as test triggers in your CI/CD/GitOps pipelines!

Why run trace-based tests?

When running integration tests, you have no way of knowing precisely at which point an HTTP transaction goes wrong in a network of microservices! With tracing enabled, Tracetest can run tests with assertions against existing trace data throughout every service in the entire transaction. You can utilize these tests as part of your CI/CD process to ensure system functionality, and to catch regressions.

Try Tracetest with Testkube

To run trace-based tests with Tracetest and Testkube, make sure you have these three things installed before starting.

1. A running Kubernetes cluster, either locally or in the cloud
2. Kubectl
3. Helm

Install Testkube

Testkube is open-source and easy to install. Start by installing the Testkube CLI by following these instructions for your operating system.

# MacOS example

brew install testkube

Install Testkube in your Kubernetes cluster via the CLI.

From here, follow the official documentation to install the Testkube cluster.

testkube init

Confirm that Testkube is running:

kubectl get all -n testkube

By default, Testkube is installed in the testkube namespace. To explore the Testkube dashboard, run the command:

testkube dashboard

Install Tracetest

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI by following these instructions for your operating system.

# MacOS example

brew install kubeshop/tracetest/tracetest

Note: Check out the download page for more info.

From here, follow the official documentation to install the Tracetest server.

tracetest server install

[Output]
How do you want to run TraceTest? [type to search]:
  Using Docker Compose
> Using Kubernetes

Select Using Kubernetes.

[Output]
Do you have OpenTelemetry based tracing already set up, or would you like us to install a demo tracing environment and app? [type to search]:
  I have a tracing environment already. Just install Tracetest
> Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.

Select Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app..

Confirm that Tracetest is running:

kubectl get all -n tracetest

By default, Tracetest is installed in the tracetest namespace.

To explore the Tracetest Web UI, run the command:

kubectl --kubeconfig ${HOME}/.kube/config --context kind-kind --namespace tracetest port-forward svc/tracetest 11633

Once the server is installed, open the Tracetest Web UI in the browser and follow the instructions for connecting the OpenTelemetry Collector with Tracetest, if it has not been connected already. If you followed the steps above, the Tracetest server will have been automatically provisioned to connect to the OpenTelemetry Collector instance running in the tracetest namespace.

If you look closely, you’ll see the OpenTelemetry Sample Configuration from the settings page above matches the collector.config.yaml that was generated by the Tracetest CLI when provisioning the Tracetest server.

# collector.config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 100ms

exporters:
  otlp/1:
    endpoint: tracetest:21321
    tls:
      insecure: true

service:
  pipelines:
    traces/1:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/1]

Create a Test in Tracetest

Start by clicking Create > Create New Test > HTTP Request > Next > Choose Example (dropdown) > Pokeshop - List (generates a sample test from the Tracetest demo) > Next > URL is prefilled with http://demo-pokemon-api.demo/pokemon?take=20&skip=0 > Create & Run.

This will trigger the test and display a distributed trace in the Trace tab to run assertions against.

Proceed to add a test spec to assert all database queries return within 500 ms. Click the Test tab and proceed to click the Add Test Spec button.

In the span selector make sure to add this selector:

span[tracetest.span.type="database"]

In the assertion field add:

attr:tracetest.span.duration < 500ms

Save the test spec and publish the test.

The database spans that are returning in more than 500ms are labeled in red.

This is an example of a trace-based test that asserts against every single part of an HTTP transaction, including all interactions with the database.

However, Tracetest cannot run this test as part of your CI/CD without integrating it with another tool.

Let's introduce how Testkube makes it possible.

Deploy the Tracetest Testkube Executor

Note: As of the latest Teskube release, the Tracetest Testkube executor has been added to the Testkube’s available executors out-of-the-box. If you have an older version of Testkube running, proceed with deploying the Tracetest Testkube executor manually.

Testkube works with the concept of Executors. An Executor is a wrapper around a testing framework, Tracetest in this case, in the form of a Docker container and runs as a Kubernetes job.

To start you need to register and deploy the Tracetest executor in your cluster using the Testkube CLI. Run the command below in your terminal.

kubectl testkube create executor --image kubeshop/testkube-executor-tracetest:latest --types "tracetest/test" --name tracetest-executor --icon-uri icon --content-type string --content-type file-uri

[Output]
Executor created tracetest-executor 🥇

Trigger a Trace-based Test in Tracetest with Testkube

In the Tracetest Web UI, click the ⚙️ button in the top right. Then click Test Definition.

This will open a YAML definition for the test run.

Save this into a file called test.yaml.

# test.yaml

type: Test
spec:
  id: RUkKQ_aVR
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
      method: GET
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - name: Database queries less than 500 ms
    selector: span[tracetest.span.type="database"]
    assertions:
    - attr:tracetest.span.duration  <  500ms

Execute the following command to create the test object in Testkube. Do not forget to provide the path to your Tracetest definition file using the --file argument, and also the Tracetest server endpoint using the TRACETEST_ENDPOINT --variable.

Remember that your TRACETEST_ENDPOINT should be reachable from Testkube in your cluster. Use your Tracetest service's CLUSTER-IP:PORT. E.g: 10.96.93.106:11633.

kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-test --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-test 🥇

Opening the Testkube Dashboard will show the test is created successfully.

Finally, to run the test, execute the following command, or run the test from the Testkube Dashboard.

kubectl testkube run test --watch pokeshop-tracetest-test

Here's what the Testkube CLI will look like if the test fails.

[Output]

Type:              tracetest/test
Name:              pokeshop-tracetest-test
Execution ID:      641885f39922b3e1003dd5b6
Execution name:    pokeshop-tracetest-test-3
Execution number:  3
Status:            running
Start time:        2023-03-20 16:12:35.268197087 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

  Variables:    1
  - TRACETEST_ENDPOINT = http://10.96.93.106:11633

Getting logs from test job 641885f39922b3e1003dd5b6
Execution completed
🔬 Executing in directory :
 $ tracetest test run --server-url http://10.96.93.106:11633 --definition /tmp/test-content737616681 --wait-for-result --output pretty
✘ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test)
    ✘ Database queries less than 500 ms
        ✘ #2b213392d0e3ff21
            ✘ attr:tracetest.span.duration  <  500ms (502ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=2b213392d0e3ff21)
        ✔ #7e6657f6a43fceeb
            ✔ attr:tracetest.span.duration  <  500ms (72ms)
        ✔ #6ee2fb69690eed47
            ✔ attr:tracetest.span.duration  <  500ms (13ms)
        ✘ #a82c304a3558763b
            ✘ attr:tracetest.span.duration  <  500ms (679ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=a82c304a3558763b)
        ✔ #6ae21f2251101fd6
            ✔ attr:tracetest.span.duration  <  500ms (393ms)
        ✔ #2a9b9422af8ba1a8
            ✔ attr:tracetest.span.duration  <  500ms (61ms)
        ✔ #010a8a0d53687276
            ✔ attr:tracetest.span.duration  <  500ms (36ms)
        ✘ #895d66286b6325ae
            ✘ attr:tracetest.span.duration  <  500ms (686ms) (http://10.96.93.106:11633/test/RUkKQ_aVR/run/2/test?selectedAssertion=0&selectedSpan=895d66286b6325ae)

And, here's the Testkube Dashboard.

If the test passes, it'll look like this in the terminal.

[Output]

Type:              tracetest/test
Name:              pokeshop-tracetest-test
Execution ID:      6418873d9922b3e1003dd5b8
Execution name:    pokeshop-tracetest-test-4
Execution number:  4
Status:            running
Start time:        2023-03-20 16:18:05.60245717 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

  Variables:    1
  - TRACETEST_ENDPOINT = http://10.96.93.106:11633

Getting logs from test job 6418873d9922b3e1003dd5b8
Execution completed
🔬 Executing in directory :
 $ tracetest test run --server-url http://10.96.93.106:11633 --definition /tmp/test-content1901459587 --wait-for-result --output pretty
✔ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
    ✔ Database queries less than 500 ms

✅ Execution succeeded
Execution completed ✔ Pokeshop - List (http://10.96.93.106:11633/test/RUkKQ_aVR/run/3/test)
    ✔ Database queries less than 500 ms

And, like this in Testkube Dashboard.

Running Scheduled Trace-based Tests

Integrating with Testkube enables you to add Tracetest to the native CI/CD/GitOps pipeline in your Kubernetes cluster. This allows for scheduled test runs on set intervals, also called synthetic tests. Now with trace-based testing available, full, in-depth assertions against trace data is available, not just a response.

By using Testkube's scheduling, you can trigger the same test you defined above every minute. It works by providing a CRON schedule. You’ll add an additional --schedule="*/1 * * * *" flag.

kubectl testkube create test --file ./test.yaml --type "tracetest/test" --name pokeshop-tracetest-scheduled-test --schedule="*/1 * * * *" --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-scheduled-test 🥇

In your Testkube Dashboard, you'll see this test run continuously and get triggered every minute.

Running Event-driven Trace-based Tests

Event-based testing in Kubernetes is a critical aspect of ensuring the reliability and performance of microservices in Kubernetes. This testing approach involves observing events that are emitted by various components and services in the system to trigger tests against the system’s components under various conditions.

The main benefit of event-based testing is that it provides a more comprehensive testing approach than traditional unit, integration, and functional testing. With event-based testing, testers can simulate real-world scenarios and test the system's response to different types of input, load, and failure, while also verifying the system's ability to recover from such events.

To effectively perform event-based testing in Kubernetes, you’ll use Testkube as an event monitoring and management system that can capture and analyze the events generated by the system.

This sample will trigger a test when a deployment is scaled.

You’ve configured the Tracetest assertions to make sure all database queries finish within 500ms. Now, define a trigger that will run the trace-based test every time the deployment scales to ensure each replica satisfies the defined assertions.

Define a Test Trigger for the Deployment resource to run the trace-based test when a deployment-scale-update event occurs:

# testkube-trigger.yaml

apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
  name: deployment-scale-update-trigger
  namespace: testkube
spec:
  resource: deployment
  resourceSelector:
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: demo
  event: deployment-scale-update
  action: run
  execution: test
  testSelector:
    name: pokeshop-tracetest-test
    namespace: testkube

Save the file, name it testkube-trigger.yaml and apply it.

kubectl apply -f ./testkube-trigger.yaml

This will configure a trigger to run the test every time the demo app deployment is scaled. Try it yourself by running:

kubectl scale deployment demo-pokemon-api --replicas=4 -n demo

Moving back to the testkube dashboard you’ll see the test was triggered by the event.

Running Tests in a Test Suite or Transaction

Running singular, isolated, event-driven tests has its own important use cases and values. But, in the wild, you’ll more often rely on chaining multiple tests together into a transaction or test suite.

Both Testkube and Tracetest support such logical constructs.

In Tracetest they’re called transactions. In Testkube they’re called test suites.

Tracetest Transactions

Running end-to-end tests is not simple. It requires configuration before the actual test can be run, such as creating a new user or removing all items from a cart. Therefore, it's important to be able to execute multiple steps as part of your transaction. Tracetest introduces the concept of Transactions to achieve this goal.

A transaction is a group of steps executed in a defined order, where each step is a test that can access information exported by previous tests.

The main benefit of using transactions is the ability to chain tests together and use values obtained in one test as input for a subsequent test.

When a test is executed within a transaction, if it generates any outputs, the test outputs will be injected into the transaction context environment. After the outputs are injected, all subsequent tests to be run within the transaction will be able to reference those values with env:VARIABLE_NAME.

Note: Outputs generated by steps don't modify the selected environment. It only modifies the transaction run context object.

Tracetest allows tests to declare outputs. An output is a value that is extracted from a trace by providing a selector to choose which spans to use to get the information from, and an expression to get the value from the selected spans.

Run a Tracetest Transaction

Start by adding a new test by clicking Create > Create New Test > HTTP Request > Next > Choose Example (dropdown) > Pokeshop - Add (generates a sample test from the Tracetest demo) > Next > URL is prefilled with http://demo-pokemon-api.demo/pokemon > Create & Run.

The request body will be populated with this JSON.

{"name":"meowth","type":"normal","imageUrl":"https://assets.pokemon.com/assets/cms2/img/pokedex/full/052.png","isFeatured":true}

Navigate to Test > Outputs and click Add Test Output. Select the create pokeshop.pokemon database span.

span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]

The attribute to export the Pokemon’s id as a value is:

attr:db.result | json_path 'id'

Finally give it a name:

add_pokemon_db_result_id

Save the test output and publish the variables. Here’s what the YAML definition of this test looks like:

type: Test
spec:
  id: RAt3JIfVg
  name: Pokeshop - Add - In Transaction
  description: Add a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon
      method: POST
      headers:
      - key: Content-Type
        value: application/json
      body: '{"name":"meowth","type":"normal","imageUrl":"https://assets.pokemon.com/assets/cms2/img/pokedex/full/052.png","isFeatured":true}'
  outputs:
  - name: add_pokemon_db_result_id
    selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
    value: attr:db.result | json_path 'id'

Now you can edit the List Pokemon test to use this variable in an assertion.

Select the HTTP span.

span[tracetest.span.type="http" name="GET /pokemon?take=20&skip=0" http.method="GET"]

Set this assertion.

attr:http.response.body contains '${env:add_pokemon_db_result_id}'

Finally, give the assertion a name.

Save the test spec and click publish.

Now you can add a transaction and see how it works together. Click Create > Create New Transaction > Give it a name > Create.

Once created, add the Add and List tests to the transaction list. You’ll see the defined variables on the right, below the execution steps. The List test is passing as it is correctly asserting that the result of the List request contains the exported variable from the Add test.

Copy the transaction definition file and save it as a file named transaction.yaml.

# transaction.yaml

type: Transaction
spec:
  id: MnUSxIf4g
  name: Add+List Pokemon
  steps:
  - RAt3JIfVg
  - UQe_xIBVg

Now you can create a test in Testkube to trigger the transaction.

kubectl testkube create test --file ./transaction.yaml --type "tracetest/test" --name pokeshop-tracetest-transaction --variable TRACETEST_ENDPOINT=http://CLUSTER-IP:PORT

[Output]
Test created testkube / pokeshop-tracetest-transaction 🥇

Trigger the transaction in the same way as you did the test.

kubectl testkube run test --watch pokeshop-tracetest-transaction

[Output]
Type:              tracetest/test
Name:              pokeshop-tracetest-transaction
Execution ID:      641b19a9183f824d354b51d1
Execution name:    pokeshop-tracetest-transaction-2
Execution number:  2
Status:            running
Start time:        2023-03-22 15:07:21.796730144 +0000 UTC
End time:          0001-01-01 00:00:00 +0000 UTC
Duration:

  Variables:    1
  - TRACETEST_ENDPOINT = http://10.96.4.232:11633

Getting logs from test job 641b19a9183f824d354b51d1
Execution completed
🔬 Executing in directory :
 $ tracetest test run --server-url http://10.96.4.232:11633 --definition /tmp/test-content1800237311 --wait-for-result --output pretty
✔ Add+List Pokemon (http://10.96.4.232:11633/transaction/MnUSxIf4g/run/11)
    ✔ Pokeshop - Add - In Transaction (http://10.96.4.232:11633/test/RAt3JIfVg/run/19/test)
    ✔ Pokeshop - List - In Transaction (http://10.96.4.232:11633/test/UQe_xIBVg/run/10/test)
        ✔ Make sure pokemon id from the ADD is contained in the LIST

✅ Execution succeeded
Execution completed ✔ Add+List Pokemon (http://10.96.4.232:11633/transaction/MnUSxIf4g/run/11)
    ✔ Pokeshop - Add - In Transaction (http://10.96.4.232:11633/test/RAt3JIfVg/run/19/test)
    ✔ Pokeshop - List - In Transaction (http://10.96.4.232:11633/test/UQe_xIBVg/run/10/test)
        ✔ Make sure pokemon id from the ADD is contained in the LIST

Run a Testkube Test Suite

Test suites in Testkube are a way to orchestrate different test steps and entirely different testing frameworks to run in a suite. Your front-end team uses Cypress for browser tests, while the back-end team uses Tracetest. You may also have Postman collections testing various parts of your apps.

With test suites, you can orchestrate different test steps and combine them to run in a sequence. Even if each team runs its tests on its own, they can ultimately be combined and triggered from one location by a test suite.

Learn More About Kubernetes Testing

Combined, Testkube and Tracetest provide a comprehensive testing solution for Kubernetes applications. By utilizing Testkube triggers, you can automatically initiate trace-based tests with Tracetest, ensuring that your services are adhering to defined SLAs.

Tracetest provides detailed distributed trace data, allowing you to gain insight into the behavior of your application at a granular level, and ultimately create assertions against this data to write bullet-proof tests.

Would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out today by downloading it today!

To explore more options Testkube gives you, check out the documentation on test triggers. They enable you to trigger tests based on Kubernetes events. Want to learn more about Testkube? Read more here.

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.