Announcing the Tracetest integration with Keptn, the control plane for cloud-native application life-cycle orchestration

Today, we're happily announcing that Tracetest now works with Keptn, a control plane and micro platform for cloud-native application delivery and operations.

Check out this hands-on Demo example of how Tracetest works with Keptn!

What is Keptn?

Keptn is an open-source CNCF project. It’s a control plane and micro platform for cloud-native application delivery and operations following best practices of putting observability data into the driving seat of making automated decisions.

Keptn augments any standard Kubernetes cluster to support delivery based on automated quality gates and self-healing operations workflows. Enabled by a fully event-driven architecture, Keptn can be easily extended to integrate with new toolchains. The declarative approach versus traditional scripts enables enterprise-wide deployments and easy maintenance.

If you are interested in Keptn, also take a look at the latest evolution called KLT (Keptn Lifecycle Toolkit). KLT uses a K8s Operator approach which brings automated deployment observability and orchestration to your GitOps deployments.

What is Tracetest?

Tracetest is an open-source CNCF project. It allows you to quickly build integration and end-to-end tests, powered by your OpenTelemetry traces.

Tracetest leverages your OpenTelemetry traces to enable trace-based testing with assertions against your trace data at every point of the request transaction. To use Tracetest, simply point it to your existing trace data source or send traces directly to Tracetest.

We make it possible to

• Define tests and assertions for every microservice that a request passes through.
• This solution works with your existing distributed tracing, allowing you to build tests based on your already-instrumented system.
• Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
• Return both the response data and a distributed trace.
• Define assertions against both the response and trace data. This will ensure that both your response and underlying processes work correctly, quickly, and without errors.
• Save tests.
• Run the tests manually or via CI build jobs with the Tracetest CLI.

Tracetest Now Works with Keptn!

Tracetest now works with Keptn, by using the Keptn Job Executor Service plugin. You can now upload a Tracetest test definition and a CLI configuration to a service and run a test with this simple job definition below.

apiVersion: v2
actions:
- name: "Run tracetest on your service"
  events:
    - name: "sh.keptn.event.test.triggered"
  tasks:
    - name: "Run tracetest"
      files:
        - data/test-definition.yaml
        - data/tracetest-cli-config.yaml
      image: "kubeshop/tracetest:latest"
      cmd:
        - tracetest
      args:
        - --config
        - /keptn/data/tracetest-cli-config.yaml
        - test
        - run
        - --definition
        - /keptn/data/test-definition.yaml
        - --wait-for-result

Why is it important?

When running integration tests, it can be difficult to pinpoint where, within a network of microservices, an HTTP transaction fails. With tracing enabled, Tracetest can run tests with assertions against existing data throughout every service involved in the transaction. These tests can be used as part of your CI/CD process to ensure system functionality and catch regressions.

By enabling trace-based testing from within your Keptn application life-cycle orchestration, you can ensure every part of your multi-stage delivery is covered to match SLOs.

Read more about the Keptn integration with Tracetest, here.

Try Tracetest with Keptn

Install Keptn in your Kubernetes cluster by following the official docs, here. This shows both a simple Helm install and a full install of Keptn. Next, you need to install the Job Executor Service 0.3.x.

Finally, on your local machine, you need:

1. Keptn CLI installed
2. and already authenticated with Keptn API.

With Keptn set up, you can configure Keptn and Tracetest.

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI:

brew install kubeshop/tracetest/tracetest

Note: Check out the download page for more info.

From here, follow the official documentation to install the Tracetest server in your Kubernetes cluster. Once the server is installed, open up Tracetest Web UI in the browser and follow the instructions for connecting your trace data store with Tracetest.

Configure Tracetest with Keptn

Keptn works with two concepts. A Project and a Service. A Project is an element that holds multiple services forming an application in stages. And, a Service is the smallest deployable unit and is deployed in all project stages according to the defined order.

Usually, these resources are managed by Keptn during Sequences — a set of tasks for realizing a delivery or operations process.

Start by creating a Shipyard to contain the Project. Then, create a skeletal shipyard.yaml file with the following content:

apiVersion: "spec.keptn.sh/0.2.2"
kind: "Shipyard"
metadata:
  name: "<shipyard_name>"
spec:
  stages:
    - name: "production"
      sequences:
        - name: "validate-app"
          tasks:
            - name: "test"

Create a new project using the shipyard.yaml file:

keptn create project <project_name> -y -s shipyard.yaml

Note: Keptn may ask you to have a Git repository for this project to enable GitOps. If so, you need to create an empty Git repository and a Git token and pass it through the flags --git-remote-url, --git-user, and --git-token. More details about this setup can be seen on Keptn docs/Git-based upstream.

Now you can create a service:

keptn create service <service_name> --project <project_name> -y

Now, we will set up a job associated with the service, listening to a task event.

Create the tracetest-cli-config.yaml configuration file for the Tracetest CLI in your current directory, identifying the Tracetest instance that should run the tests:

scheme: http
endpoint: tracetest.tracetest.svc.cluster.local:11633

Create the test-definition.yaml Tracetest test definition in your current directory:

type: Test
spec:
  id: apdCx-h4g
  name: Pokeshop - List Pokemon
  description: Get a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
      method: GET
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[name = “Tracetest trigger”]
    assertions:
    - attr:tracetest.span.duration < 500ms

Note: The Tracetest test definition will vary based on your own test definitions.

Add these files as resources for your service in the production stage:

keptn add-resource --project <project_name> --service <service_name> --stage production --resource test-definition.yaml --resourceUri data/test-definition.yaml

keptn add-resource --project <project_name> --service <service_name> --stage production --resource tracetest-cli-config.yaml --resourceUri data/tracetest-cli-config.yaml

These files will be located in the folder data and will be injected into our Keptn job that we will set up in the next step.

Create a job-config.yaml:

apiVersion: v2
actions:
- name: "Run tracetest on your service"
  events:
    - name: "sh.keptn.event.test.triggered"
  tasks:
    - name: "Run tracetest"
      files:
        - data/test-definition.yaml
        - data/tracetest-cli-config.yaml
      image: "kubeshop/tracetest:latest"
      cmd:
        - tracetest
      args:
        - --config
        - /keptn/data/tracetest-cli-config.yaml
        - test
        - run
        - --definition
        - /keptn/data/test-definition.yaml
        - --wait-for-result

This job will run Tracetest every time a test event happens, listening to the event sh.keptn.event.test.triggered. This event is emitted by the test task on the validate-app sequence.

Add this job as a resource in Keptn:

keptn add-resource --project <project_name> --service <service_name> --stage production --resource job-config.yaml --resourceUri job/config.yaml

To guarantee that the job will be called by Keptn when you execute the production sequence, you need to configure the Job Executor Service integration in your project to listen to sh.keptn.event.test.triggered events.

You do that through the Keptn Bridge, the Web UI, by going to the project, choosing the Settings option, and Integrations.

Choose the job-executor-service integration, and add a subscription to the event sh.keptn.event.test.triggered and your project.

Finally, to see the integration running, we only need to execute the following command:

keptn trigger sequence validate-app --project <project_name> --service <service_name> --stage production

Now you should be able to see the sequence running in Keptn Bridge.

What's next?

Would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out today by downloading it today! Want to learn more about Keptn? Read more here, or check out more integrations here.

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.