DEV Community: Ken Hamric

Automating Tracetest Tests via Typescript or Javascript

Ken Hamric — Wed, 21 Feb 2024 17:39:39 +0000

A customer recently asked for the ability to run Tracetest tests programmatically from JavaScript or TypeScript so they could execute setup steps directly in code, then fire off a trace-based test. The need to execute setup steps, or to automated the running of Tracetest as part of other automation sequences are common patterns, so we have released a new npm package called @tracetest/client that allows you to programmatically add and run Tracetest tests.

Want a “codeless” way to write your setup or tear down steps? You can define steps using our test suite capability. Read “Setup and Teardown of Tracetest Tests with Test Suites”.

What is Trace-based Testing and Why You Should Care?

Before we dive in, let me answer the first question: “What the heck is Tracetest?”

It’s an observability-enabled testing tool for Cloud Native architectures. Distributed tracing was introduced to empower SREs to troubleshot the complex, interconnected systems being used in todays microservice and FaaS-based architectures. Tracetest leverages these distributed traces as part of testing, providing you with better visibility and testability via a technique known as trace-based testing.

Learning by doing!

This guide will focus on how to use the new NPM package by showing a real example running against a publicly accessible test environment. This example solves a common testing problem, externally setting up data in the test environment before running a test. The scenario is:

As a user,
I want to be able to test
our delete Pokemon endpoint
but need to add a Pokemon
via an external POST call first

We will explain the system under test, further discuss the test scenario and explain code for key sections, then have you clone the repo and run the Typescript code yourself.

The Test Environment

We are going to use a pre-deployed version of our Pokeshop demo which is sending its telemetry information to Jaeger. This instance is publicly available so you can run tests against it. You can even view the UI at https://demo-pokeshop.tracetest.io/. The code for this Pokeshop Demo is available at ‣.

The Test Scenario

This is the scenario we are going to execute via TypeScript against our Pokeshop test environment:

Add an entity via a POST to a REST endpoint. For this example, we are adding a Pokemon programmatically as a setup step using the Node.js built-in fetch command.
Run a Tracetest delete-pokemon test using the [@tracetest/client](https://www.npmjs.com/package/@tracetest/client) NPM package. We will:
- Authenticate to app.tracetest.io.
- Upsert a delete-pokemon Tracetest test.
- Run the delete-pokemon Tracetest test.
- Print a summary.

The code to execute this scenario is contained in the delete_test.ts file from a repo which we will clone and run locally further in this article. First, let’s discuss the code for the key sections that are utilizing the [@tracetest/client](https://www.npmjs.com/package/@tracetest/client) NPM package.

Breakdown and Explanation of Key Sections

Instantiate Tracetest & Authenticate to app.tracetest.io with an API Token

This code does a few things:

Instantiates the module.
Defines an API token - read more about tokens in the included comment.
Defines the tracetest object using the await Tracetest(TRACETEST_API_TOKEN) call. We will use this tracetest object in future calls to interact with Tracetest.

import Tracetest from '@tracetest/client';

// To use the @tracetest/client, you must have a token for the environment. This is created on the Settings 
// page under Tokens by the administrator for the environment. The token below has been given the 'engineer'
// role in the pokeshop-demo env in the tracetest-demo org so you can create and run tests in this environment.
// Want to read more about setting up tokens? https://docs.tracetest.io/concepts/environment-tokens

const TRACETEST_API_TOKEN = 'tttoken_4fea89f6e7fa1500'; 

console.log('Lets use the TRACETEST_API_TOKEN to authenticate the @tracetest/client module...')
const tracetest = await Tracetest(TRACETEST_API_TOKEN);

Upsert the delete-pokemon Test

Open the delete_pokemon.yaml file which contains the delete-pokemon test and upsert it to the current environment. This is similar to running the Tracetest CLI command: tracetest apply test -f delete_pokemon.yaml.

// Lets pull in the delete-pokemon test from a file
let deleteTest = fs.readFileSync('delete_pokemon.yaml', 'utf-8');

const test = await tracetest.newTest(deleteTest);

This command returned a test object which is used in the next step.

Run the delete-pokemon Tracetest Test

Run the test while passing variables. The variable in this example contains the id of the Pokemon to be deleted in the delete-pokemon test.

// run deletes pokemon test
console.log('Running the delete-pokemon test...');
const run = await tracetest.runTest(test, { variables: getVariables(String(pokemonId)) });
run.wait();

Print a Summary of the Test Result

Prints a summary of the test result.

console.log(await tracetest.getSummary());

Run the TypeScript Script

Enough talking… lets see this TypeScript script run locally. You will need Node and npm installed - see the Downloading and installing Node.js and npm instructions. You should be able to run both node -v and npm -v before trying this example.

First, clone the repo:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/setup-of-tracetest-tests

Open the delete_test.ts file in this directory. You will see the key sections discussed earlier in this article. To run the example, you only need to execute these commands:

npm install typescript -g
npm i @tracetest/client
npm i # to install dev dependencies
npm run build
npm start

Upon running the program, you will see:

Great! You successfully ran delete_test.ts . The tracetest.getSummary() shows a brief summary of the test and provides a link to the test run on app.tracetest.io. To see the test response, you will need to be a member of this “Pokeshop Demo” environment. To join, click this link:

➡︎ Join the Pokeshop Demo ⬅︎

You will be prompted to sign in with either your GitHub or Google account. Once logged in, click the “Join organization” button to get access to the Pokeshop demo environment.

Great - you are now a member of the organization with access to the Pokeshop demo environment!

You can then follow the link to the test run, click on the “Test” tab and see the results for the delete-pokemon test.

Congrats - you successfully ran the example. Want to look at another example? Checkout the Programmatically triggered trace-based tests using Tracetest and TypeScript quick start guide.

What’s next?

Now that you have seen how to run Tracetest tests via your own TypeScript code, you are now ready to create your own tests and embed them in custom flows!
To use Tracetest to test your own environment, follow these steps:

Create a new organization and environment.
Use the environment setup wizard to enable your Tracetest environment to run tests against and collect distributed trace information from your system under test.
Start creating tests!

Have questions? The devs and I hang out in the Tracetest Slack channel - join, ask, and we will answer!

Setup and Teardown of Tracetest Tests with Test Suites

Ken Hamric — Wed, 21 Feb 2024 17:33:44 +0000

A customer recently asked what the best way to run setup or tear down steps before running a test. Since this is a common pattern, we felt a pair of articles discussing it would be helpful. This article will focus on using the test suite capability to chain together and pass variables between Tracetest tests.

We also support using the @tracetest/client NPM package to programmatically execute setup steps and call the Tracetest test as part of a JavaScript or TypeScript program.

Want to write your setup process in JavaScript or TypeScript instead? Read “Automating Tracetest Tests via TypeScript or JavaScript”.

What is Trace-based Testing and Why You Should Care?

Before we dive in, let me answer the first question: “What the heck is Tracetest?”

It’s an observability-enabled testing tool for Cloud Native architectures. Distributed tracing was needed to empower SREs to troubleshot the complex, interconnected systems being used in microservices and FaaS-based architectures. Tracetest can be leveraged in your preproduction environments by DevOps, QAs, or SREs to test critical flows. Since it uses your existing observability tooling, its tests have more visibility into the overall system. This technique is called trace-based testing, and has several compelling advantages:

Makes it easier to write tests, as you can see all the systems involved in the flow.
Tests can be more powerful and assert on conditions that are not visible to black box tests.
When troubleshooting failed tests, you have the full distributed trace available, decreasing MTTR.

How to Handle Setup and Teardown of Tracetest Tests with Test Suites?

Tracetest’s codeless approach for this requirement utilizes test suites to chain together several tests into one comprehensive flow. Tracetest tests have the ability to trigger activity in the system under test, triggering activity based on the following types of calls:

HTTP
GRPC
Message queue

Using this capability, you can run “setup tests” before the main test and “clean up” tests after it.

This article describes a test scenario, the two tests involved, and provides instructions so you can download and run the examples yourself.

The Test Scenario

This test scenario is a common one. Adding an entity so you can then test the deletion of it.

The system we will be testing if a Pokemon was deleted successfully. You’ll add and delete a Pokemon with two tests. The tests are:

Adds an entity as a setup step. For this example, we are adding the Pokemon Fearow!
Run a delete test to verify that the deletion removes the entity we added in the setup step. This is the main flow we are interested in testing. Hopefully we can say “Goodbye Fearow!”.

We are going to use a pre-deployed version of our Pokeshop demo which is sending its telemetry information to Jaeger. This instance is publicly available so you can run tests against it.

You can even view the UI, here. The code for this Pokeshop Demo is available at on GitHub at kubeshop/pokeshop.

To follow the examples and see the code below, clone our main repo:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/setup-of-tracetest-tests

The Delete Test

The delete test is contained in the file named delete_pokemon.yaml. This test does a DELETE call against a specific Pokemon based on the id of the Pokemon. We are passing the id of the Pokemon into this test using an environment variable using the syntax ${env:pokemon_id}.

This trace-based test has four test specifications. The first is a typical assertion you might see with any black box test framework:

Delete returns a 200 status code: Ensures the REST API returns a successful status.

The next 3, however, leverage the visibility enabled by trace-based testing to look deeper into the execution, allowing you to assert on what actually happened by looking at the data contained in the captured distributed trace:

Ensure we are deleting from the Redis cache also: A more insightful assertion, and one you cannot make with a normal, black box test.
All Database Spans: Processing time is less than 10ms: Check performance as part of the test so it will fail if a slow query is introduced.
Check that number of deleted rows from Postgres is one: Verify the delete actually occurs.

Here is the delete_pokemon.yaml test:

type: Test
spec:
  id: delete-pokemon
  name: Delete Pokemon
  trigger:
    type: http
    httpRequest:
      method: DELETE
      url: https://demo-pokeshop.tracetest.io/pokemon/${env:pokemon_id}
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    name: Delete returns a 200 status code
    assertions:
    - attr:tracetest.response.status = 200
  - selector: span[tracetest.span.type="database" db.system="redis" db.operation="del" db.redis.database_index="0"]
    name: Ensure we are deleting from the redis cache also
    assertions:
    - attr:tracetest.selected_spans.count = 1
  - selector: span[tracetest.span.type="database"]
    name: "All Database Spans: Processing time is less than 10ms"
    assertions:
    - attr:tracetest.span.duration < 10ms
  - selector: span[tracetest.span.type="database" name="delete pokeshop.pokemon" db.system="postgres" db.name="pokeshop" db.user="ashketchum" db.operation="delete" db.sql.table="pokemon"]
    name: Check that number of deleted rows from Postgres is one
    assertions:
    - attr:db.result = 1

Now, lets look at the setup step which we will run before to add the Pokemon.

Using Another Tracetest Test to Set up the Data

We can define a new test to create the Pokemon, and then chain the creation and deletion test together within a test suite. First, let’s look at the setup_pokemon.yaml test. It is fairly basic, with 3 primary areas of interest:

It is doing a POST against the /pokemon endpoint and passing body data to create the Pokemon.
We are checking for a 201 response from this POST to ensure the creation of the Pokemon worked. This is just a basic sanity check so the entire test suite will fail if something goes wrong in this setup step.
We are getting the id of the Pokemon by selecting data returned in the trace, parsing it with a json_path statement, and assigning it to the output variable pokemon_id. This is enabled with Tracetest’s test output capability.

type: Test
spec:
  id: setup-pokemon
  name: Setup Pokemon
  trigger:
    type: http
    httpRequest:
      method: POST
      url: https://demo-pokeshop.tracetest.io/pokemon
      body: "{\n  \"name\": \"fearow\",\n  \"type\":\"normal,flying\",\n  \"imageUrl\":\"https://raw.githubusercontent.com/PokeAPI/sprites/master/sprites/pokemon/22.png\",\n  \"isFeatured\": false\n}\n"
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    name: Check to see the post worked
    assertions:
    - attr:tracetest.response.status = 201
  outputs:
  - name: pokemon_id
    selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    value: attr:tracetest.response.body| json_path '$.id'
  skipTraceCollection: true

Combining the Two Tests into a Test Suite

We can then create a test suite, include the setup_pokemon.yaml test as the first to run, then call our main test, delete_pokemon.yaml. Creating test suites is covered in our docs.

When you have finished creating a test suite, the automate tab provides the YAML definition for the suite. Below is the definition for the delete_test_suite.yaml test suite we created:

type: TestSuite
spec:
  name: Delete with Setup
  description: A test of the deletion end point with setup test which adds the pokemon as a setup step.
  steps:
  - setup-pokemon
  - delete-pokemon

The setup-pokemon test outputs the pokemon_id variable, and the delete-pokemon test expects a variable with this name via the ${env:pokemon_id}. With this syntax, when you run them as part of a test suite the first test will automatically pass the value and the second test will consume it. This lets the delete-pokemon test know to delete the Pokemon that was created in the setup-pokemon setup test.

How do we Run the Example?

Enough talking - lets see this test suite run! First, clone the repo if you did not earlier:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/setup-of-tracetest-tests

We are going to do several steps so we can see the test suite run:

Join the Tracetest pokeshop-demo environment so you can run and view your tests
Install the Tracetest command line tool (CLI) so you can upload the two tests
Use the CLI to run the test suite and view the test output
Open the test suite in Tracetest and examine the results from the suite and the delete test.

Let’s Do It!

Let’s run the test suite inside of the pokeshop-demo environment which we have available as part of our tracetest-demo organization at app.tracetest.io. In order to connect and run tests in this environment, you will need to join it with this link:

➡︎ Join the Pokeshop Demo ⬅︎

You will be promoted to sign in with either your GitHub or Google account. Once logged in, click the “Join organization” button to get access to the Pokeshop demo environment.

Great - you are now a member of the organization with access to the Pokeshop demo environment! We will be running the test suite against this environment from the command line interface (CLI) tool, so you will need to install the CLI. The instructions are documented here, but for Mac, you can just type:

brew install kubeshop/tracetest/tracetest

Now, tell the command line what organization and environment we want to issue commands against by typing:

tracetest configure

You will need to select the tracetest-demo organization and the pokeshop-demo environment as shown below:

You’re ready to add the two tests to this environment so they can be run by the test suite. The tests are part of the repo you cloned, so just type the following:

tracetest apply test -f setup_pokemon.yaml
tracetest apply test -f delete_pokemon.yaml

Now that the definitions for the two tests have been created in the test environment, you can run the test suite, which chains the two tests together and executes them.

tracetest run testsuite -f delete_test_suite.yaml

Here is the output you should see after running the test suite:

This output shows that the “Delete with Setup” test suite was run successfully. It also shows the two tests that were run as part of the suite. Both the suite and the two tests show links which will allow you to view the results.

And here is what you will see if you click on the Delete Pokemon test results and click to see the “Test” tab results. It shows the 4 test specs you are running as part of the delete test.

Congratulations! You have walked through the process needed to create two tests, combine them into a test suite, and pass a test output from one to another. This is an easy way to chain tests together in order to perform setup tasks.

What’s next?

While I did not show an example, you can use the same test suite technique to run a teardown or cleanup process after a test. Rename the processes “teardown” rather than “setup” and run them after the test! 😃

Now that you have seen how to run setup steps with test suites, you are now ready to create your own tests! Follow these steps:

Create a new organization and environment.
Use the environment setup wizard to enable your Tracetest environment to run tests against and collect distributed trace information from your system under test.
Start creating tests!

To learn how to automate this same setup process in JavaScript or TypeScript instead, check out part 2 of called “Automating Tracetest Tests via Typescript or Javascript”.

Have questions? The devs and I hang out in the Tracetest Slack channel - join, ask, and we will answer!

The Lord of Playwright: The Two Traces

Ken Hamric — Thu, 08 Feb 2024 17:26:55 +0000

Test and Debug Critical End to End Flows with Playwright and OpenTelemetry

Today is a tale of two traces!

Playwright Traces and OpenTelemetry Traces are brought together to create one test to rule them all. Well actually, to test both your front end AND back end in one swing.

Using the power of Playwright to drive tests of your critical flows from the browser, this integration extends your Playwright tests to leverage OpenTelemetry and trace-based testing.

You can now:

Test, in depth! Apply assertions against the entire front end application AND back end application.
Troubleshoot failed tests, with traces. Both Playwright traces and OpenTelemetry-based distributed traces.
Stop the blame game. With a view of the the entire flow, and data from both Playwright and distributed traces, quickly determine the cause of the failure, enabling you to easily identify the developer that can help finish the quest.
Be the hero and give your developer all the data. Examine both Playwright traces and OpenTelemetry traces, all from one test. The devs will love, admire and respect you for it!
Complete the quest with better observability! As tracing data from instrumentation is used to build trace-based tests, developers will want to add more insightful and meaningful instrumentation. See your ability to fight bugs in your production system enhanced when armed with better observability data.

If you’re eager to start, these instructions provide a full example, with an instrument Cloud Native app! Want to see some examples right away of Tracetest infused Playwright tests? Join the environment and look at the three Playwright tests, and tune in for a live workshop on February 15th!

https://www.youtube.com/watch?v=mmOAZxYdTYw

The Heroes

To start, lets identify the heroes involved in this saga.

Playwright is the fastest growing web testing framework. Playwright enables end-to-end (E2E) tests to be created by synthetically driving a headless browser session and enabling assertions to verify the system performs as expected. With Playwright traces, you can review a test after the script has run and identify exactly what occurred.

OpenTelemetry is the fastest growing Cloud Native Computing Foundation (CNCF) project. It standardizes the instrumentation and collection of traces, metrics, and logs from applications, and is supported by all the major observability projects, languages, and tools. One standard to rule them all!

Lastly, located in a quiet shire at the edge of the world is a newcomer to the scene, Tracetest. Unassuming, but able to wield the power of trace-base testing and OpenTelemetry traces to shine a light in the black box, revealing any hidden gremlins, orcs or nasty spiders (Shelob anybody?) lurking in the dark places.

The Quest: Test the Untestable - Complex, Cloud Native Systems

While capable of doing much good, Cloud Native systems have introduced a level of complexity heretofore not seen in the West. Or the East.

Microservices, created by different races, in different places. Elves, Dwarves, Men… or at least C#, Go, and React developers, working from afar, from office or from home. Separated into teams, rarely talking, with distrust of the other’s strange ways and weird coding practices. Can anybody bring these groups together?

The challenge falls to the pre-deployment team, who, like the hero Aragorn, a.k.a. Strider, go by several names. DevOps Engineer. SRE. Cloud, Automation, Infrastructure, or CI/CD Engineer. Their role is to bring together these pieces into a coherent, working whole, that is prepared and ready for the rigors of a production environment and the ultimate challenge - Customers! But what can they place their hopes in?

End-to-End Tests!

Yes - true end-to-end tests are the answer. Use Playwright to cover the critical flows in the cloak of actions and assertions, testing to make sure the interactions driven from the browser will return expected results. Catch the errors in your CI/CD flow before deploying. But is this enough?

No, for evil lurks in the back end also. The Go and C# developers have complex and mysterious ways.

While the API gateway may return a 200 to the browser, deep in the system, asynchronous gRPC processes may be timing out or returning error status codes. Systems may be using wrong versions of specifications, and processes may not be listening to the correct messages from your Kafka queue. Critical back-end flows, like sending the email on sign up or writing to the database fast enough may not be occurring.

All hidden from the benevolent view of your Playwright tests by a black box, containing other black boxes, with many differing technologies and arcane logic hidden within.

Flip the Light Switch on the Black Box(es)

Much like carrying a vial of the Light of Earendil's Star, embedding a trace-based test into your Playwright test allows you to see the surrounding dangers lurking in your system. But also to set assertions that verify the correct operation of the complete system. What does this magic look like in code?

import { test, expect } from "@playwright/test";
import Tracetest, { Types } from "@tracetest/playwright";
const { TRACETEST_API_TOKEN = "" } = process.env;

let tracetest: Types.TracetestPlaywright | undefined = undefined;

test.describe.configure({ mode: "serial" });

const definition = `
  type: Test
  spec:
    id: UGxheXdyaWdodDogaW1wb3J0cyBhIHBva2Vtb24=
    name: "Playwright: imports a pokemon"
    trigger:
      type: playwright
    specs:
    - selector: span[tracetest.span.type="http"] span[tracetest.span.type="http"]
      name: "All HTTP Spans: Status  code is 200"
      assertions:
      - attr:http.status_code   =   200
    - selector: span[tracetest.span.type="database"]
      name: "All Database Spans: Processing time is less than 100ms"
      assertions:
      - attr:tracetest.span.duration < 2s
    outputs:
    - name: MY_OUTPUT
      selector: span[tracetest.span.type="general" name="Tracetest trigger"]
      value: attr:name
    `;

test.beforeAll(async () => {
  tracetest = await Tracetest({ apiToken: TRACETEST_API_TOKEN });
  tracetest.setOptions({
    "Playwright: imports a pokemon": {
      definition,
    },
  });
});

test.beforeEach(async ({ page }, { title }) => {
  await page.goto("/");
  await tracetest?.capture(title, page);
});

// optional step to break the playwright script in case a Tracetest test fails
test.afterAll(async ({}, testInfo) => {
  testInfo.setTimeout(60000);
  await tracetest?.summary();
});

test("Playwright: creates a pokemon", async ({ page }) => {
  expect(await page.getByText("Pokeshop")).toBeTruthy();

  await page.click("text=Add");

  await page.getByLabel("Name").fill("Charizard");
  await page.getByLabel("Type").fill("Flying");
  await page
    .getByLabel("Image URL")
    .fill("https://upload.wikimedia.org/wikipedia/en/1/1f/Pok%C3%A9mon_Charizard_art.png");
  await page.getByRole("button", { name: "OK", exact: true }).click();
});

test("Playwright: imports a pokemon", async ({ page }) => {
  expect(await page.getByText("Pokeshop")).toBeTruthy();

  await page.click("text=Import");

  await page.getByLabel("ID").fill(Math.floor(Math.random() * 101).toString());
  await page.getByRole("button", { name: "OK", exact: true }).click();
});

test("Playwright: deletes a pokemon", async ({ page }) => {
  await page.locator('[data-cy="pokemon-list"]');

  await page.locator('[data-cy="pokemon-card"]').first().click();
  await page.locator('[data-cy="pokemon-card"] [data-cy="delete-pokemon-button"]').first().click();
});

Track the Uruk-hai (or other Bugs) using Traces

In much the same way that Aragorn used footprints, a dropped elven brooch, and other traces to hunt the Uruk-hai across the stretches of Middle Earth, the entire team can benefit from the dual traces left as artifacts from these Tracetest enhanced Playwright tests.

Use the Playwright traces to see exactly what was occurring in the browser, step back in time to see the browser state between each action, and view a video of the entire flow.

Then, hop into the Tracetest test run. View the full distributed trace, showing the trace spans and attributes captured from both the browser and across your back-end services. See exactly where the trace-based test assertions failed, and inspect the services above and below this point.

With both types of traces, it will be much easier to test for failures across the entire system. Once a test fails, the wealth of data will enable you to identify the area in the system that is causing the problem and quickly assign the bug to the proper team. By providing that team with the same set of traces, they will easily be able to rectify the issue, improving the time to resolve and push a correction.

Shine a Light in the Black Box with OpenTelemetry & Tracetest

Much in the same way the pieces of the sword Andúril were forged together from the shards of Narsil to serve Aragorn, modern day heroes in DevOps, Observability, and SRE teams can combine OpenTelemetry, Playwright, and Tracetest to produce a tool to bring peace and harmony to Middle Earth… or at least to your distributed system.

https://www.youtube.com/watch?v=Wv38LNleUpE

Want to undertake this quest? Do you think you’re ready? You will need:

To use Playwright for your front-end tests. Using Cypress instead? Check this out.
Have observability, that hopefully works and plays well with the OpenTelemetry standard, configured for your back-end systems so you can produce distributed traces.
Use the OpenTelemetry browser instrumentation in your front-end app.
Sign up for an account on Tracetest so you can run trace-based tests.
See the instructions on how to include Tracetest libraries in your Playwright tests.

These instructions provide a full example, with an instrument Cloud Native app, an OpenTelemetry instrumented React front end, and Playwright tests with Tracetest included.

Want to see some examples of these Tracetest infused Playwright tests? We are running these tests in our demo environment. Join the environment and look at the three Playwright tests.

Have questions? Join the discussion with the nerds (yes, we like LOTR…) that created this supernatural fusion of Playwright and OpenTelemetry in the Tracetest Slack channel!

Cloud Native Observability and Cypress - an Unlikely Marriage

Ken Hamric — Fri, 26 Jan 2024 00:20:32 +0000

OpenTelemetry and distributed tracing are critical in enabling troubleshooting and monitoring of your complex, distributed, Cloud Native application.

Cypress is a powerful tool for testing front-end applications, enabling quick and confident releases of front-end changes. Imagine if you could combine Cypress with distributed traces.

Now you can! Use distributed traces directly in your Cypress tests to cover the entire distributed system, including both the front end and back end, with trace-based tests. Here’s how Tracetest and Cypress fell in love. 👇

Tracetest ❤️ Cypress

The Tracetest integration with Cypress provides:

Full observability. Cypress allows you to visually debug test runs. With a distributed trace captured with each test run, you will also have visibility of exactly what happened across the full front end and back end system.
Faster MTTR for failed tests. Having full observability will allow you to quickly identify the root cause of failed tests, determine whether the problem is in the frontend or backend, and easily identify the correct team to fix the problem. Armed with data from both Cypress and the distributed trace, the team correcting the bug will have the necessary information to find a solution quickly.
Actual end-to-end tests. The industry sometimes calls front end tests ‘end to end’, but they are not. Frontend tests can fully test the front end, and even assert on values returned by API calls, but it cannot verify deeply. In today’s asynchronous, message-based architectures, an API calls often initiates a full process, but quickly returns a status code of 200 to say ‘we have the request’. It then relies on multiple backend systems, developed by different teams, to properly execute a flow. Checking that the full process works as expected is an actually end-to-end test. Trace-based tests leverage the visibility empowered by a distributed trace to allow assertions to be placed on the systems through out the flow, providing actual end-to-end testing.
Increased confidence to deploy. With your ‘golden paths’ covered by real end-to-end tests, your teams can release with confidence, knowing the tests running as part of your CI/CD process are ensuring critical flows.

Well, the marriage has happened, and we would like to invite you to attend the reception party!

Tracetest, the leading trace-based test tool, now works directly with your current Cypress tests to fully test your instrumented application. Bring your favorite observability tracing backend, such as Jaeger, Tempo, Honeycomb, AWS X-ray, Dynatrace, Sumo Logic, or 9 others to the event, and let’s party!

Show Me The Wedding Photos!

Let’s look at the rich artifacts provided by these observability enabled Cypress tests.

First, you get the rich results provided by a Cypress test:

Second, you get a distributed trace, showing entire process flow provided by instrumentation in your frontend and backend:

Finally, using trace-based testing assertions, you can have your Cypress test also verify the proper execution of the full system:

As shown, trace-based tests can assert the following:

Ensure the JavaScript bundle in the UI loads in less than 300 ms.
All HTTP internal calls between systems returned a successful status code.
Check to make sure a particular service received a message and processed it.
See if a call to an external service returned the proper value.
See if a particular, or all, database calls take less than 100 ms.

With these assertions created and included in the Cypress test, the test now can assert across the entire system, verifying both functionality and performance.

What To Bring to the Reception Party?

There are a few requirements:

Use Cypress for your frontend tests. If you use Playwright… stayed tuned for an upcoming event, as this particular party is not for you!
Have observability, that hopefully works and plays well with the OpenTelemetry standard, configured for your backend systems so you can produce distributed traces.
Use the OpenTelemetry browser instrumentation in your frontend app.
Sign up for an account on Tracetest so you can run trace-based tests.
See the instructions on how to include Tracetest libraries in your Cypress tests.

The instructions provide a full example, with an instrument cloud native app, an OTel instrumented react front end, and Cypress tests with Tracetest included. This is an example of a Cypress test which includes a trace-based test:

import Tracetest, { Types } from '@tracetest/cypress';

const TRACETEST_API_TOKEN = Cypress.env('TRACETEST_API_TOKEN') || '';

let tracetest: Types.TracetestCypress | undefined = undefined;

const definition = `
type: Test
spec:
  id: aW1wb3J0cyBhIHBva2Vtb24=
  name: imports a pokemon
  trigger:
    type: cypress
  specs:
  - selector: span[tracetest.span.type="http"] span[tracetest.span.type="http"]
    name: "All HTTP Spans: Status  code is 200"
    assertions:
    - attr:http.status_code   =   200
  - selector: span[tracetest.span.type="database"]
    name: "All Database Spans: Processing time is less than 100ms"
    assertions:
    - attr:tracetest.span.duration < 2s
  outputs:
  - name: MY_OUTPUT
    selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    value: attr:name
`;

describe('Home', { defaultCommandTimeout: 60000 }, () => {
  before(done => {
    Tracetest({ apiToken: TRACETEST_API_TOKEN }).then(instance => {
      tracetest = instance;
      tracetest
        .setOptions({
          'imports a pokemon': {
            definition,
          },
        })
        .then(() => done());
    });
  });

  beforeEach(() => {
    cy.visit('/', {
      onBeforeLoad: win => tracetest.capture(win.document),
    });
  });

  // uncomment to wait for trace tests to be done
  after(done => {
    tracetest.summary().then(() => done());
  });

  it('create a pokemon', () => {
    cy.get('[data-cy="create-pokemon-button"]').should('be.visible').click();
    cy.get('[data-cy="create-pokemon-modal"]').should('be.visible');
    cy.get('#name').type('Pikachu');
    cy.get('#type').type('Electric');
    cy.get('#imageUrl').type('https://oyster.ignimgs.com/mediawiki/apis.ign.com/pokemon-blue-version/8/89/Pikachu.jpg');

    cy.get('button').contains('OK').click();
  });

  it('imports a pokemon', () => {
    cy.get('[data-cy="import-pokemon-button"]').click();
    cy.get('[data-cy="import-pokemon-form"]').should('be.visible');

    cy.get('[id="id"]')
      .last()
      .type(Math.floor(Math.random() * 101).toString());
    cy.get('button').contains('OK').click({ force: true });
  });

  it('deletes a pokemon', () => {
    cy.get('[data-cy="pokemon-list"]').should('be.visible');
    cy.get('[data-cy="pokemon-card"]').first().click().get('[data-cy="delete-pokemon-button"]').first().click();
  });
});

Attending the Reception Party

Tune in live for a webinar workshop on January 24th at 9am PT (12pm ET) to learn hands-on how to enhance your existing Cypress tests with trace-based testing!

A Promising Union

With the combination of distributed tracing, Tracetest, and Cypress, true end-to-end tests that matches Martin Fowler’s definition, is finally achieved.

An end-to-end test verifies that a system meets external requirements and achieves its goals, testing the entire system, from end to end.

With these tests in place, your entire dev team can release at high velocity with greater confidence.

Want to see some examples of these Tracetest infused Cypress tests? We are running these tests in our demo environment. Join the environment and look at the three Cypress tests:

Have questions? Join the discussion with the matchmakers of this union in the Tracetest Slack channel!

Tracetest - Recapping 2023

Ken Hamric — Thu, 11 Jan 2024 13:08:29 +0000

What a year 2023 was for Tracetest and Trace-based Testing! Let's start by looking at it through numbers:

Grew from <400⭐ to 768⭐!
Had 52 public releases - exactly one per week!
Added integrations for 9 additional tracing backends: DataDog, AWS X-Ray, Honeycomb, Azure Apps Insight, SigNoz, Dynatrace, Lightstep, Elastic, and New Relic. This makes a total of 15 supported tracing backends, plus any others that use the OpenTelemetry Collector.
Added 5 integrations: Github Actions, Tekton, Keptn, Testkube, and our favorite, k6. Expect many more of these in 2024!
Attended 10 different conferences across 3 different continents, including KubeCon Amsterdam and Chicago, Monitorama in Portland, DeveloperWeek in both San Mateo and Latin America, OpenSearch in Seattle, FOSDEM in Brussels, and others.
Published 3 user case studies: Sigma Software testing with k6+Tracetest, Uzufly testing from the browser with OpenTelemetry, and how Tyk is leveraging Tracetest for integration testing of their new native support of OpenTelemetry.

With all the activity, new users and use cases, releases, and improvements, it is hard to pick the most impactful, but four stand out:

k6 Integration

In February, Tracetest teamed up with k6 to provide deep assertions when running load tests. When running a load test against any complex system, being able to detect where the system is failing and verify the entire flow is critical.

The k6 integration allows trace-based tests to be run against each of the hundreds or thousands of runs in a load test, enabling deep detection across the full system.

Check out this 8 minute video guide on how to get started!

https://www.youtube.com/watch?v=cv-9-pkLqbY

Trace Analyzer

In May, the team introduced the Trace Analyzer as a means to help developers and teams instrument their applications easier and more consistently. We considered calling it the Trace Linter. Its purpose is to inspect the trace data and ensure it is following the OpenTelemetry semantic conventions, look for common problems and maintain standards, and help you find security problems. These additional validations can be run automatically as part of your CI/CD process, allowing you to verify the quality of your instrumentation across all your teams.

Inclusion in the OpenTelemetry Community Demo

OpenTelemetry is a CNCF incubating project promoting an open standards-based approach to instrumenting applications to enable observability. It is 'the' standard in the observability space, with widespread industry support.

The community behind OpenTelemetry provides an OpenTelemetry Demo which allows new users to understand how to instrument their own systems. This demo is a microservice-based architecture, written in 11 different languages, with over 12 services, and maintained by dozens of contributors.

In May of 2023, the community decided they needed better testing due to a number of breaking commits, and Tracetest was able to contribute a series of trace-based tests to ensure the quality of the releases. These tests are now run as part of the CI/CD process for the demo, making Tracetest a key component in ensuring successful releases for the project.

Tracetest Managed Platform

In October, the Tracetest commercial offering was released, and the open-source version was rebranded as Tracetest Core. Tracetest introduced several new features in addition to the ones already available in Tracetest Core.

Tracetest Core installs directly into the test environment. However, with ephemeral test instances, the results were not accessible once the tests were finished and the test instance was reclaimed. This resulted in the loss of one of the key benefits of trace-based testing - the ability to visually observe the entire process of a failed test run, including which assertions failed and what was happening in the system before and after the failure point.
Tracetest has a loyal following of customers using Tracetest to test FaaS (Function-as-a-Service) based systems. These customers do not want to manage servers, and the hosted Tracetest eliminates this unwanted additional work.
Tracetest eases both networking and installation. Tracetest Core requires a local server installation into Docker or Kubernetes. This made local development on the desktop harder, as network configuration was required. With Tracetest, you just install an agent on your local machine, in Docker, or in Kubernetes, and it immediately can listen for OpenTelemetry spans.
Tracetest Core is built for a single user and single environment. Tracetest enables organizations, with multiple environments, users, roles, and SSO.

Looking Forward

The team is excited to enter 2024! We will expand the scope of Tracetest by integrating it into the load test, synthetic monitoring, and front-end testing fields. Our first major announcement will be the introduction of true end-to-end testing using Trace-Based Testing (hint: with Cypress). Stay tuned for this thrilling and potentially groundbreaking release!

Would you like to learn more about Tracetest and what it brings to the table? Visit the Tracetest docs and try it out by downloading it today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

👨‍🚀 Tracetest Monthly Newsletter - November 2023

Ken Hamric — Mon, 11 Dec 2023 02:14:57 +0000

Join the Community Discord here to talk all things Tracetest, get instant support from our team, or simply say hi! If you like what we're doing, and haven't already, leave us a ⭐ on GitHub. Thank you for being part of the community. Your support and feedback are invaluable! ❤️

Hello, Tracetest Community! 👋

We are thrilled to share the latest updates and exciting news from the Tracetest community! Take a look at what's happened in the last month. 👇

Hot tip of the day!

Parent-child Relation and :First Filtering

There are times when a span you want to select appears several times in a trace, but you want to test a specific instance of it. To select the particular span, you can specify its parent or its position. For example, when trying to test the FeatureFlagService GetFlag operation, we see it matches 3 spans, but we want to apply an assertion against the one called by the get_product_list operation:

Using the information from the selectors docs, we can see how to select the first span that is a child of the ‘get_product_list’.

You can then create add assertions that apply to the specific span, verifying the value retrieved from the feature flag service by get_product_list is correct.

What's New with Tracetest 🎉

Tracetest Core v0.14.7 has been released: This update includes a brand new, lovely, test creation flow, skip trace capability, and more. Lots and lots of new capability released... we will cover the highlights below.

New! ‘skip trace’ allows black box tests! You can now run Tracetest on code bases you have not instrumented… yet. Run a black box test, instrument with OpenTelemetry, and then upgrade to a full end to end test.

Try our demos! We made it easy to try Tracetest, as well as play around with its features to learn more about our OpenTelemetry solution. We have added the OpenTelemetry Community Demo, an Astronomy Store, and our own Pokeshop API demo. Check out the docs and use the ‘join’ links for each demo to immediately run and create tests.

Invite Links. Aren’t these links to join the tracetest-demo organization and be registered as an engineer for the demo environments cool? Creating these magic invite links is part of the product now! Slack an invite to allow people to join a particular Org / environment with a particular role.

Run Agent on the Cloud (BETA). A Tracetest customer asked if they could use app.tracetest.io without running an agent locally? They have a large function as a service (FaaS) based system with lots of complexity, and want to use Tracetest to build tests. They have public endpoints, and wanted to send the trace data to tracetest.io via the OTel collector, having Tracetest run the agent. It is now in Beta!

Integrations 🤝

New Github Action! Making it easy to connect to Tracetest.io from Github Action Workflows. Try it!
Test Kafka Based Architectures! Our own Daniel Dias created an example and wrote a technical blog post documenting how to test event-driven systems with OpenTelemetry.

From the Community 🎙️

Tracetest was at KubeCon NA! Jorge and I attended OTel Day on Monday and talked to people at the Tracetest booth Tuesday through Thursday. 100+ great conversations with current and future users.

Didn’t Make it to Kubecon? Read about what we saw in our ‘Observability at KubeCon’ article.

KCD Austria Talk. Adnan Radic’s conference talk, Distributed tracing is awesome, testing is not… until now, has been published on Youtube. Watch it!
Tracetest is part of the OTel Demo CI/CD Pipeline! Austin Parker, one of the co-founders and maintainers of OpenTelemetry, just added Tracetest tests to the CI/CD process for the official OTel Demo. Our team wrote these tests earlier this summer to help maintain quality of this important Open Source project, but they were run manually… until now! It is now automated as part of the build system.

Contribute to the OpenTelemetry Community Demo! Our very own Adnan Rahic contributed to the OTel Demo, then wrote an article teaching others how. Read: The Power of Traces: Learn by Contributing to OpenTelemetry.

Like what we do?

Do you have a specific use case you would like us to cover? Do you want to contribute to Tracetest?

At Tracetest, we welcome all contributions!

Not sure where to start? You can:

Visit our Community Page.
Check out our contributing guidelines here.
Book a meeting with us or chat with our team on Discord.

Thank you for being a part of the Tracetest community! We are committed to providing the necessary tools, knowledge, and resources to succeed in the field of tracing and testing. Let’s stay in touch and continue exploring. Stay tuned for more exciting updates in the coming months.

Super long recap, but we got a ton done in November! See you next month!

p.s. Adnan is out this week, but he has a great reason! Congratulations Adnan!!

OpenTelemetry is not just for Monitoring and Troubleshooting any longer. Announcing Tracetest Open Beta!

Ken Hamric — Thu, 05 Oct 2023 18:54:43 +0000

We are excited to announce the Open Beta launch of Tracetest, the best way to develop and test with OpenTelemetry!

With this tool, we enable a concept that, up to now, has merely been an industry buzzword - Observability-Driven Development (ODD). The promise of observability-driven development is that Observability will be “baked in” from the beginning of the development process, not just bolted on at the end to satisfy the need for visibility in production.

How will this shift occur?

Ted Young, co-founder of the OpenTelemetry project, told me in a recent conversation:

“In general, I would like to see more tools that are focused on development and instrumentation, not production.”

Currently, most Observability tools are focused on one thing - Production. Without tooling that enables developers to seamlessly instrument code, maintain consistency, and write tests to verify functionality, Observability remains primarily a solution geared towards operations. This limits its potential to deliver value to other parts of the organization.

How will Tracetest fill the gap?

Tracetest is already the leading trace-based testing solution on the market today, and part of the OpenTelemetry Demo repo. It uses the power of distributed tracing, offering comprehensive insights into the inner workings of your code during testing. When you can observe what your code is doing and the interaction between services, writing tests becomes a trivial process.

Tracetest achieves this by providing a comprehensive service map of your system under test. In essence, Tracetest leverages distributed traces, allowing users to construct tests based on these traces. This enables the creation of specific criteria and assertions tailored to each unique end-to-end flow.

Image 1: Tracetest Assertions based on Distributed Traces

Adoptees see the time required for writing integration or end-to-end tests across their Cloud Native systems drop from hours to minutes. It is platform agnostic, working with all the major observability vendors such as Datadog, New Relic, Honeycomb, Dynatrace, and 10 others.

Image 2: Tracetest Data Store Integrations

Additionally, the individual developer experience surrounding instrumentation of your code is vastly improved with a newly introduced (and free) “localMode”.

Image 3: Tracetest localMode

With localMode, developers can run a simple agent in their console, Docker, or Kubernetes workspace and immediately begin seeing and improving their instrumentation.

Tracetest will also let you know if you are not following the OpenTelemetry semantic conventions, and you can use the Analyzer to enforce the OpenTelemetry specification standards across your dev teams.

Image 4: Tracetest Analyzer

Key Features of Tracetest Core

The open-source version of Tracetest, which we are referring to as Tracetest Core, has had a warm reception, generating excitement across the industry and garnering over 700 stars in its first 18 months.

Tracetest Core offers a unique feature that makes building integration tests much faster. By using Distributed Tracing, it reduces the required time by 98%.

Here's how it works:

Trigger: Use tailored test triggers, like API calls (HTTP or gRPC) or Kafka streams, to get system responses and initiate tracing. Tracetest Core automatically adds essential headers to these requests for comprehensive tracing.
Trace: Tracetest Core uses a polling algorithm to query the existing tracing solution and gather a complete trace for analysis. The trace is shown in both a timeline and system view.
Test: Based on response and trace data, developers can create customized assertions. These assertions can check status codes, business data, span existence, anticipated flow, and more. For example, you can validate that all database calls complete within a specific time frame.
Automate: Tracetest Core can run in your CI/CD pipelines.

With Tracetest Core, creating integration tests is no longer a time-consuming task. Instead of taking days, you can now build comprehensive integration and end-to-end tests within minutes.

Why Create a SaaS Version of Tracetest?

Our users did point out some limitations with Tracetest Core.

It’s Heavy. Not super heavy, mind you, but it needs several parts and pieces to run - the core executable, Postgres, etc. Due to this, the install took a few minutes and was limited to Docker and Kubernetes environments. It’s not the “grab it and go” test tool we think it can be.

Networking is Hard. We are not all Cisco certified engineers or experts in Docker networking. Not being able to collect trace information directly from the environment you are working in made getting started with Tracetest harder than desired and lead to more support needs.

Server Installation is Not Ephemeral-friendly. GitHub Actions or other CI/CD environments you spin up, and then spin down, as part of the test environment resulted in a use pattern we did not expect - the installation of Tracetest server into a transient environment. Spin up Tracetest, run tests, shut it down. This eliminates one of the most valuable capabilities of Tracetest - seeing your trace-based test results via the UI. By viewing a failed test in the UI, you can see not just failed assertions, but also look at the system surrounding it using the full captured trace. It also allows you to adjust your assertions at that point and resave your test.

Image 5: Tracetest Test Specs

Monetization. We are a small, dedicated team working full time on Tracetest. We draw salaries, have house notes to pay, and have children and pets to feed. Putting Tracetest on a path to sustainability is critical to the success of continuing to develop Tracetest Core. There… we said the quiet part out loud.

Image 6: Tracetest’s Chief Happiness Officer Scout likes treats

Why Use Tracetest SaaS instead of Tracetest Core

Managed infrastructure. Tracetest runs in the cloud (available for on-prem deployments as well), where you have a central location for storing, viewing and running tests. No need to manage your own Tracetest Core instance. It’s a clear benefit for distributed systems, serverless, and microservice architectures.

It is super lightweight. With Tracetest you install a lightweight agent in your working environment, whether it's on your local machine, Docker, Kubernetes, Github Actions, or similar platforms.

This agent has two jobs when a test is run:

Trigger your local code via REST, gRPC, or Kafka.
Collect the distributed trace either directly or by reaching out to your observability solution.

It has a super easy install. For MacOS, there are just two commands:

brew install kubeshop/tracetest/tracetest
tracetest start

Click here to view installation instructions for other operating systems.

You can start visualizing traces and creating tests right away, whether it's ad hoc tests for code in development or complete integration tests.

Networking is Easy (Really!) Since you install the agent directly into the environment where you are running your application, there is no complex networking. When developing in localMode, the agent listens on the common OpenTelemetry Line Protocol (OTLP) on ports 4317 & 4318 automatically.

Tracetest is persistent, not ephemeral! You can work and run tests in ephemeral environments such as GitHub Actions using the Tracetest Agent, which sends test results to Tracetest. These results will be available for you to examine the full trace, diagnose the issue, or share with the team.

It is Free! Tracetest will be the easiest, most convenient tool to instrument and test your application. As a developer, you can launch the Tracetest Agent in the free localMode and immediately begin visualizing the trace, improving the quality of your instrumentation, and building tests.

I thought you needed $$$ to feed your dog? Eventually, you will want to run these tests in automated CI/CD pipelines, create a shared environment, configure the data store you are using for traces, and run the tests. These shared environments for automated testing will eventually be part of a paid plan. You can collect data from OTLP or connect to a wide range of observability backends, such as Grafana Tempo, OpenSearch, CloudWatch, and more. For the Beta period, these shared environments are also free as we seek feedback and input to improve Tracetest.

Try it! Give us feedback!

We are looking for feedback. Go to Tracetest.io and click “Get Started”. If you have questions, you can talk to us on the Discord channel. You can also reach out to me on LinkedIn.

Why did OpenTelemetry Ditch Black-box Tests?

Ken Hamric — Fri, 25 Aug 2023 19:19:58 +0000

Black-box testing has long been the recommended technique for testing back-end architectures. This tendency to rely on them has continued even as major architectural and technology shifts have occurred.

What are those shifts?

Monolith to distributed, microservice-based systems
Logs to distributed logging to distributed tracing
Synchronous, blocking based APIs to asynchronous handling of requests
Database centric design to message based architectures

Are black-box tests still adequate?

No.

Why?

This summer, the OpenTelemetry (OTel) Demo (see below) experienced growing pains due to the rapid pace of merged PRs by numerous contributors. This resulted in releases with broken features and lower than desired quality. It was occurring in spite of test coverage with a suite of black box tests having been in place for quite some time.

This article will look at how trace-based testing techniques, which in turn rely on modern observability techniques, were used to increase the reliability and confidence in releases. The lessons learned can also help you.

OpenTelemetry CNCF Project (OTel)

The OpenTelemetry Project, commonly referred to as OTel for short, was formed in 2019 when OpenCensus and OpenTracing merged. The goal was to provide one unified standard to collect observability data. Before this standard, developers implementing observability were forced to choose a particular vendor's application performance monitoring (APM) solution and instrument their application with vendor specific tooling. This led to vendor lock-in.

From the observability vendor perspective, maintaining instrumentation across the wide spectrum of languages and technologies was a daunting task, even for the larger vendors. By moving to an open standard with OTel, developers have a standard way to instrument their code and vendors can focus on how to visualize and analyze the data.

OpenTelemetry focuses on 3 major observability signals:

Logs
Metrics
Distributed Traces

Of these, distributed tracing is considered the most valuable and was the starting point for the OpenTelemetry Project. A distributed trace records the paths taken by requests as they propagate through multi-service architectures, showing you both call order, timing, and containing the details of each step.

A distributed trace has four components:

the trace
spans representing each step
attributes associated to each span
events associated to each span

Below is a typical trace:

OpenTelemetry Demo Microservice App

In April of 2021, the OpenTelemetry Project decided to create a demo application to showcase how to instrument a heterogeneous distributed application. You can install and run the OpenTelemetry Demo in Docker with these commands:

git clone https://github.com/open-telemetry/opentelemetry-demo.git
cd opentelemetry-demo/
docker compose up --no-build

💡 Note: If you're running on Apple Silicon, run docker compose build in order to create local images vs. pulling them from the repository.

The application, in keeping with the observability theme of the OTel project, is an Astronomy Store Front. This application has evolved to have 11 different languages and 12 major services, and it demonstrates both automatic and manual instrumentation:

The application is used by developers beginning to learn observability, enterprises adopting OTel that want to be able to examine a working application, and by vendors that want to demonstrate their observability related product working with a complex application. The list of companies that work with and rely on the demo is extensive:

The project is quite popular, with over 1K stars on GitHub and over 500 forks. It has had 86 unique contributors, and the pace of change is rapid. In just the past month, 10 different open source contributors have merged 43 PRs into the repo. These developers work for different companies, are located all over the world, and are submitting changes to different areas of the application.

Let’s add all of this up:

11 different languages
12 major services
Both automatic and manual instrumentation
10 different developers around the globe merging 43 PRs in the last month
Numerous technologies including Kafka, Redis, Postgres, gRPC, HTTP

This is considerable complexity, but not unlike most modern development environments which often have teams located in multiple areas, utilizing several development languages, a wide selection of technologies, and are committing code rapidly across numerous services. With this complexity, the risk of introducing errors when releasing is high.

What technique is widely used to reduce risk and increase confidence when releasing code

Tests!

Existing OpenTelemetry Demo Tests

The OpenTelemetry Demo has had test coverage for quite some time. The front end is covered with tests created in Cypress which were written when the front end was migrated to NextJS in August of 2022. These tests can be run with the following command:

docker compose run frontendTests

The back end was covered by black box tests against the gRPC endpoints that were written in AVA, a popular Node.js-based testing framework. They were added in July of 2022 with coverage across 9 of the services. These are straightforward black box tests, exercising a gRPC endpoint and verifying that the returned results were valid. Here is one of the AVA tests that verifies the checkout service, making sure the response to the call returns the proper data:

const checkoutClient = new oteldemo.CheckoutService(
  CHECKOUT_SERVICE_ADDR,
  grpc.credentials.createInsecure()
);
checkoutOrder = promisify(checkoutClient.placeOrder).bind(checkoutClient);

// --------------- Checkout Service ---------------

test("checkout: place order", async (t) => {
  const req = data.checkout;
  const res = await checkoutOrder(req);

  t.truthy(res.order.orderId);
  t.truthy(res.order.shippingTrackingId);
  t.truthy(res.order.shippingAddress);
  t.is(res.order.shippingCost.currencyCode, "USD");
});

These existing AVA tests can be run with the following command:

docker compose run integrationTests

Front-end tests, back-end black box tests. The world is great, and there are no problems!

Trouble in Testing the Distributed Application

Except there were problems. Changes were breaking the demo, resulting in many issues utilizing the demo application for both new and established users. The issue was highlighted in a thread in the otel-community-demo channel of the CNCF Slack posted in May of 2023. This thread was started by one of the lead contributors to the repo:

There was enough complexity in the Demo application that unintended consequences were occurring, reducing confidence when deploying changes or adding new features. Looking at how the architecture of the application had grown and the number of services involved, this is understandable:

What was needed was true end-to-end tests that could verify not only the response data from gRPC calls to the application, but could also verify that the entire application was working properly at a deeper level. Enter trace-based testing and Tracetest.

Trace-based Testing

Traditionally, creating end-to-end tests that go past simple black box testing require instrumenting your application so you have visibility to deeper interactions throughout the system. When looking at these types of tests, typically 90% of the code enables visibility across the various services and technologies, while only 10% of the code in the test actually defines the triggering action or assertions.

Trace-based testing is a technique that uses the instrumentation you have already enabled by making your code observable and allowing assertions to be created against data contained in it. It leverages the work your development team has already invested in instrumenting your application for observability. In doing so, trace-based testing dramatically reduces the effort and time involved in creating end-to-end tests.

Tracetest

Tracetest is an open-source tool which enables trace-based testing. Read more about Tracetest’s launch, here.

Tests can be created in both the web-based user interface and via a code editor in a YAML format. Tracetest works with any OTel compatible vendor or open source solution, including Jaeger, Grafana Tempo, Dynatrace, Datadog, NewRelic, etc. The Tracetest server can be installed in Docker or Kubernetes, and you will also be able to use it without a server via the upcoming cloud version.

The Tracetest team had previously contributed to the OpenTelemetry Community Demo in July of 2022 by porting the Go-based front end to Next.js, so was familiar with contributing to the project and happy to help the community by adding a complete suite of true end-to-end tests against the back-end services.

Trace-based Testing Suite

The new tests were created in early July on a service by service basis and can be viewed in the OpenTelemetry Demo repo, here. You can run the tests with the following command:

docker compose run traceBasedTests

For this article, we are going to look at the frontend-service tests which focus on the checkout process. These tests are run as a test suite, and go all the way from checking the recommendation service to adding a product to a cart and then checking out.

Looking at the last test, “Frontend: Checking out shopping cart”, we can see the structure of a Tracetest test. It is divided into two parts, the trigger and the test specs. The trigger, shown below, defines how the trace-based test is initiated:

type: Test
spec:
  id: frontend-checkout-shopping-cart
  name: 'Frontend: Checking out shopping cart'
  description: Simulate user checking out shopping cart
  trigger:
    type: http
    httpRequest:
      url: http://${env:FRONTEND_ADDR}/api/checkout
      method: POST
      headers:
      - key: Content-Type
        value: application/json
      body: |
        {
          "userId": "2491f868-88f1-4345-8836-d5d8511a9f83",
          "email": "someone@example.com",
          "address": {
            "streetAddress": "1600 Amphitheatre Parkway",
            "state": "CA",
            "country": "United States",
            "city": "Mountain View",
            "zipCode": "94043"
          },
          "userCurrency": "USD",
          "creditCard": {
            "creditCardCvv": 672,
            "creditCardExpirationMonth": 1,
            "creditCardExpirationYear": 2030,
            "creditCardNumber": "4432-8015-6152-0454"
          }
        }

As we can see from the code, this is an http-based trigger that hits the /api/checkout endpoint. The information needed for the POST is contained in the body section. Running this test will cause the trigger to fire, initiating a checkout process. Since the OpenTelemetry Demo produces a trace for any request, Tracetest can capture both the response of this POST and the distributed trace which the OpenTelemetry Demo app writes to Jaeger. It can then process these results using the test specs defined in the second part of this test:

specs:
  - selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    name: "It called the frontend with success"
    assertions:
    - attr:tracetest.response.status = 200
  - selector: span[tracetest.span.type="rpc" name="oteldemo.CheckoutService/PlaceOrder" rpc.system="grpc" rpc.method="PlaceOrder" rpc.service="oteldemo.CheckoutService"]
    name: "The order was placed"
    assertions:
    - attr:app.user.id = "2491f868-88f1-4345-8836-d5d8511a9f83"
    - attr:app.order.items.count = 1
  - selector: span[tracetest.span.type="rpc" name="oteldemo.PaymentService/Charge" rpc.system="grpc" rpc.method="Charge" rpc.service="oteldemo.PaymentService"]
    name: "The user was charged"
    assertions:
    - attr:rpc.grpc.status_code  =  0
    - attr:tracetest.selected_spans.count >= 1
  - selector: span[tracetest.span.type="rpc" name="oteldemo.ShippingService/ShipOrder" rpc.system="grpc" rpc.method="ShipOrder" rpc.service="oteldemo.ShippingService"]
    name: "The product was shipped"
    assertions:
    - attr:rpc.grpc.status_code = 0
    - attr:tracetest.selected_spans.count >= 1
  - selector: span[tracetest.span.type="rpc" name="oteldemo.CartService/EmptyCart" rpc.system="grpc" rpc.method="EmptyCart" rpc.service="oteldemo.CartService"]
    name: "The cart was emptied"
    assertions:
    - attr:rpc.grpc.status_code = 0
    - attr:tracetest.selected_spans.count >= 1

These test specs define the assertions to run on specific parts of the data returned from the test. If we look at the first test spec, we can see that it, similar to a black box test, checks against the response data to make sure the HTTP POST returns a status code of 200:

- selector: span[tracetest.span.type="general" name="Tracetest trigger"]
  name: "It called the frontend with success"  
  assertions:
  - attr:tracetest.response.status = 200

More interesting is the next test specification. It looks into the trace attributes being exposed by the PlaceOrder method to make sure that the item count being checked out is equal to 1:

- selector: span[tracetest.span.type="rpc" name="oteldemo.CheckoutService/PlaceOrder" rpc.system="grpc" rpc.method="PlaceOrder" rpc.service="oteldemo.CheckoutService"]
  name: "The order was placed"
  assertions:
  - attr:app.user.id = "2491f868-88f1-4345-8836-d5d8511a9f83"
  - attr:app.order.items.count = 1

This test specification is created in the Tracetest Web UI by simply selecting the span of interest, in this case the CheckoutService/PlaceOrder span.

span[tracetest.span.type="rpc" name="oteldemo.CheckoutService/PlaceOrder" rpc.system="grpc" rpc.method="PlaceOrder" rpc.service="oteldemo.CheckoutService"]

Then, selecting the attribute of interest, attr:app.order.items.count, and creating a test specification asserting that this value should equal 1.

attr:app.order.items.count = 1

You can then name the test specification.

💡 Have you followed along and installed the OpenTelemetry Demo in Docker and ran the Tracetest tests? If so, go to http://localhost:11633/ to view the UI.

Building tests based on the visibility provided by distributed tracing instrumentation makes end-to-end tests quick and easy to build. This reduction in the time, energy, and expertise needed to build tests makes it possible to provide high quality coverage across the entire distributed application. Once you set these tests up to run either manually or via your CI/CD process with the Tracetest CLI, your developers will be able to deploy changes with confidence.

💡 Want to read more about how Trace-based Testing overcomes the limits of black box testing and see where it fits in the testing pyramid? Check out this TheNewStack article!

Conclusions and Final Remarks

The OpenTelemetry Demo, like many modern applications, contains different languages, services, and numerous technologies. For these types of complex cloud-native architectures, we have seen that black box testing is insufficient. Fortunately, we have also provided a solution!

With trace-based testing with tools like Tracetest, that leverage existing observability instrumentation, you can enable deep end-to-end testing and reduce the risk of introducing errors when releasing code. It makes developing new features in microservice infrastructures easier and more secure. But, most important, it gives developers more confidence to deploy code with confidence.

Bonus Challenge - Run the Tests Yourself!

Want to see the OpenTelemetry Demo and run Tracetest tests yourself? It takes only four commands to install and run the demo and tests in Docker. Check out the one page instructions in the docs!

Want to learn more about the OpenTelemetry Community Demo? Look at the official OpenTelemetry docs!

Want to read more about the trace-based tests by the author of the tests, Daniel? See the blog post on the OpenTelemetry blog.

Have questions for the Tracetest team? Reach out to us on Discord!

Want to check out the repo, add an issue with a desired feature, or give Tracetest a star? Go to our GitHub.

GitHub Actions + Observability + Slack: Synthetic API Tests

Ken Hamric — Thu, 01 Jun 2023 18:40:28 +0000

Synthetic monitoring is a useful technique that allows you to simulate user transactions executed against your application. This helps you monitor critical API endpoints across different layers of your systems, alerting you to any failure. By proactively testing the API at regular intervals, you can quickly detect issues, investigate them and optimize system performance for your end-users, reducing MTTR.

Characteristics of synthetic monitoring:

It is highly consistent, simulating user behavior but with a fixed, predictable test.
It can be run on a scheduled basis and/or as part of a CI/CD process.
It can be run in “pre-production”.
It is typically tied to an alerting mechanism to inform the team of failures.

Let’s see how we can use existing, commonly available tooling to build our own synthetic monitoring alerts to cover an API back end.

Using GitHub Actions to Simulate Synthetic Monitoring

Every major CI environment, including GitHub Actions, is capable of scheduling jobs. In GitHub Actions, we can schedule a workflow by using the on.schedule attribute and provide a cron expression to configure how often we want our job to run. If you are not sure how to use cron, you can use generators to write the expression for you.

Secondly, we need a test framework to rely on. Traditionally, API synthetic monitoring tools have been limited to conducting simple black box tests. While effective when testing monolithic, synchronous applications, these black box tools are limited in their ability to test deeply across the entire flow triggered by an API call. They do not provide much confidence when building tests against modern, distributed, cloud-native architectures such as microservices or Lambda/FaaS-based systems.

Luckily, there is a way to utilize observability tools, namely distributed tracing, that are relied on by these complex systems to also empower your testing. The technique is called trace-based testing, and we will be using Tracetest, an open-source tool, to build and run these tests.

For this article, we are considering that you already have a Tracetest instance running either on your CI environment or deployed somewhere. We are also using the sample Pokeshop application as the system under test. You will see an option to install it when installing Tracetest. If you don’t have Tracetest running yet, check out our installation guide, recipes, and examples.

Finally, we need a way to send alerts when tests fail. For this, we will rely on Slack!

Configure Trace-based Testing with Tracetest in GitHub Actions

What does this Pokeshop application we will be setting up a synthetic monitor for look like? It has two microservices with an asynchronous message queue tying them together. Their jobs are:

Accept the API call, validate it and put it on a message queue, returning a 200 status code on success.
The second ‘worker’ microservice gets the request off of the queue and does the ‘real’ work asynchronously.

We want our monitor to validate the full flow, ensuring both services do their jobs. We will be testing a critical API call, a POST to the Pokemon import process. All Pokemon MUST be imported properly!!!

Considering that you already set up your Tracetest instance, let’s start building our synthetic monitoring tool using it. Let’s create a full test that will verify both microservices execute properly:

# monitoring/usecases/user_listing_products.yaml

type: Test
spec:
  id: 8jEnYSwVg
  name: Pokeshop - Import
  description: Import a Pokemon
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon/import
      method: POST
      headers:
      - key: Content-Type
        value: application/json
      body: '{"id":52}'
  specs:
  - name: POST should return status code 200
    selector: span[tracetest.span.type="http" name="POST /pokemon/import" http.method="POST"]
    assertions:
    - attr:http.status_code  =  200
  - name: Validation step in API microservice should validate properly
    selector: span[tracetest.span.type="general" name="validate request"]
    assertions:
    - attr:validation.isValid  =  "true"
  - name: Import process that pulls message off queue must exist!
    selector: span[tracetest.span.type="general" name="import pokemon"]
    assertions:
    - attr:tracetest.selected_spans.count = 1
  - name: External request to pokeapi.co should return 'Meowth'
    selector: span[tracetest.span.type="http" name="HTTP GET pokeapi.pokemon" http.method="GET"]
    assertions:
    - attr:http.response.body| json_path '$.name' = 'meowth'
  - name: Should insert into pokeshop postgres DB
    selector: span[tracetest.span.type="database" name="pg.query:INSERT pokeshop"
      db.system="postgresql" db.name="pokeshop" db.user="ashketchum"]
    assertions:
    - attr:tracetest.selected_spans.count = 1

We have 2 test specifications covering the first microservice:

POST should return status code 200.
Validation step in API microservice should validate properly.

And 3 test specifications covering the second “worker” microservice:

Import process that pulls a message off the queue must exist!
External request to pokeapi.co should return Meowth.
Should insert into the pokeshop Postgres database.

Now that we have our test, let’s make it run in a GitHub Actions workflow:

# This example illustrates how you can use Tracetest to achieve synthetic monitoring
# and get notified via Slack when something fails

name: Synthetic monitoring with Tracetest

on:
  # allows the manual trigger
  workflow_dispatch:

jobs:
  run-syntehtic-monitoring:
    name: Run synthetic monitoring
    runs-on: ubuntu-latest
    steps:

      - name: Checkout
        uses: actions/checkout@v3

      - name: Start app and tracetest
        run: docker-compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up -d

      - name: Install tracetest CLI
        run: curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

      - name: Configure Tracetest CLI
        run: tracetest configure -g --endpoint http://localhost:11633 --analytics=false

      - name: Run synthetic monitoring tests
        run: |
          tracetest test run -d test-api.yaml

Automated Trace-based Testing with GitHub Actions Schedule

Now we can run our test any time we want by triggering it manually, however, synthetic monitoring requires us to run this test periodically, so, we can use on.schedule to make GitHub Actions run the workflow automatically on a set time interval:

# This example illustrates how you can use Tracetest to achieve synthetic monitoring
# and get notified via Slack when something fails

name: Synthetic monitoring with Tracetest

on:
  # allows the manual trigger
  workflow_dispatch:

  schedule:
  # Normally, we run synthetic monitoring in a time schedule. GitHub Actions allows
  # us to achieve that using a cron job. Read more about how cron jobs are configured in
  # this article: https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm
  #
  # Here are some examples of valid cron strings:
  #
  #- cron: '*/30 * * * *' # every 30 minutes
  #- cron: '*/5 * * * *' # every 5 minutes
  #- cron: '* * * * *' # every minute
  #- cron: '0 */1 * * *' # every hour
  #
  # For this example, we are going to run the job every 5 minutes
  - cron: '*/5 * * * *'

jobs:
  run-syntehtic-monitoring:
    name: Run synthetic monitoring
    runs-on: ubuntu-latest
    steps:

      - name: Checkout
        uses: actions/checkout@v3

      - name: Start app and tracetest
        run: docker-compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up -d

      - name: Install tracetest CLI
        run: curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

      - name: Configure Tracetest CLI
        run: tracetest configure -g --endpoint http://localhost:11633 --analytics=false

      - name: Run synthetic monitoring tests
        run: |
          tracetest test run -d test-api.yaml

Introduce Synthetic Monitoring Alerts with Slack

And, to cover the last aspect of synthetic monitoring, we need a way of receiving alerts when something breaks. For this, we will set up a Slack bot to notify us when the test breaks.

For this example, we are going to use Slack’s Incoming Webhook to send messages to a channel.

# This example illustrates how you can use Tracetest to achieve synthetic monitoring
# and get notified via Slack when something fails

name: Synthetic monitoring with Tracetest

on:
  # allows the manual trigger
  workflow_dispatch:

  schedule:
  # Normally, we run synthetic monitoring in a time schedule. GitHub Actions allows
  # us to achieve that using a cron job. Read more about how cron jobs are configured in
  # this article: https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm
  #
  # Here are some examples of valid cron strings:
  #
  #- cron: '*/30 * * * *' # every 30 minutes
  #- cron: '*/5 * * * *' # every 5 minutes
  #- cron: '* * * * *' # every minute
  #- cron: '0 */1 * * *' # every hour
  #
  # For this example, we are going to run the job every 5 minutes
  - cron: '*/5 * * * *'

jobs:
  run-syntehtic-monitoring:
    name: Run synthetic monitoring
    runs-on: ubuntu-latest
    steps:

      - name: Checkout
        uses: actions/checkout@v3

      - name: Start app and tracetest
        run: docker-compose -f docker-compose.yaml -f tracetest/docker-compose.yaml up -d

      - name: Install tracetest CLI
        run: curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

      - name: Configure Tracetest CLI
        run: tracetest configure -g --endpoint http://localhost:11633 --analytics=false

      - name: Run synthetic monitoring tests
        run: |
          tracetest test run -d test-api.yaml

      - name: Send message on Slack in case of failure
        if: ${{ failure() }}
        uses: slackapi/slack-github-action@v1.24.0
        with:
          # check the block kit builder docs to understand how it works
          # and how to modify it: https://api.slack.com/block-kit
          payload: |
            {
              "blocks": [
                {
                  "type": "header",
                  "text": {
                    "type": "plain_text",
                    "text": ":warning: Synthetic Monitoring Alert :warning:",
                    "emoji": true
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {
                      "type": "mrkdwn",
                      "text": "*Status:*\nFailed"
                    },
                    {
                      "type": "mrkdwn",
                      "text": "*Pipeline:*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View workflow>"
                    }
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

With this configuration, from now on, tests will run every 5 minutes and, if any of them fail, you will receive a message on Slack informing you about it:

!https://res.cloudinary.com/djwdcmwdz/image/upload/v1685632757/Blogposts/synthetic-monitoring-github-actions/Untitled_1_nnrvvg.png

You can check our synthetic monitoring example on GitHub to see how Tracetest and the demo application were configured.

Last, but not least - if would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

The Future

The Tracetest team has been discussing whether to build synthetic API monitoring directly into Tracetest to make creating and running synthetic tests easier. This would involve adding the scheduling of tests and notifications into Tracetest.

The Pros:

It would make it super convenient to add scheduled tests with notifications.
For SREs installing Tracetest, it would provide a ready tool for doing deep synthetic tests against the system they are responsible for.

The Cons:

The dev team creating Tracetest would be focused on items outside its core mission of trace-based testing, pulling time from other needed and exciting improvements.
There are a variety of different ways to notify team members, and it would be difficult to support a wide number of them.

Should we leave scheduling and notification to outside services, such as GitHub Actions paired with Slack, or would you like to see this capability embedded in Tracetest? Let us know in this four-question poll!

A Tool for Cloud Native Testing - One Year In

Ken Hamric — Wed, 26 Apr 2023 22:48:04 +0000

Starting anything from scratch is a challenge. With Tracetest reaching its one year birthday today, this seemed a good chance to look back on the first year of the project. Hopefully this information will be of interest to other founders considering starting a new open-source project.

It’s been a rollercoaster ride with amazing highs. In the first year, Tracetest reached 500 stars, integrated with 9 different products, including leading tracing provides like Datadog, Lightstep, and many more. Let’s celebrate the year, and welcome the next with open arms. Here’s our story of how Tracetest was born and the challenges seen in our first year.

Building the Product from Scratch

First, what is Tracetest? Tracetest is a testing tool that leverages the data in a distributed trace to enable integration, system and end-to-end tests to be created in a fraction of the time. End-to-end tests are typically difficult to write. They can take 3 to 10 days to build across a complex distributed system. With Tracetest you can build the same test in 20 minutes. That is 24 to 72 times faster. Seriously!

How? We use a new test type called trace-based testing. Most of the time and complexity involved in building an end-to-end test is in adding code to provide visibility into the systems under test. You have to understand the complete flow, set up connections and permissions to various data stores you may want to inspect to ensure proper operation and find means to introspect components such as microservices and key infrastructure so you can ensure what you believe is happening actually is. Creating the actual assertions, such as “status code should equal 200”, is a trivial part of the work.

With trace-based testing and Tracetest, the observability of the distributed system under test is provided by the distributed trace you have already instrumented your system to provide. Tracetest can “see” not only the response from API calls to your application, but the full flow and each system involved as documented by the captured distributed trace. With that data, the work to build a test is to simply decide the assertions you want to add.

Tracetest is open source, MIT licensed, and the first MVP release occurred April 26th, 2022. Let's look at some of the challenges and questions faced in this first year, and how we addressed them.

Building a Fully Remote Team

My first two startups were bootstrapped, with only myself doing the initial development on the first, and only myself and two co-founders, who were part time, working on the second. It was a great approach in the day, but I believe it is less applicable today. More skill sets are needed to develop a system, from back end, front end, UX, and product management, and it is harder to be able to cover all these talents. There is also a higher level of expectation in the market - we are used to lovely products that are intuitive to use, and it is hard for a generalist to have this level of talent.

There is also a time to market constraint, as good ideas are quickly copied by well funded potential competitors. Tracetest is fortunate enough to be an incubator project under Kubeshop, so this affords us the luxury of recruiting a full development team.

We began Tracetest last year with resources that were already on staff at Kubeshop, “borrowing” three developers based in Europe. Jasmin, Povilas, and Mahmoud got the project off the ground and the first MVP released in only 3 months. Working with a European-based development team while acting as the product manager was, however, hard, as I was based in Germantown, TN in the CST time zone. With a 6 to 7 hour time difference, the commitment to wake up at 4 or 5 am most days so there was overlap was just not sustainable.

We made the decision to try to recruit a development team in the Americas. While I had worked with remote developers in Europe before and had seen both the advantages and disadvantages, I did not have experience hiring and working with a team in Central and Latin America. With the help of our recruiter, Julian at Aura Recruit, we were able to build a team in just 2 months. This team is a pleasure to work with - Sebastian, Matheus, Oscar, Jorge, and Daniel!

They are mature, focused on the product in addition to the technology, and are each a leader in their own right. Each acts as a product manager and as a developer relations person, while still holding down their “day job” of being a back-end or front-end developer. If you have ever wondered about hiring a team in the Americas, I can highly recommend it.

Developing Software Based on a New Concept

Testing tools are hardly new. Testing tools that execute against a back-end application have existed since the 1990s. They typically involve defining a method to trigger a test, perhaps hitting an API, and then creating assertions against the response data.

Trace-based testing has some entirely new challenges, as you are asserting not only against the response data, but against the distributed trace. The trace is a tree like structure, with activities branching off from the main call.

One of the main challenges we had was how to write assertions against this tree, as it was not flat like simple response data is. We came up with a solution which will be familiar to front end developers - using a selector. We modeled our selectors on the CSS selector language, and the assertions in Tracetest, called test specs, have two components:

A selector specifying what span or spans to apply the assertions against.
One or more assertions to apply against any of the spans matching the selector.

This decision created some options for very powerful assertions. In one test spec, we can specify that any gRPC process in the system should return a status code of zero. This singular check applies across every call involved in the trace. It is also self maintaining, as it will test any new gRPC calls that are added into the flow for that operation.

Another decision we faced this year was how to write the tests. We knew we wanted a graphical interface to see the trace and build tests from it. We also knew we needed a way to express the test in code so it could be versioned and used in a CI/CD/GitOps flow. We debated using a language such as JavaScript or Go to express it, but these would not easily handle our assertions which apply against multiple spans. The team decided on a YAML representation, and it was a great choice. It is clean, concise and easily diffed in Git. We express the entire test in this format and the test trigger and test specs are clearly shown.

type: Test
spec:
  id: phAZcrT4B
  name: Books list with availability
  description: Testing the books list and availability check
  trigger:
    type: http
    httpRequest:
      url: http://app:8080/books
      method: GET
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="http" name="GET /books" http.target="/books"
      http.method="GET"]
    assertions:
    - attr:tracetest.span.duration  < 500ms
  - selector: span[tracetest.span.type="general" name="Books List"]
    assertions:
    - attr:books.list.count = 3
  - selector: span[tracetest.span.type="http" name="GET /availability/:bookId" http.method="GET"]
    assertions:
    - attr:http.host = "availability:8080"
  - selector: span[tracetest.span.type="general" name="Availablity check"]
    assertions:
    - attr:isAvailable = "true"

A third challenge was the user interface. With a trace-based test, there is a lot going on! In addition to the normal flows in a test tool, such as seeing a list of tests and test runs and defining a test and seeing the response and having assertions applied against it, we had an entire trace to show and enable test specs to be created against. We have kept Olly, our UX designer, busy. We opted to create our initial versions ‘wrong’, accepting that we needed to see and use it before we could hope to get an elegant flow. It has taken multiple tries, but we now have a version that ‘flows’ as you create tests and view the results.

UX - April 26th, 2022 - v0.1 MVP

UX - August 26th, 2022 - v0.6

UX - v0.11, April 26th, 2023

Getting the Word Out

New concepts are, by definition, new. There is no existing model, and potential users have no prior experience with which to base understanding the new concept. My previous startup, which we began in 2008 and sold in 2016, was named CrossBrowserTesting.com. As you may imagine, it was used to do cross browser testing, providing the ability to do live testing, take screenshots, and do automated selenium testing across a wide range of browsers and devices. It was simple to explain, and with several competitors in the marketplace there was a general understanding of what to expect from the product.

Tracetest is different. While the name is, once again, very direct, what exactly does it mean to test based on a trace? In addition to this, distributed tracing, upon which it is based, is a concept that is newer and many teams do not have exposure to. This results in a need to spend more energy explaining the problem space and particularly our approach to solving it. Personally, it has resulted in my developing new skills, as explaining and promoting the use of the project fell to me for most of the year.

In my previous startup, we wrote only 20 blog posts in the first 8 years, at Tracetest we have written 38 in the past year alone, a pace of almost one per week. We have also been producing videos, which was a real challenge as a newcomer to the medium, resulting in many, many retakes. Like any skill, practice makes you better, and while I will be getting no Emmy’s awarded, time at task has made the process easier. We are fortunate to have added a professional developer relations expert to the team, Adnan, so the pace and quality of the work has increased.

Conferences and speaking engagements are also a new approach for me personally, and we have been to several this year. Adnan recently spoke at FOSDEM, and we both were present at KubeCon + CloudNativeCon Europe in Amsterdam just last week. It is very motivating to present the project in person and receive direct feedback from real experts in the distributed, cloud-native world. We were overwhelmed by the response and crowds at the booth last week, and are reenergized to keep paving a way to making trace-based testing mainstream.

Wrapping up the First Year

Wow. It has been a busy, exciting year. As we were demoing Tracetest to hundreds of interested people last week at KubeCon, I was proud of the work the team had done. Tracetest works, and it will revolutionize testing for distributed systems. It flows well and is a pleasure to use. We have had 1000s of downloads, and https://github.com/kubeshop/tracetest reached 500 stars on Wednesday at the conference! Getting to 500 stars before the one year anniversary was a goal, and we were excited to exceed expectations. There is still work to be done. We are focused on adding new integrations so the tool works with any distributed tracing solution, whether vendor based such as Datadog, New Relic, Elastic or Lightstep, or open source such as OpenSearch or Jaeger. We are also excited to begin work on some new capabilities being suggested by our users to extend the capabilities of Tracetest, such as the Ability to export application behavior as policies for runtime or Define common trace "patterns / issues / problems".

Like what we are doing and want to help make our second year even better than the first? Give us a star on Github, create an issue there with your thoughts on the direction to take the product, or talk to us on our Discord channel. Looking forward to year #2!

Spring update: Integrations, Improving UX, and Ease of Provisioning

Ken Hamric — Tue, 11 Apr 2023 16:23:21 +0000

Spring has sprung, and Q1 is over! It is a perfect time to look back, see what you’ve done well, and contemplate how to improve moving forward. I want to share some of the highlights I’m most proud of from the Tracetest team in the first quarter of 2023. And also share our roadmap for Q2 moving forward.

What is Tracetest? It is an open-source test tool built for any application using a distributed, cloud-native architecture. It uses the visibility exposed by your existing instrumentation, namely distributed tracing, to allow you to build integration, system, and end-to-end tests that can reach across the entire system. This technique, known as trace-based testing, allows you to assert the proper behavior of all the different microservices, lambda functions, asynchronous processes, and infrastructure components involved in a modern application. It is the ‘easy button’ for your full system tests, allowing you to create tests in orders of magnitude less time, increase your test coverage and improve quality in today’s increasingly fast-paced development and deployment cycles.

We began the quarter with a couple of major focuses:

Integrations
Improving ease of use

Let’s look at what we accomplished in each category.

Integrations with New Distributed Tracing Solutions

Tracetest works by leveraging the distributed tracing capability you have already instrumented into your application. As part of this, we need to support a wide spectrum of observability and APM tools. We came into the quarter with integrations with multiple tracing solutions, including Jaeger, OpenSearch, Grafana Tempo, SignalFX, and of course, the OpenTelemetry Collector. This quarter, we added more of the most popular tracing solutions.

Lightstep

Lightstep is a leading cloud-native reliability platform. Adriana Villela, a developer advocate from Lightstep, was one of the first users of Tracetest way back in May 2022. Since then, she has been a vocal proponent of trace-based testing and a friend of the Tracetest project. The team worked with her to enable Tracetest to interoperate with Lightstep, enabling an integration via the OpenTelemetry collector. Adriana even created a video showing Tracetest and the OTel Demo App running on Hashicorp Nomad with Lightstep as the back end (of course)!

Elastic

Elastic, the company famous for developing the ELK stack, has a very popular APM solution. Michael Hyatt, a solutions architect for Elastic, reached out to see if we would add Elastic as a trace data source, as the Elastic datastore can accept traces whether instrumented with Elastic APM agents or the default OpenTelemetry instrumentation. Michael was super helpful in assisting the team as we developed and tested the Elastic integration, and even wrote the blog post which appeared on both the Tracetest and Elastic sites.

New Relic

New Relic needs no introduction, as it is a well-known observability platform. New Relic supports using the OpenTelemetry collector, so we were able to add it as a trace data source and document the process. The team also documented a use case contained in this recipe:

Running Tracetest With New Relic

AWS X-Ray

The integration with AWS X-Ray was initiated due to conversations with two different teams with very similar use patterns. They were both using serverless architectures with AWS and needed a better way to test the critical flows. Few tools serve this area well, but Tracetest is a natural choice for testing modern distributed FaaS flows. The team quickly built an integration to retrieve trace data from AWS X-Ray. As importantly, the team focused on documenting use cases, as there are several options for deploying in AWS. Some of the recipes and use cases include:

Cloud Native Tool Integrations

We also wanted to integrate with tools in the Open Source arena that are focused on modern, cloud-native applications. The tools we decided to integrate with this quarter both came from user inquiries asking “Do you work with k6?” and “Can Keptn use trace-based testing?”.

k6

The initial thought to add a k6 integration came from Kubecon Detroit, as we had 5 or 6 different people asking “Do you work with k6?”. These k6 users could see the value in having load tests that could assert that the complete flow and all the micro-services involved perform correctly and speedily under load, rather than just having a “black box” test that barely tests past the API gateway. As part of the rollout of this capability, we participated in a wonderful k6 “Office hours” community call with the k6 team.

Keptn

The Keptn integration was added based on two GitHub issues raised in the Keptn community ([integration] Tracetest #48 and SLO for Traces #508). The desire was to use the OpenTelemetry instrumentation which has been adopted at the core of Keptn to enable tests to be built that used data from those traces. The results of these tests will be used as service level indicators as part of Keptn’s control plane to decide whether to deploy new code. The implementation was done using the Keptn Job Executor Service, which you can read about here. Another great experience was getting invited to talk about the integration at the Keptn Community Meeting in February.

Keptn Community & Developer Meeting - Feb 1, 2023 - YouTube

Meeting notes: https://docs.google.com/document/d/1y7a6uaN8fwFJ7IRnvtxSfgz-OGFq6u7bKN6F7NDxKPg/edit Learn more: https://keptn.sh Get started with tutorials: ...

youtube.com

Testkube

Testkube was a natural choice for building an integration. It’s one of 4 other projects under our open-source startup accelerator, Kubeshop. Testkube has an adjacent “job to be done” with Tracetest.

Testkube is a Kubernetes-native testing framework that allows you to write tests with a number of other test frameworks, such as Cypress, k6, Postman, and others. You then run them all via Testkube. It allows you to manage the test orchestration and execution of the various tests and integrate running them into your CI/CD/GitOps pipeline.

Both Tracetest and Testkube are designed for testing cloud-native applications, and, by integrating with Testkube, we gain some useful capabilities:

Orchestration with other tests to make test suites.
The ability to trigger Tracetests based on particular Kubernetes events.
The ability to schedule tests, enabling synthetic monitoring of your back-end distributed system.

The use cases were documented in a blog post and a recipe was added showing how to use Testkube and Tracetest together. We also participated in a workshop with the Testkube team and showcased how both tools work in action.

Improving Ease of Use

Creating a powerful tool introducing a totally new way to test distributed applications is not enough. The tool has to be one you WANT to use. This is continually top of mind for the entire Tracetest team, and we had several areas of improvement for this quarter.

Transactions and Chaining Tests, v2

Software development is a process, and perfection on the first pass often results in paralysis. With the release of v0.8 a couple of months earlier, we introduced the ability to chain tests, have variables in tests, and define environments such as Test, QA, or Prod. To call this a huge release would be accurate, and we knew there were UX flows that could be improved. We documented these and tackled a ‘Chaining V2’ epic at the beginning of the quarter. Of the 15 issues tackled in this epic, the enabling of ad-hoc tests by prompting for undefined variables was my favorite. This allows you to leave a particular input “blank”, run a test, and have Tracetest prompt you each time for the value to run the test with. This is particularly useful when developing and you want to test with numerous values interactively.

Moving Settings Into the User Interface

With the initial version of Tracetest, we had a configuration file that contained all settings: how to connect to the trace data store, how often to poll for a trace and when to timeout, and a few other settings. This presented several challenges:

The server had to be rebooted to accept any changes to the configuration.
The sections in the configuration file were hard to document and understand.
Errors in the configuration file could cause the server not to start.

All these issues impacted usability, particularly for new users. In December, we moved the configuration of the trace data stores into the application, making a critical, error-prone step much easier to successfully complete.

This quarter, we completed the process by adding three new tabs - Analytics, Trace Polling, and Demo. We also added the ability to alter the server configuration via the CLI, making sure any of the changes which can be done on the UI can be done in an automated manner.

Multiple Ways to Provision a New Install

Speaking of being able to configure a server from the command line, the team added multiple ways to configure a server. These include using a provisioning file to “hydrate” the server and passing the provisioning file as a base64 encoded string via an environment variable, perfect when deploying Tracetest in an ECS cluster so there is no need to mount a file. The only “required” data to start the Tracetest server is the Postgres connection information, and it can now be passed as environment variables as well as via a configuration file. This capability was covered in detail in our March community call:

Looking Forward

For Q2, the team will be doubling down on integrations, with plans to add Azure Application Insights, Google Cloud Trace, and Honeycomb as supported trace data stores. We are also looking for a couple of tool integrations to add and would love your ideas and input.

One area we are considering investing in is enabling Tracetest tests to be run as a synthetic monitor against your API, enabling you to schedule Tracetest tests and be alerted when they fail. This would enable deep end-to-end tests to be easily created and used to “keep an eye on” and alert the SRE and DevOps teams about potential failures. We would love your thoughts on using trace-based tests for synthetic monitoring - do you need it!? Let us know in Discord!

Adnan and I will be at the Tracetest booth in KubeCon Europe this month! Come by, see us, and discuss any and all things OpenTelemetry, distributed tracing, testing, trace-based testing, and Tracetest. Would love to show you the tool, get your thoughts, and hear your ideas. Reach out to myself or Adnan on LinkedIn and we will set up a time for a chat!

Lastly, our birthday is coming up. Tracetest will be 1 year old on April 26th!!! We have a goal of hitting 500 stars by our birthday. We are currently at 474, so we need your help at github.com/kubeshop/tracetest.

Announcing the Tracetest integration with k6: Deep Load Testing of your Cloud Native System

Ken Hamric — Sun, 05 Mar 2023 21:28:13 +0000

Today, we're happily announcing that Tracetest now works with k6, a modern load-testing tool used by thousands of developers worldwide. The combination of k6 and Tracetest allows your load tests to check across your entire microservice, FaaS, or Kubernetes-based applications.

This checks that not only the top surface of the application returns the correct information and remains performant, but allows you to test and verify across the full distributed application, making sure none of the many services involve begin failing or taking too long under load.

Load Testing Across a Distributed Application

Oscar, one of the developers of the k6-tracetest extension, shows why the combo is so potent:

What is k6?

k6 allows you to build performant load tests that take a fraction of the resources legacy load-testing tools require. The tests are written in an easy-to-follow JavaScript form and can be extended with a number of extensions. The tests execute via a Go-JavaScript compiler so they take a minimal amount of resources when being executed, allowing the scaling of load tests with fewer resources.

What is Tracetest?

Tracetest is an open-source CNCF project. It allows you to quickly build integration and end-to-end tests, supercharged by your existing distributed traces. The use of your existing distributed traces as part of the test enables assertions to be created across multiple microservices or parts of your distributed application.

Tracetest makes it possible to:

Define tests and assertions against every single microservice that a request goes through.
Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.
Define multiple transaction triggers, such as a GET against an API endpoint, a gRPC request, etc.
Return both the response data and a full trace.
Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.
Save tests.
Run the tests manually or, via CI, build jobs with the Tracetest CLI.

Tracetest Now Works with k6!

Tracetest now works with k6, allowing you to conduct load tests that verify your entire distributed application.

Most load-testing tools will trigger your application and apply tests against the response from the call. As you scale up the number of tests running concurrently, the system under test will be exercised. Assertions can be defined against any of the response data, whether it is the status codes, parts of the response body, or expected timing.

By including Tracetest in the k6 load test, you can test against both the response AND the information contained in the trace. This allows you to define powerful assertions against all the processes involved in servicing the call invoked by a load test. Some examples of checks it enables include:

Checking the microservices on “the other side” of a message queue to verify it can handle the load. Typically when load-testing complex systems with message queues you cannot easily assert against the entire system, particularly when it has asynchronous calls.
In one assertion, assert that all database queries across all microservices should return in less than 100ms, and verify this under load.
Or that every gRPC process in the entire stack continues returning status codes of zero as the load increases.
Showing you graphically, via the trace, where any detected failures occur across any of the runs.

The ability to observe and assert against the entire system under load, empowered by your current APM or observability platform, makes Tracetest and k6 a powerful combination.

Try Tracetest with k6

Tracetest is open-source and easy to install. First, install Tracetest CLI. Check out the download page for more info. From here, follow the official documentation to install the Tracetest server.

Once the server is installed, open up Tracetest Web UI in the browser and define where Tracetest gets the traces from, whether Jaeger, Lightstep, Elastic, or any trace vendor. You can even send traces to Tracetest via the OpenTelemetry collector. Check out the Configuration Overview for help!

Once you have Tracetest working and can run trace-based tests against your distributed system, you will want to create a load test with k6. Use the excellent k6 docs to get started. You can then utilize the k6-tracetest extension to run deep load tests. We have full documentation on how to use k6 and Tracetest together at https://docs.tracetest.io/tools-and-integrations/k6.

The Tracetest team recently discussed the integration during the k6 Office Hours:

What's next?

This is the first version of the integration between k6 and Tracetest. Have any ideas for improving it? Interested in enabling trace-based testing for other load testing tools? Let us know by creating an issue in Github, or start a conversation with the team behind Tracetest in our Discord community. You can also reach out to us to schedule a call to discuss your particular needs or get assistance.

Excited about the Tracetest project and trace-based testing? Give Tracetest a star on GitHub!