Introducing "badtrace". Generate "bad" OpenTelemetry traces easily

#testing #monitoring #tooling #showdev

I needed a tool that would deliberately generate OpenTelemetry traces that were "bad". I needed this for training, enablement and demo purposes so that I could easily fire a trace with a known issue into my Observability system and thus show "why it was bad" and "how it looks".

Yes, tools like tracepusher and telemetrygen already exist, but they typically:

a) Generate "good" or "healthy" traces
b) Generate single traces

Hence, "badtrace" was born.

It works based on "scenarios" (you can implement your own in a few lines of Python).

python app.py \
  --endpoint=http://localhost:4318 \
  --service-name=badtrace \ # optional: defaults to "badtrace"
  --trace-count=1 \ # optional: defaults to 1
  --insecure=true \
  --scenario=scenario1

Scenarios

As I write this, there are currently 4 scenarios (please come and help implement more) which can be toggled by changing scenario1 above to scenario2, scenario3 or scenario4 etc.

Scenario 1 creates a short, small trace. This is quick (in end-to-end time) and short in terms of number of spans. In reality this would be a very low value trace (and thus a candidate to filter or heavily sample).

Scenario 2 creates a trace where some of the spans have errored. This models a trace where some of the operations error (imagine calling third party services and some of them fail).

Scenario 3 creates a "chatty" client to server trace with the client repeatedly calling the same endpoint over and over. This models things like the N+1 query problem where (for example) multiple SELECT statements occur to a database (instead of batching up the SELECTS into fewer statements).

Scenario 4 is similar to 3. It creates a "chatty" client but this time, the connections occur to many different servers.