Automatic OpenTelemetry for Kubernetes Jobs (Hands on)

Use the tracepusher Kubernetes Operator to automatically generate OpenTelemetry traces for Kubernetes Job and CronJob resources.

Step 0: Create a cluster & Start Jaeger

If you don't already have a cluster, create one now:

kind create cluster

# Or install jaeger on Kubernetes with the jaeger operator
# https://www.jaegertracing.io/docs/1.53/operator/
docker run --rm --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.53

Step 1: Install tracepusher Custom Resource Definitions on cluster

kubectl apply -f https://raw.githubusercontent.com/agardnerIT/tracepusher/main/operator/crds.yml

Step 2: Create a JobTracer

Modify the OpenTelemetry collector URL as appropriate for your setup and then apply the file. One JobTracer per namespace please!

This file instructs tracepusher to monitor for all Job resources in the given namespace (you can ignore on a job-by-job basis with annotations that you'll see next):

cat <<EOF | kubectl create -f -
---
apiVersion: tracers.tracepusher.github.io/v1
kind: JobTracer
metadata:
  name: tracer
  namespace: default
spec:
  # Specify your OpenTelemetry collector endpoint
  # Here I've given the docker URI
  collectorEndpoint: "http://host.docker.internal:4318"
EOF

Step 3: Create a Normal k8s Job / CronJob

Create a Job / CronJob as normal. The tracepusher annotations are optional and will override the behaviour of the above JobTracer

cat <<EOF | kubectl create -f -
---
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
  namespace: default
  #annotations:
  #  tracepusher/ignore: "true"
  #  tracepusher/collector: "http://example.com:4318"
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 0
EOF