Access the Jaeger REST API programatically in OpenShift

Have you ever tried to access Jaeger programmatically in OpenShift? It can be a little bit difficult. Sometimes, it is needed because you want, for instance, access to the REST API which is not documented because it is for internal usage. Or you want (as it is my case) to test something from the UI.

Before reading this article

This article is about how to get access to the Jaeger query service when using the “production” deployment strategy and deploying via the Jaeger Operator. Some steps can be the same if you are using other deployment methods. Even, these steps can be not needed if you disabled the security:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: disable-oauth-proxy
spec:
  ingress:
    security: none

Also, take into account this is for OpenShift. So, if you are using Kubernetes without any security proxy or something, you’ll not need to follow these steps to access the Jaeger query endpoints. Or (if you are using another proxy) you’ll need to configure your credentials to access the API. I’ll assume you’re using the OpenShift Oauth Proxy.

Getting access to the Jaeger query service

Let’s imagine I deployed the following CRD (kubectl create -f simple-prod.yaml) in the israel namespace:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    elasticsearch:
      nodeCount: 1
      resources:
        requests:
          cpu: 200m
          memory: 1Gi
        limits:
          memory: 1Gi

The result of running curl against the created OpenShift route will be a 403 error:

$ curl https://$(kubectl get route simple-prod -o jsonpath='{.spec.host}') -I
HTTP/1.1 403 Forbidden
Set-Cookie: _oauth_proxy=; Path=/; Domain=simple-prod-israel.apps.mycluster.iblancasa.com; Expires=Thu, 05 May 2022 11:09:34 GMT; HttpOnly; Secure; SameSite
Date: Thu, 05 May 2022 12:09:34 GMT
Content-Type: text/html; charset=utf-8
Set-Cookie: 3f978cf9c2d3ee8d70cad1d11aef9dcd=8d2f11be8db90157d1e73d19b515f187; path=/; HttpOnly; Secure; SameSite=None

If you didn’t specify the -I option to receive just the headers, you will receive the full HTML code from the logging webpage. Let’s see how to get the “correct” answer from the server.

The first step is to create a Service Account. From the OpenShift documentation:

A service account is an OpenShift Container Platform account that allows a component to directly access the API.

I’ll name my Service Account as “automation-access”. So, my file sa.yaml contains:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: automation-access

And create it in OpenShift using kubectl create -f sa.yaml:

$ kubectl create -f sa.yaml
serviceaccount/automation-access created

Now, we need to create a Cluster Role Binding. From the Kubernetes documentation:

A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.

As a simplification, we can use Cluster Role Bindings to establish relationships between, for instance, a Service Account and Cluster Role. In our case, it would be enough with system:auth-delegator for the jaeger-operator Service Account from the observability namespace and cluster-reader for the Service Account we created before. Our crb.yaml file will be similar to this one:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jaeger-operator-with-auth-delegator
  namespace: observability
subjects:
- kind: ServiceAccount
  name: jaeger-operator
  namespace: observability
roleRef:
  kind: ClusterRole
  name: system:auth-delegator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: automation-access-bind
subjects:
- kind: ServiceAccount
  name: automation-access
  namespace: israel
roleRef:
  kind: ClusterRole
  name: cluster-reader
  apiGroup: rbac.authorization.k8s.io

And apply it to our OpenShift cluster using: kubectl create -f crb.yaml:

$ kubectl create -f crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/jaeger-operator-with-auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/automation-access-bind created

Finally, we need to set some extra options to our Jaeger deployment (jaeger-sar.yaml):

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  query:
    options:
      query.bearer-token-propagation: true
  ingress:
    openshift:
      sar: '{"namespace": "israel", "resource": "pods", "verb": "get"}'
      delegateUrls: '{"/":{"namespace": "israel", "resource": "pods", "verb": "get"}}'
    options:
      pass-access-token: true
      pass-user-bearer-token: true
      scope: "user:info user:check-access"
      pass-basic-auth: false

Then, we apply it with kubectl apply -f jaeger-sa.yaml:

$ kubectl apply -f jaeger-sar.yaml
Warning: resource jaegers/simple-prod is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
jaeger.jaegertracing.io/simple-prod configured

Ok! We’re almost there. The next step is to get the token that will allow us to access the Jaeger query service programmatically. To get the token, we need to get the name of the Kubernetes secret to use. We can get that information from the following command:

$ kubectl get sa automation-access -o jsonpath='{.secrets}'
[{"name":"automation-access-token-q2p2x"},{"name":"automation-access-dockercfg-xvzsq"}]

The result is the name of 2 secrets. We’ll use the one called auomation-access-token-<ID>. If you use yq, the command can be:

$ kubectl get sa automation-access -o yaml | yq eval '.secrets[] | select( .name == "*-token-*")'.name 
automation-access-token-q2p2x

And, finally, we extract the token from the secret:

$ kubectl get secret automation-access-token-q2p2x -o jsonpath='{.data.token}' |  base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6Iks3TXcwUWhyb3BVdUVJblhkQmVQT25Fek1KVVlfa2RPV3dDZWpJVXpmekUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJpc3JhZWwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiYXV0b21hdGlvbi1hY2Nlc3MtdG9rZW4tcTJwMngiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYXV0b21hdGlvbi1hY2Nlc3MiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NGY1ZDE0MC1lMDEwLTQ5NWYtYWE4Yy0wNmQ3ZjIwNjg3OGIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6aXNyYWVsOmF1dG9tYXRpb24tYWNjZXNzIn0.J5USv-olD1vU0A5tL8TT1SDoH-jf5f58hSIEPnkkiZzTiOjHHx2Vj2v1p0bm26epfTQwLV3TfIsgcsV7tnpC0cR64MmDQe9EuBtNonJqV45nTkSoboWuBfeSxXqjk7Tuj_bcZcutcA0vsVHzAglb-Z0wavx9ETfQJueDWTzF6Cry5HxVxYn6ytogHfbaEgiLN9HNeZBWW4xi7JAzY-2viECrFq1yEIJKzWDihZScFkyX2fu1UDpFLH2ewA8N6bdqSd5FTEna0JS0e8yxQcGo-oXykDW2G_YcLDBSrCpQaji390FgMp_6dEl1yHSazGgCaKwulwOU4Rr6usf807mdBQ

This command gets the token from the secret and decodes it. We need to provide this token in the Authorization header when doing a curl call:

$ export TOKEN=$(kubectl get secret automation-access-token-q2p2x -o jsonpath='{.data.token}' |  base64 -d)
$ curl https://$(kubectl get route simple-prod -o jsonpath='{.spec.host}') -I -H "Authorization: Bearer $TOKEN"
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: Thu, 05 May 2022 12:44:06 GMT
Gap-Auth: system:serviceaccount:israel:[e-mail]
Gap-Upstream-Address: localhost:16686
Vary: Accept-Encoding
Set-Cookie: 3f978cf9c2d3ee8d70cad1d11aef9dcd=2e6b6fcf2050268c74e1aabcf71e9bab; path=/; HttpOnly; Secure; SameSite=None
Cache-control: private

Thank you for reading! I hope this article helped you! :)