Kubernetes: ServiceAccounts, JWT-tokens, authentication, and RBAC authorization

For the authentification and authorization, Kubernetes has such notions as User Accounts and Service Accounts.

User Accounts  -  common user profiles used to access a cluster from the outside, while Service Accounts are used to grant access from inside of the cluster.

ServiceAccounts are intended to provide an identity for a Kubernetes Pod to be used by its container to authenticate and authorize them when performing API-requests to the Kubernetes API-server.

Content

• Default ServiceAccount
• default token
• JWT token
• JWT token and authentification
• ServiceAccounts, and RBAC
• RoleBindig for ServiceAccount
• ServiceAccounts and security
• Useful links

Default ServiceAccount

Every Kubernetes Namespace has its own default ServiceAccount (SA) which is created when creating a namespace.

Let’s check the default namespace:

$ kubectl --namespace default get serviceaccount
NAME SECRETS AGE
default 1 176d

For each ServiceAccount a token is generated and stored as a Kubernetes Secret.

Check the default SA:

$ kubectl --namespace default get serviceaccount default -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: “2020–05–25T12:04:49Z”
name: default
namespace: default
resourceVersion: “296”
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: 19cc2b5f-fbc3–403e-a7c7-d62361a4038a
secrets:
- name: default-token-292g9

Here is the token for this SA — the default-token-292g9 Secret:

…
secrets:
- name: default-token-292g9

default token

Now, check the Secret’s content:

$ kubectl get secret default-token-292g9 -o yaml
apiVersion: v1
data:
ca.crt: LS0…sdA==
token: ZXl…TWc=
kind: Secret
metadata:
annotations:
kubernetes.io/service-account.name: default
kubernetes.io/service-account.uid: 19cc2b5f-fbc3–403e-a7c7-d62361a4038a
creationTimestamp: “2020–05–25T12:04:49Z”
name: default-token-292g9
namespace: default
resourceVersion: “294”
selfLink: /api/v1/namespaces/default/secrets/default-token-292g9
uid: 07a46645–0083–45a0-a640–6e6a78ebd9b1
type: kubernetes.io/service-account-token

At first, its type is the kubernetes.io/service-account-token.

Another interesting part here is the data that keeps two records - ca.cert и token.

If a token is not from the default namespace — there will be a third field specifying a namespace to which this token belongs.

Theca.cert is signed by the cluster's master key so the cluster is playing the Certificate Authority role, and allows a pod or an application to verify the API-server.

And now, let’s go to investigate the tokenpart.

JWT token

To make it easier to work from the terminal — save the data.token value to a variable:

$ token=”ZXl…TWc=”

Use the base64 get its content:

$ echo $token | base64 -d
eyJ[…]iJ9.eyJ[…]ifQ.g5I[…]3Mg

Here I’ve removed some data with the […], but we can see that the value is divided into three parts with dots:

• the header — describes how the token was signed
• the payload — actual data of the token, such as expiration date, who issued it, etc see the RFC-7519
• the signature — is used to verify that the token wasn’t modified and can be used to validate the sender

See the documentation>>>.

To check the token’s content we can use the jwt utility or on the jwt.io website.

In our case, the payload section has the following lines:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "default-token-s8m4t",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "b4514006-4c9a-4c30-92c8-1cc1c058b31c",
  "sub": "system:serviceaccount:default:default"
}

here in the sub filed we can see the ServiceAccount name, i.e. - who is presenting this token to the Kubernetes API-server so the server will know from who this token came.

Okay, but what about a password? In the sub there is a "login" - but where is his "password"?

And here is the third part is playing — the signature.

JWT token and authentification

I wasn’t able to see these details in any from the googled materials, see the Useful links section of this post, although as for me — this is the most interesting part of the scheme.

Let’s go back to the first section of the token — the header, which in our case has the RS256 algorithm type defined i.e. RSA (Rivest-Shamir-Adleman) — the asymmetric algorithm with private and public keys and uses SHA-256 algorithm for the signature.

Let’s check our token on the jwt.io:

Invalid Signature  — as we not provided the private and public keys to verify the token.

Because the masters’ private key on AWS Elastic Kubernetes Service is stored on the ConrolPlane nodes and we can’t access them — let’s use minikube for the testing.

Run a local cluster:

$ minikube start

In its default namespace we can see already existing token:

$ kubectl get secrets
NAME TYPE DATA AGE
default-token-s8m4t kubernetes.io/service-account-token 3 2m44s

Grab the token field and decode it with base64:

$ kubectl get secrets -o jsonpath=’{.items[0].data.token}’ | base64 -d
eyJhbGciO[…]61O_LxbM_-tiLjyjeCZw

Go back to the jwt.io, paste the string received above:

Still Invalid Signature  — but go to your minikube and take its public certificate - the ~/.minikube/ca.crt file:

$ cat ~/.minikube/ca.crt
 — — -BEGIN CERTIFICATE — — -
MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
…
0g+FhVM92T+yV38vYLO/HaKeiOzIcgHHkAoLJZd/K/Mu7crwIuGlcCVhrjcHoa3p
Md34ZTeqxA4J3w==
 — — -END CERTIFICATE — — -

Paster it to the Public Key or Certificate field.

Find the private key of the minikube cluster - actually, it is also used to sing the ca.crt and tokens, the ~/.minikube/ca.key file:

$ cat ~/.minikube/ca.key
 — — -BEGIN RSA PRIVATE KEY — — -
MIIEowIBAAKCAQEAtDRDag2D7UBaBmWQwTKVLjuKTuat4eD/oThRgfi5bcCnwooG
…
xnL96EHthflb3NaS4GKuJYzNAPhfOdMw96Ce8KtNYpMYjRhNF9TN
 — — -END RSA PRIVATE KEY — — -

Paste it to the Private Key field:

Signature Verified  — yup, it works! The authenticity of the bearer of the token is verified.

So, going back to the ServiceAccounts:

• for a ServiceAccount a token is created which keep the SA name
• the token is signed by the master key of the Kubernetes cluster
• a pod make a request to the API server using this token to authenticate him
• the API server validates the token by using its public key and verify that the token wasn’t modified and is relly issued by this Kubernetes clutserм

Now, let’s go to see in practice how this is working and how Kubernetes RBAC is used here.

ServiceAccounts, and RBAC

For each Pod that has no ServiceAccount specified the default ServiceAccount is attached and its default token is mounted.

Go back to our EKS cluster and run a Pod:

$ kubectl run -i --tty --rm ca-test-pod --image=radial/busyboxplus:curl
kubectl run — generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run — generator=run-pod/v1 or kubectl create instead.
If you don’t see a command prompt, try pressing enter.
[root@ca-test-pod-5c96c78d7f-wqlsq:/]$

Check it volumeMounts, serviceAccount, and volumes:

$ kubectl get pod ca-test-pod-5c96c78d7f-wqlsq -o yaml
apiVersion: v1
kind: Pod
…
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-292g9
readOnly: true
…
serviceAccount: default
serviceAccountName: default
…
volumes:
- name: default-token-292g9
secret:
defaultMode: 420
secretName: default-token-292g

Inside of the pod check the /var/run/secrets/kubernetes.io/serviceaccount directory content:

[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ ls -1 /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token

And recall the content of the data section of the default-token-292g9 Secret:

$ kubectl get secret default-token-292g9 -o yaml
apiVersion: v1
data:
ca.crt: LS0t[…]
namespace: ZGVmYXVsdA==
token: ZXlKaGJ
…

Try to perform a request to the API-server without authentification — use the special Service kubernetes, add the -k or --insecure to the curl to skip server's certificate validation

[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ curl -k [https://kubernetes](https://kubernetes)
{
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {
},
“status”: “Failure”,
“message”: “forbidden: User \”system:anonymous\” cannot get path \”/\””,
“reason”: “Forbidden”,
“details”: {
},
“code”: 403
}

Cool — we got the 403, Forbidden.

Now, add two variables — one with the ca.crt and with the token:

[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

And run curl again - let's try to get a list of the pods in our namespace, this time without --insecure and with authorization by using the Authorization header:

[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ curl — cacert $CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes/api/v1/namespaces/default/pods/"
{
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {
},
“status”: “Failure”,
“message”: “pods is forbidden: User \”system:serviceaccount:default:default\” cannot list resource \”pods\” in API group \”\” in the namespace \”default\””,
“reason”: “Forbidden”,
“details”: {
“kind”: “pods”
},
“code”: 403
}

At this time, we are able to see our user — the User "system:serviceaccount:default:default", but it has no permissions to perform requests as by default all users and ServiceAccounts have no privileges (the principle of the least privileges, POLP).

RoleBindig for ServiceAccount

To give our SericeAccount permissions we need to create a RoleBinding or ClusterRoleBinding as for normal users.

Create a RoleBinding mapping to the default ClusterRole view, see User-facing roles:

$ kubectl create rolebinding ca-test-view --clusterrole=view --serviceaccount=default:default
rolebinding.rbac.authorization.k8s.io/ca-test-view created

And run curl again:

[root@ca-test-pod-5c96c78d7f-wqlsq:/]$ curl — cacert $CERT -H “Authorization: Bearer $TOKEN” “https://kubernetes/api/v1/namespaces/default/pods/"
{
“kind”: “PodList”,
“apiVersion”: “v1”,
“metadata”: {
“selfLink”: “/api/v1/namespaces/default/pods/”,
“resourceVersion”: “66892356”
},
“items”: [
{
“metadata”: {
“name”: “ca-test-pod-5c96c78d7f-wqlsq”,
“generateName”: “ca-test-pod-5c96c78d7f-”,
“namespace”: “default”,
“selfLink”: “/api/v1/namespaces/default/pods/ca-test-pod-5c96c78d7f-wqlsq”,
“uid”: “f0d77cfe-38ab-48e9-aaf3-f344f1d343f3”,
“resourceVersion”: “66888089”,
“creationTimestamp”: “2020–11–17T16:08:09Z”,
“labels”: {
“pod-template-hash”: “5c96c78d7f”,
“run”: “ca-test-pod”
},
…
“qosClass”: “BestEffort”
}
}
]

ServiceAccounts and security

Remember, that having access to Secrets and ServiceAccounts any pod can have any token attached and thus can be able to perform actions allowed by such a token.

For example, by using the ServiceAccount of the ExternalDNS — such a pod can make a mess in our AWS Route53.

That’s why it is important to divide access to resources by using RBAC rules and roles for users, for example by allowing access to resources from only one namespace.

Useful links

• Using RBAC with Service Accounts in Kubernetes
• Kubernetes Access Control: Exploring Service Accounts
• Kubernetes Tips: Using a ServiceAccount
• What is JSON Web Token?
• kubernetes.io: Service Accounts
• kubernetes.io: Authenticating
• kubernetes.io: Using RBAC Authorization
• The Dark Arts of IAM & RBAC — Read-only Kubernetes Access
• Kubernetes Authentication
• Istio End-User Authentication for Kubernetes using JSON Web Tokens (JWT) and Auth0
• Kubernetes RBAC 101: Authentication
• How to Authorize Non-Kubernetes Clients With Istio on Your K8s Cluster
• Kubernetes Client Authentication on Amazon EKS
• Verifying EKS digital certificates
• 3 Realistic Approaches to Kubernetes RBAC

Originally published at RTFM: Linux, DevOps и системное администрирование.

---