DEV Community

Cover image for I Deployed Athenz/k8s-athenz-syncer locally! (Feat. Fixing Legacy Code)
Jeongwoo Kim
Jeongwoo Kim

Posted on

I Deployed Athenz/k8s-athenz-syncer locally! (Feat. Fixing Legacy Code)

#athenz #kubernetes #k8sathenzsyncer

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

Athenz provides the following API endpoints for getting Athenz domain and its role/policy information:

  • /v1/domain/{domainName}/group/admin
  • /v1/domain/{domainName}/group/viewer

However, if your applications are running in Kubernetes, constantly querying these endpoints can be inefficient. Ideally, you want a synchronization mechanism to cache this data within the cluster. Furthermore, you might want to decouple your applications from a direct dependency on Athenz to improve resilience.

Let's be realistic—building a custom client to fetch, cache, and manage these resources within Kubernetes is a hassle. Why reinvent the wheel when you just want to consume the data?

That's why I looked into Athenz/k8s-athenz-syncer. It is an existing tool designed to sync Athenz data into Kubernetes Custom Resources (CRDs) called AthenzDomain, effectively handling the heavy lifting for us. In this post, I’ll walk through how to deploy this syncer, fix a few build issues I encountered, and explore how it can save us from writing unnecessary boilerplate code.

Result

I successfully deployed k8s-athenz-syncer, and the AthenzDomain CRD was registered as follows:

01

Next, I created a domain named home.syncer and a role can-i-see-this-role-in-crd to see if syncer can really sync the AthenzDomain CRD:

02

Finally, I created a Kubernetes namespace home-syncer (Note: Athenz domains replace dots . with dashes - in K8s namespaces). I verified that the syncer successfully generated the AthenzDomain resource, with its roles and policies correctly synced:

03

Setup

Setup: Working directory

Let's set up the working directory. Feel free to use your own, but here is an idempotent script for a quick start:

test_name=deploy_k8s_athenz_syncer
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir
Enter fullscreen mode Exit fullscreen mode

Setup: Athenz and Local Kubernetes Cluster

[!WARNING]
The following script only works on macOS. Let me know in comments if you want to use other platforms.

The following script will set up a local Kubernetes cluster and install the Athenz server:

git clone https://github.com/mlajkim/dive-manifest.git manifest
make -C manifest setup
Enter fullscreen mode Exit fullscreen mode

Test

Let's verify that the Athenz server is running:

kubectl get pods -n athenz
Enter fullscreen mode Exit fullscreen mode

Setup: Set UI for web page

To visualize the results later, let's set up port forwarding to access the Athenz UI in a browser:

kubectl -n athenz port-forward deployment/athenz-ui 3000:3000
Enter fullscreen mode Exit fullscreen mode

Test

Open the UI in browser:

open http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

Setup: Clone k8s-athenz-syncer

[!NOTE]
Once the PR https://github.com/AthenZ/k8s-athenz-syncer/pull/45 is released, we will use mlajkim's fork.

git clone -b fix/deprecated-Dockerfile-images-and-CRD-definition-API https://github.com/mlajkim/k8s-athenz-syncer.git syncer
Enter fullscreen mode Exit fullscreen mode

Setup: Build image

Let's build the image locally:

(cd syncer && docker build -t local/k8s-athenz-syncer:latest .)
Enter fullscreen mode Exit fullscreen mode

Setup: Load image to kind cluster

Since we are using Kind, we need to load the locally built k8s-athenz-syncer image into the cluster:

kind load docker-image local/k8s-athenz-syncer:latest

# Image: "local/k8s-athenz-syncer:latest" with ID "sha256:bb5bcf2d9c362a46444f9476791f6c9e3f81ce6abf6ebc07d3228b9b7da53fa8" not yet present on node "kind-control-plane", loading...
Enter fullscreen mode Exit fullscreen mode

Setup: Deploy manifests

[!TIP]
For the detailed explanation of each command, please refer to the following

[!WARNING]
We will create a custom deployment.yaml later, as the sample provided in the OSS repository requires configurations that are a bit complex for a quick demo.

kubectl create ns kube-yahoo
kubectl apply -f ./syncer/k8s/athenzdomain.yaml
kubectl apply -f ./syncer/k8s/serviceaccount.yaml
kubectl apply -f ./syncer/k8s/clusterrole.yaml
kubectl apply -f ./syncer/k8s/clusterrolebinding.yaml
Enter fullscreen mode Exit fullscreen mode

Test

[!TIP]
It is okay to see No resources found at this stage, as the syncer managing the AthenzDomain is not yet deployed.

The commands above registered the AthenzDomain CRD (shortened to domain). Let's quickly check:

kubectl get domain

# No resources found
Enter fullscreen mode Exit fullscreen mode

Setup: Create a secret to represent k8s-athenz-syncer

Unlike a standard production setup which uses CopperArgos to auto-distribute X.509 certificates, we will simply use the root certificate for this quick demo to represent k8s-athenz-syncer as an Athenz service.

kubectl create secret generic k8s-athenz-syncer-cert \
  -n kube-yahoo \
  --from-file=cert.pem=./athenz/certs/athenz_admin.cert.pem \
  --from-file=key.pem=./athenz/keys/athenz_admin.private.pem \
  --from-file=ca.pem=./athenz/certs/ca.cert.pem

# secret/k8s-athenz-syncer-cert created
Enter fullscreen mode Exit fullscreen mode

Setup: Create our custom deployment

[!NOTE]
Note the following configurations:

  • We set zms-url=https://athenz-zms-server.athenz:4443/zms/v1 because we are sharing the same Kubernetes cluster with the Athenz server
  • We set update-cron=5s to see the synchronization results quickly

As mentioned earlier, we are using a custom manifest that mounts the Secret we just created:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-athenz-syncer
  namespace: kube-yahoo
  labels:
    app: k8s-athenz-syncer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-athenz-syncer
  strategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 0%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: k8s-athenz-syncer
    spec:
      containers:
      - name: syncer
        image: local/k8s-athenz-syncer
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 1
            memory: 1Gi
          requests:
            cpu: 1
            memory: 1Gi
        args:
        - --zms-url=https://athenz-zms-server.athenz:4443/zms/v1
        - --update-cron=5s
        - --cert=/var/run/athenz/cert.pem
        - --key=/var/run/athenz/key.pem
        - --cacert=/var/run/athenz/ca.pem
        - --exclude-namespaces=kube-system,kube-public,kube-k8s-athenz-syncer,default,local-path-storage,kube-node-lease,athenz,ajktown-api,kube-yahoo
        volumeMounts:
        - name: athenz-certs
          mountPath: /var/run/athenz
          readOnly: true
      serviceAccountName: k8s-athenz-syncer
      volumes:
      - name: athenz-certs
        secret:
          secretName: k8s-athenz-syncer-cert
EOF

# deployment.apps/k8s-athenz-syncer created
Enter fullscreen mode Exit fullscreen mode

Verify: Does it work?

Please refer to the Result section above to see the verification steps and outcome.

What I learned

Here's what I've learned:

  • I discovered that when members change in a Trusted Domain (delegated role), the Modified Date and ETag of the Provider Domain do NOT change.
  • I learned that CustomResourceDefinition's v1beta1 API has been deprecated
  • I learned that you can enforce type in CRD

What's next?

I will dive into jag token and how Athenz integrates with it, and what Athenz lacks so far.

Dive Hours: 24.5 Hours

[!NOTE]
aegis that utilizes syncer's CRD and kubernetes RBAC enforcer has been stopped as they do not sync super well yet.

  • 1/1 Thu: 6.75 Hours
  • 1/2 Fri: 4.75 Hours
  • 1/3 Sat: 6.5 Hours
  • 1/11 Sun: 6.5 Hours

With the separate PRs of the following:

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

cats_thumbs_up

Top comments (0)