DEV Community: Nabeel Sulieman

Fast Multi-Platform Builds on GitHub

Nabeel Sulieman — Mon, 09 Feb 2026 15:15:00 +0000

If you want to build multi-architecture Docker containers in GitHub Actions, the standard recommendation you'll find online is to install BuildX and QEMU. The downside of this approach is that QEMU emulation is about 10x slower than native hardware. Building my simple Hello World project went from 30 seconds to 3 minutes.

In this post, I will show you several ways to speed up your builds. The options you have are:

Switching to runners that have cross-platform remote builds pre-configured
Building single-architecture images in a matrix and merging them manually
Using GitHub Actions instances as remote builders with Tailscale
Setting up your own machines for remote builds
Using a Kubernetes cluster for remote builds

Switching to a different Runner Provider

This is the simplest solution, but it requires signing up for a new service and will cost some money. There are alternative runner providers that have their runners preconfigured with BuildX and remote builders with native hardware.

One such provider that I tried is Namespace.so. Signing up was fast and easy, and switching to them in my workflows only required changing the runs-on field in my YAML.

Building Single Images and Merging

This option is probably the simplest way to get cross-platform builds without leaving GitHub. You build an individual image for each of the architectures that you want, then merge them with a buildx command like:

docker buildx imagetools create -t nabsul/myproject:v1.0.0 nabsul/myproject:v1.0.0-amd64 nabsul/myproject:v1.0.0-arm64

In this example, I use a matrix of jobs to reduce duplicate YAML, and then a merge job to create the final image.

GitHub Remote Builders with Tailscale

Honestly, this is cool in a nerdy way, but I wouldn't recommend it for production. For each hardware architecture, I spin up a job that starts a buildkitd server. I then join my tailnet with a pre-determined hostname that allows the builder to find the machine. The final step in the job is printf "HTTP/1.1 200 OK\r\nContent-Length: 16\r\n\r\nShutting down..." | nc -l -p 8080 which just waits until someone hits port 8080 and then shuts down.

The main build step configures itself with the remote builders from the previous step. It then joins the tailnet and uses those remote instances to do a cross-platform build. After the build is done, I use a curl command to cause the other jobs to end.

Like I said, this is a pretty cool setup, but there's just so much that can go wrong. If the curl fails, you'll get jobs that hang for a long time, and you'll have to worry about tailnet configurations and security.

Kubernetes Remote Builders

If you happen to have a Kubernetes cluster that has both Intel and ARM nodes in it, you can use them as remote builders. In this example, I create a temporary namespace for each build, run the builds there, and then clean up afterwards.

Overall this is not a bad option if you already have a Kubernetes cluster being used for other purposes. But you probably don't want to be creating one just for the purpose of your builds.

TCP Remote Builders

You can also just run individual VMs of your different hardware types and use them for remote builds. In this example, I created one ARM and one Intel VM and secured them with TLS certs. I then configured BuildX to use those remote runners for the build. You could also leverage Tailscale for this and avoid the need for TLS certificates.

Conclusion

So there you have it, several options to get faster multi-architecture builds on GitHub Actions. Personally, I currently lean towards Namespace.so simply because it only costs me about $2 a month and I'm lazy. If Namespace started to get expensive, I would probably go with the separate builds and merge pattern. And if my builds were starting to get expensive on GitHub, I might look into setting up builders at home and doing remote builds over Tailscale.

Talos Kubernetes in Five Minutes

Nabeel Sulieman — Sun, 28 Sep 2025 22:20:00 +0000

Original post: https://nabeel.dev/2025/09/28/talos-in-five

Talos Linux is an OS designed specifically for running Kubernetes.
It is locked down with no SSH access. All operations are done through a secured API.
The documentation is (understandably) catered to setting up multi-node Kubernetes clusters that are resilient to failure.
But what if you want the cheapest possible Kubernetes cluster, for testing purposes for example, where reliability isn't super important?

In this article I'll show you how to set up a simple single-node Talos cluster in less than five minutes.
By following these instructions, you can have a full Kubernetes cluster running on a single VM,
without the extra costs of control planes and load balancers that cloud providers normally add onto their Kubernetes services.

Summary

The basic outline of steps to create a single-node cluster is:

Get a Talos ISO image
Create a blank Talos VM instance
Update your config to allow workloads on control plane nodes
Initialize the Talos VM and bootstrap the cluster
Install MetalLB
Install Envoy Gateway

Step 0: Get a Talos ISO Image

Okay, this is where I cheat a little. I'm not counting the time it takes to download and upload a Talos VM image as part of the 5 minutes.
This step depends on which cloud provider (or home lab setup) you have.
The good news is that the official documentation is quite good.
Find the section that matches your setup and follow those instructions.

Essentially, you are going to be downloading a Talos Linux ISO.
If you are using a cloud provider (Azure, AWS, OCI, DigitalOcean, etc.),
you will then need to upload that image so that VMs can be created from that image.
I have done this on DigitalOcean and Oracle Cloud. It takes a bit of time, maybe 10-15 minutes,
but it's not hard and you only need to do it once to create as many VMs as you like going forward.

Step 1: Create a Talos VM

Next you will need to create a Talos Linux VM (or server if you're installing on bare metal).
As with the previous section, you will need to follow the instructions based on the infrastructure you are using.
I've been most recently using DigitalOcean and automating everything with PowerShell.
For me, creating a new blank Talos VM looks like this:

doctl compute droplet create --region sfo3 --image $talosImageId --size s-2vcpu-4gb --enable-private-networking --ssh-keys $sshKeyId $vmName --wait

After creating your blank VM, DO NOT follow any other instructions from the documentation!
Specifically, do not execute any of the talosctl commands described there.
This is where we will diverge from the official documentation.

Once your VM or machine is created, make note of its IP address for the following steps.

Step 2: Bootstrap the Cluster

Now we are going to initialize our Talos Kubernetes cluster.
Do this with the following commands:

talosctl gen config $vmName "https://${VM_IP}:6443" --additional-sans $VM_IP -o $CONFIG_DIR
export TALOSCONFIG="$CONFIG_DIR/talosconfig"
talosctl config endpoint $VM_IP
talosctl config node $VM_IP

This will create a directory and populate it with an auto-generated cert and some default configuration files.
Note the following:

--additional-sans ensures that the certificate is valid for the VM's public IP address
Set the TALOSCONFIG environment variable so you don't have to add --talosconfig mydir/talosconfig every time you use talosctl

Talos normally configures separate control plane and worker nodes.
This is good practice for production clusters, but is expensive when you just want to test or kick the tires.
Instead, we want to create a single control plane VM that will also be our worker.
To do this, edit controlplane.yaml in the Talos config directory.
Scroll to the end of the file and uncomment (remove the #) the line # allowSchedulingOnControlPlanes: true.

Now we are ready to initialize the VM. By default, a freshly created VM waits for someone to configure it.
Once you run this command, the VM is locked down to only work with the certificate that you generated with the talosctl gen config command.
Technically, there's a risk that someone could randomly beat you to configuring the VM and take ownership.
The likelihood of this happening is very low, but if it did, you would see a failure in the apply-config command,
and you would simply delete the VM.
There are more secure ways to do this, specifically generating an ISO that is preconfigured to only respond to your cert.
However, that is beyond the scope of this simple tutorial.

talosctl apply-config --insecure --nodes $VM_IP --file "$CONFIG_DIR/controlplane.yaml"

Give the VM a few seconds (I wait 10) to apply the configuration, then run talosctl bootstrap.
You can then run talosctl health or talosctl dashboard to watch the cluster come alive in real-time.

At this point, your Kubernetes cluster is alive and you just need to generate the kubeconfig to use it:

talosctl kubeconfig $CONFIG_DIR
export KUBECONFIG=$CONFIG_DIR/kubeconfig

You should now be able to run commands like kubectl get pods --all-namespaces or k9s.

Step 3: Install MetalLB

Have you ever created a LoadBalancer type service in AKS, EKS, etc., to create a load balancer that routes traffic to your cluster?
MetalLB will give that same functionality, but for free on your bare VM.
You can install MetalLB as the documentation prescribes with:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.2/config/manifests/metallb-native.yaml
kubectl wait --timeout=5m --for=condition=available --all deployments -n metallb-system

After that, we need to configure an IPAddressPool so MetalLB is aware of the IP address we want it to use:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.100/32  # Replace with your VM's public IP

Replace 192.168.1.100 with the public IP address of your VM, save the YAML to metallb-ipaddresspool.yaml and then run kubectl apply -f metallb-ipaddresspool.yaml.
Congratulations, you now have MetalLB installed and ready to work with your Gateway Controller.

Step 4: Install Envoy Gateway Controller

Finally, you will probably want to use the Kubernetes Gateway API
to route traffic through the public IP address to services running in your cluster.
I found that Envoy Gateway was the easiest solution to achieve this.
The quick start documentation worked flawlessly, but in summary:

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.5.1 -n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available

You can test that everything works as it should with the following:

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.5.1/quickstart.yaml -n default
curl --verbose --header "Host: www.example.com" http://$VM_IP/get

Conclusion

And there you have it! A simple single-node Kubernetes cluster in less than the time it took to read this article.
You can create as many as you like, tear them down, and create more when you need them.
I ended up automating all of this in a PowerShell script, and the time to run is 3-4 minutes.
This script likely won't work right out of the box for you, but it should be fairly easy to adapt it if you like:

echo "Getting VM parameters..."
$sshKey = (doctl compute ssh-key list -o json | ConvertFrom-Json | where {$_.name.Contains('dummy')}).id
$imageId = (doctl compute image list -o json | ConvertFrom-Json | where {$_.name.Contains('Talos')}).id
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$vmName = "kcert-test-$timestamp"
mkdir $vmName | Out-Null
echo "Creating droplet..."
$vmJson = (doctl compute droplet create --region sfo3 --image $imageId --size s-2vcpu-4gb --enable-private-networking --ssh-keys $sshKey $vmName --wait -o json)
$vm = $vmJson | ConvertFrom-Json
$vmIp = $vm[0].networks.v4 | where {$_.type -eq 'public'} | Select-Object -ExpandProperty ip_address
echo "VM created with IP address: $vmIp"
echo $vmIp > $vmName/ip.txt
echo "Initializing Talos cluster at $vmIp"
talosctl gen config $vmName "https://${vmIp}:6443" --additional-sans $vmIp -o $vmName
$env:TALOSCONFIG = (Resolve-Path "$vmName/talosconfig").Path
talosctl config endpoint $vmIp
talosctl config node $vmIp
$yaml = Get-Content -Path "${vmName}/controlplane.yaml"
$yaml = $yaml -replace '# allowSchedulingOnControlPlanes:', 'allowSchedulingOnControlPlanes:'
Set-Content -Path "${vmName}/controlplane.yaml" -Value $yaml
talosctl apply-config --insecure --nodes $vmIp --file "${vmName}/controlplane.yaml"
echo "Sleeping for 10 seconds to allow the node to initialize..."
Start-Sleep -Seconds 10
talosctl bootstrap
echo "Sleeping for 10 seconds to allow the cluster to stabilize..."
Start-Sleep -Seconds 10
talosctl health
talosctl kubeconfig $vmName
$env:KUBECONFIG = (Resolve-Path "$vmName/kubeconfig").Path
echo "Setting up MetalLB"
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.2/config/manifests/metallb-native.yaml
kubectl wait --timeout=5m --for=condition=available --all deployments -n metallb-system
@"
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
  - $vmIp/32
"@ | kubectl apply -f -
echo "Setting up Envoy"
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.5.1 -n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
echo "Here are your environment variables:"
$envVars = @(
    "`$env:KUBECONFIG = '$env:KUBECONFIG'",
    "`$env:TALOSCONFIG = '$env:TALOSCONFIG'",
    "`$env:VMIP = '$vmIp'"
)
$envVars | ForEach-Object { echo $_ }
$envVars | Out-File -FilePath "$vmName/env.txt" -Encoding utf8

NATS: You Need it Now!

Nabeel Sulieman — Sun, 18 Sep 2022 15:58:09 +0000

Original Post: https://nabeel.dev/2022/09/17/nats

If you are running Kubernetes, or really any kind of microservice architecture, you will eventually run into challenges with communication and synchronization between your instances. To solve this, I recommend deploying an instance of NATS as part of your initial infrastructure setup.
NATS is great because:

It's tiny, light-weight, and easy to run
A single instance will likely be sufficient for the needs of your whole cluster
It will be there, ready and waiting, when you need it
It solves the problem of one-to-many communication
It can be used to build extensible event-driven systems

What is NATS?

NATS is a light-weight, easy to deploy service that provides pub-sub functionality with very little fuss. It is a tiny application, written in Go, that listens on a port for connections from clients.

The NATS executable is a few MB in size and runs out of the box with sensible defaults. It has no dependencies or required configuration parameters. As a Kubernetes service, it can be deployed very easily with this yaml. With that simple deployment, your microservices can use NATS by connecting to nats://nats:4222.

Clients can send and receive messages to each other by publishing and subscribing to subjects. For example, two clients could be subscribed to subject x. If any client publishes a message to subject x, all subscribed clients will receive that message.

NATS Use-Cases

NATS can replace and streamline many service-to-service communication scenarios. The following sections describe a few of them:

Broadcast to All Instances of a Distributed Service

This was my first use for NATS. I had a deployment with multiple instances running in the cluster, and when a configuration change is made, I need all instances to reload their configurations from a database.

To solve this problem, every instance of my service subscribes to myapp.refresh. When the configuration changes, I publish a message to that subject, and all instances will take action by reloading their configurations.

Ping-Pong

Want to get some information or a status report from all instances of your service? To fetch information about all running instances:

All instances listen to myapp.ping
Start listening to some unique temporary subject called myapp.pong.[UNIQUE_GUID] for example
The single instance will publish a message such as replyto=myapp.pong.[UNIQUE_GUID] to the myapp.ping subject
Every instance listening to myapp.ping will then respond to the myapp.pong.[UNIQUE_GUID] subject with the relevant information

You can listen to the myapp.pong.[UNIQUE_GUID] subject for a certain amount of time and then unsubscribe from it. It should only take a few milliseconds to receive messages from all listening instances.

Event-Driven Systems

The beauty of NATS is that multiple clients can subscribe to the same subject without any fancy configuration or setup. This can be very handy when building a future-proof system that can easily be extended. Take the following scenario for example:

Imagine you are running a microservice-based e-commerce system.
One microservice handles payments and another one handles the front-end UI that customers see. The front-end might send a message requesting that a payment be processed (using NATS or a REST API), and then it might listen on a predetermined subject (payments.updates.[TXN_ID] for example) for a notification that the payment has completed.

Imagine now that you want to add a quota system that automatically updates inventory numbers whenever a purchase is made. You might be tempted to add that logic to either your front-end or payment microservice. However, this functionality doesn't logically fit into either of these services. With NATS, you could create a new microservice that subscribes to payments.updates.* to receive notifications of all payment updates. It could then perform the desired action, and we did all this without modifying any of the existing systems.

Performance Concerns

A simple instance of NATS should be fine for most workloads. Some possible concerns might be:

Speed

Although using NATS involves an extra network hop compared to direct communication, you have to remember that this is all done over an already open TCP connection (no handshake overhead),
and that this will all likely be communication between machines that are quite close to each other physically. The round-trip times I typically observe in my Digital Ocean cluster is less than 70ms (which means a one-way of about 30ms).

Volume

You might be worried that a single instance won't be able to handle the number of services and messages that you need to send.
But remember, a single instance of NATS should be able to easily handle thousands of simultaneous connections. Furthermore, NATS is fairly stateless and should not demanding in terms of CPU or memory. It simply receives a message, forwards it to all the subscribers, and then forgets about it.

Reliability

What happens if NATS goes offline? What about network issues? These are valid concerns, but:

Statistically speaking, the smaller your cluster the more rare outages are
For non-mission-critical applications a small outage is likely not going to cause major issues
NATS doesn't really make this problem worse, it exists either way

If your reliability needs really aren't met by a simple NATS instance, there are solutions: ACKs and retries, periodic refreshes from persisted source of truth, or running NATS in a high-availability configuration
(also see Jetstream documentation).

Conclusions

NATS is an easy to use service that provides extremely useful functionality for today's distributed microservices. It provides the right balance simplicity vs. performance to be useful for many applications, and it can grow as your needs do. I also highly recommend checking out this Changelog podcast episode about NATS.

Replacing YAML with TypeScript

Nabeel Sulieman — Sat, 30 Apr 2022 17:39:07 +0000

Original Post: https://nabeel.dev/2022/04/30/yaml-alternative

TL;DR

Are you tired of copy-pasting and editing a ton YAML files? In this post I suggest using TypeScript to define your services and either Handlebars templates or the Kubernetes NodeJS client to more easily manage your deployments. You can find some sample code that demonstrates this at https://github.com/nabsul/k8s-yaml-alternative.

Introduction

I've been using YAML files checked into a Git repo to manage my Kubernetes deployment for many years now. But as the number of deployments grows, this approach starts to run into problems:

A lot of the YAML is boilerplate that is repeated over and over again.
Global changes are tedious, requiring editing all the files in your repo.

I've recently started experimenting with managing my deployments in a different way. In this post I will describe the steps I take to reach this design. The example that I will be using is a cluster running three application:

An NGINX service with two replicas exposing port 80
A single instance of NATS exposing ports 4222 and 8222
A custom application that doesn't have ports but requires some environment variables

Start with your Own Definition

I usually start building out my deployments by first deciding what language/system I will use. For my cluster this has been the YAML files, but it could also have been something like Terraform or Helm charts. With that approach, my application requirements take a back seat and the focus becomes: "What does this system require to work?". Take for example a Kubernetes deployment specified in YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1
  namespace: k8s-ts-test
  labels:
    app: service1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: nginx:latest
        ports:
        - containerPort: 80
          name: port80

What parts of of the above specification do I really care about or control? Number of replicas, image tag, and the application port. That's it, three lines out of twenty. Everything else is boilerplate.

Instead of trying to fit our application into what Kubernetes wants, let's start with a specification that only includes the details that we care about. Based on the initial list of services that I want to run, we can define everything in as follows:

const services = { 
    service1: {
        image: 'nginx:latest',
        replicas: 2,
        ports: [80]
    },
    service2: {
        image: 'nats:latest',
        replicas: 1,
        ports: [4222, 8222]
    },
    service3: {
        image: 'app:latest',
        replicas: 1,
        env: {
            VAR1: 'Some Value',
            VAR2: 'another value',
        }
    }
}

Starting with the specification like this keeps me focused on the details that I care about. You can clearly see how many applications I plan to deploy, and how those applications differ from each other. We can worry about YAML/Terraform/Helm details later. Moreover, I can change my mind about the YAML/Terraform/Helm question and still keep this definition as the starting point for everything I do later.

From Custom Definition to YAML

Converting this custom specification into regular YAML can easily be done using Handlebars templates. You can see the all of the templates here, but here is a small sample:

    spec:
      containers:
      - name: {{name}}
        image: {{image}}
        {{#if ports}}
        ports:
        {{#each ports}}
        - containerPort: {{this}}
          name: port{{this}}
        {{/each}}

The handlebars library is used in /generate.ts to create all of the YAML in one go. With this approach notice that:

All of my deployments are guaranteed to look the same.
If I need to make a change to all of my services (API version or namespace change for example) I can easily do it with one template change.

You can generate /generated.yaml by running npm run generate. You can then deploy all of the services in one go with kubectl apply -f generated.yaml.

Skipping YAML Altogether

In this repo I also demonstrate how to eliminate the need for YAML completely. Instead of generating YAML and running kubectl, I use the @kubernetes/client-node npm package to directly deploy to Kubernetes. This requires a little more work than YAML generation, but has several advantages. The biggest advantage is that the npm package includes TypeScript definitions for V1Secret, V1Service, and so on. This makes your deployment strongly typed and reduces the possibility of errors compared to authoring YAML. You can follow the /deploy.ts script to see how this all works, but at the heart of it is a simple loop:

for (var [n, s] of Object.entries(services)) {
    console.log(`Deploying ${getName(namespace, n)} started`)
    await makeSecret(namespace, n, s)
    await makeDeployment(namespace, n, s)
    await makeService(namespace, n, s)
    console.log(`Deploying ${getName(namespace, n)} complete\n`)
}

What Next?

I'm only starting to rethink how I want to manage and deploy my Kubernetes clusters. I think this is a good start, and I hope to share more learnings as I continue to experiment. As next steps I'm going to be looking into removing more of the "click-ops" that I do when creating my cluster:

Logging into the digital ocean dashboard to create a new cluster
Manually setting up the load balancer
Finding the load balancer IP address and configuring DNS to point to the new cluster
Configuring all the infrastructure (docker repo, AWS) and application secrets

As for challenges I foresee: In this example, the applications are defined generically such that you could use them even if you wanted to skip Kubernetes altogether. However, I have two Kubernetes-specific applications that I run in the cluster: KCert and ecr-login-renewal. These applications require special Kubernetes configurations around service accounts and permissions. I'm not yet sure how to cleanly encode those.

If you like this idea, give it a try in your own setup. If you're comfortable writing code in NodeJS/TypeScript, try out the Kubernetes client approach. Or if you're just looking to simplify your templates, give Handlebars templates a try.

My "Artisinal" Ingress

Nabeel Sulieman — Tue, 19 Apr 2022 00:07:42 +0000

Original Post: https://nabeel.dev/2022/04/17/myingress-intro

Summary

I built a replacement for nginx and cert-manager in my Kubernetes cluster. It leverages NATS and CockroachDB, and is written in .NET Core C#.

It's simple, easy to setup, and easy to understand. It features a web-interface for management and configuration. It's also horizontally scalable out of the box and aims to follow all the best practices for high availability and observability.

Finally, it's pre-alpha at the moment and I'm not ready to open-source the project. However, I'm looking for like-minded people who might be interested in turning this into something more broadly useful.

Background

For the past year or so, I've been working on replacing cert-manager in my Kuberentes cluster. It started with a cert-manager outage due to a DNS bug, and by learning how to manually manage certificates. I then automated all of that with KCert.

When I got KCert done, I realized there was another part of my setup that could be improved: the NGINX Ingress controller. So I decided to continue the effort and replace that as well.

And that is how I created My "Artisinal" Ingress. I call it artisinal because I built it to my own personal taste. So far I'm pleased with the result. I'm using it in my personal Kubernetes cluster, which is serving the page you are reading right now.

Design Decisions

When I decided to build a replacement for NGINX Ingress Controller, I came up with several goals:

Simple Setup

I'm not a fan of Helm or CRDs (more accurately: I love the idea of CRDs, but I think they're often used unnecessarily). They are frequently used to create overly complex systems, and make it extremely difficult to debug when things goes wrong. Sometimes your only option is to delete the whole cluster and start again.

For example, take a look at the cert-manager and NGINX Ingress Controller installation:

The installation of cert-manager requires 7000+ lines of yaml
cert-manager runs three pods in the cluster (Is cert management really that complex?)
NGINX Ingress controller can be installed via Helm or almost 700 lines of yaml

Is all this necessary? What if something goes wrong? Will you know how to debug and fix any of this?

No State in Kubernetes

New versions of Kubernetes are released quite frequently. The easiest way for me to stay up to date is to create a brand-new cluster, move all my services there, and destroy the old one.

This requires copying everything over from the old cluster to the new one. While my deployement and service definitions are checked into a git repository, secrets and certificates are not. Those objects need to be manually copied over.

For this reason I decided to eliminate the need to copy certificates and ingress configurations. I decided to store the state of my ingress controller in a central CockroachDB store. This really could have been anything: S3, Azure Key Vault, etc. But the main idea is to store all of this information outside of the Kubeneretes cluster.

This approach has two advantages:

I don't have to copy Kubernetes certificates from one cluster to another
I can deploy multiple Kubernetes clusters that rely on the same source of truth

Must be Easy to Scale Horizontally

My first ingress controller in Kubernetes was Traefik, and I was happy with it for a long time. Then I discovered that it doesn't support multiple instances (what they called high-availability).
Certificates were stored on the local disk and couldn't be shared across multiple instances of the service. The paid version of Traefik did not have this limitation, and that did not sit well with me. I even tried to fix that myself, and eventually gave up and moved to NGINX.

For this reason, I set out from the start to design my ingress controller to scale seamlessly. Using CockroachDB as my data store is the first part of solving this, but there is also the problem of keeping all nodes synchronized when things change. I decided to leverage NATS for this purpose.
Using NATS made it easy for all instances of the service to stay synchronized and exchange messages.

Built in Certificate Management

I thought about using KCert
together with this new system, but decided against it. It feels like they should be two separate systems, but especially with ACME HTTP challenges, it becomes difficult to cleanly separate the two.

The other issue is that KCert is specifically geared towards working with Kubernetes ingresses and certificates. However, I had decided that I want to store that information outside of Kubernetes for this project. I therefore couldn't use KCert without decoupling it completely from Kubernetes and making it much more complex.

I therefore decided to build certificate management directly into my new controller.
This wasn't too much work, since I could reuse the code I wrote in KCert.

Good Observability Practices

I wanted to make sure that the system is easy to debug, monitor, and maintain. For the monitoring piece, I tried Azure's Application Insights, Datadog, and Honeycomb. All of these options are great, but:

I'm sure there are other great options out there.
Pulling in all those client libraries doesn't feel right.

I'm therefore leaning towards a more generic approach: I will use Open Telemetry, which is the standard the industry is converging to. Most monitoring systems support Open Telemetry, either natively or through side-car shims.

Other Considerations

If I were optimizing for broad community adoption, I would have written this in Go or Rust. However, I really enjoy writing in C# and I can practice Go and Rust at work. For this reason I decided to go with .NET Core C# and used YARP.

Additionally, I've been looking for an excuse to learn and use both CockroachDB and NATS. I use CockrochDB's cloud service as my data store and NATS to keep my load balancer instances synchronized.

Where's the Code?

The project is currently private and I'm probably not going to open-source it any time soon. I expect that I will open-source it at some point,
but for now I want to have the freedom to make drastic design changes. Open-sourcing it would leave me worried about affecting anyone using the code.

I would however love to collaborate on this idea if there is interest. If you are interested in seeing the code and helping turn it into a usable open-source project, please reach out! The easiest ways to contact me are LinkedIn and Twitter.

KCert - V1 Release

Nabeel Sulieman — Thu, 03 Mar 2022 00:39:27 +0000

Original post: https://nabeel.blog/2022/02/27/kcert-v1

It's been about a year since I introduced my certificate manager: KCert. For my specific needs KCert was a great improvement over the ubiquitous cert-manager. You can read about the differences/advantages in the project's README. However, I did not take the time to review the design to make it more generally useful.

This month I finally got around to revisiting KCert. I have redesigned several aspects of the tool, and I think what I have now is much more refined, simplified, and easy to use.

The changes I've made are:

Get Started Fast

A basic installation of KCert is now super fast. You should be able to get started in a matter of minutes. Just edit three lines of the provided deploy.yml file and use kubectl apply to deploy it to your cluster. You can now start creating ingresses and KCert will issue the needed certificates. It's that simple!

Less Reliance on a UI

Before the refactor, setting up KCert required entering your initial settings via the web UI. I came to the conclusion that this is not consistent with how Kubernetes is usually managed. Everything is now configured in the more standard "config as code" approach.

The web UI still exists, but it is mostly now a read-only view of the tool's status. The only actions you can take in there are sending a test email and manually renewing certificates
(and I'm considering removing that second feature).

Watching for Ingress Changes

Before this release, certificates were created in the web UI with a browser-based form. The secret name and hosts are entered and KCert creates a Kubernetes secret based on that information. That is all gone now.

Instead, KCert now watches for changes to ingresses marked with the kcert.dev/ingress=managed label. Whenever a change occurs, KCert will check if it needs to issue new certificates accordingly. KCert supports issuing multi-host (but not wildcard) certificates. If multiple TLS definition exist across different Ingress definitions, a single certificate will be created for all referenced hosts.

Conclusion

This project has been dormant for a while, but I'm really excited about this latest update. I think it's finally in a state that should make it useful to many people.

So give it a try!

KCert: Simple Let's Encrypt for Kubernetes

Nabeel Sulieman — Sun, 28 Mar 2021 21:40:14 +0000

Original post here

Last month I wrote about a tool I've been using in my cluster to manager Let's Encrypt certificates. For me, building this tool has been a fantastic learning experience, and as the author of the tool I am extremely pleased with the result. I doubt I will ever go back to using cer-manager.

Today I would like to announce that KCert is stable enough for broader usage. I would love for people to try it out, submit feedback, and help me turn this into a tool that is useful to more people than just myself.

What is KCert?

KCert is meant to replace cert-manager in your Kubernetes cluster. It offers a simple alternative to the complex system that is cert-manager. Instead of thousands of lines of yaml, multiple services and custom resource types, KCert runs as a simple, single-instance service in your cluster. It will automatically renew certs before they expire and can send you email notifications of actions taken. It also has a web UI for manual configuration and management.

How Reliable is the Tool?

I've been running KCert in my own cluster since December. Over the past few months, I've been tweaking the UI and reorganizing the way the tool works to make it as simple as possible. Of course, as the author of the tool I am biased and my experience is not the same as a new user. I've very keen to see if other people will be as excited about this tool as I am.

What If I Have Trouble Using KCert?

If you have any issues or questions using KCert, please submit your question or feature request in GitHub.

Potential Caveats

This tool works great for me in my cluster and environment. However, as a side-project, I've only been working on it part time.This is not a perfectly polished, production-grade tool. There are no automated tests and I've only tested it in my own cluster with the following setup:

Kubernetes 1.19.3 on Digital Ocean
Standard nginx ingress controller

I suspect that folks with different configurations will encounter bugs. If you take the time to describe the issues you face,
I will happily try to resolve them. Specifically, as requests come in, I expect to expand KCert to support:

Other types of Ingress controllers (HAProxy, Traefik, Kong, etc.)
Other architectures such as ARM for Raspberry Pi

Getting Started

Trying out KCert is extremely simple and low-risk. Simply apply the deploy.yml file to your cluster. Unlike cert-manager, the file is less than 100 lines and should be easy to follow and understand.

deploy.yml will create a few resources in a new namespace called kcert, as well as a global ClusterRole and ClusterRoleBinding for accessing TLS certs. Deleting KCert is as simple is deleting those resources.

Please let me know how it goes!

DigitalOcean Hackathon Submission: Meal Match

Nabeel Sulieman — Mon, 11 Jan 2021 06:26:07 +0000

What I built

An App to help families and friends decide what to eat.

Category Submission:

Random Roulette

App Link

https://mealmatch.mydemo.dev/

Screenshots

Description

Day after day, families, couples, and any group of people wanting to eat together face the age-old question:

What's for dinner tonight?

This app can help. Build your own list of options, or search and copy results from Yelp. Share a link and see what people like! The vote can be set to automatically end when enough matches are found.

Link to Source Code

https://github.com/mealmatch/app

Permissive License

MIT

Background

My brother-in-law came up with this idea: Wouldn't it be great if there was something like a dating app, but for deciding what to eat?

How I built it

I started building this app in .net core because that is what I'm most comfortable with. The front page needed a bit of client-side magic, so I threw in some VueJS to handle that. The VueJS was simple enough that I could add it without any addition build (webpack) step.

I also wanted to try to build an app that respects privacy and doesn't even ask users for their emails. Vote sessions are assigned a random GUID. The user can bookmark the URL if they want to return to the page.

For data storage I used Azure Table Storage. It's a basic NoSQL database that's easy to use.

For getting Yelp results, I created a developer account at Yelp and used their REST API. I tried my best to follow their usage guidelines and make clear which results came from their data.

I tested the app locally until I got everything working. From there I created a Docker file to build it, put it on GitHub and created the Digital Ocean App instance.

I was pleasantly surprised at how seamless Digital Ocean App was. The platform automatically built my docker image and deployed it. Adding the needed environment variables and a custom domain was intuitive and took very little effort.

Additional Resources/Info

https://dotnet.microsoft.com/
https://vuejs.org/
https://www.yelp.com/developers/documentation/v3/business_search
https://azure.microsoft.com/en-us/services/storage/tables

How to Effectively use a Single Monitor

Nabeel Sulieman — Fri, 27 Mar 2020 18:49:52 +0000

Original post: https://nabeel.dev/2020/03/27/single-monitor/

Many people I know love working with multiple monitors. Personally, I'm not a huge fan. The only times I find myself using a second monitor are
when I'm on a video conference (or watching a video) and want to keep working while I listen. Other than that, I'm generally a single-monitor user.

I feel like using a single monitor helps keep me focused. Instead of moving my head left and right between monitors, I use keyboard shortcuts to quickly switch windows on my screen. With many people working from home these days, I imagine many don't have the space for two monitors anyways.

Here are some tricks I use to make the most of my single monitor:

Virtual Desktops

I use virtual desktops the way many people use multiple monitors. In Windows, you can press win+tab to view all your windows, and at the top you'll see a list of virtual desktops. Press the + sign to create as many as you need.

Switching between virtual desktops is fast and easy: hold the ctrl and win keys down, and press left or right to move between desktops. If you're not already comfortable with the alt+tab keyboard shortcut, then I highly recommend mastering it to quickly switch between windows.

Split the Screen

The most common justification I hear for two monitors is: "I need to see two documents side-by-side." For most cases there's a simple alternative: Split Screen. In fact, I would argue that split screen is better than two monitors,
since the two documents can be placed closer together (or even overlapping!)
to make side-by-side comparisons even easier.

In Windows I use win+left and win+right all the time for this purpose. That keyboard shortcut will automatically size a window to occupy
the left or right half of your screen respectively.

Invest in a Good Monitor

In my home office, I use a 27" Dell LED monitor. Not all 27" monitors are created equal. Consider getting a high-quality monitor and not the cheapest one possible. And definitely consider getting one high-quality monitor over two low-quality monitors.

Your Laptop is your Second Monitor

My laptop is my primary work machine. I have docking stations at work and at home, so I just plug in and start working wherever I am. Usually my laptop's screen is turned off while I'm plugged into the docking station. For the few situations where I want a second monitor, I simply turn that screen on.

End

Connecting Kubernetes to AWS ECR

Nabeel Sulieman — Mon, 23 Mar 2020 21:22:57 +0000

I'm pleased to announce the release of k8s-ecr-login-renew
(GitHub / Docker). It's a small tool written in Go that simplifies working with Amazon's Elastic Container Registry (ECR). It addresses the fact that ECR Docker login credentials expire every 12 hours. k8s-ecr-login-renew solves this by:

Fetching Docker login credentials from an AWS
Creating/Updating a Docker login secret in Kubernetes
Running as a cron job to prevent the Docker secret from expiring

The source code and Docker image are published here:

Source Code: https://github.com/nabsul/k8s-ecr-login-renew
Docker Image: https://hub.docker.com/repository/docker/nabsul/k8s-ecr-login-renew

I'm also quite proud of the README and example code. My hope is that they will make getting started extremely easy.

Enjoy!

PS: Feedback is welcome and desired! Please also let me know if you found this tool useful (or if you had trouble using it).

Analyzing WordPress Hook Usage with Azure Data Lake

Nabeel Sulieman — Fri, 31 May 2019 19:05:09 +0000

Note: I wrote this post in 2017, so keep in mind that the code may need updating and things like Azure prices have probably changed.

WordPress provides a large number of hooks that allow plugins to extend and modify its behavior. A few months ago, I was curious about which of these hooks are popular, and which of them are hardly ever used. I was also looking for an excuse to give Microsoft’s Data Lake Analytics a spin. U-SQL looked especially attractive as it brought back fond memories of petabyte-scale data crunching at Bing.

With that in mind, I set out to build some tools that would calculate the usage of WordPress’s hooks. Breaking that up into smaller steps, I came up with:

Crawl all published plugins on WordPress.org
Extract which hooks are used by each plugin
Extract a list of WordPress hooks
For each WordPress hook, calculate its usage

On the technical side, I set the following goals for this project:

The code should be developed in C# and U-SQL
The project should use .NET Core so that it’s cross-platform (Windows, Linux, Mac)
The project should be usable in Visual Studio, VS Code or from the command line

In this article I talk about the approach and algorithms in general. For the nitty-gritty details, you can check out the source code here: https://github.com/nabsul/WordPressPluginAnalytics.

See the README.md file for instructions on building and running the code.

Crawling for Plugins

I decided to crawl the WordPress.org plugins directory to extract a list of all the plugins. All of the plugins can also be accessed from a common SVN repository, but with different branches and tag folders, I felt that would be slightly more tedious than crawling the html pages to extract the official link to each zip file. The HtmlAgilityPack library makes parsing HTML and extracting information very easy. I use it to parse each page of plugins for the links to each individual plugin page, and then I parsed each plugin page for the zip file URL.

Once I have the zip file URL, I uploaded it to Azure Blob Storage. I considered skipping this and working directly with the data from WordPress.org, but I felt this approach allowed me to have a stable snapshot of the original data to experiment on without repeatedly hitting wordpress.org for the same data.

Running the process sequentially takes nearly 5 hours from a Digital Ocean droplet, but about 90% of that time is just waiting on I/O. Therefore, adding some parallelism to this process made a lot sense. This was done very simply by fetching all 12 plugins per page in parallel. This brought the run time down to just over an hour.

Extracting Data

Now that I have my raw data, the next step is to extract useful information out of it. I used System.IO.Compression.ZipArchive to iterate over each PHP file in the zip file. I then considered writing my own code to parse each PHP file, but quickly gave up on the idea when I realized how complicated that would get. So I looked around and found Devsense.Php.Parser. Using this library, I was able to work directly on tokenized data and avoided all the hassle of parsing text myself.

With that library, I extracted each hook usage and creation in the PHP files. I only count instances where the hook name is a constant string, since it would be impossible to predict the hook name for code like add_action( "updated_$myvar", ...).

The final result needed to be in a format that can be easily analyzed with U-SQL and Azure Data Lake Analytics. U-SQL comes with built in TSV extractors, so if you upload your raw data in that format, you don’t need custom C# code to process it. Data Lake Analytics can automatically uncompress gzipped files, which is great since my TSV files compress to about 10% of their uncompressed size.

Extracting the plugins takes less than 1 hour, so I didn’t bother to run parts of that code in parallel.

Running the Analysis

The final step of the process is running a U-SQL script to analyze the data and generate the final report. You can upload the data manually or using the command line tool included in the project. You should have two extraction files: One for the WordPress source code and one for all the plugins. The final step is to run the U-SQL script. Again, you can edit and submit the script manually, or if you followed the naming conventions used in the program you can submit the job using the command line tool.

U-SQL is a SQL-like language. If you’re familiar with SQL, the code in the script should all make sense. The raw data is read from the uploaded files. The WordPress data is filtered by hooks created and the plugins are filtered by hooks used. Hook usage is counted using a GROUP BY statement. The hooks from WordPress and the plugins are then cross-referenced using a JOIN. The graph of the job looks like this:

The Cost of Data Lake Analytics

The job should take a couple of minutes to run and costs around $0.03 (US). However, I learned a few important lessons on the pricing of Data Lake jobs. First, when running on a few GB of data make sure you run with a parallelism of 1. Increasing the parallelism on a small data set is just a waste of money. For example, my 3-cent job cost 12 cents when I ran it with a parallelism of 5. I also suspect that compressing my data files helped reduce the cost of jobs. Compressed data should mean less data travelling over the network, which can often result in significantly faster (and cheaper) jobs.

The second and more important point is about using custom code and libraries in your scripts: It is possible to upload and use custom .NET DLLs in your U-SQL scripts, but I highly recommend avoiding that unless it’s absolutely necessary. I experimented with uploading the individual plugin zip files to Data Lake storage and using a custom extractor library that directly processed the zip files and tokenizes the PHP. The cost of running such a job was around $5. This is way more than the cost of working on TSV files but it does makes sense since doing the Zip extraction and PHP parsing on Microsoft’s Azure infrastructure will consume far more CPU cycles than if you do most of the pre-processing separately.

As you can see, unlike simpler services like storage, the cost of using this type of service can vary widely depending on how you design your data pipelines. It is therefore important to spend some time researching and carefully considering these decisions before settling on an approach.

Viewing the Results

The final result of running the script is a small TSV formatted report with the follow pieces of information:

Hook Name: The name of the hook (prefixed with action_ and filter_ to differentiate those two types of hooks)
Num Plugins: Number of plugins using the hook
Num Usages: Number of times the hook is used

The data can be imported to a spread sheet for further analysis and charting:

https://1drv.ms/x/s!AoNGbuElNYPMjMUVzq5931eX9YzSuA

Conclusions

Overall, I felt like there was definitely a learning curve to Azure Data Lake services, but it wasn’t all too bad. I’m definitely curious how all of this could be done in the Hadoop ecosystem, which I’m much less familiar with. If anyone would like to try replicating these results in Hadoop, I would greatly appreciate a tutorial and/or shared source code.

This code could easily be expanded to perform other types of analysis. For example, it might be interesting to see the usage of various WordPress functions and classes. It also might be interesting to reduce the list of plugins to the most popular ones to get more realistic usage information for the hooks.