Provisioning k3s clusters over Tailscale

It has been a while since I wrote the last article about this deployment platform. The shape of it has changed since then, and I have been meaning to write down why.

The old pipeline still exists in spirit. GitHub Organizations, GitHub Actions, k3s and SOPS are still there. The part that changed first was provisioning, the one I was least happy with. If you are new to this, the previous article describes the platform this one builds on and the decisions behind it.

The previous setup used Ansible playbooks. They worked, but they were a second codebase with its own abstractions, and every new provider meant another set of assumptions about networking, users and firewall tools. Most of each playbook repeated the same work. Create a user, copy a key, install packages, configure a private network, bring up k3s. None of it was interesting, and all of it had to be right before the cluster could exist.

So I wrote a small tool called tailk3s. You give it a list of servers and provider logins, and it gives you a k3s cluster on a Tailscale network. The repository is at github.com/ude-p/tailk3s.

The cluster file

You describe the cluster in an HCL file. If you have used Terraform, the format will look familiar. Each block is one server.

node "cp1" {
  host      = "203.0.113.10"
  role      = "server"
  init_user = "ubuntu"
  init_key  = env("SSH_KEY")
}

node "w1" {
  host      = "203.0.113.20"
  role      = "agent"
  join      = "cp1"
  init_user = "ubuntu"
  init_key  = env("SSH_KEY")
}

host is where the machine can be reached. role decides what it does. A server runs the Kubernetes control plane, which holds the API and the cluster state. An agent is a worker. It runs the applications. join tells an agent which server to join.

init_user and init_key are the login the provider gave you. tailk3s uses them once, for the first connection, and removes the account when it is done.

You run one of two commands. tailk3s plan --config cluster.hcl shows what would change without touching anything. tailk3s up --config cluster.hcl performs it. You can also describe a single node with flags and skip the file entirely.

There is no state file to lose. Every step checks what the machine already looks like and skips work that is already done. Running up twice does nothing the second time.

What Tailscale does

A Kubernetes cluster needs its machines to reach each other. The control plane has an API that the workers call, the control plane servers talk to each other, and pods on different machines need a route between them. Those addresses can be public, but most clusters keep that traffic on a private network so it is not exposed to the internet. On a cloud provider, that normally means a VPC, private subnets and firewall rules, and every provider does it differently.

Tailscale builds that private network for you, across providers. It is built on WireGuard, a modern VPN protocol. Each machine joins a network called a tailnet and gets a stable private address in the 100.x range, for example 100.117.36.53. It can then reach the other machines at their addresses, whether they sit in the same data centre or on different providers. When a direct connection is possible, the machines talk directly. When it is not, Tailscale relays the traffic, so you do not have to open ports or arrange a route.

The part that matters for this tool is identity. Every machine in the tailnet has a name, and it can be given a tag such as dev-server. A policy file, the ACL, decides which names and tags may reach which addresses and which users may connect. tailk3s uses that policy for two things. It gives every node a tag, and it adds one SSH rule that lets the orchestrator and administrators log in to those tags. I never copy an SSH key to a machine, and I never write provider-specific firewall rules.

By the time Kubernetes is being installed, the network already exists. Every machine has the same kind of private address it would have had inside a single VPC, and that address does not change when the cluster moves to another provider.

How it works

The work happens in three phases. tailk3s prepares the tailnet, gets each machine onto it, and then finishes the cluster over it.

The tool runs on your machine and joins the tailnet itself, as a node tagged tag:tailk3s. You do not need Tailscale installed on your laptop. It needs one thing from you, a Tailscale API key. It uses the key to read and update the access policy and to create the keys the machines join with.

First, prepare the tailnet.

• Read the cluster file and check it. Machines that start a cluster are handled before the ones that join it, because a joining machine needs a control plane to exist first.
• Update the access policy. Every node will get a tag, which is a label the policy can match on. The policy needs an owner for each tag and one SSH rule that lets the tool and administrators log in to machines wearing them. If the policy already says this, the tool writes nothing.
• Create a joining key for each machine that is not on the tailnet yet. The key is a single-use ticket, and it expires after ten minutes.

Then, get each machine onto the tailnet.

• Connect with the provider login and prepare the machine. Install the base packages, set the hostname, create the admin user, install Tailscale, and join the tailnet with the node's tag. This is the only time the provider password or key is used.

Finally, finish the cluster over the tailnet.

• Connect again through Tailscale SSH. Close the public firewall. Only ports 80 and 443 stay open to the internet, and everything else is allowed on the tailnet interface. Then install k3s, configure it, and wait for the API to answer.
• If the machine joins an existing cluster, read the join token from a control plane and pass it along. The machines never need SSH access to each other.
• Write a kubeconfig file on your machine so kubectl works, and an SSH config entry so you can connect by name.
• Delete the provider login. The admin account has no password, and Tailscale SSH is the way in.

A diagram makes this easier to see.

                      Tailscale API
                    (policy, auth keys)
                           ^
                           |
  laptop ── tailk3s ───────┘
     │  provider SSH       │  tailscale SSH
     v                     v
   cp1 ─────── tailnet ─────── cp2
     \                         /
      \─────── w1 ────────────/

Two details are worth calling out. Node names carry a short hash of the address, so a node name can be reused across clusters without collisions. And the provider login is deleted at the end. The admin account has no password and is reachable only through Tailscale SSH. That is the point, but it also means the provider console is the recovery path if the tailnet is ever misconfigured.

High availability

HA is a file change. Add another role = "server" node that joins the first one. The first server starts with cluster-init: true, which selects embedded etcd. Every later server joins the same etcd cluster. Agents receive every control plane address in their server: list, so they are not tied to one machine being up. The API server's certificate includes the tailnet address and the hostname, which is what makes the tailnet the transport rather than an afterthought.

There is no external load balancer to configure. k3s clients use the list of servers, and the network underneath is the tailnet.

What this does not give you is machine-level reconciliation. tailk3s is a CLI, not a controller. If a server dies, provisioning a replacement is one command, but nothing does it for you. I am comfortable with that boundary. The machines are provisioned deliberately, and the workloads are reconciled continuously.

What it gets right

The obvious win is that there is no vendor lock-in. Any provider that gives you a Linux server and SSH works. Moving from one to another is a change to host values, not a rewrite of networking and firewall rules.

There are a few less obvious wins.

• No private networking to build. No VPC, peering, NAT gateways or per-provider firewall rules.
• No configuration management layer. No playbooks, no inventory, no state file.
• Secure by default. Public ports are closed, the provider login is deleted, and there are no SSH keys to distribute or rotate.
• Fast. A node is a few minutes, and a cluster is the time it takes to install k3s.
• Readable. The cluster is one file, and plan is honest about what it will do.

What it does not do

It is not a general infrastructure tool. It does not create VPCs, load balancers, DNS records, volumes or provider accounts. You bring the server and an SSH login. It brings the cluster.

It assumes a Linux machine with systemd, and it has only been exercised on Ubuntu. That is a deliberate constraint, not an accident, but it is still a constraint.

It depends on Tailscale. If you do not want a tailnet, this tool is not for you. I think that is a reasonable dependency, because it replaces an entire category of provider-specific work, but it is a dependency.

Finally, it is a one-shot provisioner. It does not watch the nodes. Drift in k3s configuration is corrected when you run it again, not while you sleep. Flux handles that for the workloads. Nothing handles it for the machines.

Flux

The cluster is now disposable. What matters is the state you want it to have, and that state should live in Git. Flux is the tool that makes the cluster follow Git.

The idea is called GitOps. You keep the Kubernetes YAML files in a repository. Flux runs inside the cluster, watches that repository, and continuously compares what the cluster has with what the repository says. When the two differ, it changes the cluster to match. People call that reconciliation. You do not run a deployment command and hope it worked. You change a file, review the change, merge it, and Flux applies it.

Flux is installed with one command, called a bootstrap. It installs its own components into the cluster, commits them to the repository, and keeps them running from then on.

flux bootstrap github \
  --owner=my-org \
  --repository=infrastructure \
  --path=clusters/prod \
  --personal

The layout I use has one directory per cluster, with shared pieces in a base directory. A file in each directory lists the applications and the cluster software that belongs to that cluster.

The value is in the loop, not the tool. If someone edits a resource by hand, Flux notices and changes it back. If a component is deleted, it is recreated. The cluster stops drifting away from what is written down.

Operators

Kubernetes runs what you declare. You write a YAML file asking for three copies of a container, and the cluster keeps that true. The components that watch these declarations and act on them are called controllers. An operator is a controller built for one specific piece of software. It teaches Kubernetes how to run that software, including the parts that need care, such as failover or backups.

Every cluster gets the same foundation. These are installed through Flux before any application lands.

• Tailscale operator. It lets a Kubernetes service be reached over your tailnet. A service is normally reachable only from inside the cluster. Exposing it usually means a public load balancer or a public hostname. With this operator, an internal tool can get a tailnet address instead.
• Longhorn. Persistent storage for pods. A pod that writes to a disk on one node needs its data to follow it when it moves. Longhorn keeps copies of each volume on several nodes, so a pod can be rescheduled without losing its data. It also handles snapshots and backups. The previous article ended by saying Longhorn was a maybe. It is now the default.
• CloudNativePG. Postgres, run by an operator. You declare a Postgres cluster, and it creates the instances, handles replication and failover, and takes backups. Point-in-time recovery means you can restore the database to a specific moment, not just to the last nightly backup.
• kube-prometheus-stack. Monitoring. It collects metrics from the nodes and the cluster, stores them over time, evaluates alert rules, and provides dashboards.
• cert-manager. TLS certificates. It requests certificates from Let's Encrypt, installs them, and renews them before they expire, so HTTPS becomes a resource you declare rather than a manual task.

Each one solves a cluster-level concern. None of them belongs in an application repository, and all of them are easier to reason about when they are part of the same Git history as the cluster.

appd

Flux deploys what is in Git. Someone still has to write it. So I wrote appd to do that part. It is at github.com/ude-p/appdeploy.

A normal web application needs several Kubernetes objects. A Deployment runs the container and keeps the requested number of copies alive. A Service gives those copies a stable address inside the cluster. ConfigMaps and Secrets hold configuration. An Ingress or Route publishes it on a hostname. Writing all of that for every service is repetitive, and Helm charts bring their own template language and values files.

appd lets you describe the application once in HCL and generates those objects for you. The common blocks are app, stateful, job, cron, route, config, secret, volume and manifest.

config "api" {
  env { source = ".env" prefix = "CONFIG_" }
}

secret "api" {
  env { source = ".env" prefix = "SECRET_" }
}

app "api" {
  image    = "ghcr.io/example/api:1.4.0"
  replicas = 2

  env {
    config = config.api
    secret = secret.api
  }

  port {
    service = 80
  }

  health {
    path = "/health"
    port = 3000
  }
}

route "api" {
  to   = app.api
  host = "api.example.com"
  path "/" {}
}

appd run -o prints the YAML instead of applying it, which is the path that works with Flux. appd run applies it directly to the current cluster. appd explain documents the schema, and appd init writes a starter file.

The motivation was the same as before. I was tired of copying the same Deployment, Service, ConfigMap, Secret and Ingress between repositories, and I was tired of Helm values files that exist only to change an image tag. HCL is a better fit for the shape of the problem. It has blocks, it has expressions, and it does not need a templating function to concatenate two strings.

I am also reducing the amount of configuration a user has to write, and rewriting the deployment path so that appd renders and Flux applies. Deploying directly from a CLI is convenient, but it bypasses the property that made Flux worth installing in the first place.

The intended split is simple. appd describes an application, and Flux owns the cluster.

What I think comes next

Provisioning is becoming boring, and that is a good thing. A cluster that takes minutes to create and a Git repository that reconciles it means the interesting work moves up a layer. How fast you can get a correct environment, and how quickly you can understand what broke.

On the second point, I want to build an operator that watches Kubernetes resources and explains them. Not another alerting rule. The system I have in mind correlates events, logs, metrics and ownership, and answers the question you actually ask during an incident. What changed, what depends on it, and what is the smallest safe fix. The model can read. It can propose a patch. It can run a server-side dry-run. It should not silently apply changes to production. The safeguards are the design. Read-only RBAC by default, scoped escalation, an audit trail, a human approval for writes, and an explicit ban on automatic remediation of stateful workloads.

The other area I keep coming back to is GPUs. Kubernetes is a reasonable scheduler for GPU workloads, but it does not hide the hard parts. Device plugins, MIG partitioning, topology, NVLink, storage throughput, and the cost of leaving a card idle. I am learning this area now. I do not have deep experience in it yet, but I am looking for an opportunity where GPU infrastructure is the main problem, in an organization that treats it as a first-class discipline rather than a side project.

For now, the pipeline is smaller and the clusters are faster, and that is enough for one post.