DEV Community: Joe Block

Creating a Talos cluster with a Cilium CNI on Proxmox

Joe Block — Sun, 04 Jan 2026 16:27:42 +0000

I’ve been meaning to set up a talos cluster in my homelab for a while and set one up over the holiday break. Here’s how I did it.

All the blogs and videos I looked at used the nginx ingress, which would be fine, except that the nginx ingress is a dead man walking and will be unsupported starting in March of 2026. No patches, no security updates, completely unsupported.

Based on some advice on the hangops slack (Thanks, Brandon!) I wanted to use Cilium since it supports the Gateway API and can also do ARP announcements like MetallB.

This is part one of a series I'm writing as I get my homelab cluster up and running.

Talos Homelab Setup Series

01 Creating a Talos cluster with a Cilium CNI on Proxmox
02 Add SSL to Kubernetes using Cilium, cert-manager and LetsEncrypt with domains hosted on Amazon Route 53

Pre-requisites

proxmox will make it easier to rebuild your cluster if you make a mistake, but these instructions will work with bare metal as well.
cilium, kubectl & helm cli tools. If you don't want to brew install them or are not using a Mac, installation instructions are at cilium.io, helm.sh and kubectl.

Software Versions

Here are the versions of the software I used while writing this post. Later versions should work, but this is what these instructions were tested with.

Software	Version
`helm`	4.0.1
`kubectl`	1.34
`kubernetes`	1.34.1
`talos`	1.11.5

Let's get started.

Create a Proxmox VM control plane node

There are a ton of videos and blogs describing getting started with proxmox, so I'm not going to go into a lot of detail here. The TL;DR is:

Go to the Talos image factory.
Create an image.
1. Start with the Cloud Server option
2. Select the latest Talos version
3. Pick nocloud from the cloud type screen since it explicitly mentions proxmox in the description
4. Select your architecture
5. You should now be on the System Extensions page. Pick qemu-guest-agent and util-linux-tools
6. Pick auto for the bootloader.
You should now see Schematic Ready. Copy the ISO link
Download the ISO into your Proxmox host's ISO storage
Start a new VM with at least 4 CPUs, 4 GB of RAM and at least a 16GB drive. Make sure you enable the qemu agent. If you're planning to use this for real workloads later, you'll want to go bigger - have a look at Sidero's System Requirements page. I went with 100G per their recommendations.
Wait until the Talos Dashboard appears, then copy the new node's IP address
On your DHCP server, find the IP from step 6, and set that as a static assignment. The server will reboot multiple times during installation, and if it changes IP you will have to update your kubeconfig and talosconfig files. If it changes IP addresses after you add a worker node, things will break so spare yourself future aggravation and give it a static assignment from the beginning.

Once the Talos dashboard on the VM console shows that is in maintenance mode you can start to configure it. Again, make sure your DHCP is assigning it a static IP, that will save you aggravation later.

We're going to make a single node cluster to simplify things since it's just for learning. You can add worker nodes later very easily once you want to put real workloads in the cluster.

Configure the cluster control plane

Setup some environment variables

Set a CONTROL_PLANE_IP environment variable to make copying commands from the post easier.

export CLUSTER_NAME=sisyphus
export CONTROL_PLANE_IP=10.0.1.51

Find out what disks are on the server

The Talos installer needs to know what device is the node's hard drive, so use talosctl to get the available disks.

talosctl get disks --insecure --nodes "$CONTROL_PLANE_IP"

You'll see something like

NODE        NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL           SERIAL
10.0.1.51   runtime     Disk   loop0   2         73 MB    true
10.0.1.51   runtime     Disk   sda     2         17 GB    false       virtio      true                QEMU HARDDISK
10.0.1.51   runtime     Disk   sr0     2         317 MB   false       ata         true                QEMU DVD-ROM    QEMU_DVD-ROM_QM00003

Our node's hard drive is sda, so

export DISK_NAME=sda

Create a cluster patch file

We're going to use Cilium as the CNI and also have it replace kube-proxy, so let's create the cluster with no CNI and disable kube-proxy. To do that, we're going to create a patch file we can use when we generate the cluster's configuration with talosctl.

# cluster-patch.yaml
cluster:
  network:
    cni:
      name: none
  proxy: # Disable kube-proxy, Cilium will replace it too
    disabled: true

Generate the talos configuration

talosctl gen config $CLUSTER_NAME "https://$CONTROL_PLANE_IP:6443" --install-disk "/dev/$DISK_NAME" --config-patch @cluster-patch.yaml
export TALOSCONFIG="$(pwd)/talosconfig"

Initialize the cluster

I like to test things in short-lived clusters so I don't have to worry about breaking things that my internal services depend on. I like naming nodes clustername-role-number so that when I look at their proxmox console, it's nice and clear what cluster the node is and what its role is.

Here's how to create a patch file that sets the node name when we apply our configuration. We also want to enable scheduling on the control plane node since we're setting up a single-node cluster.

Create a controlplane-1-patch.yaml that includes the hostname you want and sets allowSchedulingOnControlPlanes to true.

# controlplane-1-hostname-patch.yaml
machine:
  network:
    hostname: sisyphus-cn-1
cluster:
  allowSchedulingOnControlPlanes: true

Apply the configuration to your control plane node to initialize the cluster with the merged controlpane.yaml and controlplane-1-patch.yaml files.

talosctl apply-config --insecure \
  --nodes $CONTROL_PLANE_IP \
  --file controlplane.yaml \
  --insecure \
  --config-patch @controlplane-1-patch.yaml

Bootstrap etcd in the cluster

ONLY DO THIS ONCE! Wait until you see etcd is waiting to join the cluster in the bottom portion of the dashboard. Depending on how fast your proxmox host is, this can take 5-10 minutes.

There will be some error messages and it will look like nothing is happening, be patient it will get back to ready. I think this is because we're configuring the cluster to not include a CNI so we can use Cilium, and/or because we disable kube-proxy because Cilium replaces that functionality too.

The first time I stood a cluster up without CNI it took long enough that I thought I'd broken the configuration - it wasn't till I kicked it off and then went to cook dinner that I gave it enough time to settle down.

So be patient, at least you only have to do this once per cluster.

Create a kubeconfig file

talosctl kubeconfig sisyphus-kubeconfig --nodes $(CONTROL_PLANE_IP)
export KUBECONFIG="$(pwd)/sisyphus-kubeconfig"

Confirm that the cluster came up

kubectl get nodes

You'll see something similar to

NAME              STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION   CONTAINER-RUNTIME
sisyphus-cn-1     Ready    control-plane   5m      v1.34.1   10.0.1.51     <none>        Talos (v1.11.5)   6.12.57-talos    containerd://2.1.5

Install cilium

First install the CRDs

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.2.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml
# Confirm gateway classes
kubectl get crd gatewayclasses.gateway.networking.k8s.io gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io

Confirm the gateway classes are present

kubectl get crd gatewayclasses.gateway.networking.k8s.io gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io

You should see something like this:

NAME                                       CREATED AT
gatewayclasses.gateway.networking.k8s.io   2026-01-02T04:20:13Z
gateways.gateway.networking.k8s.io         2026-01-02T04:20:14Z
httproutes.gateway.networking.k8s.io       2026-01-02T04:20:15Z

Set up the cilium helm repo

helm repo add cilium https://helm.cilium.io/
helm repo update

Install cilium

cilium install \
  --version 1.18.1 \
  --helm-set=ipam.mode=kubernetes \
  --helm-set=kubeProxyReplacement=true \
  --helm-set=securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --helm-set=securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --helm-set=cgroup.autoMount.enabled=false \
  --helm-set=cgroup.hostRoot=/sys/fs/cgroup \
  --helm-set=l2announcements.enabled=true \
  --helm-set=externalIPs.enabled=true \
  --set gatewayAPI.enabled=true \
  --helm-set=devices=e+ \
  --helm-set=operator.replicas=1

You'll see something like

ℹ️  Using Cilium version 1.18.1
🔮 Auto-detected cluster name: sisyphus
🔮 Auto-detected kube-proxy has not been installed
ℹ️  Cilium will fully replace all functionalities of kube-proxy
I0101 21:25:52.110400   48637 warnings.go:110] "Warning: spec.SessionAffinity is ignored for headless services"

This took several minutes to come up on my sisyphus cluster controller with 2 cores and 4GB RAM

Confirm cilium status

Confirm that Cilium is fully up and has no errors.

❯ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium-envoy             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 1
                       cilium-envoy             Running: 1
                       cilium-operator          Running: 1
                       clustermesh-apiserver
                       hubble-relay
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.18.1
Image versions         cilium             quay.io/cilium/cilium:v1.18.1@sha256:65ab17c052d8758b2ad157ce766285e04173722df59bdee1ea6d5fda7149f0e9: 1
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.34.4-1754895458-68cffdfa568b6b226d70a7ef81fc65dda3b890bf@sha256:247e908700012f7ef56f75908f8c965215c26a27762f296068645eb55450bda2: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.18.1@sha256:97f4553afa443465bdfbc1cc4927c93f16ac5d78e4dd2706736e7395382201bc: 1

Update talosconfig

The beginning of your talosconfig file will start with something like this:

context: sisyphus
contexts:
    sisyphus:
        endpoints:
            - 10.0.1.51
        ca:

Update it to include a nodes entry

context: sisyphus
contexts:
    sisyphus:
        endpoints:
            - 10.0.1.51
        nodes:
            - 10.0.1.51
        ca:

This will keep you from contantly having to specify --nodes for your talosctl commands.

Confirm that the cluster is showing healthy

talosctl health

Check external connectivity to cluster services

First, test a `LoadBalancer` service

Make a playground directory and put the following files in it.

Create the playground namespace

# 01-create-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: playground
  name: playground

Create an IP Pool and Announcement Policy

# 02-cilium-setup.yaml
# Create our list of IPs
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "default-pool"
spec:
  blocks:
  - start: "10.0.1.160" # Use IPs that are outside of your DHCP range but on
    stop: "10.0.1.170"  # the same /24 as your talos VM.
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: l2-announcement-policy
spec:
  serviceSelector: {}
  nodeSelector: {}
  # On a multi-node cluster, you may not want the control-plane nodes
  # making arp announcements. Uncomment the nodeSelector stanza here
  # to disable that.
  # nodeSelector:
  #   matchExpressions:
  #     - key: node-role.kubernetes.io/control-plane
  #       operator: DoesNotExist
  externalIPs: true
  loadBalancerIPs: true
  # Different hardware will show different network device names.
  # This list of regexes (in Golang format) will find all the common
  # naming schemes I've seen for network devices so that Cilium can
  # find a network interface to make arp announcements.
  interfaces:
    - ^eth+
    - ^enp+
    - ^ens+
    - ^wlan+
    - ^vmbr+
    - ^wlp+

Create a deployment and service in the playground

Talos is focused on giving you a default secure cluster out of the box, so you can't just use kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0 - talos requires you to configure the securityContext. Here's an example deployment for nginx that runs as a non-root user and specifies the pod's resource requirements to satisfy the Pod Security Admission configuration that ships with talos. More info about that here.

# 03-playground-nginx.yaml
apiVersion: v1
kind: Service
metadata:
  name: playground-nginx-service
  namespace: playground
  annotations:
    # Tells Cilium to manage this IP via LB IPAM
    cilium.io/lb-ipam-pool-name: "default-pool"
    # Optional: For L2/BGP to announce this IP
    cilium.io/assign-internal-ip: "true" # Or use externalIPs:
    # If using externalIPs:
    # kubernetes.io/ingress.class: "cilium" # For Ingress
    # For a specific IP
    # lbipam.cilium.io/ips: "192.168.1.50" # The specific IP you want Cilium to answer on
spec:
  type: LoadBalancer
  selector:
    app: playground-nginx-pod
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: playground-nginx-app
  namespace: playground
spec:
  selector:
    matchLabels:
      app: playground-nginx-pod  # <-- must match pod labels exactly
  replicas: 1
  template:
    metadata:
      labels:
        app: playground-nginx-pod
    spec:
      # Talos is very security oriented, so we have to set up the
      # security context explicitly
      # ---------- Pod‑level security settings ----------
      securityContext:
        runAsNonRoot: true
        runAsUser: 101 # non‑root UID that the image can run as
        seccompProfile:
          type: RuntimeDefault
        # Uncomment if you need a shared FS group for volume writes
        # fsGroup: 101
      # ---------- Containers ----------
      containers:
        - name: nginx-playground-container
          image: nginxinc/nginx-unprivileged:latest # Alpine, but we force a non‑root UID
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          # talos requires us to specify our resources instead of
          # letting k8s YOLO them
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL

Deploy the playground

If you pass a directory name to kubectl with -f, it will apply (or delete) all resources found in the .yaml files in that directory.

kubectl apply -f playground

See what IP the playground is using

It will almost certainly be the first IP address in your IP Pool, but confirm that.

kubectl get service -n playground

You'll see something like:

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
playground-nginx-service   LoadBalancer   10.111.180.4   10.0.1.160    80:30597/TCP   1m15s

Confirm Connectivity

curl http://THE_EXTERNAL_IP

You should see the nginx default page! Don't delete the playground yet, we're going to use it to confirm that the Cilium Gateway API is working.

Test Cilium's Gateway API

We configured Cilium to provide Gateway API services to the cluster when we installed it. Let's confirm that it's working correctly.

Make a new gateway-tests directory.

Create a Gateway

Create gateway-tests/01-create-gateway.yaml. For ease of testing we're going to configure the gateway to allow it to be used by services in any namespace. We're also assigning it a specific IP address so we can give it a stable FQDN. We don't want that changing.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: playground-gateway
spec:
  gatewayClassName: cilium
  addresses:
    - type: IPAddress
      value: 10.0.1.160
  listeners:
  - name: http
    protocol: HTTP # Case matters!
    port: 80
    allowedRoutes:
      namespaces:
        from: All

Update the `playground-nginx-service`

Before we can add a HTTPRoute, we're going to need to update the playground-nginx-service so it isn't a LoadBalancer, so create gateway-tests/02-playground-service.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: playground-nginx-service
  namespace: playground
spec:
  selector:
    app: playground-nginx-pod
  ports:
    - name: http
      port: 80 # This is what the service is listening on, and what will be routed to
      targetPort: 8080 # Port the pods are listening on, don't route directly here!

Create a HTTPRoute

Create gateway-tests/03-http-route.yaml to route all incoming requests for ip-160.mydomain.com to our playground-nginx-service.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: playground-http-route
  namespace: playground
spec:
  parentRefs:
  - name: playground-gateway
    namespace: default
  hostnames:
  - "ip-160.mydomain.com"
  rules:
  - backendRefs:
    - name: playground-nginx-service
      port: 80

Create the Gateway test resources

kubectl apply -f gateway-tests

Confirm it worked

curl http://ip-160.mydomain.com

It should display a "Welcome to nginx!" document that looks like this:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Congrats, you have a working talos cluster.

I will cover setting up SSL by creating certificates with cert-manager, LetsEncrypt and Route 53 in a follow-up post.

Adding a worker node

I originally planned to make that another post, but it's only two steps.

Create another VM

The talos website recommends at least 2 CPUs and 2 GB RAM. I set mine to 50G of disk. You can use the same ISO you used when creating the control plane node.

Make sure it has a static IP assignment in DHCP.

Add it to the cluster

Create a patch file with the node name you want

# worker-1-patch.yaml
machine:
  network:
    hostname: clustername-worker-1

When the worker node console gets to Maintenance, you can add it with a one line talosctl command

talosctl apply-config --insecure --nodes "$WORKER_IP" --file worker.yaml --config-patch @worker-1-patch.yaml

My proxmox cluster isn't on beefy hardware so it took a couple of minutes for the new node to join the cluster and start accepting workload.

Problems I ran into

When I was making this post, I ran into a couple of problems because I was redoing everything to run on a single node blog cluster and made some mistakes tidying things up for a post.

`curl` fails to connect

While testing the LoadBalancer service, even though the service LoadBalancer shows that it has an external IP, when you test with curl, it gives an error message similar to this:

$ curl http://10.0.1.160/
curl: (7) Failed to connect to 10.0.1.160 port 80 after 16 ms: Could not connect to server

And when you check the pods

kubectl get pods -n playground

it shows the worker pod as pending, not crash loop backoff, not creating, just pending.

NAMESPACE     NAME                                          READY   STATUS    RESTARTS      AGE
playground    pod/playground-nginx-app-6d7ddb5b95-lv82x     0/1     Pending   0             12s

When I ran into this with the a blog cluster while writing this post, it was because I made the test cluster a single node cluster and forgot to set allowSchedulingOnControlPlanes to true, so there was no place to schedule the pods. You can fix this by applying an updated configuration.

talosctl apply-config --insecure \
  --nodes $(CONTROL_PLANE_IP) \
  --file controlplane.yaml \
  --config-patch @controlplane-1-patch.yaml

`curl` connects, but you get a 404

While testing the gateway, curl connects to the IP but you get a 404 error.

curl -iv http://ip-160.mydomain.com/index.html
* Host ip-160.mydomain.com:80 was resolved.
* IPv6: (none)
* IPv4: 10.0.1.160
*   Trying 10.0.1.160:80...
* Established connection to ip-160.mydomain.com (10.0.1.160 port 80) from 10.0.1.121 port 61150
* using HTTP/1.x
> GET /index.html HTTP/1.1
> Host: ip-160.mydomain.com
> User-Agent: curl/8.17.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< date: Sun, 04 Jan 2026 00:36:07 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host ip-160.mydomain.com:80 left intact

I ran into this during testing because I forgot to update the gateway yaml file to allow all namespaces after I moved all the test manifests into the playground namespace.

Talos reboots so fast that when I make settings changes to a single node cluster I always reboot the control plane node with talosctl reboot -n $CONTROL_PLANE_IP.

You changed cilium settings but they don't appear to have any effect

Some changes to cilium settings require you to restart cilium pods to pick them up.

kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system rollout restart ds/cilium-envoy
kubectl -n kube-system rollout restart deployment/cilium-operator

Tips

If you plan on standing up and tearing down VMs, copy the MAC of the first one. Go to the proxmox datacenter UI, select the VM, then select Hardware and double click Network Device for details) and set each replacement to that MAC. Your DHCP server uses a machine's MAC to determine if it should get a static assignment, so recycling the MAC keeps you from having to update DHCP each time you bring up a new VM.

This is one of the few times it's a good idea to reuse a MAC - having two VMs or physical machines with the same MAC running simultaneously will cause problems with on your network.

2023 Hacktoberfest work

Joe Block — Sun, 12 Nov 2023 16:12:26 +0000

Intro

I'm an SRE living in Denver.

Hacktoberfest 2023

In total, this year I had 29 PRs merged during this year's Hacktoberfest as a contributor. I reviewed and merged 17 PRs by other contributors on projects I maintain.

Maintenance work

This year I did maintenance work on awesome-zsh-plugins, the sysadmin-reading-list, the [zsh-quickstart-kit](https://github.com/unixorn/zsh-quickstart-kit and git-extra-commands where I reviewed & merged 17 contributor PRs

New open source contributions

I also wrote prometheus-moosefs-tricorder, a prometheus exporter for moosefs.

And finally, I hit 3260 consecutive days of GitHub contributions during this Hacktoberfest.

There's a bug in the streak detection where it occasional trims 12-18 months off of my streak graphic.

Fix Securifi Peanut issue with zigbee2mqtt

Joe Block — Fri, 07 Oct 2022 07:20:45 +0000

Securifi Peanut plugs have issues with zigbee2mqtt.

I have several Securifi Peanut Zigbee switches. Overall, they’re nice little smart plugs and make good Zigbee routers to strengthen your Zigbee mesh, but they have one annoying issue - zigbee2mqtt doesn’t recognize them perfectly, though there’s a simple fix that I’m going to document here.

Adding Peanut Smart Plugs to zigbee2mqtt

The Peanut Smart Plug does not provide a modelId in its database entry, so zigbee2mqtt can't identify it to know how to handle it. Fortunately, it's an easy fix, though you'll have to do it every time you add a new Peanut plug.

Pair your new Peanut(s) to your zigbee2mqtt instance
Once you've added all the peanut plugs, stop zigbee2mqtt. We're going to need to do some surgery on its database.db file and that can't be done with the service running.
Backup your database.db file. If you mess up the edit, you'll want to be able to revert and try again easily.
Edit your database.db file. Add a "modelId":"PP-WHT-US" to each of your Peanut entries. For example, change ""manufId":4098, to "manufId":4098,"modelId":"PP-WHT-US",
Once you've finished editing database.db, restart the zigbee2mqtt service.

You should now see proper entries with capabilities in zigbee2mqtt and be able to turn the switches on and off, both from zigbee2mqtt and from Home Assistant. Now would be a good time to go to zigbee2mqtt's OTA tab and check if your Peanut plug(s) have any firmware updates.

AWS IAM Self Tagging EC2 Instances

Joe Block — Sun, 13 Jun 2021 21:21:03 +0000

For a variety of reasons, I needed to enable some EC2 instances to write/update a single EC2 tag, but the instaces needed to only be able to tag themselves.

This was more annoying than I expected, so I'm documenting the IAM policy here.

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "ec2:DeleteTags",
              "ec2:CreateTags",
              "ec2:DescribeInstances"
          ],
          "Resource": "*",
          "Condition": {
              "StringEquals": {
                  "aws:ARN": "${ec2:SourceInstanceARN}"
              },
              "ForAllValues:StringEquals": {
                  "aws:TagKeys": "THAT_ONE_ALLOWED_TAG"
              }
          }
      }
  ]
}

Some notes:

The AWS IAM editor in the webui will complain about SourceInstanceARN. Ignore it and click next anyway.
Then it will complain that the policy doesn't add any permissions. It lies. Ignore it and save the policy.

You can attach this policy to an IAM role and the instances will then be able to tag themselves, but only with the THAT_ONE_ALLOWED_TAG tag.

Building Multi Architecture Docker Images with buildx

Joe Block — Sun, 13 Jun 2021 15:37:07 +0000

I've got a mix of architectures in my basement cluster - some Odroid HC2s that are arm7, some Raspberry Pi 4s that are arm64, and am soon going to add an Intel node as well. It's more hassle than it's worth to have to specify different images for the different architectures. I already build my own copies of images, so I decided to start building all my images as multiarchitecture images.

This turned out to be a lot easier than I was expecting - recent stable builds (2.0.4.0 (33772) or higher) of Docker Desktop can build for other architectures by running virtual machines in QEMU, so I can do the whole build on my MacBook Pro instead of baking each architecture separately and stitching them together with a manifest file.

Install the latest Docker Desktop for macOS
Enable experimental mode either by setting DOCKER_CLI_EXPERIMENTAL=enabled in your environment or by adding "experimental" : "enabled" to ~/.docker/config.json
Do docker buildx ls to see the current builders
docker buildx create --name multiarch
docker buildx use multiarch
docker buildx inspect --bootstrap

Now you're ready to build. Go into one of your docker projects, then do docker buildx --platform linux/amd64,linux/arm64,linux/arm/v7 -t username/demo --push .

If you're not ready to push your image to docker hub, do --load instead of --push to have it build the image and copy it out of the buildx system and into your local docker.

Use a Raspberry Pi as a print server

Joe Block — Sun, 06 Jun 2021 20:09:51 +0000

I have an old HP 4050N. For a variety of reasons, I want to have it behind a print server instead of having our laptops print directly to it. Here's how I set that up.

Prerequisites

A Raspberry Pi (or honestly any Linux box) running docker. I like the Raspberry Pi and Odroid HC2 for this sort of thing because they have very low power consumption.
A printer that is supported by CUPS

Setup

There are several docker images out there you can use, I made one (the source is on github at unixorn/docker-cupsd) because I wanted one that was multi-architecture - mine has AMD64, ARM7 and ARM64 all baked into the same image so you don't have to change the image label based on what system you're running it on. It works fine on Raspberry Pi, Odroids and Intel servers.

The unixorn/cupsd image is a bit on the large side because I crammed a lot of printer drivers into it, you may want to look for docker images that only support single printer families.

We're going to store printers.conf in a directory outside the container so that we don't lose our printer configuration every time we upgrade our container.

I run the cupsd server on an Odroid HC2 because I have /var/lib/docker on the 2TB drive attached to it. I could have put it on one of the Raspberry Pis in my cluster, but didn't want it spooling print jobs and causing excessive wear on a rPi's microSD card.

Make a directory to store your printer configuration. We'll use /docker/cupsd/etc
export CUPSD_DIR='/docker/cupsd/etc'
touch $CUPSD_DIR/printers.conf
Run the cupsd server with

sudo docker run -d --restart unless-stopped \
  -p 631:631 \
  --privileged \
  -v /var/run/dbus:/var/run/dbus \
  -v /dev/bus/usb:/dev/bus/usb \
  -v "$CUPSD_DIR/printers.conf:/etc/cups/printers.conf" \
  unixorn/cupsd

You can now connect to http://SERVER:631 and add printers using the web UI.

When adding the printers to your Mac, select Internet Printing Protocol and put in the IP or DNS name of your print server machine.

The queues are printers/printername, not printername.

Home Assistant Printer Power Management

Joe Block — Sun, 06 Jun 2021 19:54:13 +0000

I've got an old HP laser printer in my basement. We barely print 10 pages a month between the two of us, so we only turn it on when we're going to print. That's a hassle though, because inevitably we forget to shut it off sometimes and it stays on overnight or even for days, and while it has a powersave mode, the 4050N is so old that even that burns a good amount of power.

Enter Home Assistant.

Prerequisites

You have HA configured to connect to a MQTT server

The watcher script and associated tooling all presume that we can send messages to a MQTT topic that HA is watching.

Your printer is connected to a cupsd server running in a container

Your computers should be configured to print to the cupsd server instead of directly to the printer.

I run cupsd in a container on one of my Odroids. I could run it on the same Odroid HC2 that I run Home Assistant (HA) on, but there's no compelling reason to do so and I'm reserving that node for strictly HA containers like Home Assistant itself and my MQTT server. I picked an Odroid because it has a SATA drive attached and my /var/lib/docker is on the hard drive and not an microSD card - there's no reason you can't run it on a Raspberry Pi other than to prevent excessive wear on the microSD card.

You could modify the watcher script if you're running cupsd directly instead of in a container, but I run my cupsd in a container, so that's what the script is designed for.

There are plenty of articles about setting up cupsd, but I wrote about setting up cupsd here.

Your printer is plugged into an outlet controlled by HA

We want to be able to toggle the power from Home Assistant.

Printer Power Control

mosquitto helper script

I don't like to install anything more on my docker hosts than I absolutely have to, so instead of installing mosquitto directly on the printserver machine, I run mosquitto_pub inside a container with the following c-mosquitto_pub helper script. You can download it from github here. Put this in /usr/local/bin.

#!/usr/bin/env bash
#
# Use docker to run mosquitto_pub
#
# Copyright 2019, Joe Block <jpb@unixorn.net>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

exec docker run -t --rm eclipse-mosquitto mosquitto_pub $@

cupsd Watcher

Once I had cupsd configured to share the printer (as Franklin), I wrote a quick script that checks the print queue to see if it is empty or not. If there are jobs in the queue, it writes ON to an MQTT topic, hass/printers/franklin. If the queue is empty, it writes OFF. The examples here all assume your printer is named Franklin, replace Franklin with your printer's name.

Actually, I lied. When there are jobs, it writes OFF and then ON.

Why? Because I don't want HA to switch the printer off immediately once the queue drains - the printer has enough RAM that there may still be several pages left to print when it has accepted all of the job from the server.

Instead, I've configured HA to restart a timer every time it sees the MQTT topic hass/printers/franklin switch from OFF to ON, and only turn the printer off after the queue has been empty for five continuous minutes.

Here's the ha-check-for-print-jobs script source - you can download it from github here.

Put the script in /usr/local/bin on the same server you're running the cupsd container on - it is designed to run a tool inside that container.

#!/usr/bin/env bash
#
# ha-check-for-print-jobs
#
# Check if there are print jobs on $PRINT_Q. If there are, write
# MQTT messages to a watched topic so HA knows to turn on the power
# to the printer.
#
# Copyright 2021, Joe Block <jpb@unixorn.net>
#
# License: Apache 2

set -o pipefail

# Make all these overridable easily in your cron setup
PRINT_Q=${PRINT_Q:-'Franklin'}
CONTAINER=${CONTAINER:-'cupsd-server'}
MQTT_HOST=${MQTT_HOST:-'mqtt.example.com'}
MQTT_TOPIC=${MQTT_TOPIC:-'hass/printers/franklin'}

# We are run out of cron every minute, but I don't want it to take an
# entire minute to turn on the power because I'm impatient and the printer
# takes a bit to start up. When we print and walk downstairs, I want it
# to have already started printing by the time I get there. If I was
# patient, I wouldn't have bothered to write this tool :-)
#
# So, when we get run by cron, we check the queue CHECK_COUNT times, with
# CHECK_DELAY seconds between each run.
CHECK_COUNT=${CHECK_COUNT:-'11'}
CHECK_DELAY=${CHECK_DELAY:-'5'}

export PATH="$PATH:/usr/local/bin:/usr/local/sbin"

if [[ -f /tmp/printerdebug ]]; then
  DEBUG='true'
fi

# Only spam syslog when DEBUG is set
debugout() {
  if [[ -n "$DEBUG" ]]; then
    echo "$@"
  fi
}

validate-settings(){
  debugout "CONTAINER: $CONTAINER"
  debugout "PRINT_Q: $PRINT_Q"
  debugout "MQTT_HOST: $MQTT_HOST"
  debugout "MQTT_TOPIC: $MQTT_TOPIC"

  valid='true'
  if [[ -z "$CONTAINER" ]]; then
    echo "CONTAINER is unset"
    valid='false'
  fi
  if [[ -z "$PRINT_Q" ]]; then
    echo "PRINT_Q is unset"
    valid='false'
  fi
  if [[ -z "$MQTT_HOST" ]]; then
    echo "MQTT_HOST is unset"
    valid='false'
  fi
  if [[ -z "$MQTT_TOPIC" ]]; then
    echo "MQTT_TOPIC is unset"
    valid='false'
  fi
  if [[ "$valid" == "false" ]]; then
    echo "Configure your settings."
    exit 1
  fi
}

print-job-checker() {
  printjobs=$(docker exec -t "$CONTAINER" lpq -P "$PRINT_Q" | grep -c 'no entries')

  if [[ "$printjobs" == '1' ]]; then
    debugout "No jobs in print queue, notifying HA"
    c-mosquitto_pub -h "$MQTT_HOST" -t "$MQTT_TOPIC" -m OFF
  else
    echo "jobs found in print queue, notifying HA"

    docker exec -t "$CONTAINER" lpq -P "$PRINT_Q"

    # Set the status off, then back to on, so that the HA timer restarts
    # and HA doesn't turn off the printer in the middle of a job
    c-mosquitto_pub -h "$MQTT_HOST" -t "$MQTT_TOPIC" -m OFF
    c-mosquitto_pub -h "$MQTT_HOST" -t "$MQTT_TOPIC" -m ON
    debugout "re-enabling printer $PRINT_Q..."
    docker exec -t "$CONTAINER" lpadmin -p "$PRINT_Q" -o printer-error-policy=retry-current-job
  fi
}

validate-settings

# We run the print-job-checker every 5 seconds to minimize the UI delay on the
# macOs end

for i in $(seq $CHECK_COUNT)
do
  print-job-checker
  debugout "waiting..."
  sleep $CHECK_DELAY
done

Home Assistant Setup

I configured my HA to watch a MQTT topic as a binary sensor. You can download this snippet here.

binary_sensor:
  - platform: mqtt
    name: "Franklin Print Queue"
    payload_on: "ON"
    state_topic: "hass/printers/franklin"

Now, when the watcher writes ON and OFF to the hass/printers/franklin queue, that binary sensor will change status and we can trigger an automation for it.

This automation will turn the printer power on every time the binary sensor is turned on, and turn it off five minutes after the last time the binary sensor switched from ON to OFF.

The outlet my printer is plugged into is controlled by HA and rather unimaginatively named switch.printerpower.

Add this stanza to your automations.yaml file. Download it here.

# Franklin power is controlled by MQTT
  - alias: 'Turn on Franklin when there are jobs in the queue'
    trigger:
      platform: state
      entity_id: binary_sensor.franklin_print_queue
      to: 'on'
    action:
      service: homeassistant.turn_on
      entity_id: switch.printerpower

  - alias: 'Turn off printer 5 minutes after print queue drains'
    trigger:
      platform: state
      entity_id: binary_sensor.franklin_print_queue
      to: 'off'
      for:
        minutes: 5
    action:
      service: homeassistant.turn_off
      entity_id: switch.printerpower

Test the pieces

Print server check

Confirm that you've got the print queue configured correctly by running docker exec -it cupsd-server lpq -P Franklin. If there are no jobs, it should print something like

Franklin is ready
no entries

Automation test

Reload your automations, and you should now be able to test that the automations are correct by running c-mosquitto_pub -h mqtt.yourdomain.com -t hass/printers/franklin -m OFF or -m ON and watch HA turn the power to your printer off and on.

Once that is working, print a job, and if you run ha-check-for-print-jobs the printer power should get turned on.

Run it all automatically

Now that you've confirmed that the power is being cycled properly when the MQTT queue recieves messages and that the print job checker is seeing the printer queue, we can add the checker job to cron.

Add

* * * * * PRINT_Q=Franklin MQTT_HOST=mqtt.example.com MQTT_TOPIC=hass/printers/franklin CONTAINER=cupsd_server /usr/local/bin/ha-check-for-print-jobs | logger -t printserver

to your /etc/crontab, and you're good to go. Now every minute, the checker script will get run by cron, and it will check every five seconds for print jobs and exit before the next invocation by cron.

Setting up Shinobi and a Wyze G2 Camera

Joe Block — Sun, 14 Mar 2021 08:59:26 +0000

I wanted to set up a security camera outside, but I didn't want to be dependent on an outside cloud service - if my internet goes out, I don't want to lose my ability to record footage.

Wyze cameras are nice and cheap, and you can reflash them to support RTSP in addition to streaming to the Wyze cloud.

Setup

Prerequisites

A camera that supports the RTSP protocol (I'm using a Wyze G2)
A spare microSD card for reflashing the Wyze G2 camera
An x86 machine running docker. As of 2021-03-14, Shinobi only publishes an amd64 version of the shinobisystems/shinobi docker image.
A reasonable amount of disk space - the Wyze G2 I'm using generates around 330 megs per hour of stored 1080p video.

Camera Setup

I wanted a camera that supported the Real Time Streaming Protocol (RTSP) because that is an open standard which works with a wide variety of tooling, both Open Source and commercial.

I looked at a variety of camera options, and Wirecutter's Best Wifi Home Security Camera listed the Wyze G2 as runner-up. It and the first choice (Eufy 2K Indoor cam) both support RTSP, but the Wyze was in stock (and half the price at $26) so I went with it.

I did have to reflash the Wyze G2 to enable a beta firmware that supports both Wyze's cloud and RTSP. Conveniently, it can stream to both simultaneously, so I can watch the streams with the Wyze app when away from home and still record everything to my homelab cluster.

Reflash the Wyze Camera

Wyze now has a beta firmware that simultanously supports both their cloud offering and RTSP. Note that they've explicitly stated that the RTSP branch will get features later than the mainline firmware. I personally don't care, but it is something to consider if you're going to want to use bleeding edge features.

The official instructions for reflashing the G2 camera are on the Wyze Cam RTSP page and clear, so I'm not going to rehash them here. You'll need a FAT32 formatted microSD card to do the firmware reflash.

After you reflash the camera, you'll need to configure a username/password combination for the camera stream using the Wyze phone app.

Before you configure the camera, I recommend that you go into your router's configuration and assign the camera a static IP so that your DVR doesn't lose the stream connection when the camera or router are rebooted. You can also hardcode an IP address into the G2 camera's configuration, but I prefer to keep all the static IP assignments for my network in one place, the DHCP configuration on my router.

You'll end up with a rtsp url that looks like rtsp://username:password@192.168.1.CAMERAIP/live.

DVR Setup

I don't use any IOT devices that require a cloud service to function. In this case, I especially do not want to be unable to record security footage just because the internet is down, so I set up shinobi as a local DVR to record my security footage.

Start shinobi

I'm running shinobi in a docker container. As of 2021-03-14, there is only an AMD64 build of this docker image so I'm running it on the an Intel machine in my homelab cluster.

Here's a shinobi-start script:

#!/usr/bin/env bash
#
# Start shinobi
#
# Copyright 2021, Joe Block <jpb@unixorn.net>
#
# License: Apache 2.0

SHINOBI_D=${SHINOBI_D:-'/data/shinobi'}

set -o pipefail
if [[ -n "$DEBUG" ]]; then
  set -x
fi

for dvr_d in config customAutoLoad database plugins video
do
  mkdir -p "$SHINOBI_D/$dvr_d"
done

exec docker run -d -p 8080:8080 \
  --name='shinobi' \
  -v ${SHINOBI_D}/config:/config:rw \
  -v ${SHINOBI_D}/customAutoLoad:/home/Shinobi/libs/customAutoLoad:rw \
  -v ${SHINOBI_D}/database:/var/lib/mysql:rw \
  -v ${SHINOBI_D}/plugins:/plugins:rw \
  -v ${SHINOBI_D}/videos:/home/Shinobi/videos:rw \
  -v /dev/shm/Shinobi/streams:/dev/shm/streams:rw \
  -v /etc/localtime:/etc/localtime:ro \
  -v /etc/timezone:/etc/timezone:ro \
  --restart always \
  shinobisystems/shinobi

Run this with SHINOBI_D=/path/to/local/dvr/files shinobi_start and it will create any missing required directories for you and start shinobi.

Configure shinobi

First, set up a new admin account

Login at http://your.shinobi.server:8080/super with username admin@shinobi.vido and password admin
Create a new admin account
Don't forget to reset the password for the admin@shinobi.video account!

Add the camera

Login at http://your.shinobi.server:8080
Click on the + icon in the toolbar at the top of the page
Set mode to record
Change the name to something human friendly like "Mailbox Camera"
Set input type (in the connection section) to H.264 / H.265 / H.265+
Set the full URL path to the rtsp stream url you got from the camera
Optionally set Skip Ping to Yes
Set Stream Type to HLS (includes audio)
Set Record File Type to MP4
Set Video codec to copy
Set Audio Codec to Auto
Save

You can optionally set retention times for the camera data.

It took about 30-45 seconds before my camera stream was visible in shinobi.

Growing EBS Volumes in Place

Joe Block — Tue, 20 Aug 2019 13:10:27 +0000

Yesterday I had to grow a live filesystem on a server in EC2, without downtime. I do this just infrequently enough to not quite remember all the details without poking around the internet, so I’m documenting it all in one place.

Grow the volume

Log into the EC2 console and find your instance. In the description tab, look at the block devices (bottom right as of August 2019) and find the volume you need to grow and get its volume ID.
Find that volume in the EBS volumes list. Now is a good time to name it something useful like "InstanceName /data01" if you haven't already named it.
Click Modify Volume, then give it a new size. It may take a minute or two to finish growing the volume, you'll see a percentage displayed.

Resize the filesystem

Log into the instance and start a tmux or screen session to do all the work in. Getting disconnected in the middle of resizing the filesystem would be bad.
Use lsblk to confirm that the EBS block device has increased to the size you expect.
If you have partitioned your drive, do sudo growpart /dev/xyz1 0 to grow the partition.
Check /etc/fstab to see what format the filesystem is.
If you're using xfs, sudo xfs_growfs /dev/DEVICE. If you're using ext2, ext3 or ext4, do sudo resize2fs /dev/DEVICE. If you're using ext2 or ext3, seriously consider replacing this filesystem with an ext4 one during your next downtime window.
Wait. Depending on how much larger the EBS volume has become and the instance type, it can take several minutes for the filesystem to finish growing.
Confirm the new size with df

DEV Community: Joe Block

Creating a Talos cluster with a Cilium CNI on Proxmox

Talos Homelab Setup Series

Pre-requisites

Software Versions

Create a Proxmox VM control plane node

Configure the cluster control plane

Setup some environment variables

Find out what disks are on the server

Create a cluster patch file

Generate the talos configuration

Initialize the cluster

Bootstrap etcd in the cluster

Create a kubeconfig file

Confirm that the cluster came up

Install cilium

First install the CRDs

Confirm the gateway classes are present

Set up the cilium helm repo

Install cilium

Confirm cilium status

Update talosconfig

Confirm that the cluster is showing healthy

Check external connectivity to cluster services

First, test a LoadBalancer service

Create the playground namespace

Create an IP Pool and Announcement Policy

Create a deployment and service in the playground

Deploy the playground

See what IP the playground is using

Confirm Connectivity

Test Cilium's Gateway API

Create a Gateway

Update the playground-nginx-service

Create a HTTPRoute

Create the Gateway test resources

Confirm it worked

Adding a worker node

Create another VM

Add it to the cluster

Problems I ran into

curl fails to connect

curl connects, but you get a 404

You changed cilium settings but they don't appear to have any effect

Tips

2023 Hacktoberfest work

Intro

Hacktoberfest 2023

Maintenance work

New open source contributions

Fix Securifi Peanut issue with zigbee2mqtt

Adding Peanut Smart Plugs to zigbee2mqtt

AWS IAM Self Tagging EC2 Instances

Building Multi Architecture Docker Images with buildx

Use a Raspberry Pi as a print server

Prerequisites

Setup

Home Assistant Printer Power Management

Prerequisites

You have HA configured to connect to a MQTT server

Your printer is connected to a cupsd server running in a container

Your printer is plugged into an outlet controlled by HA

Printer Power Control

mosquitto helper script

cupsd Watcher

Home Assistant Setup

Test the pieces

Print server check

Automation test

Run it all automatically

Setting up Shinobi and a Wyze G2 Camera

Setup

Prerequisites

Camera Setup

Reflash the Wyze Camera

DVR Setup

Start shinobi

Configure shinobi

First, set up a new admin account

Add the camera

First, test a `LoadBalancer` service

Update the `playground-nginx-service`

`curl` fails to connect

`curl` connects, but you get a 404