Joining two Talos Kubernetes clusters with Cilium Cluster Mesh

The setup I started with

My lab ran on a Mac mini with Ubuntu 26.04 x86_64, 4 CPU, 16 GiB of RAM, and
Docker 29.4.3. I wanted to find out what Cilium Cluster Mesh actually needed
beyond two working Kubernetes clusters. I ended up with four Talos containers,
two Kubernetes clusters, several Docker network attachments, and a node address
problem that was more instructive than the successful installation.

Both clusters used Talos Linux 1.12.5, Kubernetes 1.35.2, and Cilium 1.19.6. I
managed them with talosctl 1.12.5, kubectl 1.35.1, and cilium CLI 0.19.7.

Each cluster had one control plane and one worker. I assigned 2 GiB of RAM and
1.5 CPU to each control-plane container. Each worker received 1.5 GiB of RAM and
1 CPU. Both clusters ran as Docker containers on the same Ubuntu host.

I had already bootstrapped Talos with no built-in CNI and with kube-proxy
disabled:

cluster:
  network:
    cni:
      name: none
  proxy:
    disabled: true

I installed Cilium separately with ipam.mode=kubernetes,
kubeProxyReplacement=true, VXLAN tunnel routing, and devices=eth0, as
described in a separate post.

The two clusters had this address plan:

Cluster	Cluster ID	Node subnet	Control plane	Worker	Pod CIDR	Service CIDR
mesh-a	1	10.51.0.0/24	10.51.0.2	10.51.0.3	10.241.0.0/16	10.101.0.0/16
mesh-b	2	10.52.0.0/24	10.52.0.2	10.52.0.3	10.242.0.0/16	10.102.0.0/16

At first the missing SSH access annoyed me. I am used to opening a node and
looking at interfaces, routes, processes and logs with ordinary Linux commands.
In Talos I had to work out first which talosctl command or resource shows the
state I wanted. Once I got used to it I liked it more than I expected: changes
do not dissolve into shell history, they stay as patches I can apply again.

What Cluster Mesh required from the two clusters

A working Kubernetes API in each cluster was not enough. I needed unique Cilium
cluster IDs, non-overlapping address ranges, and direct IP reachability between
the participating nodes.

I assigned cluster ID 1 to mesh-a and cluster ID 2 to mesh-b. The ID tells
Cilium which cluster owns a remote identity or endpoint. Reusing an ID would
make information from two independent clusters ambiguous.

The Pod CIDRs also had to remain distinct. A route to a pod address inside
10.241.0.0/16 must have one unambiguous destination. If both clusters
allocated pods from that range, an address alone would not tell the datapath
whether to keep a packet inside the local cluster or send it to the remote one.

I applied the same separation to the Service CIDRs, with 10.101.0.0/16 in
mesh-a and 10.102.0.0/16 in mesh-b. Service addresses are
virtual addresses. Reusing their ranges would add ambiguity when the clusters
exchange service information.

The last requirement was reachability. Cluster Mesh could exchange metadata,
but cross-cluster pod traffic would still fail if the nodes had no route to each
other. My two clusters lived on separate Docker networks, so their initial
isolation was part of the container topology rather than a Kubernetes policy.

Cilium Cluster Mesh provided cross-cluster connectivity, identity exchange,
remote service discovery, global services, and cross-cluster load balancing in
this lab. It was not an L7 service mesh. I had no sidecar proxies or L7 routing
rules, and I did not install Istio or Linkerd.

Making the nodes reachable and hitting the node IP trap

![Node address selection: the failure mode after attaching a second Docker network, and the Talos patch that pins the subnet]

I attached the containers from each cluster to the other cluster's Docker
network. After that change, the nodes could route directly between the
10.51.0.0/24 and 10.52.0.0/24 networks.

That solved one problem and introduced another.

Before the change, each Talos container had an obvious address from the
perspective of kubelet. After I added another network interface, the container
had more than one candidate node address. Kubelet could select an address from
the wrong Docker network.

A wrong node address does not always break anything visibly. The API can still
list the node while Cilium tries to reach it through the wrong network, and
local pods keep running while cross-cluster traffic fails.

I fixed the selection in mesh-a by restricting kubelet to its node subnet:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 10.51.0.0/24

I applied the corresponding restriction to mesh-b:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 10.52.0.0/24

This told kubelet which interface addresses were valid without relying on
interface discovery order. I picked machine.kubelet.nodeIP.validSubnets
deliberately. Once the containers were attached to the second Docker network,
every node had more than one plausible address. Pinning a single IP would have
been too rigid, while restricting the subnet kept the cluster's address plan
intact and let me recreate containers without editing anything. This is the
documented Talos mechanism for multihomed nodes, not a workaround I invented.

The control-plane containers had a second address-sensitive component: etcd. I
pinned its advertised subnet in mesh-a:

cluster:
  etcd:
    advertisedSubnets:
      - 10.51.0.0/24

I used 10.52.0.0/24 for the mesh-b control plane.

A wrong kubelet address can disrupt communication with one node. A wrong etcd
advertised address can affect the datastore used by the control plane. An etcd
member tells clients and peers how to reach it through its advertised address.
If it publishes an address that the intended peers cannot reach, control-plane
communication can fail at a lower layer than ordinary pod networking.

Each cluster had one control-plane node, so no redundant etcd member could
absorb an address mistake. I had to correct the advertised subnet.

The dead end was assuming that attaching more Docker networks only added
routes. It also changed the set of addresses from which kubelet and etcd could
choose. I did not track how long the diagnosis took.

I kept Cilium configured with devices=eth0 and VXLAN tunnel routing. VXLAN
added encapsulation inside a setup that already used Docker networking. I did
not test MTU behaviour inside the nested network path, so I cannot say how much
packet size headroom this setup retained.

Turning on the mesh

I published clustermesh-apiserver as a NodePort on port 32379 in both clusters.
This gave the opposite cluster a reachable endpoint without adding a separate
load balancer to the lab.

I ran one clustermesh-apiserver replica per cluster. Each cluster had one
control-plane node, so extra replicas would only consume more memory in this
topology. I did not measure the component's memory or CPU overhead.

I disabled KVStoreMesh. For this two-cluster experiment, I did not add that
extra layer to the design.

The clusters had been created independently, so they did not share a certificate
authority. I connected them with --allow-mismatching-ca. The name can sound
like a request to bypass certificate checks, but that is not what it did here.
The option added the remote cluster's CA to the trust set. TLS verification
remained enabled.

After connecting the clusters, I checked the Cluster Mesh status from both
contexts:

cilium clustermesh status --context admin@mesh-a
cilium clustermesh status --context admin@mesh-b

I did not preserve the full command logs, so the only fragment I can report is:

abbreviated output, not a saved log

2/2 clusters configured
2/2 clusters connected

I recorded the same result from both contexts.

If my goal had been to run one small application, this overhead would not be
justified. Cluster Mesh adds CIDRs to plan, routes between networks,
certificates, one more component to keep alive in clustermesh-apiserver, and a
separate class of problems to diagnose when something between the clusters
breaks. As a lab it was worth it, because I wanted to watch Cilium discover
remote backends and balance requests through a global service, not to find the
shortest path to a running service.

A service that lived in two clusters

I created the mesh-demo namespace in both clusters. In mesh-a, I ran a
deployment named mesh-a-backend. In mesh-b, I ran mesh-b-backend.

Both deployments used the same test image and command:

image: registry.k8s.io/e2e-test-images/agnhost:2.53
command:
  - /agnhost
args:
  - netexec
  - --http-port=8080

I then created a Service named global-demo in both clusters. The relevant part
of each Service manifest was:

metadata:
  name: global-demo
  namespace: mesh-demo
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"
spec:
  ports:
    - port: 80
      targetPort: 8080

The shared name and namespace represented one global service. The annotations
told Cilium to publish and share its backends across the mesh.

I created a client pod named mesh-client in mesh-a with
curlimages/curl:8.12.1. From that pod, I ran the following request 20 times:

curl http://global-demo/hostname

I read the returned hostnames from those 20 curl calls. Twelve responses came
from mesh-a-backend, and eight came from mesh-b-backend. The command reached
both the local and remote service backends through the same service name.

![Twenty requests to one global service name: twelve served locally in mesh-a, eight from mesh-b over Cluster Mesh]

The 12/8 split was not a failed attempt at a 10/10 split. I only sent 20
requests, and the global service did not promise an exactly even count over
that sample. The useful finding was that both backend groups appeared in the
responses.

I did not run the failover test where I set the local backend replica count to
zero and check whether all requests move to the remote cluster.

I also did not test service.cilium.io/affinity=local, so I have no recorded
distribution to compare with the 12/8 result.

I ran cilium connectivity test as a broader datapath check. The test namespace
needed this Pod Security Admission label because some diagnostic components
required privileged access to the datapath:

pod-security.kubernetes.io/enforce: privileged

The result was 80 of 81 tests passed, 473 of 474 actions passed, and 51 skipped.
The single failure came from the node-to-node encryption test. The ping
succeeded, but the test failed while stopping tcpdump, which exited with code
197.

I did not interpret that as a packet delivery failure. The network action
completed, but diagnostic cleanup failed. The suite still finished at 80 of 81.

I did not measure latency or the resource cost introduced by Cluster Mesh.

What belongs in this lab

The most useful thing I took from this was where the layers end. Two healthy
Kubernetes clusters do not mean Cluster Mesh can work between them. The node
addresses, the routing between the Docker networks, the Pod and Service CIDRs
and the certificates all have to be right first. Cilium only connects clusters
and serves a global service after that groundwork holds.

I would not call this production, and the reason is not the single control plane
per cluster. Both clusters sit on one physical host. If the Mac mini, Docker or
the Ubuntu network stack goes down, both clusters go with it. The setup
exercises cross-cluster networking, but it gives me no independent failure
domains, and independent failure domains are usually the reason to run more
than one cluster in the first place.

If I built this again I would write down the whole address plan first, prepare
the nodeIP.validSubnets and etcd.advertisedSubnets patches, and only then
create the nodes and attach the extra Docker networks. I would also put every
step into scripts/up.sh from the beginning. The first time through I did part
of it by hand, so I had to reconstruct the exact sequence after the lab was
already running.

My next experiment is a third cluster named mesh-c on k0s, with cluster ID 3,
Pod CIDR 10.243.0.0/16, and Service CIDR 10.103.0.0/16. I have not installed
k0s yet. Running it directly on the same Ubuntu host would touch iptables,
routing, cgroups, and eBPF state shared with the Docker clusters, so I still
need to find out whether a privileged container provides enough isolation or
whether mesh-c requires a separate VM.

The Talos patches, Cilium values and the Global Service manifests are in the repo:

wepaleen / talos-cilium-clustermesh

Lab: two Talos Linux clusters joined with Cilium Cluster Mesh. No CNI, no kube-proxy, global services across clusters.

Talos Cilium Cluster Mesh lab

This repository reproduces a two-cluster Cilium Cluster Mesh lab built with Talos Linux containers on one Docker host.

The lab uses:

• Talos Linux 1.12.5
• Kubernetes 1.35.2
• Cilium 1.19.6
• cilium CLI 0.19.7
• talosctl 1.12.5
• kubectl 1.35.1
• Docker 29.4.3

The host used for the original run was a Mac mini running Ubuntu 26.04 x86_64 with 4 CPU and 15 GiB of RAM.

Topology

Cluster	Cilium ID	Node subnet	Control plane	Worker	Pod CIDR	Service CIDR
mesh-a	1	10.51.0.0/24	10.51.0.2	10.51.0.3	10.241.0.0/16	10.101.0.0/16
mesh-b	2	10.52.0.0/24	10.52.0.2	10.52.0.3	10.242.0.0/16	10.102.0.0/16

Each cluster has one control plane and one worker. Control-plane containers use 2 GiB of RAM and 1.5 CPU. Worker containers use 1.5 GiB of RAM and 1 CPU.

Repository layout

talos/          Safe Talos configuration patches, never generated configs
cilium/         Cilium values for mesh-a and mesh-b
clustermesh/    Cluster Mesh enable and connect script
manifests/      Global Service demonstration
docs/

…

View on GitHub