Not being able to SSH into a server is something most sysadmins learn to recognise as a failure. In Talos Linux it isn't a failure — it's the design. No shell, no SSH, no package manager; not even busybox. What's left is one thing: a gRPC API and the talosctl client that speaks to it.
Let me be upfront: I don't run Talos in production. My own infrastructure still consists of classic, mutable Linux servers, and it will stay that way for the foreseeable future. But a small incident in early July pushed me to read this documentation out of need rather than curiosity. This piece is my notes from that reading: what Talos's immutability claim actually costs, how the management surface works, which traps it carries, and who it suits.
The Seven Lines That Got Me Here
On the night of 3 July 2026, this blog's CI pipeline ate itself. The setup was this: an ephemeral GitHub Actions runner running inside a container, tearing itself down after every job, brought back up by restart: always. Clean on paper. In practice, restart: always preserved the same container, which meant the .runner configuration file the previous job had left on disk was still sitting there. The runner comes back up, config.sh looks at the disk, says "already configured" and exits, the container dies, the restart policy brings it back. An infinite loop, with an alert flood on top.
The root fix landed on 6 July and was seven lines added to a single file (693b738e): rm -f .runner .credentials .credentials_rsaparams on every start.
Seven lines. But the source of the fault wasn't a seven-line omission; it was state that survived a restart with nobody owning it. I thought that container was "ephemeral"; the disk thought otherwise. And this is precisely what Talos exists to make impossible: to narrow the writable surface so far that nothing anyone assumed was transient can accidentally become permanent.
How Serious Is "No Shell"
Sidero Labs's philosophy document doesn't hedge here: no shell, no SSH, no GNU utilities, "not even a rollup tool such as busybox". The resulting SquashFS image is under 80 MB. There are no passwords in the system either; all networked communication is encrypted and key-authenticated, and the certificates are short-lived and automatically rotating.
Talos isn't derived from another distribution. It runs the Linux kernel, but everything downstream of it is written from scratch in Go. PID 1 is machined, not systemd. The components talk to each other over gRPC through a socket on the local machine, meaning even the "internal" interfaces are explicitly defined and subject to the public git record.
The filesystem layers are built to carry the same claim. The root filesystem is a read-only SquashFS mounted as a loop device into memory; on top of it come tmpfs filesystems for runtime needs; the handful of directories that must persist (like /etc/kubernetes) are overlayfs mounts backed by XFS on /var. Files that genuinely have to be writable, such as /etc/hosts and /etc/resolv.conf, are bind-mounted individually instead of opening up all of /etc. There are six partitions on disk: EFI, BIOS, BOOT, META, STATE and EPHEMERAL. The last name is not an accident; the documentation says they call the writable partition "ephemeral" precisely because they want to make sure none of us ever uses it for unique, non-replicated data.
That struck me as an honest design. You can apply immutability discipline on a mutable Linux server too — I wrote about that in detail in immutable infrastructure discipline on Linux servers — but there the discipline depends on your will to sustain it. In Talos the discipline is baked into the physics of the filesystem. The difference is the one between "I shouldn't" and "I can't".
The Management Surface: Two Ports and a CLI
Once the shell is gone, what replaces it? Two TCP ports. On control plane nodes apid listens on 50000 and trustd on 50001; on worker nodes only apid on 50000 is open. These ports are not currently configurable, and the documentation says so explicitly, linking to the tracking issue. Kubernetes's own ports are required on top of these; they're a separate account.
Authorization is certificate-based: the user's role is encoded in the organization field of the client certificate's subject. There are four predefined roles. os:admin grants access to every method, os:operator adds rebooting, shutting down and etcd backups on top of read access, os:reader grants only "safe" methods (it can list files but cannot read their contents), and os:etcd:backup allows nothing but taking an etcd snapshot. Handing out a role is as simple as talosctl config new --roles=os:reader reader; RBAC itself is switched on in the machine configuration with machine.features.rbac: true. One thing not to miss: RBAC is already enabled by default in new clusters created with talosctl v0.11 and later; it's the older clusters where it's off.
For sysadmins, the real adaptation starts here: every habit has an API equivalent, and none of them feel like the original. talosctl list instead of ls, talosctl read instead of cat. If you want to know what a node exposes at all, talosctl get resourcedefinitions hands you the machine's entire internal resource inventory.
Let me clear up a common misreading here: Talos does not take kubectl away from you. The cluster's Kubernetes layer is right where you left it, and the talosctl kubeconfig command exists precisely for that, downloading the admin kubeconfig from the node. talosctl get members is not a substitute for kubectl get nodes but a different layer: it lists the members confirmed and approved in Talos's own cluster discovery. A machine can appear as a Member on the Talos side while being NotReady on the Kubernetes side, so you learn to ask the two questions of two different tools.
A node that hasn't yet received a machine configuration is in "maintenance" mode, where you must send commands with --insecure, and not every command is available in that mode. (Don't confuse this flag with the talosctl upgrade --insecure I'll come to shortly: the --insecure on apply-config and get is for connecting to the maintenance service that has no authentication, and it is very much still there.)
One Surface: The Machine Configuration
A Talos node's state is fully defined by a single YAML manifest. No scripting, no procedural steps. There are three ways to update it: talosctl apply-config to apply from a file, talosctl edit machineconfig to open the current configuration in an editor and write it back, and talosctl patch machineconfig to pass a JSON patch.
The interesting part isn't the commands, it's the modes. In the default automatic mode Talos inspects the change and reboots if it must, applying immediately if it doesn't. --mode=no-reboot returns an error if the change touches any field that can't be updated without a reboot. --mode=staged defers the change to the next boot. But the one that stopped me was --mode=try: the change is applied immediately and automatically reverted within one minute unless another configuration update arrives.
Anyone who has cut their own connection while changing network configuration remotely knows what that sentence is worth. It's the Cisco world's reload in 5 reflex, brought down to the operating system level. Which fields don't require a reboot is documented as well: .machine.network, .machine.kubelet, .machine.sysctls, .machine.registries, .cluster and a number of others can be applied immediately. There's one exception worth noting: .machine.registries applies immediately, but containerd's CRI plugin won't pick up registry authentication settings without a reboot. Underneath the sentence "everything is declarative" sit distinctions like this one, and a fair share of the operational cost lives there.
Upgrades: Fine Design, Plus a Silent Trap
An OS upgrade is an API call too. Every Talos version has its own installer image, and talosctl upgrade --nodes <ip> --image ghcr.io/siderolabs/installer:<version> tells the node which image to move to. Upgrades use an A-B image scheme: the previous kernel and OS image stay on disk, Talos rolls back on its own if the new version fails to boot, and you can roll back manually with talosctl rollback. Because the reboot goes through kexec, the extra time is barely noticeable.
There's a rule that has held since 1.0 and is expensive to overlook: an OS upgrade does not, by default, upgrade Kubernetes. The Kubernetes side is managed separately with talosctl upgrade-k8s. The official recommendation on upgrade paths is equally clear: configuration migration is only tested between adjacent minor releases, so you should step through the latest patch release of every intermediate minor. Skipping two minors at once means being the first to try an untested migration path.
Now for the real matter. Talos v1.13 introduced a new streaming upgrade API called LifecycleService.Upgrade, and talosctl upgrade now uses it by default. The legacy --force, --insecure, --preserve and --stage flags are deprecated and will be removed in Talos 1.18. The detail that lowers the panic level: those flags only do anything when falling back to the MachineService.Upgrade endpoint for older Talos versions. So far, an ordinary API migration. But the warning box in the upgrade documentation says this: the new upgrade API does not pass .machine.install.extraKernelArgs to the installer. On a node that boots with GRUB and has .machine.install.grubUseUKICmdline set to false, the upgrade completes successfully and reports no error — yet the node boots with a kernel command line that no longer matches its machine configuration.
The biting part of this trap is the default: grubUseUKICmdline is false for machine configurations generated before v1.12. In other words, long-running GRUB clusters that never opted into anything are the affected ones. There are two ways out: either fall back to the old API with the --legacy flag (supported until 1.18, so a stopgap rather than a permanent answer), or build the arguments into the UKI when generating boot assets and set grubUseUKICmdline: true. Nodes booting with systemd-boot, and GRUB nodes with grubUseUKICmdline: true, are unaffected.
An upgrade that silently does the wrong thing is always more expensive than one that fails loudly. If there's a place to fault Talos, it's here — but at least Talos is the one raising the flag; the warning sits on its own upgrade page.
How Do You Debug Without a Shell
Let's take the most common form of it first: logs. journalctl is gone, and talosctl logs <service> takes its place, pulling a service's records over the API. What you should actually do in production is stop looking for logs on the node at all: the .machine.logging.destinations field in the machine configuration streams records to an external collector over UDP or TCP in json_lines format (currently the only supported format). Because that field is one of the no-reboot fields listed above, you can change it on a running cluster.
Now to the shell question itself. This is the first question every sysadmin looking at Talos asks, and for a long time the answer was "over the API, as far as it goes". v1.13 changed that: the talosctl debug command starts a privileged debug container from an image you supply and drops you into it. When you exit, the container is removed and the cluster carries on. The node's root filesystem shows up inside the container under /host.
You can get the image there in two ways: have the machine pull it from a registry, or push the image to the machine over the API. The second is meant for the case where you're debugging network connectivity, which is a well-considered detail: it heads off the absurdity of "to repair a node that can't reach the internet, it needs to reach the internet". The docs also note that the image architecture must match the machine's; push an amd64 image to an ARM node and the container simply won't start.
I think this command is a sign of Talos maturing. The total absence of a shell was a naive claim; a temporary, auditable shell you can open when you need one is a design that can survive production.
Security Defaults: "Enabled" and "Enforcing" Are Not the Same
Talos's security story is strong, but don't trust its defaults before reading them. Two examples.
The first is SELinux. Talos has shipped SELinux support since 1.10, and it is enabled by default in the images. Stop reading there and you'll feel reassured. The rest of the sentence: the default mode is permissive, meaning policy violations are logged rather than blocked. The reason is stated honestly: some CNI and CSI solutions and extensions aren't yet compatible with enforcing, and enforcing mode has so far only been tested with the Flannel CNI that Talos ships by default. The documentation also notes the feature is still experimental. You can see what state your node is actually in with talosctl get SecurityState; the SELINUXSTATE column gives you a plain answer such as enabled, permissive. The policy's scope is limited as well: it's designed to protect the OS from workloads (denying access to the machine configuration and preventing debuggers from attaching to system processes), but securing the Kubernetes components themselves is out of scope. SELinux and AppArmor also can't run at the same time; enabling one means disabling the other, which requires touching the kernel command line in an Image Factory schematic.
The second is the machine-wide image signature verification that arrived with v1.13. The ImageVerificationConfig document covers every image pull on the node: the ones Talos initiates for its own components and the ones Kubernetes initiates for your workloads. Rules are evaluated top-down, the first matching rule wins and the rest are ignored. Matching happens on the image reference without its tag or digest, so for ghcr.io/siderolabs/installer:v1.13.0, the reference matched is ghcr.io/siderolabs/installer.
The default here deserves attention: images that match no rule are pulled without verification. You can't write the policy and then relax in the belief that unsigned images can no longer get in; if you want a closed policy, you need a final rule carrying deny: true. This is a pattern I run into often in security features: the feature is on, the rules are written, and the door is still ajar. Sidero's documentation offers a starting policy with keyless verification rules for registry.k8s.io, ghcr.io/siderolabs and factory.talos.dev; taking that as a base and adding the registries of your own workloads is a reasonable start.
Who It Fits, and Who It Doesn't
Talos is designed to do one job: run Kubernetes. The decision framework has to be built around that sentence.
- If anything other than Kubernetes will run on the node, Talos is the wrong choice. A side service, an agent, a cron job, a database. There's room for none of it. If you have a requirement outside the rule, pick another OS instead of bending the rule.
-
If your team's reflexes are tied to SSH, don't underestimate the transition cost. The problem isn't technical, it's habitual. A team used to logging into a node at 3am and running
journalctlwill respond more slowly until it has learned thetalosctlequivalents. - If you're on bare metal, AWS or GCP, you're on the best-supported tier; Sidero counts those three as Tier 1 and tests them automatically. Platforms like Hetzner, Vultr and Oracle Cloud are Tier 3, meaning they aren't tested by the core team; testing is left to the community.
- Check the version policy. The v1.13 line supports Kubernetes 1.31 through 1.36 and runs on amd64 and arm64. The support matrix states that community support for 1.13 ends with the 1.14.0 release; the planned date is 30 August 2026 and it isn't finalised yet. As of the day this was written, the most recent patch is v1.13.9, published on 19 August 2026. Against that, 1.14.0 reached rc.2 on 25 August; if you're building from scratch it seems sensible to wait for GA, and if you're migrating existing clusters, to stay on the last 1.13 patch.
-
Rehearse the upgrade from day one. The
extraKernelArgstrap above bites in exactly the "we didn't try it on one node, we upgraded the fleet directly" scenario.
Let me put my own situation in that frame too: there is no Kubernetes on my VPS, the blog is served from a container, and the machine does several other jobs at the same time. Talos doesn't fit me, but that says something about my workload rather than about Talos.
What Stays With Me
What Talos really proposes isn't removing the shell. Removing the shell is the consequence; the proposal is moving every change onto a single surface that can be reviewed and reverted. If everything you can do to a node passes through one YAML file and one API, the question "who changed what by hand, and when" simply disappears.
My runner failure in July wasn't a Linux problem. It was a file with no owner, appearing in no manifest, surviving a restart. I wrote the seven lines that delete it and the issue closed, but the real question came from reading Talos's documentation: how many other such files are there in my systems, and which of them am I assuming are temporary? You can ask that question without migrating to Talos. You may not like the answer.
Top comments (0)