DEV Community: ude-p

Provisioning k3s clusters over Tailscale

ude-p — Sat, 17 Oct 2026 20:40:08 +0000

It has been a while since I wrote the last article about this deployment platform. The shape of it has changed since then, and I have been meaning to write down why.

The old pipeline still exists in spirit. GitHub Organizations, GitHub Actions, k3s and SOPS are still there. The part that changed first was provisioning, the one I was least happy with. If you are new to this, the previous article describes the platform this one builds on and the decisions behind it.

The previous setup used Ansible playbooks. They worked, but they were a second codebase with its own abstractions, and every new provider meant another set of assumptions about networking, users and firewall tools. Most of each playbook repeated the same work. Create a user, copy a key, install packages, configure a private network, bring up k3s. None of it was interesting, and all of it had to be right before the cluster could exist.

So I wrote a small tool called tailk3s. You give it a list of servers and provider logins, and it gives you a k3s cluster on a Tailscale network. The repository is at github.com/ude-p/tailk3s.

The cluster file

You describe the cluster in an HCL file. If you have used Terraform, the format will look familiar. Each block is one server.

node "cp1" {
  host      = "203.0.113.10"
  role      = "server"
  init_user = "ubuntu"
  init_key  = env("SSH_KEY")
}

node "w1" {
  host      = "203.0.113.20"
  role      = "agent"
  join      = "cp1"
  init_user = "ubuntu"
  init_key  = env("SSH_KEY")
}

host is where the machine can be reached. role decides what it does. A server runs the Kubernetes control plane, which holds the API and the cluster state. An agent is a worker. It runs the applications. join tells an agent which server to join.

init_user and init_key are the login the provider gave you. tailk3s uses them once, for the first connection, and removes the account when it is done.

You run one of two commands. tailk3s plan --config cluster.hcl shows what would change without touching anything. tailk3s up --config cluster.hcl performs it. You can also describe a single node with flags and skip the file entirely.

There is no state file to lose. Every step checks what the machine already looks like and skips work that is already done. Running up twice does nothing the second time.

What Tailscale does

A Kubernetes cluster needs its machines to reach each other. The control plane has an API that the workers call, the control plane servers talk to each other, and pods on different machines need a route between them. Those addresses can be public, but most clusters keep that traffic on a private network so it is not exposed to the internet. On a cloud provider, that normally means a VPC, private subnets and firewall rules, and every provider does it differently.

Tailscale builds that private network for you, across providers. It is built on WireGuard, a modern VPN protocol. Each machine joins a network called a tailnet and gets a stable private address in the 100.x range, for example 100.117.36.53. It can then reach the other machines at their addresses, whether they sit in the same data centre or on different providers. When a direct connection is possible, the machines talk directly. When it is not, Tailscale relays the traffic, so you do not have to open ports or arrange a route.

The part that matters for this tool is identity. Every machine in the tailnet has a name, and it can be given a tag such as dev-server. A policy file, the ACL, decides which names and tags may reach which addresses and which users may connect. tailk3s uses that policy for two things. It gives every node a tag, and it adds one SSH rule that lets the orchestrator and administrators log in to those tags. I never copy an SSH key to a machine, and I never write provider-specific firewall rules.

By the time Kubernetes is being installed, the network already exists. Every machine has the same kind of private address it would have had inside a single VPC, and that address does not change when the cluster moves to another provider.

How it works

The work happens in three phases. tailk3s prepares the tailnet, gets each machine onto it, and then finishes the cluster over it.

The tool runs on your machine and joins the tailnet itself, as a node tagged tag:tailk3s. You do not need Tailscale installed on your laptop. It needs one thing from you, a Tailscale API key. It uses the key to read and update the access policy and to create the keys the machines join with.

First, prepare the tailnet.

Read the cluster file and check it. Machines that start a cluster are handled before the ones that join it, because a joining machine needs a control plane to exist first.
Update the access policy. Every node will get a tag, which is a label the policy can match on. The policy needs an owner for each tag and one SSH rule that lets the tool and administrators log in to machines wearing them. If the policy already says this, the tool writes nothing.
Create a joining key for each machine that is not on the tailnet yet. The key is a single-use ticket, and it expires after ten minutes.

Then, get each machine onto the tailnet.

Connect with the provider login and prepare the machine. Install the base packages, set the hostname, create the admin user, install Tailscale, and join the tailnet with the node's tag. This is the only time the provider password or key is used.

Finally, finish the cluster over the tailnet.

Connect again through Tailscale SSH. Close the public firewall. Only ports 80 and 443 stay open to the internet, and everything else is allowed on the tailnet interface. Then install k3s, configure it, and wait for the API to answer.
If the machine joins an existing cluster, read the join token from a control plane and pass it along. The machines never need SSH access to each other.
Write a kubeconfig file on your machine so kubectl works, and an SSH config entry so you can connect by name.
Delete the provider login. The admin account has no password, and Tailscale SSH is the way in.

A diagram makes this easier to see.

                      Tailscale API
                    (policy, auth keys)
                           ^
                           |
  laptop ── tailk3s ───────┘
     │  provider SSH       │  tailscale SSH
     v                     v
   cp1 ─────── tailnet ─────── cp2
     \                         /
      \─────── w1 ────────────/

Two details are worth calling out. Node names carry a short hash of the address, so a node name can be reused across clusters without collisions. And the provider login is deleted at the end. The admin account has no password and is reachable only through Tailscale SSH. That is the point, but it also means the provider console is the recovery path if the tailnet is ever misconfigured.

High availability

HA is a file change. Add another role = "server" node that joins the first one. The first server starts with cluster-init: true, which selects embedded etcd. Every later server joins the same etcd cluster. Agents receive every control plane address in their server: list, so they are not tied to one machine being up. The API server's certificate includes the tailnet address and the hostname, which is what makes the tailnet the transport rather than an afterthought.

There is no external load balancer to configure. k3s clients use the list of servers, and the network underneath is the tailnet.

What this does not give you is machine-level reconciliation. tailk3s is a CLI, not a controller. If a server dies, provisioning a replacement is one command, but nothing does it for you. I am comfortable with that boundary. The machines are provisioned deliberately, and the workloads are reconciled continuously.

What it gets right

The obvious win is that there is no vendor lock-in. Any provider that gives you a Linux server and SSH works. Moving from one to another is a change to host values, not a rewrite of networking and firewall rules.

There are a few less obvious wins.

No private networking to build. No VPC, peering, NAT gateways or per-provider firewall rules.
No configuration management layer. No playbooks, no inventory, no state file.
Secure by default. Public ports are closed, the provider login is deleted, and there are no SSH keys to distribute or rotate.
Fast. A node is a few minutes, and a cluster is the time it takes to install k3s.
Readable. The cluster is one file, and plan is honest about what it will do.

What it does not do

It is not a general infrastructure tool. It does not create VPCs, load balancers, DNS records, volumes or provider accounts. You bring the server and an SSH login. It brings the cluster.

It assumes a Linux machine with systemd, and it has only been exercised on Ubuntu. That is a deliberate constraint, not an accident, but it is still a constraint.

It depends on Tailscale. If you do not want a tailnet, this tool is not for you. I think that is a reasonable dependency, because it replaces an entire category of provider-specific work, but it is a dependency.

Finally, it is a one-shot provisioner. It does not watch the nodes. Drift in k3s configuration is corrected when you run it again, not while you sleep. Flux handles that for the workloads. Nothing handles it for the machines.

Flux

The cluster is now disposable. What matters is the state you want it to have, and that state should live in Git. Flux is the tool that makes the cluster follow Git.

The idea is called GitOps. You keep the Kubernetes YAML files in a repository. Flux runs inside the cluster, watches that repository, and continuously compares what the cluster has with what the repository says. When the two differ, it changes the cluster to match. People call that reconciliation. You do not run a deployment command and hope it worked. You change a file, review the change, merge it, and Flux applies it.

Flux is installed with one command, called a bootstrap. It installs its own components into the cluster, commits them to the repository, and keeps them running from then on.

flux bootstrap github \
  --owner=my-org \
  --repository=infrastructure \
  --path=clusters/prod \
  --personal

The layout I use has one directory per cluster, with shared pieces in a base directory. A file in each directory lists the applications and the cluster software that belongs to that cluster.

The value is in the loop, not the tool. If someone edits a resource by hand, Flux notices and changes it back. If a component is deleted, it is recreated. The cluster stops drifting away from what is written down.

Operators

Kubernetes runs what you declare. You write a YAML file asking for three copies of a container, and the cluster keeps that true. The components that watch these declarations and act on them are called controllers. An operator is a controller built for one specific piece of software. It teaches Kubernetes how to run that software, including the parts that need care, such as failover or backups.

Every cluster gets the same foundation. These are installed through Flux before any application lands.

Tailscale operator. It lets a Kubernetes service be reached over your tailnet. A service is normally reachable only from inside the cluster. Exposing it usually means a public load balancer or a public hostname. With this operator, an internal tool can get a tailnet address instead.
Longhorn. Persistent storage for pods. A pod that writes to a disk on one node needs its data to follow it when it moves. Longhorn keeps copies of each volume on several nodes, so a pod can be rescheduled without losing its data. It also handles snapshots and backups. The previous article ended by saying Longhorn was a maybe. It is now the default.
CloudNativePG. Postgres, run by an operator. You declare a Postgres cluster, and it creates the instances, handles replication and failover, and takes backups. Point-in-time recovery means you can restore the database to a specific moment, not just to the last nightly backup.
kube-prometheus-stack. Monitoring. It collects metrics from the nodes and the cluster, stores them over time, evaluates alert rules, and provides dashboards.
cert-manager. TLS certificates. It requests certificates from Let's Encrypt, installs them, and renews them before they expire, so HTTPS becomes a resource you declare rather than a manual task.

Each one solves a cluster-level concern. None of them belongs in an application repository, and all of them are easier to reason about when they are part of the same Git history as the cluster.

appd

Flux deploys what is in Git. Someone still has to write it. So I wrote appd to do that part. It is at github.com/ude-p/appdeploy.

A normal web application needs several Kubernetes objects. A Deployment runs the container and keeps the requested number of copies alive. A Service gives those copies a stable address inside the cluster. ConfigMaps and Secrets hold configuration. An Ingress or Route publishes it on a hostname. Writing all of that for every service is repetitive, and Helm charts bring their own template language and values files.

appd lets you describe the application once in HCL and generates those objects for you. The common blocks are app, stateful, job, cron, route, config, secret, volume and manifest.

config "api" {
  env { source = ".env" prefix = "CONFIG_" }
}

secret "api" {
  env { source = ".env" prefix = "SECRET_" }
}

app "api" {
  image    = "ghcr.io/example/api:1.4.0"
  replicas = 2

  env {
    config = config.api
    secret = secret.api
  }

  port {
    service = 80
  }

  health {
    path = "/health"
    port = 3000
  }
}

route "api" {
  to   = app.api
  host = "api.example.com"
  path "/" {}
}

appd run -o prints the YAML instead of applying it, which is the path that works with Flux. appd run applies it directly to the current cluster. appd explain documents the schema, and appd init writes a starter file.

The motivation was the same as before. I was tired of copying the same Deployment, Service, ConfigMap, Secret and Ingress between repositories, and I was tired of Helm values files that exist only to change an image tag. HCL is a better fit for the shape of the problem. It has blocks, it has expressions, and it does not need a templating function to concatenate two strings.

I am also reducing the amount of configuration a user has to write, and rewriting the deployment path so that appd renders and Flux applies. Deploying directly from a CLI is convenient, but it bypasses the property that made Flux worth installing in the first place.

The intended split is simple. appd describes an application, and Flux owns the cluster.

What I think comes next

Provisioning is becoming boring, and that is a good thing. A cluster that takes minutes to create and a Git repository that reconciles it means the interesting work moves up a layer. How fast you can get a correct environment, and how quickly you can understand what broke.

On the second point, I want to build an operator that watches Kubernetes resources and explains them. Not another alerting rule. The system I have in mind correlates events, logs, metrics and ownership, and answers the question you actually ask during an incident. What changed, what depends on it, and what is the smallest safe fix. The model can read. It can propose a patch. It can run a server-side dry-run. It should not silently apply changes to production. The safeguards are the design. Read-only RBAC by default, scoped escalation, an audit trail, a human approval for writes, and an explicit ban on automatic remediation of stateful workloads.

The other area I keep coming back to is GPUs. Kubernetes is a reasonable scheduler for GPU workloads, but it does not hide the hard parts. Device plugins, MIG partitioning, topology, NVLink, storage throughput, and the cost of leaving a card idle. I am learning this area now. I do not have deep experience in it yet, but I am looking for an opportunity where GPU infrastructure is the main problem, in an organization that treats it as a first-class discipline rather than a side project.

For now, the pipeline is smaller and the clusters are faster, and that is enough for one post.

Building a Deployment Platform on Self-Managed Infrastructure with k3s

ude-p — Wed, 24 Jun 2026 19:49:17 +0000

At the beginning of this year, our production workloads were spread across AWS, Contabo and GoDaddy.

Most of the applications had predictable traffic patterns and fairly stable resource requirements. As more products were added, infrastructure costs became harder to ignore, particularly on AWS. The deployment workflows, infrastructure layouts and operational processes supporting those products had also become increasingly inconsistent.

Part of my responsibility became consolidating that environment, reducing unnecessary infrastructure costs and introducing a more consistent deployment workflow.

Over the last few months, that work resulted in a deployment platform built around GitHub Organizations, GitHub Actions, Ansible, k3s, SOPS and an internal application deployment operator written in Go.

This article covers the current version of that platform, the decisions behind it, and some of the problems encountered while building it.

Repository Management

Repositories were moved into a GitHub Organization and branch protection became consistent across projects.

Rulesets were configured to prevent direct pushes to main, require pull request reviews, enforce successful CI checks before merging and restrict who could bypass those requirements.

Since deployments would eventually be tied to merges into main, repository rules became part of the deployment platform itself. A failed CI check blocks a deployment. A missing review blocks a deployment. Direct pushes to main are no longer possible.

The result is that every change reaching production passes through the same validation path regardless of which repository it originated from.

Continuous Integration

The next step was standardizing CI across repositories using GitHub Actions. Most of our services are written in Go, so the workflow was fairly similar:

Run formatting checks
Run linters
Execute tests
Build the application
Build and publish container images

Once CI became consistent across repositories, deployments could assume the same validation process had already taken place regardless of which application was being deployed.

Infrastructure Architecture

Most applications had predictable resource consumption and did not require the elasticity that AWS is particularly good at providing.

A comparison between AWS infrastructure costs and equivalent VPS resources on Contabo made the migration decision fairly straightforward, and several workloads were moved onto VPS infrastructure.

The resulting architecture was built around Contabo's private networking feature. Each server was connected to a private network and cluster communication was configured to use private IP addresses. Kubernetes control plane traffic, pod networking, database connections and application-to-application communication remained on the private network, while public access was limited to services that actually needed internet exposure.

The architecture ended up looking something like this:

Internet
    |
Traefik Ingress
    |
k3s Cluster
    |
+---------------+---------------+---------------+
|               |               |               |
Control Plane   Worker 1      Worker 2      Worker N
     \             |             |             /
      \____________Private Network____________/

Using private networking meant node-to-node communication never needed to traverse the public internet, and new nodes could join the cluster using private addresses rather than public endpoints.

Infrastructure Provisioning

As the number of servers increased, infrastructure provisioning became repetitive.

Every machine required:

User creation
SSH configuration
Hostname configuration
Firewall configuration
Package installation
Cluster bootstrap steps

The first few servers were configured manually. Ansible appeared shortly afterwards.

Most of the provisioning process eventually became playbooks covering server configuration, SSH setup and k3s installation. The value of this became obvious when one of the worker nodes failed.

Replacing the node involved provisioning a replacement server, updating inventory and rerunning the playbooks rather than manually rebuilding configuration.

Deploying k3s

I evaluated kubeadm briefly before choosing k3s.

For the size of infrastructure I was managing, k3s solved most of the problems I cared about without introducing additional operational overhead.

The control plane, service discovery, ingress controller and storage provisioner were already packaged together.

I wasn't particularly interested in assembling Kubernetes components individually if a distribution already existed that solved the same problem.

Within a few hours the cluster was operational and workloads started moving over.

For a small engineering team operating a handful of products, the defaults provided by k3s have been difficult to argue against.

Deployment Automation

Around this period I started getting tired of maintaining Kubernetes manifests.

Most applications required the same collection of resources:

Deployments
Services
Ingresses
ConfigMaps
Secrets

The differences between applications were usually small, but they still resulted in maintaining large amounts of repetitive YAML.

I initially used Helm, but over time I found myself spending more effort maintaining templates and manifests than I wanted to.

Eventually I moved deployment definitions into Go using the Kubernetes client libraries and controller-runtime.

Working directly with the Kubernetes API provided a much deeper understanding of Kubernetes resources than maintaining YAML manifests. Deployments, Services, ConfigMaps and Secrets became resources constructed and managed directly in code.

Writing deployment definitions directly in Go became repetitive over time, so I started building an internal application deployment operator that abstracts much of the Kubernetes resource configuration behind a simpler deployment definition.

The operator is still evolving, but it has become the primary way applications are deployed into the cluster.

I'll cover its design and implementation in a future article.

Secret Management

Keeping environment variables synchronized across applications and environments became increasingly difficult as the platform grew.

I looked at Vault and Infisical, but both introduced another service to operate. For the size of the team and operational requirements, SOPS with Age encryption was a better fit.

Environment files are encrypted with SOPS and committed to Git. During deployment, the files are decrypted and converted into Kubernetes ConfigMaps and Secrets.

The Age private key is stored outside the repository and is only available to trusted deployment environments responsible for decrypting configuration during deployments.

The deployment workflow looks roughly like this:

.env
  |
SOPS + Age Encrypt
  |
Git Repository
  |
Deployment Environment
  |
Age Private Key
  |
Decrypt
  |
Kubernetes Secrets / ConfigMaps

Kubernetes RBAC

Giving GitHub access to deploy applications introduced another problem: authentication to the cluster.

Kubernetes RBAC provides several ways to solve this, typically through service accounts and scoped permissions.

The implementation worked, but managing service account tokens quickly became an operational concern.

At the moment I use a combination of restricted Kubernetes identities and trusted administrative machines for deployment operations.

I am currently evaluating ArgoCD as a GitOps layer on top of the deployment platform, allowing the cluster to reconcile changes from Git rather than accepting deployments pushed directly from GitHub.

Stateful Workloads

Stateful workloads required a different approach from application workloads.

Running databases as StatefulSets works, but using local-path storage introduces an important limitation, the persistent volume is tied to the node where it was created. If that node goes down, Kubernetes can reschedule the pod, but the data does not automatically move with it.

For now, databases still run as StatefulSets on local-path storage, with backups and restore procedures treated as the primary recovery path. This keeps the architecture simple, but it also means node failure recovery depends on either recovering the original node or restoring the database from backup.

Most small teams eventually move toward dedicated database infrastructure, managed database services or distributed storage solutions such as Longhorn as their operational requirements grow.

Current Architecture

Today the platform runs on a k3s cluster hosted on Contabo infrastructure and connected through private networking.

GitHub Actions handles CI, Ansible provisions infrastructure, SOPS manages secrets and applications are deployed through an internal application deployment operator built on top of the Kubernetes API.

Profiling gorilla/websocket fan-out bottlenecks in Go

ude-p — Tue, 19 May 2026 21:20:45 +0000

I have been working on a realtime multiplayer server in Go.

The transport layer is WebSocket using gorilla/websocket, and most outbound payloads are protobuf messages generated with protobuf-go-lite.

The core server loop is simple enough:

accept client input
advance simulation
build world snapshot
broadcast snapshot to connected clients

Each simulation instance runs at 60 ticks per second, so every tick has roughly 16ms available for input processing, simulation, synchronization, and outbound networking.

At small connection counts, everything felt fine. Once I started testing hundreds of clients connected to the same simulation instance, the problems became obvious. Joining became slower, movement updates started lagging behind, snapshots backed up, and tick duration became unstable.

At that point the useful question stopped being:

why is the server slow?

and became:

what part of the realtime path gets worse as connection count grows?

The answer was mostly fan-out.

The server shape

Each WebSocket connection owns a session object.

The session manages:

the websocket connection
outbound queues
heartbeat state
write synchronization
inbound rate limiting

All websocket writes are intentionally serialized.

gorilla/websocket allows:

one concurrent reader
one concurrent writer

In practice, it is still easy to accidentally write from multiple goroutines:

the normal write loop
heartbeat responses
disconnect handlers
internal events

So every socket write goes through one locked path:

func (s *UserSession) write(data []byte) error {
    s.writeMu.Lock()
    defer s.writeMu.Unlock()

    if err := s.Conn.SetWriteDeadline(time.Now().Add(wsWriteTimeout)); err != nil {
        return err
    }

    return s.Conn.WriteMessage(websocket.BinaryMessage, data)
}

That lock was not the main bottleneck, but it matters for correctness. I do not want multiple goroutines calling WriteMessage on the same connection.

The simulation state is owned and mutated by a single goroutine.

Every simulation instance has one command channel handling:

joins
leaves
player input
disconnects
internal events

Each tick roughly looks like this:

drain commands
advance simulation
build synchronization state
broadcast updates

The last step became the expensive one.

The shape that broke first

The simulation work itself was not the first thing to fail.

The first bad shape was the broadcast path.

Each simulation tick builds a snapshot containing replicated world state:

updatedObjects := scratch.updatedObjects[:0]

for objectID, object := range simulation.objects {
    if !object.shouldReplicate {
        continue
    }

    updatedObjects = append(
        updatedObjects,
        syncSnapshotObject(objectID, object),
    )
}

Only objects marked for network replication are included in the snapshot.

That snapshot then gets sent to every connected client subscribed to the simulation instance.

The important detail here is that the snapshot payload is usually identical for every client in that room.

The original broadcast path effectively looked like this:

for _, recipient := range recipients {
    snapshot := buildSnapshot()
    data, _ := snapshot.MarshalVT()

    recipient.Send(data)
}

At small scales, this feels harmless.

At larger scales, the shape becomes expensive very quickly.

With 500 connected clients running at 60 ticks per second:

500 recipients x 60 snapshots/sec
= 30,000 websocket snapshot writes/sec

But the bigger problem was not the socket writes themselves.

The expensive part was repeatedly serializing the same snapshot payload for every connected client.

That means every tick was doing:

protobuf marshaling per recipient
allocations per recipient
buffer growth per recipient

even though the payload itself was identical.

The first profiling lesson was simple:

if the payload is identical for every recipient, serializing it per client is wasted work

Profiling setup

The server uses Pyroscope for continuous profiling:

runtime.SetMutexProfileFraction(5)
runtime.SetBlockProfileRate(5)

pyroscope.Start(pyroscope.Config{
    ApplicationName: "server",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
    },
})

Tick duration is also exported through OpenTelemetry:

tickDuration.Record(
    context.Background(),
    time.Since(tickStartedAt).Seconds(),
)

The metrics I cared about for this issue were:

active websocket sessions
room tick p50/p95/p99
protobuf marshal cost
allocation pressure
websocket backlog growth
goroutine buildup
socket write duration

The profiling run made the problem obvious.

Reliable vs unreliable messages

One important change was separating outbound traffic by semantics.

Not every websocket message deserves the same delivery behavior.

Snapshots are latest-state messages. If the client misses one snapshot, the correct behavior is usually to apply a newer one, not replay old movement history.

Some messages are different:

object created
object deleted
session ended
important gameplay events

Those are ordered state transitions and should not be dropped.

The session ended up with two outbound paths:

Reliable(frame OutboundFrame)
Unreliable(frame OutboundFrame)

Both paths operate on an OutboundFrame.

An OutboundFrame is the object the server uses to represent a websocket payload that is ready to be written. In this case, it carries the marshaled protobuf bytes and the release logic needed when those bytes come from a pool.

Reliable messages use a bounded queue:

Outbound chan OutboundFrame

Snapshots use a latest-only slot:

LatestUnreliable OutboundFrame
UnreliableNotify chan struct{}

The unreliable path intentionally avoids replacing a snapshot that is already waiting to be written:

func (s *UserSession) Unreliable(frame OutboundFrame) {
    s.outboundMu.Lock()

    if s.LatestUnreliable.Data != nil {
        s.outboundMu.Unlock()
        frame.Release()
        return
    }

    s.LatestUnreliable = frame
    s.outboundMu.Unlock()

    select {
    case s.UnreliableNotify <- struct{}{}:
    default:
    }
}

This part is easy to get wrong.

The simulation loop may call Unreliable 60 times per second. If every new snapshot immediately replaced the pending one, a busy session could keep overwriting the pending frame before the writer loop gets a chance to consume it.

So the rule is simple:

if no unreliable snapshot is pending, publish one
if one is already pending, drop the newer one and let the writer catch up

That keeps each session bounded to at most one pending unreliable snapshot.

Reliable messages queue because they represent state transitions. Unreliable snapshots do not queue because they represent latest world state.

A slow client may miss movement snapshots, but it should not force the server to retain stale movement history.

Marshaling once per broadcast

The fix was to move protobuf marshaling out of the recipient loop.

The current broadcast path marshals once:

snapshotFrame := MarshalOutboundFrame(
    snapshotEnvelope,
    snapshotRecipients,
)

Then shares the same frame across sessions:

for _, recipient := range recipients {
    recipient.Unreliable(snapshotFrame)
}

That changes the scaling shape from:

marshal protobuf once per recipient
to:
marshal protobuf once per broadcast

The network work is still multiplied by recipient count.

The serialization work is not.

Sharing broadcast frames safely

Once protobuf marshaling moved out of the recipient loop, the server needed a safe way to share the same websocket payload across many sessions.

The broadcast path now produces one shared OutboundFrame:

type OutboundFrame struct {
    Data    []byte
    refs    *atomic.Int32
    release func()
}

The frame contains:

the marshaled websocket payload
a reference counter
a cleanup callback for returning pooled buffers

The important detail is that the byte slice comes from a pool.

The server marshals protobuf data into a reusable buffer:

buf := outboundFrameBufferPool.Get().(*writeBuffer)

if cap(buf.data) < size {
    buf.data = make([]byte, size)
}

data := buf.data[:size]

_, err := envelope.MarshalToVT(data)

That same byte slice then gets shared across every recipient:

for _, recipient := range recipients {
    recipient.Unreliable(snapshotFrame)
}

At that point ownership matters.

The pooled buffer cannot go back into the pool until every session has either written or dropped the frame.

So the frame carries a reference count:

func (f OutboundFrame) Release() {
    if f.refs == nil {
        return
    }

    if f.refs.Add(-1) == 0 && f.release != nil {
        f.release()
    }
}

Each session releases the frame after writing or dropping it:

err := s.write(frame.Data)
frame.Release()

Without ownership tracking, pooled buffers become dangerous very quickly. One session can still be writing bytes while another goroutine has already returned the buffer to the pool for reuse.

The optimized broadcast path eventually became:

build snapshot once per tick
marshal protobuf once per broadcast
write into pooled buffer
share frame across recipients
release pooled buffer after all recipients finish
drop stale unreliable snapshots

None of these changes remove the cost of socket writes. Every connected client still needs its own websocket write.

What they remove is the avoidable work around the write:

repeated protobuf marshaling
repeated buffer allocation
stale snapshot queue buildup
unnecessary garbage collector pressure

Profiling result

This was the test shape:

simulation tick rate: 60Hz
connected clients: ~500
transport: gorilla/websocket
payload format: protobuf
snapshot delivery: unreliable latest-only

The profiling result looked like this:

Metric	Unoptimized	Optimized
Snapshot builds per tick	~500	1
Snapshot marshals per tick	~500	1
Snapshot marshals/sec	~30,000	~60
Pooled frame buffers	no	yes
Snapshot delivery	queued per recipient	latest-only
Room tick p99	~243ms	~18–22ms
Allocation pressure	very high	much lower
WebSocket backlog growth	severe during bursts	reduced
CPU time in protobuf marshal path	dominant hotspot	mostly removed from broadcast multiplier

The important part is not just the p99 number.

The important part is the shape change.

The optimized version still performs one websocket write per connected client, but it no longer rebuilds and serializes identical snapshot payloads per recipient.
That moved protobuf encoding and buffer allocation out of the recipient loop.

Takeaways

The issue was not that gorilla/websocket is slow.

The issue was that fan-out multiplies small costs very aggressively.

At one client:

protobuf marshaling is noise
allocations are noise
queue pressure is noise

At hundreds of clients and 60 ticks per second, those costs become visible very quickly in:

CPU usage
allocations
websocket backlog growth
tick latency

The main changes were:

separate reliable events from snapshots
make snapshots latest-only
marshal broadcast payloads once
share pooled frames across sessions
use ref-counted ownership for shared buffers

There is still a larger scaling problem left.

Right now every replicated object is still sent to every subscribed client. That means the network shape is still:

objects x recipients

The next step is proper interest management:

visibility filtering
area-of-interest replication
per-client relevance filtering

But I would not start there.

First:

profile the current system
remove multiplied work
fix ownership problems
stabilize the hot path

Then decide whether the architecture actually needs a larger cut.