DEV Community

Cover image for GenieACS on k3s for 30,000 Devices: What Happens at 95% CPU
Shuvo Halder
Shuvo Halder

Posted on Edited on

GenieACS on k3s for 30,000 Devices: What Happens at 95% CPU

I run GenieACS on k3s for an ISP network with almost 30,000 online devices.

My setup is simple on paper:

  • 3 physical servers
  • Each one: Intel Xeon 88-core CPU, 256 GB RAM
  • k3s: 1 control-plane node, 2 worker nodes
  • GenieACS handling TR-069 requests from the CPEs

With that much CPU and RAM, it is easy to think we have headroom for everything. We do not. When CPU load goes above 95%, the failure mode is not "a little slow". It is packet drops — and then the devices make it worse by retrying.

This is what I have learned from running it in production.

GenieACS on k3s for 30,000 devices — CPU at 95% causes packet drops

I run GenieACS — the open-source TR-069 auto-configuration server — on k3s, the lightweight Kubernetes distribution. That combination works well for an ISP, but only if the receive path and the CPU headroom are planned for bursts, not for the average load.

30k devices do not send traffic evenly

A CPE does not sit idle. It sends Inform messages, it checks in after a reboot, it asks for config, it reports a fault. Most of the time that traffic is spread out.

The bad moments are the bunched-up ones:

  • A power cut in an area, then thousands of ONTs come back at the same time
  • A firmware or config push touches a large group at once
  • A worker node restarts, and all its GenieACS pods move and reconnect

One Inform is cheap. Ten thousand Informs in the same minute is not. That burst is what pushes CPU to 95% and keeps it there.

Why 95% CPU means packet drops, not just delay

People think high CPU means requests queue up and finish late. That is only half the story.

On Linux, packets arrive at the NIC first, then the kernel has to drain the ring buffer and hand them to the application. That work runs as softirq. If the CPU is already at 95% running GenieACS, MongoDB, and the rest of the stack, there is almost no CPU left to drain the queue in time.

So this happens:

  1. The NIC ring buffer fills up
  2. The kernel cannot keep up with softirq processing
  3. New packets get dropped before GenieACS ever sees them
  4. The CPE does not get a response, times out, and tries again
  5. Retries add even more load to an already saturated CPU

That last step is the trap. A drop does not remove load. It creates a retry, which is more load.

You can see it in the symptoms: CWMP sessions timing out, devices flapping between online and offline, Inform retries climbing, while the dashboard still says "the app is up". The app is up. The network path to it is falling over — that is what packet drops look like from the device side.

What I check first when CPU goes high

I do not restart anything first. A restart causes a reconnect storm, and that is exactly what I am trying to avoid.

I check, in this order:

kubectl top nodes
kubectl top pods -n genieacs --sort-by=cpu
kubectl get pods -n genieacs -o wide
Enter fullscreen mode Exit fullscreen mode

That tells me if it is one pod, one node, or the whole cluster.

Then I check if packets are actually being dropped, not just delayed:

# on the worker node
ethtool -S eth0 | grep -i drop
nstat -az | grep -i drop
cat /proc/net/softnet_stat
Enter fullscreen mode Exit fullscreen mode

If rx_dropped, rx_missed_errors, or the drop counters in softnet_stat are climbing fast, I know it is a receive-path problem, not just a slow app.

I also check MongoDB. GenieACS leans on MongoDB hard. In my experience, a slow database looks like a GenieACS CPU problem from the outside, because CWMP workers pile up waiting for the database. So I check MongoDB CPU, slow queries, and connection count before I blame the app alone.

What actually helps in this setup

These are the things that made a real difference for me, running Kubernetes with k3s on bare metal. Not theory — things I would do again.

1. Keep the control-plane node clean

My control-plane node runs etcd, the API server, and the k3s control components. I do not run GenieACS workloads on it. I taint it and keep CWMP pods on the two workers only.

If a workload eats the control-plane CPU, kubectl gets slow, scheduling gets slow, and I lose visibility exactly when I need it most.

2. Split GenieACS, do not run it as one big pod

GenieACS is not one service. CWMP, NBI, FS, and UI do different work and have different traffic patterns.

I keep CWMP separate and give it the most CPU. NBI and UI can live elsewhere. If they share a pod and share a CPU limit, one heavy UI query can starve CWMP.

Use pod anti-affinity so CWMP replicas land on different workers, and do not put all of them on the same node.

3. Set CPU requests and limits carefully

This one hurts if you get it wrong.

Too low a limit, and Kubernetes throttles the CWMP pod at the worst moment — it looks like high CPU, but it is actually the limit.

Too high a request, and the scheduler cannot place the pod anywhere else.

I set a realistic request based on kubectl top during a busy hour, not during idle time, and I leave headroom on the node. For CWMP I prefer a Guaranteed QoS class where I can afford it, because a throttled CWMP pod drops more than a slow one.

4. Tune the receive path on the host, not in the pod

You cannot fix a full NIC ring buffer from inside a container. This is host work.

On the worker nodes I check and raise the ring buffer where the NIC supports it:

ethtool -g eth0
ethtool -G eth0 rx 4096
Enter fullscreen mode Exit fullscreen mode

And I tune the backlog so a short burst does not drop straight away:

sysctl -w net.core.netdev_max_backlog=10000
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
Enter fullscreen mode Exit fullscreen mode

This does not add CPU. It buys a little time for the kernel to drain the queue. It helps with bursts. It does not save you if CPU stays at 100% for a long time. That part still needs scaling or less load.

5. Stagger the devices

If I can control it, I do not let all 30k devices Inform at the same interval at the same second.

A longer, staggered Inform interval, plus a planned rollout for config and firmware changes instead of a single big push, flattens the peak. The total work is the same. The peak is lower. Peak is what kills you at 95%.

6. Scale CWMP horizontally before you scale the hardware

With 88 cores per node, adding another server is not my first move. Adding another CWMP replica, spread across both workers, is.

CWMP requests are short-lived HTTP requests. That makes them easier to scale out than a single big process. A HorizontalPodAutoscaler on CPU can help here, but I set it to scale early — around 70% — not at 95%. By 95%, it is already too late.

What I would not do

  • I would not restart a worker node during peak load to "clear" CPU. The pods come back, the devices reconnect, and the storm gets bigger.
  • I would not raise the CPU limit to unlimited and hope. That just lets one pod eat the node and starve everything else, including the networking path.
  • I would not blame GenieACS alone before checking MongoDB, the NIC counters, and which node is actually hot.

The short version

88 cores and 256 GB RAM per server sound like a lot. And they are. But 30,000 devices can still saturate a k3s worker if all their retries land at once.

Above 95% CPU, my problem is not that GenieACS is slow. It is that packets never reach it, devices retry, and the retries push CPU even higher.

Keep the control-plane clean, split CWMP out, set honest CPU requests and limits, tune the NIC receive path on the host, stagger the devices, and scale CWMP early.

FAQ: GenieACS on k3s at 30k devices

Does high CPU always cause packet drops in GenieACS on k3s?
Not always, but above 95% CPU the risk is high. The kernel has little CPU left for softirq work, the NIC ring buffer fills up, and packets drop before GenieACS sees them.

Is k3s enough for 30,000 TR-069 devices?
Yes, in my setup it is — 1 control-plane node and 2 workers with 88 cores and 256 GB RAM each. The key is splitting CWMP out, keeping the control-plane clean, and scaling CWMP early, around 70% CPU.

What should I check first: GenieACS, MongoDB, or the network?
Check all three, in this order: kubectl top for CPU, ethtool -S and softnet_stat for packet drops, then MongoDB CPU and slow queries. A slow database often looks like a GenieACS CPU problem.

Have you run TR-069 or GenieACS at this scale? What broke first for you — CPU, MongoDB, or the network?

More from me: my DEV profile — and my earlier automation post, Automate Apache/Nginx Virtual Server host in Seconds.


— Shuvo Halder
DevOps & System Engineer | Kubernetes, Docker, OpenStack & Proxmox

Top comments (0)