Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts

By Jorrick Sleijster · Senior Data Platform Engineer, Adyen

I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking.
In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes.

We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck.

Our setup: why we're different

At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen:

Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute.

Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours.

Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and transformations with relatively few network connections. Others are extremely network-intensive, like Trino pods querying thousands of small files on HDFS, each requiring a new connection. This creates a large number of short-lived connections that stress the connection tracking table.

Furthermore, our machines are more powerful than most Kubernetes cluster machines within Adyen:

• Machines with 64 physical cores and 512GB of RAM
• Machines with 128 physical cores and 2TB of RAM

At the time, our environment consisted of Kubernetes 1.31.7 and Cilium 1.16.5. We provide these specific versions to help interested readers correlate our findings with the relevant codebases.

To support our unique workloads, we progressively tuned Cilium by increasing DNS proxy timeouts, expanding the connection tracking table capacity, raising API rate limits, and streamlining "security labels". This configuration allowed us to overcome challenge after challenge, except for one:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0fecf4844d3f8f2df218f09d91f9698bb424e2166551952f46fd7638f4757cf2": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded

Kubernetes would create a pod, spin up the sandbox, and then attempt to set up the networking with Cilium. This operation would time out. From that point onwards, it also timed out any other pod attempting to spawn on the same node. Furthermore, we noticed that on those affected nodes, the API duration for DELETE /v1/endpoint and PUT /v1/endpoint, the routes responsible for adding and deleting a Cilium endpoint, also increased. You can see this clearly in the image below where the endpoint call latency starts to increase linearly over time from 4:20 pm onwards, indicating that endpoint creation and deletion never finish.

It is important to note that these clusters are only used for analytical processing and big data workloads. The issue was identified, analysed, and fully resolved before any SLO was breached. Furthermore, as our big data environments are isolated from our core transactional flows, this issue never impacted our real-time payment processing pipeline or merchant transactions.

The symptom: something's wrong, but what?

First, we found no related timeouts in the Cilium Agent logs. We enabled debug logging, hoping to find hidden errors. Nothing. The logs gave us the big picture but didn't reveal the bottleneck.

We did learn some valuable things from debugging the broken nodes:

• Listing all endpoints on a broken node would time out: cilium endpoint list
• Listing the logs of an endpoint worked fine and showed the endpoint was stuck in the regenerating phase: cilium endpoint log <ENDPOINT_ID>
• Requesting the endpoint state with kubectl get ciliumendpoint showed it was still regenerating, while the Kubernetes manifest of the endpoint showed a ready status.
• Checking the filesystem at /var/run/cilium/state showed endpoint folders with a _next suffix, indicating they didn't complete successfully.

This validated our suspicion that the issue was in the Cilium agent, not at the kubelet or CNI plugin level. But we still didn't know why.

Under the hood: how Cilium sets up networking for a pod

Before we dive into the debugging journey, it's helpful to understand what happens when a pod spawns and Cilium sets up its networking.

The process starts when Kubernetes assigns a pod to a specific node. The kubelet initiates the pod setup and calls the configured CNI plugin (Cilium, in our case). The Cilium CNI plugin performs several operations:

1. Allocates an IP using IPAM (IP Address Management)
2. Creates a link device (in our setup, a veth pair. One end in the host namespace, one in the pod)
3. Configures the pod network by setting the IP address, configuring routes and setting sysctl parameters
4. Creates a Cilium endpoint via the Cilium agent API
5. Retrieves or allocates a security identity using pod labels
6. Calculates network policy for this endpoint
7. Generates, compiles, and injects eBPF code into the kernel
8. Returns success to the kubelet

The key thing to understand is that when creating a Pod, and thus an endpoint, the CNI plugin calls the Cilium agent, which runs as a DaemonSet on each node. The agent does the heavy lifting: managing eBPF maps, handling connection tracking, applying network policies, and more.

Here's a simplified view of the flow:

A vital part of this process occurs during endpoint creation: Cilium triggers the connection tracking table's garbage collection to verify that the pod's IP is free of residual connections from its previous run. The scrubIPsInConntrackTable function executes this operation, scanning the entire conntrack table to identify and delete relevant entries.

The consequences of failing garbage collection are significant. Stale entries accumulate without limit, and while our table can accommodate up to 16 million entries, the real bottleneck is the mandatory scan that every new pod requires. This cleaning step is essential to mitigate IP address reuse conflicts, ensuring that a fresh pod doesn't inherit any open connections from a prior one.

This is where our story really begins.

Note: Arthur Chiao's excellent deep dive into Cilium's CNI implementation provided much of our understanding of the CNI flow. While Arthur Chiao wrote it a few years ago, the core concepts remain relevant, and it is an invaluable resource for anyone wanting to understand how Cilium works under the hood.

Down the rabbit hole: tracing the root cause

What is the agent actually doing?

We needed to see what the Cilium agent was doing when it hung. Enter pprof, Go's built-in profiler. We enabled pprof in Cilium's configuration and captured traces from a broken node right after spawning 50 pods.

The CPU and memory profiles didn't reveal much at first. But when we opened the execution traces, the timeline view showed exactly what each goroutine was doing, and everything became clear.

We saw long-running goroutines spending their entire time in syscalls. Zooming in closer revealed the pattern:

The agent was making BPF syscalls in a tight loop: nextKey(), lookup(), nextKey(), lookup(), over and over again. The agent uses these syscalls to iterate through an eBPF map:

1. nextKey(currentKey) - Gets the next key in the BPF map after currentKey
2. lookup(key) - Retrieves the value associated with key

Let's do some napkin math. We selected a 35-millisecond fragment and counted 14,916 syscall occurrences. That's 426,171 syscalls per second. Since getting each element from an eBPF map takes two syscalls (next + get), we were iterating through roughly 213,000 entries per second.

As Cilium is a user-space process, accessing or manipulating the map forces a context switch on every syscall. A context switch is the process of the CPU temporarily halting the user-space process (Cilium) to execute code in the kernel (to handle the BPF syscall) and then resuming the user-space process. This involves saving and restoring the entire state of the CPU registers and memory space, which adds significant overhead and is a major source of latency.

Here's the critical insight: this is a sequential operation. You can't parallelize it because you need the current key to get the next key. Even on our powerful server CPUs, this was the maximum speed we could achieve. And there was one more important detail in the traces: this was happening in the scrubIPsInConntrackTable function, the one that cleans the connection tracking table when creating an endpoint.

Why so many syscalls? The connection tracking table

That's when we remembered something we'd seen in cilium status --verbose:

BPF Maps:   dynamic sizing: on (ratio: 0.005000)
  Name                          Size
  TCP connection tracking       16777216
  Non-TCP connection tracking   14232516
  ...

Our TCP connection tracking table had a maximum size of 16 million entries, two times the default of eight million for a machine with 2TB of RAM. We had deliberately configured **cilium\_bpf\_map\_dynamic\_size\_ratio: 0.0050\** in our Helm Chart months earlier, fully expecting the connection tracking capacity to scale proportionally with memory across our node types. At the time, this was a planned scaling adjustment to prevent connection tracking exhaustion on our 512GB RAM machines under high-throughput workloads. This configuration worked as intended to support our unique workload evolution, but as our traffic grew, it created a new scaling challenge in how the larger table interacted with the garbage collection auto-scaling algorithm on our high-resource 2TB RAM nodes.

But how many entries were actually in the table? We listed them with **cilium bpf ct list global\*. This showed about 7 million entries. But here’s what made it interesting: when we checked the timestamp of the *expires field against the system uptime, we found that the vast majority were already expired, some by several hours. This pointed us directly toward the garbage collection mechanism.

Now the napkin math gets interesting:

• Maximum iteration speed: ~200,000 entries per second
• Current table size: 7 million entries
• Time to walk the current table: 35 seconds
• Maximum table size: 16 million entries (at worst)
• Time to walk the full table: 80 seconds

That means every time we created or deleted an endpoint, we would spend as much as 35 seconds iterating through the connection tracking table, and in the worst-case scenario, a maximum of 80 seconds. Keep in mind our CNI timeout constraint is 90 seconds.

But wait, if the entries were expired, why wasn’t the garbage collector cleaning them up?

Why aren't expired entries cleaned up? Garbage collection gone wrong

Cilium has garbage collection for the connection tracking table. Looking at the metrics, GC was triggering quite often:

But when we looked at what the garbage collector was actually deleting, we saw a problem:

The garbage collector was running frequently but deleting almost nothing most of the time. Only occasionally would it delete significant numbers of entries.

Digging into the code revealed why. Cilium has two types of GC operations:

1. Endpoint-specific cleanup: When creating or deleting an endpoint, clean entries matching that endpoint's IP
2. Periodic expired entry cleanup: Runs on an interval to remove all expired entries

Pod creation and deletion triggered almost all of the frequent GC runs in the metrics (the type 1 endpoint-specific cleanups). The periodic cleanup (type 2), which removes expired entries, was barely running at all.

Why? Because the GC interval auto-scales based on how much it deletes:

// Simplified from Cilium source
func GetInterval(interval time.Duration, maxDeleteRatio float64) time.Duration {
    if maxDeleteRatio > 0.25 {
        // Deleted > 25% of entries → GC more frequently
        interval = time.Duration(float64(interval) * (1.0 - maxDeleteRatio))
    } else if maxDeleteRatio < 0.05 {
        // Deleted < 5% of entries → GC less frequently
        interval = time.Duration(float64(interval) * 1.5)
    }

    if interval > ConntrackGCMaxLRUInterval {
        interval = ConntrackGCMaxLRUInterval  // 12 hours
    }
    return interval
}

Here's the problem with a 16-million-entry table:

• To delete more than 5% (and avoid slowing down), you need to delete 800,000+ entries
• To delete more than 25% (and speed up), you need to delete 4+ million entries
• Starting interval: 5 minutes
• Maximum interval: 12 hours

Imagine this scenario:

1. The node initially experiences a low connection volume when it is newly onboarded or runs only light workloads.
2. The garbage collector deletes less than 5% of entries during a run, which fails to trigger more frequent cycles.
3. The garbage collection interval increases progressively from minutes until it reaches the 12-hour maximum. It would start with 7.5 minutes, increase to 11.25 minutes, then to 16.875 minutes, and so on, until eventually reaching the 12-hour limit.
4. Heavy data workloads eventually land on the node and generate a high volume of network traffic.
5. New connections fill the tracking table for up to 12 hours before the garbage collector runs again.

By the time garbage collection eventually executes, the connection tracking table may have already accumulated up to 16 million stale entries. While the GC run might delete enough records to temporarily restore speed, the excessive delay between cycles is inherently problematic. This lag allows the table to accumulate a large number of stale entries again, forcing every pod created during the long interval after the previous GC to endure the significant performance penalty of a sequential scan.

Why does it cascade? Mutex locks, timeouts, and retries

While a single slow pod spawn is highly inconvenient, the issue escalated significantly when multiple pods tried to spawn simultaneously.

We captured a goroutine dump from the Cilium agent using gops and analysed it with a script that groups similar stack traces. The results were revealing:

• 57 occurrences: createEndpoint() → WaitForFirstRegeneration() → waiting on RWMutex
• 57 occurrences: regenerateBPF() → runPreCompilationSteps() → invoked
• 56 occurrences:

scrubIPsInConntrackTable() → garbageCollectConntrack() → waiting for Lock

There's a global mutex on the connection tracking table. When we spawn 50 pods at once:

• Pod 1 acquires the lock and starts the 80-second table iteration
• Pods 2-50 queue up waiting for the lock
• Pod 1 finishes after 80 seconds
• Pod 2 acquires the lock, starts another 80-second iteration
• But Pod 2's timer started 80 seconds ago → timeout at 90 seconds
• Pod 2 times out
• Pods 3-50 never stand a chance

Here's where it gets worse. When the CNI times out after 90 seconds:

• The timeout returns an error to the caller
• But the underlying work doesn't stop, the agent keeps iterating
• The container runtime (containerd) immediately calls DeleteEndpoint()
• Delete also needs to walk the conntrack table
• Now the system queues up both create and delete operations

And then Kubernetes retries:

• The kubelet's podWorkerLoop retries after 60-90 seconds (with jitter)
• Each retry adds another endpoint creation and deletion request to the queue
• The queue grows faster than it drains

We could see this in the logs. For one pod (cilium-node-breaker-5), we saw:

• 15:54:30 - Create endpoint (attempt 1)
• 15:56:00 - Delete endpoint (timeout)
• 15:57:12 - Create endpoint (attempt 2)
• 15:59:57 - Create endpoint (attempt 3)
• 16:02:39 - Create endpoint (attempt 4)

The node enters a contention cycle: new work arrives faster than old work completes, and the queue never drains.

Here's the full picture:

The fix: one line to rule them all

After all that investigation, the fix was anticlimactic in its simplicity. We couldn't rely on the auto-scaling GC interval because it would inevitably grow too long on quiet nodes. Hence, we prevented the GC interval from auto-scaling by setting a fixed value:

conntrackGCInterval: 60s

That’s it. One configuration line ensures garbage collection runs at least every minute, regardless of how much it deletes. We applied the change at 9:00 am and completed the DaemonSet rollout by 10:00 am. The results speak for themselves:

The conntrack table size dropped dramatically and stayed stable. More importantly, the API call durations returned to normal:

Since the fix, we haven't seen a single instance of the timeout error. Pod spawn times are reliable again.

Lessons learned

Scaling parameters can have long-tail interactions. Our proactive tuning of bpf_map_dynamic_size_ratio to support workload scaling on 512GB RAM machines successfully resolved initial capacity limits. However, as our analytical workloads evolved and traffic increased, the larger table size dynamically allocated on our 2TB RAM machines revealed a subtle interaction with the CNI's GC auto-scaling algorithm. These scaling parameters can take months to show their full impact as traffic patterns grow, particularly in environments with adaptive background loops.

Observability and full-stack understanding are critical. While logs showed symptoms, we needed profiling and tracing to reveal the root cause across the entire stack. The container runtime (timeouts, delete behaviour), the CNI plugin (timeout values), the Cilium agent (mutex locks, GC logic), and the Linux kernel (eBPF maps, syscall performance) were all relevant to understanding how we got to the pod spawn timeouts. On top of that, adding napkin math with real numbers proved very powerful. Once we had the key numbers, 200k syscalls/sec, 12M table entries and a 90-second timeout, we determined the root cause far before we understood the full chain. Always measure your system's actual performance characteristics, not just theoretical limits.

Auto-scaling algorithms need bounds. Cilium's GC interval auto-scaling makes sense for most deployments: if you're deleting lots of entries, run GC more often; if you're deleting few entries, save CPU by running GC less often. But the algorithm didn't account for varying workloads, where a machine has a low connection volume for a prolonged period, after which, with a single pod introduction, it could get a very high connection volume. Nor did the algorithm account for very large tables where "5% of entries" is an enormous absolute number. The 12-hour maximum interval was too long for our workload. Auto-scaling without careful consideration of edge cases can backfire.

Timeouts don't stop work. When the CNI timed out, we assumed the work would stop. It didn't. The agent kept processing in the background while new requests queued up. This is a common pattern in distributed systems: timeouts protect the caller but don't necessarily cancel the operation. Be explicit about cancellation when needed.

Treat conntrack health as a first-class operational metric. The difference between a healthy cluster and a contention cycle showed up clearly in some metrics we weren't watching:

• GC duration - cilium_datapath_conntrack_gc_duration_seconds - jumped from 1s to 80s
• Table size - cilium_datapath_conntrack_gc_entries - 7M entries, mostly expired

Proactively alerting on these metrics is something we now recommend for any Cilium deployment with dynamic workloads, alongside setting conntrackGCInterval: 60s. Don't optimize for CPU savings during quiet periods at the expense of pod spawn timeouts during busy periods.

Conclusion

A single configuration line ultimately resolved the mysterious timeout error that impacted our ability to spawn new pods on our big data platform: conntrackGCInterval: 60s. Our investigation revealed that the root cause of our pod timeouts was Cilium's auto-scaling garbage collection algorithm, allowing the cleanup interval to grow to 12 hours, leading to a massive accumulation of expired entries and a linear-time iteration trap.

This experience provided major takeaways regarding system resilience and the necessity of a full-stack understanding. We learned that scaling parameters and resource allocations can have long-tail interactions that only surface months later as workloads evolve. Furthermore, we discovered that auto-scaling algorithms require strict bounds to prevent unexpected performance degradation in edge cases, such as the varying connection volumes we see on our high-resource machines. The investigation also highlighted that timeouts often only protect the caller, without stopping the underlying work, potentially triggering a contention cycle of retries that we could only diagnose through deep observability into mutex locks, syscalls, and eBPF internals.

As we move forward, we must ask ourselves: are the adaptive behaviours in our infrastructure truly protecting us, or are they masking inefficiencies that only appear at peak capacity? By treating conntrack health as a first-class operational metric and prioritising reliability over minor CPU savings, we can build more robust systems. And remember, if you ever see mysterious timeouts in your CNI: sometimes the answer hides in 426,000 syscalls per second.