<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ayodeji Ogundare</title>
    <description>The latest articles on DEV Community by Ayodeji Ogundare (@ayodejidev).</description>
    <link>https://dev.to/ayodejidev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F840299%2F5fb14c88-e0d8-4649-8acf-c05907d4a84c.jpg</url>
      <title>DEV Community: Ayodeji Ogundare</title>
      <link>https://dev.to/ayodejidev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ayodejidev"/>
    <language>en</language>
    <item>
      <title>Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts</title>
      <dc:creator>Ayodeji Ogundare</dc:creator>
      <pubDate>Sun, 12 Jul 2026 23:21:33 +0000</pubDate>
      <link>https://dev.to/adyen/inside-cilium-cni-solving-mysterious-kubernetes-pod-setup-timeouts-2gk5</link>
      <guid>https://dev.to/adyen/inside-cilium-cni-solving-mysterious-kubernetes-pod-setup-timeouts-2gk5</guid>
      <description>&lt;p&gt;&lt;strong&gt;&lt;em&gt;By Jorrick Sleijster · Senior Data Platform Engineer, Adyen&lt;/em&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking.&lt;br&gt;
In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes.&lt;/p&gt;

&lt;p&gt;We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck.&lt;/p&gt;
&lt;h2&gt;
  
  
  Our setup: why we're different
&lt;/h2&gt;

&lt;p&gt;At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data extraction from HDFS&lt;/strong&gt;. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pod churn.&lt;/strong&gt; Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Wide variety of workloads.&lt;/strong&gt; Some workloads are very CPU-intensive, like Spark pods executing complex joins and transformations with relatively few network connections. Others are extremely network-intensive, like Trino pods querying thousands of small files on HDFS, each requiring a new connection. This creates a large number of short-lived connections that stress the connection tracking table.&lt;/p&gt;

&lt;p&gt;Furthermore, our machines are more powerful than most Kubernetes cluster machines within Adyen:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Machines with 64 physical cores and 512GB of RAM
&lt;/li&gt;
&lt;li&gt;Machines with 128 physical cores and 2TB of RAM&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At the time, our environment consisted of Kubernetes 1.31.7 and Cilium 1.16.5. We provide these specific versions to help interested readers correlate our findings with the relevant codebases. &lt;/p&gt;

&lt;p&gt;To support our unique workloads, we progressively tuned Cilium by increasing DNS proxy timeouts, expanding the connection tracking table capacity, raising API rate limits, and streamlining "security labels". This configuration allowed us to overcome challenge after challenge, except for one:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0fecf4844d3f8f2df218f09d91f9698bb424e2166551952f46fd7638f4757cf2": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Kubernetes would create a pod, spin up the sandbox, and then attempt to set up the networking with Cilium. This operation would time out. From that point onwards, it also timed out any other pod attempting to spawn on the same node. Furthermore, we noticed that on those affected nodes, the API duration for &lt;strong&gt;DELETE /v1/endpoint&lt;/strong&gt; and &lt;strong&gt;PUT /v1/endpoint&lt;/strong&gt;, the routes responsible for adding and deleting a Cilium endpoint, also increased. You can see this clearly in the image below where the endpoint call latency starts to increase linearly over time from 4:20 pm onwards, indicating that endpoint creation and deletion never finish.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F68ihuic2shsxaw0gub70.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F68ihuic2shsxaw0gub70.png" alt="Graph showing payment processing times on Adyen's platform from 9:50 to 17:00." width="800" height="176"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It is important to note that these clusters are only used for analytical processing and big data workloads. The issue was identified, analysed, and fully resolved before any SLO was breached. Furthermore, as our big data environments are isolated from our core transactional flows, this issue never impacted our real-time payment processing pipeline or merchant transactions.&lt;/p&gt;
&lt;h2&gt;
  
  
  The symptom: something's wrong, but what?
&lt;/h2&gt;

&lt;p&gt;First, we found no related timeouts in the Cilium Agent logs. We enabled debug logging, hoping to find hidden errors. Nothing. The logs gave us the big picture but didn't reveal the bottleneck.&lt;br&gt;&lt;br&gt;
We did learn some valuable things from debugging the broken nodes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Listing all endpoints on a broken node would time out: cilium endpoint list
&lt;/li&gt;
&lt;li&gt;Listing the logs of an endpoint worked fine and showed the endpoint was stuck in the regenerating phase: cilium endpoint log &amp;lt;ENDPOINT_ID&amp;gt;
&lt;/li&gt;
&lt;li&gt;Requesting the endpoint state with  kubectl get ciliumendpoint showed it was still regenerating, while the Kubernetes manifest of the endpoint showed a ready status.
&lt;/li&gt;
&lt;li&gt;Checking the filesystem at /var/run/cilium/state showed endpoint folders with a _next suffix, indicating they didn't complete successfully.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This validated our suspicion that the issue was in the Cilium agent, not at the kubelet or CNI plugin level. But we still didn't know why.&lt;/p&gt;
&lt;h2&gt;
  
  
  Under the hood: how Cilium sets up networking for a pod
&lt;/h2&gt;

&lt;p&gt;Before we dive into the debugging journey, it's helpful to understand what happens when a pod spawns and Cilium sets up its networking.&lt;br&gt;&lt;br&gt;
The process starts when Kubernetes assigns a pod to a specific node. The kubelet initiates the pod setup and calls the configured CNI plugin (Cilium, in our case). The Cilium CNI plugin performs several operations:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Allocates an IP&lt;/strong&gt; using IPAM (IP Address Management)
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Creates a link device&lt;/strong&gt; (in our setup, a veth pair. One end in the host namespace, one in the pod)
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Configures the pod network&lt;/strong&gt; by setting the IP address, configuring routes and setting sysctl parameters
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Creates a Cilium endpoint&lt;/strong&gt; via the Cilium agent API
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrieves or allocates a security identity&lt;/strong&gt; using pod labels
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calculates network policy&lt;/strong&gt; for this endpoint
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Generates, compiles, and injects eBPF&lt;/strong&gt; &lt;strong&gt;code&lt;/strong&gt; into the kernel
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Returns success&lt;/strong&gt; to the kubelet&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The key thing to understand is that when creating a Pod, and thus an endpoint, the CNI plugin calls the Cilium agent, which runs as a DaemonSet on each node. The agent does the heavy lifting: managing eBPF maps, handling connection tracking, applying network policies, and more.&lt;/p&gt;

&lt;p&gt;Here's a simplified view of the flow:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk0ee5w9fctgn7sfzimnf.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk0ee5w9fctgn7sfzimnf.jpeg" alt="Flowchart of network setup process involving cabling, CNC plugin, Citrux agent, and kernel." width="640" height="484"&gt;&lt;/a&gt;&lt;br&gt;
A vital part of this process occurs during endpoint creation: Cilium triggers the connection tracking table's garbage collection to verify that the pod's IP is free of residual connections from its previous run. The scrubIPsInConntrackTable function executes this operation, scanning the entire conntrack table to identify and delete relevant entries.&lt;/p&gt;

&lt;p&gt;The consequences of failing garbage collection are significant. Stale entries accumulate without limit, and while our table can accommodate up to 16 million entries, the real bottleneck is the mandatory scan that every new pod requires. This cleaning step is essential to mitigate IP address reuse conflicts, ensuring that a fresh pod doesn't inherit any open connections from a prior one.&lt;/p&gt;

&lt;p&gt;This is where our story really begins.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Note: &lt;a href="https://arthurchiao.art/blog/cilium-code-cni-create-network/" rel="noopener noreferrer"&gt;Arthur Chiao's excellent deep dive into Cilium's CNI implementation&lt;/a&gt; provided much of our understanding of the CNI flow. While Arthur Chiao wrote it a few years ago, the core concepts remain relevant, and it is an invaluable resource for anyone wanting to understand how Cilium works under the hood.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Down the rabbit hole: tracing the root cause
&lt;/h2&gt;
&lt;h3&gt;
  
  
  What is the agent actually doing?
&lt;/h3&gt;

&lt;p&gt;We needed to see what the Cilium agent was doing when it hung. Enter pprof, Go's built-in profiler. We enabled pprof in Cilium's configuration and captured traces from a broken node right after spawning 50 pods.&lt;/p&gt;

&lt;p&gt;The CPU and memory profiles didn't reveal much at first. But when we opened the execution traces, the timeline view showed exactly what each goroutine was doing, and everything became clear.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhwnq2etl1h8r10bx7jt9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhwnq2etl1h8r10bx7jt9.png" alt="A dashboard showcasing transaction timelines and payment activity analysis by Adyen" width="800" height="311"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We saw long-running goroutines spending their entire time in syscalls. Zooming in closer revealed the pattern:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhxk28wskjv2zmy21kmyn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhxk28wskjv2zmy21kmyn.png" alt="Timeline of payment processing activities on an Adyen platform dashboard." width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
The agent was making BPF syscalls in a tight loop: nextKey(), lookup(), nextKey(), lookup(), over and over again. The agent uses these syscalls to iterate through an eBPF map:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;nextKey(currentKey)&lt;/code&gt; - Gets the next key in the BPF map after currentKey
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lookup(key)&lt;/code&gt; - Retrieves the value associated with key&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Let's do some napkin math. We selected a 35-millisecond fragment and counted 14,916 syscall occurrences. That's 426,171 syscalls per second. Since getting each element from an eBPF map takes two syscalls (next + get), we were iterating through roughly 213,000 entries per second. &lt;/p&gt;

&lt;p&gt;As Cilium is a user-space process, accessing or manipulating the map forces a context switch on every syscall. A context switch is the process of the CPU temporarily halting the user-space process (Cilium) to execute code in the kernel (to handle the BPF syscall) and then resuming the user-space process. This involves saving and restoring the entire state of the CPU registers and memory space, which adds significant overhead and is a major source of latency.&lt;/p&gt;

&lt;p&gt;Here's the critical insight: this is a sequential operation. You can't parallelize it because you need the current key to get the next key. Even on our powerful server CPUs, this was the maximum speed we could achieve. And there was one more important detail in the traces: this was happening in the scrubIPsInConntrackTable function, the one that cleans the connection tracking table when creating an endpoint.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why so many syscalls? The connection tracking table
&lt;/h3&gt;

&lt;p&gt;That's when we remembered something we'd seen in &lt;code&gt;cilium status --verbose&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BPF Maps:   dynamic sizing: on (ratio: 0.005000)
  Name                          Size
  TCP connection tracking       16777216
  Non-TCP connection tracking   14232516
  ...

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our TCP connection tracking table had a maximum size of 16 million entries, two times the default of eight million for a machine with 2TB of RAM. We had deliberately configured &lt;code&gt;**cilium\_bpf\_map\_dynamic\_size\_ratio: 0.0050\&lt;/code&gt;** in our Helm Chart months earlier, fully expecting the connection tracking capacity to scale proportionally with memory across our node types. At the time, this was a planned scaling adjustment to prevent connection tracking exhaustion on our 512GB RAM machines under high-throughput workloads. This configuration worked as intended to support our unique workload evolution, but as our traffic grew, it created a new scaling challenge in how the larger table interacted with the garbage collection auto-scaling algorithm on our high-resource 2TB RAM nodes.&lt;/p&gt;

&lt;p&gt;But how many entries were actually in the table? We listed them with &lt;code&gt;**cilium bpf ct list global\&lt;/code&gt;*&lt;em&gt;. This showed about 7 million entries. But here’s what made it interesting: when we checked the timestamp of the *expires&lt;/em&gt; field against the system uptime, we found that the vast majority were already expired, some by several hours. This pointed us directly toward the garbage collection mechanism.&lt;/p&gt;

&lt;p&gt;Now the napkin math gets interesting:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Maximum iteration speed: ~200,000 entries per second
&lt;/li&gt;
&lt;li&gt;Current table size: 7 million entries
&lt;/li&gt;
&lt;li&gt;Time to walk the current table: 35 seconds
&lt;/li&gt;
&lt;li&gt;Maximum table size: 16 million entries (at worst)
&lt;/li&gt;
&lt;li&gt;Time to walk the full table: 80 seconds&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That means every time we created or deleted an endpoint, we would spend as much as 35 seconds iterating through the connection tracking table, and in the worst-case scenario, a maximum of 80 seconds. Keep in mind our CNI timeout constraint is 90 seconds.&lt;/p&gt;

&lt;p&gt;But wait, if the entries were expired, why wasn’t the garbage collector cleaning them up?&lt;/p&gt;

&lt;h3&gt;
  
  
  Why aren't expired entries cleaned up? Garbage collection gone wrong
&lt;/h3&gt;

&lt;p&gt;Cilium has garbage collection for the connection tracking table. Looking at the metrics, GC was triggering quite often:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg4tc9slc1l11luqg4i9s.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg4tc9slc1l11luqg4i9s.png" alt="Garbage collection activity over time showing traffic spikes and moderate loads." width="800" height="298"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;But when we looked at what the garbage collector was actually deleting, we saw a problem:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp0f6xf8fo6dujvsnz47p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp0f6xf8fo6dujvsnz47p.png" alt="Graph showing payment activity spikes over a 24-hour period with Adyen transaction data" width="800" height="218"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The garbage collector was running frequently but deleting almost nothing most of the time. Only occasionally would it delete significant numbers of entries.&lt;/p&gt;

&lt;p&gt;Digging into the code revealed why. Cilium has two types of GC operations:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Endpoint-specific cleanup&lt;/strong&gt;: When creating or deleting an endpoint, clean entries matching that endpoint's IP
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Periodic expired entry cleanup&lt;/strong&gt;: Runs on an interval to remove all expired entries&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pod creation and deletion triggered almost all of the frequent GC runs in the metrics (the type 1 endpoint-specific cleanups). The periodic cleanup (type 2), which removes expired entries, was barely running at all.&lt;/p&gt;

&lt;p&gt;Why? Because the GC interval auto-scales based on how much it deletes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c"&gt;// Simplified from Cilium source&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;GetInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;maxDeleteRatio&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;maxDeleteRatio&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;0.25&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c"&gt;// Deleted &amp;gt; 25% of entries → GC more frequently&lt;/span&gt;
        &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;maxDeleteRatio&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;maxDeleteRatio&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;0.05&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c"&gt;// Deleted &amp;lt; 5% of entries → GC less frequently&lt;/span&gt;
        &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="m"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;ConntrackGCMaxLRUInterval&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ConntrackGCMaxLRUInterval&lt;/span&gt;  &lt;span class="c"&gt;// 12 hours&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here's the problem with a 16-million-entry table:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;To delete more than 5% (and avoid slowing down), you need to delete 800,000+ entries
&lt;/li&gt;
&lt;li&gt;To delete more than 25% (and speed up), you need to delete 4+ million entries
&lt;/li&gt;
&lt;li&gt;Starting interval: 5 minutes
&lt;/li&gt;
&lt;li&gt;Maximum interval: 12 hours&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Imagine this scenario:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The node initially experiences a low connection volume when it is newly onboarded or runs only light workloads.
&lt;/li&gt;
&lt;li&gt;The garbage collector deletes less than 5% of entries during a run, which fails to trigger more frequent cycles.
&lt;/li&gt;
&lt;li&gt;The garbage collection interval increases progressively from minutes until it reaches the 12-hour maximum. It would start with 7.5 minutes, increase to 11.25 minutes, then to 16.875 minutes, and so on, until eventually reaching the 12-hour limit.
&lt;/li&gt;
&lt;li&gt;Heavy data workloads eventually land on the node and generate a high volume of network traffic.
&lt;/li&gt;
&lt;li&gt;New connections fill the tracking table for up to 12 hours before the garbage collector runs again.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;By the time garbage collection eventually executes, the connection tracking table may have already accumulated up to 16 million stale entries. While the GC run might delete enough records to temporarily restore speed, the excessive delay between cycles is inherently problematic. This lag allows the table to accumulate a large number of stale entries again, forcing every pod created during the long interval after the previous GC to endure the significant performance penalty of a sequential scan.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F94jcqk5yzbkii1aisnt3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F94jcqk5yzbkii1aisnt3.png" alt="Graph of real-time payment transaction data from an Adyen terminal or system." width="800" height="230"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does it cascade? Mutex locks, timeouts, and retries
&lt;/h3&gt;

&lt;p&gt;While a single slow pod spawn is highly inconvenient, the issue escalated significantly when multiple pods tried to spawn simultaneously.&lt;br&gt;&lt;br&gt;
We captured a goroutine dump from the Cilium agent using gops and analysed it with a script that groups similar stack traces. The results were revealing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;57 occurrences:
createEndpoint() → WaitForFirstRegeneration() → waiting on RWMutex
&lt;/li&gt;
&lt;li&gt;57 occurrences:
regenerateBPF() → runPreCompilationSteps() → invoked
&lt;/li&gt;
&lt;li&gt;56 occurrences:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;scrubIPsInConntrackTable() → garbageCollectConntrack() → waiting for Lock&lt;/p&gt;

&lt;p&gt;There's a &lt;strong&gt;global mutex on the connection tracking table&lt;/strong&gt;. When we spawn 50 pods at once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pod 1 acquires the lock and starts the 80-second table iteration
&lt;/li&gt;
&lt;li&gt;Pods 2-50 queue up waiting for the lock
&lt;/li&gt;
&lt;li&gt;Pod 1 finishes after 80 seconds
&lt;/li&gt;
&lt;li&gt;Pod 2 acquires the lock, starts another 80-second iteration
&lt;/li&gt;
&lt;li&gt;But Pod 2's timer started 80 seconds ago → timeout at 90 seconds
&lt;/li&gt;
&lt;li&gt;Pod 2 times out
&lt;/li&gt;
&lt;li&gt;Pods 3-50 never stand a chance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's where it gets worse. When the CNI times out after 90 seconds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The timeout returns an error to the caller
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;But the underlying work doesn't stop&lt;/strong&gt;, the agent keeps iterating
&lt;/li&gt;
&lt;li&gt;The container runtime (containerd) immediately calls DeleteEndpoint()
&lt;/li&gt;
&lt;li&gt;Delete also needs to walk the conntrack table
&lt;/li&gt;
&lt;li&gt;Now the system queues up both create and delete operations&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And then Kubernetes retries:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The kubelet's &lt;em&gt;podWorkerLoop&lt;/em&gt; retries after 60-90 seconds (with jitter)
&lt;/li&gt;
&lt;li&gt;Each retry adds another endpoint creation and deletion request to the queue
&lt;/li&gt;
&lt;li&gt;The queue grows faster than it drains&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We could see this in the logs. For one pod (cilium-node-breaker-5), we saw:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;15:54:30 - Create endpoint (attempt 1)
&lt;/li&gt;
&lt;li&gt;15:56:00 - Delete endpoint (timeout)
&lt;/li&gt;
&lt;li&gt;15:57:12 - Create endpoint (attempt 2)
&lt;/li&gt;
&lt;li&gt;15:59:57 - Create endpoint (attempt 3)
&lt;/li&gt;
&lt;li&gt;16:02:39 - Create endpoint (attempt 4)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The node enters a contention cycle: new work arrives faster than old work completes, and the queue never drains.&lt;/p&gt;

&lt;p&gt;Here's the full picture:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmji8web9wny63om0t06j.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmji8web9wny63om0t06j.jpeg" alt="Flowchart illustrating a payment process with Adyen terminals, OLV plugin, and checkout steps." width="640" height="563"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  The fix: one line to rule them all
&lt;/h2&gt;

&lt;p&gt;After all that investigation, the fix was anticlimactic in its simplicity. We couldn't rely on the auto-scaling GC interval because it would inevitably grow too long on quiet nodes. Hence, we prevented the GC interval from auto-scaling by setting a fixed value:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;conntrackGCInterval: 60s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That’s it. One configuration line ensures garbage collection runs at least every minute, regardless of how much it deletes. We applied the change at 9:00 am and completed the DaemonSet rollout by 10:00 am. The results speak for themselves:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tcxyh7wdf7wue3s37uo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5tcxyh7wdf7wue3s37uo.png" alt="Line chart displaying payment transaction data over time at an Adyen endpoint" width="800" height="244"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The conntrack table size dropped dramatically and stayed stable. More importantly, the API call durations returned to normal:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcn5ki0o5tdja92zmowuk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcn5ki0o5tdja92zmowuk.png" alt="Bar chart illustrating transaction volume and payment data for Adyen services over time." width="800" height="245"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6pf9ns3pjjy82or0iudm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6pf9ns3pjjy82or0iudm.png" alt="Graph showing transaction volume and payment activity data from an Adyen system." width="800" height="243"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Since the fix, we haven't seen a single instance of the timeout error. Pod spawn times are reliable again.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lessons learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Scaling parameters can have long-tail interactions.&lt;/strong&gt; Our proactive tuning of bpf_map_dynamic_size_ratio to support workload scaling on 512GB RAM machines successfully resolved initial capacity limits. However, as our analytical workloads evolved and traffic increased, the larger table size dynamically allocated on our 2TB RAM machines revealed a subtle interaction with the CNI's GC auto-scaling algorithm. These scaling parameters can take months to show their full impact as traffic patterns grow, particularly in environments with adaptive background loops.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Observability and full-stack understanding are critical.&lt;/strong&gt; While logs showed symptoms, we needed profiling and tracing to reveal the root cause across the entire stack. The container runtime (timeouts, delete behaviour), the CNI plugin (timeout values), the Cilium agent (mutex locks, GC logic), and the Linux kernel (eBPF maps, syscall performance) were all relevant to understanding how we got to the pod spawn timeouts. On top of that, adding napkin math with real numbers proved very powerful. Once we had the key numbers, 200k syscalls/sec, 12M table entries and a 90-second timeout, we determined the root cause far before we understood the full chain. Always measure your system's actual performance characteristics, not just theoretical limits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Auto-scaling algorithms need bounds.&lt;/strong&gt; Cilium's GC interval auto-scaling makes sense for most deployments: if you're deleting lots of entries, run GC more often; if you're deleting few entries, save CPU by running GC less often. But the algorithm didn't account for varying workloads, where a machine has a low connection volume for a prolonged period, after which, with a single pod introduction, it could get a very high connection volume. Nor did the algorithm account for very large tables where "5% of entries" is an enormous absolute number. The 12-hour maximum interval was too long for our workload. Auto-scaling without careful consideration of edge cases can backfire.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Timeouts don't stop work&lt;/strong&gt;. When the CNI timed out, we assumed the work would stop. It didn't. The agent kept processing in the background while new requests queued up. This is a common pattern in distributed systems: timeouts protect the caller but don't necessarily cancel the operation. Be explicit about cancellation when needed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Treat conntrack health as a first-class operational metric&lt;/strong&gt;. The difference between a healthy cluster and a contention cycle showed up clearly in some metrics we weren't watching: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GC duration - cilium_datapath_conntrack_gc_duration_seconds - jumped from 1s to 80s
&lt;/li&gt;
&lt;li&gt;Table size - cilium_datapath_conntrack_gc_entries - 7M entries, mostly expired&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Proactively alerting on these metrics is something we now recommend for any Cilium deployment with dynamic workloads, alongside setting &lt;code&gt;conntrackGCInterval: 60s&lt;/code&gt;. Don't optimize for CPU savings during quiet periods at the expense of pod spawn timeouts during busy periods.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;A single configuration line ultimately resolved the mysterious timeout error that impacted our ability to spawn new pods on our big data platform: &lt;code&gt;conntrackGCInterval: 60s&lt;/code&gt;. Our investigation revealed that the root cause of our pod timeouts was Cilium's auto-scaling garbage collection algorithm, allowing the cleanup interval to grow to 12 hours, leading to a massive accumulation of expired entries and a linear-time iteration trap.&lt;/p&gt;

&lt;p&gt;This experience provided major takeaways regarding system resilience and the necessity of a full-stack understanding. We learned that scaling parameters and resource allocations can have long-tail interactions that only surface months later as workloads evolve. Furthermore, we discovered that auto-scaling algorithms require strict bounds to prevent unexpected performance degradation in edge cases, such as the varying connection volumes we see on our high-resource machines. The investigation also highlighted that timeouts often only protect the caller, without stopping the underlying work, potentially triggering a contention cycle of retries that we could only diagnose through deep observability into mutex locks, syscalls, and eBPF internals.&lt;/p&gt;

&lt;p&gt;As we move forward, we must ask ourselves: are the adaptive behaviours in our infrastructure truly protecting us, or are they masking inefficiencies that only appear at peak capacity? By treating conntrack health as a first-class operational metric and prioritising reliability over minor CPU savings, we can build more robust systems. And remember, if you ever see mysterious timeouts in your CNI: sometimes the answer hides in 426,000 syscalls per second.&lt;/p&gt;

</description>
      <category>cilium</category>
      <category>kubernetes</category>
      <category>data</category>
      <category>adyen</category>
    </item>
    <item>
      <title>Building a GlobalStream: Replication Challenges and Optimizations</title>
      <dc:creator>Ayodeji Ogundare</dc:creator>
      <pubDate>Fri, 10 Oct 2025 10:50:13 +0000</pubDate>
      <link>https://dev.to/adyen/building-a-globalstream-replication-challenges-and-optimizations-46jc</link>
      <guid>https://dev.to/adyen/building-a-globalstream-replication-challenges-and-optimizations-46jc</guid>
      <description>&lt;h4&gt;
  
  
  By Gaurav Singh &amp;amp; Luciano Sabença, Streaming Platform Team
&lt;/h4&gt;




&lt;p&gt;Global data streaming requires careful tradeoffs between performance, compliance, and reliability. At Adyen, replication is essential: data must remain close to producers to reduce latency and meet regulations, while also being centralized for analytics.&lt;/p&gt;

&lt;p&gt;In this blog, we share our experience optimizing MirrorMaker 2 for large-scale, cross-continent Kafka replication, the parameter tuning we applied, and the hidden behaviors in Kafka Connect that ultimately unlocked the required throughput.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Replicate?
&lt;/h2&gt;

&lt;p&gt;There are many things you need to take into account when designing a data streaming platform for a company such as Adyen. One of the most important aspects of it is data location and replication. Due to performance, reliability, and compliance, keeping data close to producers is usually a good idea. After all, you don't want to wait in line at the cashier to confirm the payment for that fat burger you just bought on that nice beach in Australia while your payment was being sent to Europe. However, it's also a common practice to have all data in a centralized cluster(s) in centralized locations for analytical purposes. These two need a bridge, which is where mirroring comes in.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flcj3oho4kjf3sgwjy5rg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Flcj3oho4kjf3sgwjy5rg.png" alt="An example replication flow diagram" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Mirror Maker
&lt;/h2&gt;

&lt;p&gt;When it comes to copying data between Kafka clusters, Mirror Maker is the default open source tool. MirrorMaker 2 is the current version, and it's built on top of Kafka Connect, a platform designed to make it easier to integrate Kafka with other tools such as databases and distributed file systems, and - why not? - another Kafka cluster. Setting up Mirror Maker is fairly straightforward, but tuning it to the performance Adyen requires isn't. Let's go over the journey into the depths of Kafka Connect and parameter tuning to find a solution!&lt;/p&gt;

&lt;h2&gt;
  
  
  Playing the Volume Game
&lt;/h2&gt;

&lt;p&gt;Kafka is a highly flexible tool and serves as the backbone for many different architectural patterns. It can be used for batch processing, real time processing, or any combination of these two extremes. With default parameters, Kafka producers and consumers are optimized for low-latency use cases: batch sizes are not large, and records are sent immediately to the server (i.e., linger.ms = 0). Given that our volume is on the order of several hundred thousand messages per second, any optimization has a huge impact on replication latency. In our case, the default settings were not good enough to keep the latency bounded within a few seconds; in fact, our replication latency was on the order of minutes. Since we are transferring this data across continents, we don't expect very low latency for replication, but such high latency indicates that we were at the maximum possible throughput for this configuration, so we needed to revisit those parameters.&lt;/p&gt;

&lt;p&gt;We made the following adjustments on the producer side of MirrorMaker:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;batch.size&lt;/code&gt;&lt;/strong&gt;: Increased from the default 16384 to 409600. This leads to larger batches per request, and less requests to replicate the same number of messages&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;linger.ms&lt;/code&gt;&lt;/strong&gt;: Increased from 0 to 200. This parameter sets how long the Kafka driver can wait before dispatching the messages. 0 means send immediately. Giving more time to the driver allows it to batch more messages in the same batch.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;compression.type&lt;/code&gt;&lt;/strong&gt;: Changed from 'none' to 'zstd'. Activating compression means more messages per batch.&lt;br&gt;
These changes didn't yield the desired results. There was another bottleneck. We then turned our attention to the consumer parameters. After all, MirrorMaker is both a consumer from the source cluster and a producer to the destination. &lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We adjusted:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;fetch.min.bytes&lt;/code&gt;&lt;/strong&gt;: Increased from 1 to 5000. So we wait a bit more to ensure that more data is available for consumption, thus decreasing the number of consumer requests and optimizing throughput. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;&lt;code&gt;max.fetch.records&lt;/code&gt;&lt;/strong&gt;: Increased from 500 to 2000 with the objective of fetching more records per requests and sending more requests to the producer.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal here was clear: have the consumers fetch as much data as possible at once and send it in very large batches to the destination cluster.  &lt;em&gt;The idea is to optimize network traffic and amortize large distance latencies when measured per unit of replicated byte.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Despite these efforts, the throughput remained stubbornly the same. Each time we restarted with new parameters, the record rate would spike briefly before settling back to its previous baseline, indicating that our changes were not having a lasting impact. This led us to believe there was a deeper, underlying issue we were missing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Missing Piece
&lt;/h2&gt;

&lt;p&gt;There was, however, a missing piece of the puzzle to make it work as we expect: how Kafka Connect changes are persisted and saved to the nodes. As we mentioned before, MM2 is built on top of Kafka Connect. Kafka Connect uses some internal Kafka topics to store and persist its state and configuration. This gives us three possible layers of configuration:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kafka driver's parameters&lt;/li&gt;
&lt;li&gt;Mirrormaker configuration&lt;/li&gt;
&lt;li&gt;Internal Kafka Connect state&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We needed to be sure that whenever any configuration was changed in any of these three places it would be reflected in the others. It soon became clear that this wasn't really the case: whilst our MM2 consumer was able to consume a quite decent amount of data, the producer wasn't really able to keep the throughput high and wasn't creating big enough batches for our use-case. &lt;/p&gt;

&lt;p&gt;After digging a bit into the internal configuration topics and on MM docs, we understood the root cause: configuration changes on the connectors are not updated while any instance of the connector is up and running. Instead, they are read from the internal mm configuration topics and applied to the connectors. This was our case: we are running MM2 in distributed mode, so we didn't stop all nodes and start them again. Although the behaviour makes sense, it prioritizes consistency when the application is running; it's not a very intuitive behavior and it almost led us to explore weird paths in our search for better throughput.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Results: A Tale of Two Graphs
&lt;/h2&gt;

&lt;p&gt;The results were dramatic once we understood the necessity of a complete restart. As seen in the graph below, the number of records per request skyrocketed. We were also able to validate via logs and via the data in the topic that the parameters being used by the producers and consumers were the same as we originally intended.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4uvyamjs3c6rikk656ft.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4uvyamjs3c6rikk656ft.png" alt="A graph showing the number of records per request" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;As a consequence of those parameters, the total number of requests sent by the producer dropped significantly and each request now was able to replicate more messages.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs15bqbonm9lheh2x5bxn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs15bqbonm9lheh2x5bxn.png" alt="Second graph showing the number of records per request" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This experience was a valuable lesson in the intricacies of managing a distributed system like Kafka Connect. Understanding the operational details of how configurations are managed was the key to unlocking the needed performance. &lt;em&gt;This journey highlights that sometimes the most significant gains come not from simply tweaking parameters, but from a deeper understanding of the tools themselves.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>tech</category>
      <category>payments</category>
      <category>adyen</category>
      <category>kafka</category>
    </item>
    <item>
      <title>Guide: Upgrade and customization for Salesforce Commerce Cloud cartridge</title>
      <dc:creator>Ayodeji Ogundare</dc:creator>
      <pubDate>Fri, 01 Sep 2023 13:52:28 +0000</pubDate>
      <link>https://dev.to/adyen/guide-upgrade-and-customization-for-salesforce-commerce-cloud-cartridge-4b51</link>
      <guid>https://dev.to/adyen/guide-upgrade-and-customization-for-salesforce-commerce-cloud-cartridge-4b51</guid>
      <description>&lt;p&gt;In this guide, discover the importance and process of upgrading your &lt;a href="https://github.com/Adyen/adyen-salesforce-commerce-cloud/releases"&gt;Salesforce Commerce Cloud (SFCC) cartridge&lt;/a&gt; to the latest version to create a hassle-free integration and seamless checkout experience benefitting both you and your customers. We'll highlight best practices for customizations and walk you through the process of adding custom code to a new cartridge, ensuring a smooth transition to the latest releases.&lt;/p&gt;

&lt;p&gt;As an online business owner, ensuring a smooth and seamless checkout experience for your shoppers is vital to success. To achieve this, customizations are often necessary to tailor your checkout process to meet specific needs. To take advantage of new features, bug fixes, and security patches, you’ll need to stay up-to-date with the latest cartridge version.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;strong&gt;This guide is specifically tailored to users of the Salesforce Commerce Cloud Storefront Reference Architecture (SFRA). If your e-commerce platform does not utilize SFRA, some of the outlined steps and procedures may not be directly applicable.&lt;/strong&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The benefits of upgrading your cartridge
&lt;/h2&gt;

&lt;p&gt;Upgrading your cartridge to the &lt;a href="https://github.com/Adyen/adyen-salesforce-commerce-cloud/releases"&gt;latest version&lt;/a&gt; offers several benefits that can significantly enhance your checkout experience. Here's why you should consider upgrading:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Access to new features: Each cartridge release comes with exciting new features that can improve the overall functionality and usability of your checkout process. This ensures that you offer your customers the best shopping experience.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Bug fixes and security patches: As with any software, cartridges might have bugs or security vulnerabilities. Regular updates and upgrades address these issues, not only ensuring that your checkout process remains secure, but also improving the authorization rate.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Compatibility with integrations: Third-party integrations and platforms continually evolve, and cartridge updates often ensure compatibility with the latest versions of these integrations. Upgrading your cartridge reduces the risk of conflicts and ensures a seamless connection with your preferred tools.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Continued Cartridge support: We provide support and assistance for the latest versions, making it easier to troubleshoot issues and seek guidance whenever needed.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Customization guide for smoother upgrade&amp;nbsp;(SFRA)
&lt;/h2&gt;

&lt;p&gt;Merchants use customizations to communicate their brand value to shoppers. Customizations are a great way to offer a unique shopper experience. Unfortunately, if customizations are implemented inside of the cartridge, they obscure future cartridge upgrades and introduce technology bottlenecks. We've created a &lt;a href="https://help.adyen.com/academy/how-to-videos/plugins/how-to-customize-your-sfcc-cartridge"&gt;4-minute video&lt;/a&gt; to help you decouple your customizations and upgrade a customized cartridge to the latest version. You can skip the &lt;a href="https://help.adyen.com/academy/how-to-videos/plugins/how-to-customize-your-sfcc-cartridge"&gt;video&lt;/a&gt; if you prefer a step-by-step text guide.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Add custom code to a new cartridge
&lt;/h3&gt;

&lt;p&gt;To make future upgrades and troubleshooting easier, add your custom code to the &lt;strong&gt;int_custom_cartridge&lt;/strong&gt; &lt;a href="https://github.com/Adyen/adyen-salesforce-commerce-cloud/tree/main/src/cartridges/int_custom_cartridge"&gt;folder&lt;/a&gt; in your &lt;strong&gt;src/cartridges&lt;/strong&gt; directory. If this folder is not available in your version, create a new folder and name it; &lt;strong&gt;int_custom_cartridge&lt;/strong&gt; and add your custom code. This setup will allow you to temporarily remove custom code during upgrades or troubleshooting.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Add your custom code to the int_custom_cartridge folder
# Modify the code as needed to customize the checkout experience. For example, in JavaScript:

# File: int_custom_cartridge/scripts/customization.js

function customizeCheckout() {
  // Your custom code here
  console.log("Custom checkout code executed.");
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Include custom cartridge in package.json file
&lt;/h3&gt;

&lt;p&gt;Update the name property in the &lt;strong&gt;package.json&lt;/strong&gt; file to &lt;strong&gt;int_custom_cartridge&lt;/strong&gt; or the name you chose for your custom cartridge.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Update package.json to support the custom cartridge
# File: int_custom_cartridge/package.json
{
  "name": "int_custom_cartridge",
  "version": "1.0.0",
  "description": "Custom cartridge for checkout customization",
  // Other package.json properties
}

# Update root package.json to contain the new cartridge in the build step
# Make sure you are in the root directory of your project and run

npm install
npm run build
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 3: Add the new cartridge to your cartridge path
&lt;/h3&gt;

&lt;p&gt;In the Business Manager, navigate to &lt;strong&gt;Administration&lt;/strong&gt; &amp;gt; &lt;strong&gt;Sites&lt;/strong&gt; &amp;gt; &lt;strong&gt;Manage Sites&lt;/strong&gt; &amp;gt; &lt;strong&gt;[yourSite]&lt;/strong&gt; &amp;gt; &lt;strong&gt;Settings&lt;/strong&gt;. In the Cartridges field, add the new cartridge before the Adyen cartridges. Click “&lt;strong&gt;Apply&lt;/strong&gt;” to save your changes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Modify or add end-to-end tests (optional)
&lt;/h3&gt;

&lt;p&gt;To ensure your custom code works correctly, modify or add end-to-end tests as needed. Running these tests will validate the functionality of your customizations.&lt;/p&gt;

&lt;h2&gt;
  
  
  Upgrade guide for different integrations
&lt;/h2&gt;

&lt;p&gt;Depending on your integration type (Default or Customized), the upgrade process varies:&lt;/p&gt;

&lt;h3&gt;
  
  
  Default integration
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Check which SFRA version is required for the upgrade and migrate if necessary, aiming to use the highest available SFRA version&lt;/li&gt;
&lt;li&gt;Download the desired Adyen cartridge version from &lt;a href="https://github.com/Adyen/adyen-salesforce-commerce-cloud"&gt;GitHub&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Transpile, compile, and upload the compatible, auto-generated code using &lt;code&gt;npm run build&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Re-upload the metadata
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Download the desired Adyen cartridge version from GitHub
# Replace '23.1.0' with the version you want to upgrade to
git clone https://github.com/adyen/adyen-salesforce-commerce-cloud.git --branch 23.1.0

# Re-upload the metadata
# Make sure you are in the root directory of your project
cd adyen-salesforce-commerce-cloud

# Transpile, compile, and upload the compatible, auto-generated code
npm run build

# Zip the site_import folder
zip -r adyen_integration.zip package/metadata/site_import/sites

# In Business Manager, go to Administration &amp;gt; Site Development &amp;gt; Site Import &amp;amp; Export
# Import the zipped file (site_import.zip)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Customized integration
&lt;/h3&gt;

&lt;p&gt;For a quick demo of how to upgrade the SFCC cartridge with customization, watch this &lt;a href="https://help.adyen.com/academy/how-to-videos/plugins/how-to-upgrade-sfcc-with-customization"&gt;2 mins video&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Steps:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Extract any custom code from the cartridge and place it in the custom cartridge (int_custom_cartridge) folder. Create a new folder if not available in your version.&lt;/li&gt;
&lt;li&gt;Check SFRA version requirements and migrate if needed&lt;/li&gt;
&lt;li&gt;Download the desired Adyen cartridge version from GitHub&lt;/li&gt;
&lt;li&gt;Update the package.json as explained above and run npm run build&lt;/li&gt;
&lt;li&gt;Re-upload the metadata&lt;/li&gt;
&lt;li&gt;Update the cartridge path in the Business Manager to include your customizations&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;In today's competitive e-commerce landscape, the checkout experience can make or break a customer's decision to complete a purchase. As an online business owner, customizing your checkout process is essential to cater to your unique requirements and create a branded experience that resonates with your audience. However, these customizations must not come at the cost of missing out on crucial updates and improvements.&lt;/p&gt;

&lt;p&gt;By following this guide to upgrading your cartridge to the latest version, you can strike the perfect balance between customization and innovation. Embrace the latest features, bug fixes, and security patches to ensure a seamless and secure checkout journey for your shoppers.&lt;/p&gt;

&lt;p&gt;Upgrading your cartridge to the latest version is more than just a technical necessity - it's an investment in your business's success. Stay ahead of the competition by accessing new capabilities that improve the functionality of your checkout process and enhance customer satisfaction. Take your e-commerce venture one step closer to excellence.&lt;/p&gt;

&lt;p&gt;So, are you ready to unlock the full potential of your checkout process? Your dedication to providing the best checkout experience will not only increase conversion rates but also foster customer loyalty.&lt;/p&gt;

&lt;p&gt;Upgrade today and let your checkout experience set the gold standard for online shopping!&lt;/p&gt;

</description>
      <category>sfcc</category>
      <category>onlinepayments</category>
      <category>tech</category>
      <category>ecommerce</category>
    </item>
  </channel>
</rss>
