<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Darshan Turakhia</title>
    <description>The latest articles on DEV Community by Darshan Turakhia (@darshan_turakhia).</description>
    <link>https://dev.to/darshan_turakhia</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3848023%2F326f61e3-2775-436f-9506-0b35ed05d998.png</url>
      <title>DEV Community: Darshan Turakhia</title>
      <link>https://dev.to/darshan_turakhia</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/darshan_turakhia"/>
    <language>en</language>
    <item>
      <title>How an emptyDir With medium: Memory Made Our Export Pods OOMKilled While the Heap Sat at 210 MB</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Thu, 19 Nov 2026 10:50:34 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-an-emptydir-with-medium-memory-made-our-export-pods-oomkilled-while-the-heap-sat-at-210-mb-1p4n</link>
      <guid>https://dev.to/darshan_turakhia/how-an-emptydir-with-medium-memory-made-our-export-pods-oomkilled-while-the-heap-sat-at-210-mb-1p4n</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;02:47 UTC, Wednesday. PagerDuty: &lt;code&gt;reports-export restarts &amp;gt; 3 in 30m&lt;/code&gt;. I open the Kubernetes dashboard and the restart counter on the deployment reads 17. Last reason for each one: &lt;code&gt;OOMKilled&lt;/code&gt;, exit code 137.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;Three of us end up in the channel. The export service builds CSV and XLSX files for enterprise customers, and the on-call for the finance team says three scheduled month-end exports are missing. Customers are asking where their files are.&lt;/p&gt;

&lt;p&gt;First theory, from everyone at once: memory leak in the last release. It shipped on Monday and added a new XLSX writer. That fits the timeline, so I start there.&lt;/p&gt;

&lt;p&gt;It stops fitting quickly. The container limit is 1 GiB. I pull the Node heap metrics we export from &lt;code&gt;process.memoryUsage()&lt;/code&gt; and the heap never crosses 210 MB on any pod, including the ones that were killed. RSS is 340 MB at the last scrape before each death. A leak in the new writer would show up as a climbing heap, and it is a flat line.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;Dead end one: native memory. Streaming XLSX libraries sometimes buffer in Buffers outside the V8 heap. That would show in RSS, and RSS is 340 MB. Still nowhere near 1 GiB.&lt;/p&gt;

&lt;p&gt;Dead end two: the kernel log. I read the node's &lt;code&gt;dmesg&lt;/code&gt; for the OOM kill record, expecting to see a process using 1 GiB of anonymous memory. The record is there, and it is odd:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;memory: usage 1048576kB, limit 1048576kB, failcnt 8841 Memory cgroup stats for /kubepods/burstable/pod4c1e.../a7f3...: anon 351068160 file 12582912 shmem 702545920 Killed process 29114 (node) total-vm:1204816kB, anon-rss:338120kB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;anon&lt;/code&gt; is 335 MB, which matches the Node process. &lt;code&gt;shmem&lt;/code&gt; is 670 MB. Something holds 670 MB of shared memory inside this container's cgroup, and it is not our code's heap.&lt;/p&gt;

&lt;p&gt;Shared memory in a container is usually &lt;code&gt;/dev/shm&lt;/code&gt; or a tmpfs mount. I check the pod spec and find it. When the platform team enabled &lt;code&gt;readOnlyRootFilesystem&lt;/code&gt; in September, the export service lost the ability to write to &lt;code&gt;/tmp&lt;/code&gt;, so somebody added a scratch volume:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;- name: scratch emptyDir: medium: Memory containers: - name: reports-export volumeMounts: - name: scratch mountPath: /tmp resources: limits: memory: 1Gi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;medium: Memory&lt;/code&gt; backs the volume with tmpfs. I exec into a live pod and run &lt;code&gt;df -h /tmp&lt;/code&gt;. It reports 4.2 GB size (half the node's RAM, since no &lt;code&gt;sizeLimit&lt;/code&gt; was set) and 640 MB used, with 22 leftover files from earlier exports.&lt;/p&gt;

&lt;p&gt;The exporter writes each finished file to &lt;code&gt;/tmp/exports/&amp;lt;job-id&amp;gt;.xlsx&lt;/code&gt;, uploads it to S3, and deletes it. Except the delete only runs on the success path. Any job that failed the upload, which happens a few times a day when the S3 call times out, leaves its file behind. Those files are in RAM, and tmpfs pages are charged to the memory cgroup of the container that wrote them.&lt;/p&gt;

&lt;p&gt;So the heap graph was a decoy. Every dashboard we had measured the process. The kernel measures the container, and ours had a RAM disk in it.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;Month-end is when the large exports run. A single 180 MB XLSX plus the pod's baseline 340 MB is fine. But the S3 timeouts cluster at month-end too, because the same batch hammers the bucket. Each failed upload strands another 150 to 200 MB file in tmpfs. Four stranded files, one new export in progress, and the cgroup hits 1 GiB. The kernel cannot reclaim tmpfs pages (there is no swap), so it kills the process.&lt;/p&gt;

&lt;p&gt;One more detail made the incident look like a leak: &lt;code&gt;emptyDir&lt;/code&gt; lives as long as the pod, not the container. When the OOM killer restarted the container, the stranded files were still there, so the replacement started its life already carrying them. That is why restarts bunched up instead of clearing the problem.&lt;/p&gt;

&lt;p&gt;To confirm, I delete the stranded files from one pod by hand and watch it. Its memory working set drops from 910 MB to 330 MB and it stops restarting. A second pod I leave alone is killed again at 03:31.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;Two changes. First, the scratch volume goes back to node disk, with an explicit size cap and an ephemeral-storage limit so the kubelet evicts instead of the kernel OOM-killing a healthy process:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;- name: scratch emptyDir: sizeLimit: 2Gi containers: - name: reports-export resources: requests: ephemeral-storage: 1Gi limits: memory: 1Gi ephemeral-storage: 2Gi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second, the code now cleans up on every path, and on startup it wipes anything left in the scratch directory from a previous container:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;runExport&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; \\\&lt;span class="sb"&gt;`/tmp/exports/\\${job.id}.xlsx\\\`; try { await buildWorkbook(job, path); await uploadToS3(path, job.key); } finally { await fs.rm(path, { force: true }); } } // at boot await fs.rm('/tmp/exports', { recursive: true, force: true }); await fs.mkdir('/tmp/exports', { recursive: true });
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I also added a 30-second timeout and two retries on the S3 upload, which is what had been producing the stranded files in the first place.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Impact: 17 restarts across 5 pods between 22:10 Tuesday and 03:40 Wednesday, and 3 month-end exports delayed by about 4 hours. The jobs were re-run from the queue, so nothing was lost.&lt;/li&gt;
&lt;li&gt;  Alert added: container &lt;code&gt;working_set_bytes&lt;/code&gt; divided by limit, plus a separate panel for &lt;code&gt;container_memory_cache&lt;/code&gt; and shmem, so a gap between heap and container memory is visible in one place.&lt;/li&gt;
&lt;li&gt;  A policy check in CI now rejects &lt;code&gt;medium: Memory&lt;/code&gt; on any &lt;code&gt;emptyDir&lt;/code&gt; without a &lt;code&gt;sizeLimit&lt;/code&gt; and a note about what the memory limit has to cover.&lt;/li&gt;
&lt;li&gt;  Runbook line: if OOMKilled and the process heap is flat, read &lt;code&gt;shmem&lt;/code&gt; in the cgroup stats before looking for a leak.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A RAM-backed volume makes a file feel free, and it isn't: it draws from the same 1 GiB as the heap. We spent two hours hunting a leak in code that was fine, while the memory that mattered was sitting in a directory.&lt;/p&gt;

</description>
      <category>programming</category>
      <category>backend</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How BM25 Ranks Search Results, and What k1 and b Actually Do</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Sun, 15 Nov 2026 13:49:02 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-bm25-ranks-search-results-and-what-k1-and-b-actually-do-4faf</link>
      <guid>https://dev.to/darshan_turakhia/how-bm25-ranks-search-results-and-what-k1-and-b-actually-do-4faf</guid>
      <description>&lt;p&gt;If you have ever tuned search relevance in Elasticsearch or OpenSearch, you have tuned BM25 whether you knew it or not. It has been the default scoring function in Lucene since version 6, and the formula is short enough to read in one sitting. I think it is worth doing, because most of the "why did this document rank first?" mysteries disappear once you see what each term does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Starting from TF-IDF
&lt;/h2&gt;

&lt;p&gt;The older idea is TF-IDF. Score a document for a query term by multiplying two numbers: how many times the term appears in the document (term frequency), and how rare the term is across the whole collection (inverse document frequency). A word like "the" appears everywhere, so its IDF is near zero. A word like "memtable" appears in few documents, so a match carries real weight.&lt;/p&gt;

&lt;p&gt;That works until you look at two failure cases. First, a document that says "redis" thirty times scores about ten times higher than one that says it three times, which is rarely what a reader wants. Second, a long document has more chances to contain any given word by accident, so length quietly inflates scores.&lt;/p&gt;

&lt;h2&gt;
  
  
  What BM25 changes
&lt;/h2&gt;

&lt;p&gt;BM25 keeps the IDF half and rewrites the term frequency half:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tf × (k1 + 1) / (tf + k1 × (1 − b + b × len / avglen))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two parameters do the work.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;k1&lt;/code&gt; sets how quickly repetition stops helping. With &lt;code&gt;k1 = 1.2&lt;/code&gt; and an average-length document, a term seen three times gets about 71% of the maximum possible contribution, and a term seen thirty times gets about 96%. The curve flattens. Set &lt;code&gt;k1&lt;/code&gt; to 0 and the term only counts as present or absent.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;b&lt;/code&gt; controls length normalization. At &lt;code&gt;b = 0&lt;/code&gt; a long document is not penalized at all. At &lt;code&gt;b = 1&lt;/code&gt; its term frequencies are scaled fully by how much longer than average it is. The usual default is 0.75.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seeing it on four sentences
&lt;/h2&gt;

&lt;p&gt;Reading the formula is one thing, so I wrote a small calculator where you paste documents and a query and see both scores. Using these four lines and the query "redis caching":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. redis is an in-memory data store used for caching
2. redis cluster shards keys across nodes, and redis sentinel handles failover
3. postgres is a relational database with strong consistency guarantees
4. caching strategies include write-through, write-behind, and cache-aside
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;TF-IDF scores documents 1 and 2 identically, at 1.386. Document 2 never mentions caching, but it says "redis" twice, and raw counts make up the difference. BM25 with default settings scores document 1 at 1.386 and document 2 at 0.927, so the document matching both terms wins. Drop &lt;code&gt;k1&lt;/code&gt; to 0 and document 2 falls to 0.693. Push it to 5 and repetition counts for more again, and document 2 climbs to 1.128.&lt;/p&gt;

&lt;p&gt;One quirk I did not expect: a term that appears in every document gets zero weight under textbook TF-IDF, but a small positive weight under Lucene's BM25 IDF, which is &lt;code&gt;ln(1 + (N − n + 0.5) / (n + 0.5))&lt;/code&gt;. The log argument always stays above 1, so the weight never goes negative.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things that trip people up
&lt;/h2&gt;

&lt;p&gt;Scores are only comparable within one query. A BM25 score of 12 means nothing in isolation, so a fixed cutoff across different queries is a bad idea.&lt;/p&gt;

&lt;p&gt;Statistics are per shard in a distributed engine. With a small index spread over several shards, IDF can differ from shard to shard and shuffle close rankings. Elasticsearch has a distributed-frequency search type for this, at the cost of an extra round trip.&lt;/p&gt;

&lt;p&gt;Tune with evidence. The defaults of 1.2 and 0.75 hold up across many corpora. If you change them, do it against a labeled set of queries. For short fields such as titles, a lower &lt;code&gt;b&lt;/code&gt; often works better, since length differences there say little about relevance.&lt;/p&gt;

&lt;p&gt;BM25 matches literal terms only. "Car" will not find "automobile", which is why many production systems pair it with an embedding index and merge the results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where to go next
&lt;/h2&gt;

&lt;p&gt;I wrote up the longer side-by-side, including a table of the differences, in a &lt;a href="https://nodique.com/comparisons/bm25-vs-tf-idf" rel="noopener noreferrer"&gt;BM25 vs TF-IDF comparison on Nodique&lt;/a&gt;, and the calculator I used above is the &lt;a href="https://nodique.com/tools/bm25-calculator" rel="noopener noreferrer"&gt;BM25 Calculator&lt;/a&gt;, which runs entirely in the browser. Try breaking it: stuff a document with one repeated word, or add a long paragraph of filler, and watch how &lt;code&gt;k1&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt; respond.&lt;/p&gt;

</description>
      <category>database</category>
      <category>algorithms</category>
      <category>search</category>
      <category>programming</category>
    </item>
    <item>
      <title>Why a fast link can still feel slow: TCP congestion control in practice</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Thu, 12 Nov 2026 12:46:11 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/why-a-fast-link-can-still-feel-slow-tcp-congestion-control-in-practice-22ld</link>
      <guid>https://dev.to/darshan_turakhia/why-a-fast-link-can-still-feel-slow-tcp-congestion-control-in-practice-22ld</guid>
      <description>&lt;p&gt;Most backend engineers never touch the TCP congestion control setting on a server. It defaults to something reasonable, and the connection works. Then someone transfers a large file across regions, sees 15 Mbit/s on a link that should do 500, and the question of what the sender is actually doing becomes interesting fast.&lt;/p&gt;

&lt;p&gt;Here is the mental model I use.&lt;/p&gt;

&lt;h2&gt;
  
  
  One number controls everything
&lt;/h2&gt;

&lt;p&gt;A TCP sender keeps a number called the congestion window, &lt;code&gt;cwnd&lt;/code&gt;: how many bytes it may have sent but not yet had acknowledged. It can't see the network, so it guesses. Start small, grow while ACKs come back, shrink when something goes missing.&lt;/p&gt;

&lt;p&gt;Growth has two modes. In slow start the window doubles every round trip, starting around 10 segments on modern stacks. Once it crosses a threshold, &lt;code&gt;ssthresh&lt;/code&gt;, growth turns linear, about one segment per round trip. When loss shows up, the window gets cut. Classic Reno halves it. Plot that over time and you get the familiar sawtooth.&lt;/p&gt;

&lt;p&gt;Throughput is roughly &lt;code&gt;cwnd / RTT&lt;/code&gt;. That one division explains most of the confusing behavior.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a fast link can feel slow
&lt;/h2&gt;

&lt;p&gt;On a 100 Mbit/s path with a 100 ms round trip, filling the pipe takes about 830 segments in flight. Reno halves after a loss and then adds one segment per round trip, so it needs around 400 round trips, roughly 40 seconds, to climb back. If the link loses a packet every few seconds, the connection spends its whole life below capacity.&lt;/p&gt;

&lt;p&gt;Loss rate matters more than people expect. The Mathis approximation puts steady-state throughput at about &lt;code&gt;MSS / (RTT * sqrt(p))&lt;/code&gt;. A hundredfold drop in loss gives roughly a tenfold gain in speed. A 0.1% loss rate on a transatlantic path can cripple a transfer while the link itself sits mostly idle.&lt;/p&gt;

&lt;p&gt;I find it easier to believe this after stepping through it by hand. I put together a small &lt;a href="https://nodique.com/tools/tcp-congestion-window-simulator" rel="noopener noreferrer"&gt;TCP congestion window simulator&lt;/a&gt; where you choose the starting window and which round trips lose a packet. Switching a loss from "three duplicate ACKs" to "timeout" is the instructive bit: the threshold ends up identical, but the timeout resets the window to one segment and you watch it crawl back through slow start.&lt;/p&gt;

&lt;h2&gt;
  
  
  Loss-based versus model-based
&lt;/h2&gt;

&lt;p&gt;Reno and CUBIC both treat a dropped packet as the signal to back off. CUBIC regrows along a cubic curve instead of a line, which recovers much faster on long paths, and it cuts to 70% instead of 50%. It has been the Linux default for years.&lt;/p&gt;

&lt;p&gt;Both share a flaw: they only learn the buffer is full when it overflows. On a link with a deep buffer, that means a standing queue and added latency for everyone sharing it.&lt;/p&gt;

&lt;p&gt;BBR takes a different route. It estimates the bottleneck bandwidth and the minimum round-trip time, then paces its sending to match instead of waiting for loss. On lossy links it can hold throughput where CUBIC collapses, and it tends to keep queues shorter. The catch is fairness. BBR v1 could take more than its share against CUBIC flows and caused heavy retransmission in shallow buffers; later versions use loss and ECN signals to behave better.&lt;/p&gt;

&lt;p&gt;The side-by-side details, including which signals each uses and how they recover, are in this &lt;a href="https://nodique.com/comparisons/tcp-reno-vs-cubic-vs-bbr" rel="noopener noreferrer"&gt;Reno vs CUBIC vs BBR comparison&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would actually do
&lt;/h2&gt;

&lt;p&gt;Leave CUBIC alone unless you have a measured problem. If you do, and you control the sending side of a lossy or high-latency path, try BBR:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sysctl net.ipv4.tcp_congestion_control
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then measure. &lt;code&gt;ss -i&lt;/code&gt; shows live &lt;code&gt;cwnd&lt;/code&gt;, RTT, and retransmits per socket, and &lt;code&gt;iperf3&lt;/code&gt; with &lt;code&gt;-P&lt;/code&gt; runs parallel streams, which is a quick way to check whether a single connection is the limit. Remember the setting only affects data this host sends. Changing a web server to BBR speeds up downloads to clients and does nothing for uploads from them.&lt;/p&gt;

&lt;p&gt;Parallel streams working around a single-flow ceiling is the same reason download accelerators work. Each connection runs its own sawtooth, and their sum gets closer to the link's capacity than one stream does.&lt;/p&gt;

</description>
      <category>networking</category>
      <category>backend</category>
      <category>linux</category>
      <category>performance</category>
    </item>
    <item>
      <title>The One-Second Bug Hiding in Your HTTP Cache</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Mon, 09 Nov 2026 12:43:15 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/the-one-second-bug-hiding-in-your-http-cache-90d</link>
      <guid>https://dev.to/darshan_turakhia/the-one-second-bug-hiding-in-your-http-cache-90d</guid>
      <description>&lt;h1&gt;
  
  
  The One-Second Bug Hiding in Your HTTP Cache
&lt;/h1&gt;

&lt;h2&gt;
  
  
  The bug that only shows up once a second
&lt;/h2&gt;

&lt;p&gt;A few years back I was debugging a CDN cache that kept serving stale content for exactly the kind of resource it should have invalidated instantly: a JSON feed that updated multiple times per minute during a live event. Every other asset on the site cached fine. This one kept handing out a version that was a few seconds old, even right after a fresh write.&lt;/p&gt;

&lt;p&gt;The cause turned out to be embarrassingly simple once I found it: the origin server was validating freshness with &lt;code&gt;Last-Modified&lt;/code&gt; alone, and &lt;code&gt;Last-Modified&lt;/code&gt; only has one-second resolution. If the feed got rewritten twice inside the same second, both writes carried an identical timestamp as far as HTTP was concerned, and a client that showed up in that window had no way to tell the two versions apart. It wasn't a caching bug exactly. It was a precision bug wearing a caching bug's clothes.&lt;/p&gt;

&lt;p&gt;That's the kind of thing you only really internalize once you've been burned by it, so here's the version I wish I'd read first.&lt;/p&gt;

&lt;h3&gt;
  
  
  Two headers, one job
&lt;/h3&gt;

&lt;p&gt;Both &lt;code&gt;ETag&lt;/code&gt; and &lt;code&gt;Last-Modified&lt;/code&gt; exist to answer the same question on a repeat request: has this resource changed since you last saw it, and if not, can the server skip re-sending the body? The server attaches one or both on the first response. The client echoes them back on the next request as &lt;code&gt;If-None-Match&lt;/code&gt; (for &lt;code&gt;ETag&lt;/code&gt;) and &lt;code&gt;If-Modified-Since&lt;/code&gt; (for &lt;code&gt;Last-Modified&lt;/code&gt;). If nothing's changed, the server answers with a &lt;code&gt;304 Not Modified&lt;/code&gt; and no body at all, and the client just reuses what it already has.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Last-Modified&lt;/code&gt; does this with a timestamp. &lt;code&gt;ETag&lt;/code&gt; does it with an opaque token that's usually, though not always, derived from the content itself, like a hash of the response body. That difference in mechanism is where all the interesting behavior comes from.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where the two actually diverge
&lt;/h3&gt;

&lt;p&gt;A timestamp is only ever as precise as its clock, and HTTP dates are specified down to the second. Two writes within the same second are indistinguishable to &lt;code&gt;Last-Modified&lt;/code&gt;, full stop, no matter how different the content actually is. A content-derived &lt;code&gt;ETag&lt;/code&gt; doesn't have that ceiling. Two different byte sequences essentially never collide on a hash, so a change from a millisecond ago is just as visible as one from last year.&lt;/p&gt;

&lt;p&gt;There's also a strength distinction that only &lt;code&gt;ETag&lt;/code&gt; has. A strong &lt;code&gt;ETag&lt;/code&gt; (&lt;code&gt;"abc123"&lt;/code&gt;) promises byte-for-byte identity: two responses sharing one are guaranteed identical all the way down, which matters if you're resuming a partial download with &lt;code&gt;If-Range&lt;/code&gt; and need an exact guarantee. A weak &lt;code&gt;ETag&lt;/code&gt; (&lt;code&gt;W/"abc123"&lt;/code&gt;) only promises semantic equivalence. A server might hand out the same weak token for a gzip-compressed and an uncompressed version of the same logical content, even though the bytes genuinely differ. &lt;code&gt;Last-Modified&lt;/code&gt; can't express either promise; a timestamp only ever supports the weaker claim.&lt;/p&gt;

&lt;p&gt;And when a client sends both headers on the same request, the server is supposed to trust &lt;code&gt;If-None-Match&lt;/code&gt; and ignore &lt;code&gt;If-Modified-Since&lt;/code&gt; entirely rather than trying to split the difference. That's not an arbitrary tie-break. If you can do the more precise check, falling back to the coarser one buys you nothing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Last-Modified hasn't disappeared
&lt;/h3&gt;

&lt;p&gt;Given all that, the obvious question is why anyone still bothers with &lt;code&gt;Last-Modified&lt;/code&gt;. The answer is cost. A file served straight off disk already carries a modification time as filesystem metadata, so the server gets it for free. A genuinely content-derived &lt;code&gt;ETag&lt;/code&gt; means hashing the response body, which isn't free on every single request for a large file. For a case where the resource changes at most once a second and the content is large enough that hashing has a real cost, &lt;code&gt;Last-Modified&lt;/code&gt; alone is still a reasonable choice. It's just a smaller slice of real traffic than it used to be, now that hashing is cheap and most frameworks compute an &lt;code&gt;ETag&lt;/code&gt; automatically anyway.&lt;/p&gt;

&lt;h3&gt;
  
  
  The part that's easy to get wrong in production
&lt;/h3&gt;

&lt;p&gt;If you're running behind multiple replicas, a &lt;code&gt;Last-Modified&lt;/code&gt; timestamp depends on consistent file metadata across all of them. A deploy that recreates a file on disk without actually changing its content can shift the modification time and make the server report a change that never happened. A content-derived &lt;code&gt;ETag&lt;/code&gt; sidesteps this, as long as every replica computes the hash the same way. That consistency is its own trap, though: if one replica's hash includes something request-specific that another's doesn't, you've reintroduced the exact false-mismatch problem you were trying to avoid.&lt;/p&gt;

&lt;p&gt;I ended up writing a small interactive breakdown of the actual precedence rules: what wins when both headers are present, how weak comparison works, and where the one-second blind spot shows up concretely instead of just as an assertion. It's &lt;a href="https://nodique.com/comparisons/etag-vs-last-modified" rel="noopener noreferrer"&gt;over on Nodique&lt;/a&gt; if you want to see the mechanics side by side. There's also a &lt;a href="https://nodique.com/tools/conditional-request-simulator" rel="noopener noreferrer"&gt;simulator&lt;/a&gt; where you can set a server's validators and a client's conditional headers and watch which rule actually fires, which made the precedence order click for me a lot faster than reading the RFC text did.&lt;/p&gt;

&lt;p&gt;Send both headers when it's cheap to compute them. Reach for &lt;code&gt;ETag&lt;/code&gt; alone when precision actually matters: content that can change faster than once a second, or anything served from replicas you can't fully trust to agree on a timestamp.&lt;/p&gt;

</description>
      <category>http</category>
      <category>webdev</category>
      <category>networking</category>
      <category>programming</category>
    </item>
    <item>
      <title>The part of Google's Zanzibar paper everyone skips</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Thu, 05 Nov 2026 13:46:36 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/the-part-of-googles-zanzibar-paper-everyone-skips-3oj4</link>
      <guid>https://dev.to/darshan_turakhia/the-part-of-googles-zanzibar-paper-everyone-skips-3oj4</guid>
      <description>&lt;p&gt;When people explain Google's Zanzibar, they almost always lead with the same idea: permissions as a graph of relationships instead of a role list, so "can Anne edit this doc" becomes "is there a path from Anne to this doc through some chain of group memberships and shares." That's a good idea, and it's also the easy 20% of the 2019 paper, the part you can sketch on a whiteboard in thirty seconds. The other 80%, the part that took years of actual production hardening, is about keeping that graph check both fast and correct while the graph itself is being written to, constantly, by other people's requests. That part almost never makes it into the summary, and it's the more interesting half.&lt;/p&gt;

&lt;p&gt;Here's the specific failure that forces the engineering. Someone gets removed from a group that has access to a sensitive folder. A second later, someone else drops a new file into that folder. If the permission check for the removed person happens to hit a cache that's a few seconds stale, they pass a check for content that didn't even exist when their access was revoked. The paper calls this the "new enemy problem," and the name fits. A stale check can do worse than let someone keep access they already had: it can hand them access to brand-new content that didn't even exist yet when they lost permission, purely because the system answered from information that was already wrong.&lt;/p&gt;

&lt;p&gt;Most systems just accept that risk under the label "eventual consistency" and move on. Zanzibar doesn't get to, because at Google's scale a permission check that's merely "usually correct" amounts to a security hole with a clock attached to it. The fix is a token, called a Zookie, attached to every write against the permission graph. A write returns one; a later check can pass that token in and demand an answer no staler than the data as of that exact write, and the system is built to actually honor that, not just attempt it. The token encodes a timestamp from Spanner's globally synchronized clock, which is what turns "no staler than this" from a best effort into something provable.&lt;/p&gt;

&lt;p&gt;What that buys is almost boring to state plainly: a check carrying a Zookie from a specific removal event is guaranteed to see that removal, every single time, not most of the time. The interesting engineering is less the token itself and more the fact that the whole checking architecture is built around making that guarantee without ever falling back to recomputing everything from scratch on every request, the other honest way to get correctness, and the one that doesn't survive contact with Google-scale traffic.&lt;/p&gt;

&lt;p&gt;There's a second, related problem that sits underneath the first one: checking membership in a group with hundreds of millions of members without walking the entire thing on every request. Zanzibar's answer is a precomputed index called Leopard, kept current by consuming a stream of every change to the permission graph and updating incrementally instead of rebuilding from a cold start. Same philosophy as the Zookie, really: refuse the tradeoff between correctness and speed, and build the infrastructure that earns both instead.&lt;/p&gt;

&lt;p&gt;I went back through the paper recently in more depth than the usual "it's a graph, neat" treatment gives it, mostly chasing down exactly how the Zookie threads through the check algorithm and what Leopard is actually indexing. I wrote up the longer version, including the check algorithm itself, here: &lt;a href="https://nodique.com/guides/what-is-zanzibar" rel="noopener noreferrer"&gt;What is Zanzibar?&lt;/a&gt;. If you're deciding between OpenFGA and SpiceDB, the two open-source systems that implement this model, &lt;a href="https://nodique.com/comparisons/openfga-vs-spicedb" rel="noopener noreferrer"&gt;this comparison&lt;/a&gt; is worth reading before picking one, because the thing that actually differs between them has nothing to do with the relationship-graph data model, both get that part right, and everything to do with how closely each one tracks that same consistency guarantee versus falling back to a plain cache toggle.&lt;/p&gt;

&lt;p&gt;The part of this that generalizes past Zanzibar specifically: when a system gets summarized by its data model, it's worth asking what the data model was actually in service of. The graph wasn't the hard part here. Answering fast, at enormous scale, without ever being allowed to shrug and say "probably correct," was.&lt;/p&gt;

</description>
      <category>security</category>
      <category>distributedsystems</category>
      <category>architecture</category>
      <category>computerscience</category>
    </item>
    <item>
      <title>The Lamport Timestamp Trap: Ordered Numbers Aren't Causality</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Mon, 02 Nov 2026 12:41:40 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/the-lamport-timestamp-trap-ordered-numbers-arent-causality-gb9</link>
      <guid>https://dev.to/darshan_turakhia/the-lamport-timestamp-trap-ordered-numbers-arent-causality-gb9</guid>
      <description>&lt;p&gt;I spent an embarrassing amount of time last year debugging a reconciliation bug that turned out to be a logic error I'd made months earlier, in my own head, about what an ordering guarantee actually promises. The short version: I assumed that because event A's timestamp was smaller than event B's, A must have caused or at least preceded B in some meaningful sense. It hadn't. The two events had nothing to do with each other. They just happened to land on different integers.&lt;/p&gt;

&lt;p&gt;That's the trap built into Lamport timestamps. It's worth understanding precisely because the mechanism is so simple you can hold the whole thing in your head in about thirty seconds, which makes it easy to assume you've understood everything it does.&lt;/p&gt;

&lt;p&gt;Here's the setup. Every node in a distributed system keeps one counter, starting at zero. Two rules update it. Any local event, a write, a log entry, a state change, bumps the counter by one. When a message arrives carrying the sender's counter value, the receiver sets its own counter to one more than the larger of its current value and the value in the message. That's the whole mechanism. No clock synchronization, no coordination protocol, no NTP. Leslie Lamport worked this out in 1978, and almost fifty years later it's still the cheapest way to get a number attached to events in a distributed system that respects causality, at least in one direction.&lt;/p&gt;

&lt;p&gt;That direction is the entire point. If event A happened before event B, meaning a real chain of causation connects them, whether that's two events on the same node in sequence or a message physically carrying A's effects over to wherever B occurs, then A's timestamp comes out smaller than B's. Every time. No exceptions.&lt;/p&gt;

&lt;p&gt;What nobody tells you clearly enough, and what I had to learn the expensive way, is that the guarantee runs in exactly one direction. A smaller number is never proof that two events were related. Two nodes can go their entire operational lives without exchanging a single message, do a different number of local events each, and still end up with timestamps that look perfectly ordered to anyone glancing at a log. There's no causal thread connecting them at all. The numbers differ because counting happened at different rates on different machines, and counting produces an order whether or not the things being counted were ever in a relationship.&lt;/p&gt;

&lt;p&gt;This is exactly where I got bitten: log correlation across services. If you're attaching Lamport timestamps to log lines from multiple systems to interleave them into one readable timeline after the fact, which is a genuinely good use for them (cheaper than full distributed tracing when all you need is a sane-looking sequence), it's tempting to read causal meaning into that sequence. Don't. The ordering is consistent with causality where causality exists. It says nothing reliable about the places it doesn't.&lt;/p&gt;

&lt;p&gt;One more wrinkle if you're building anything that needs a strict total order, say a replicated log where every replica has to apply operations in exactly the same sequence: Lamport timestamps alone can tie. Two unrelated events on different nodes can land on the identical integer. The standard fix is boring and effective: break ties by comparing node IDs alongside the timestamp. That doesn't manufacture a causal relationship out of nothing. It just gives every node an agreed, repeatable way to sort events that would otherwise be ambiguous.&lt;/p&gt;

&lt;p&gt;If you want to watch this gap between "ordered" and "related" happen instead of taking my word for it, &lt;a href="https://nodique.com/guides/what-is-a-lamport-timestamp" rel="noopener noreferrer"&gt;Nodique has a guide on Lamport timestamps&lt;/a&gt; that walks through both rules and the one-directional guarantee in more depth than I have room for here. There's also a &lt;a href="https://nodique.com/tools/lamport-timestamp-simulator" rel="noopener noreferrer"&gt;simulator&lt;/a&gt; that lets you trigger events across a few simulated nodes, send messages between them (or deliberately not), and then pick any two logged events to check whether a real causal link exists underneath the numbers. Set up two nodes, never send a message between them, trigger a couple of local events on each, and compare timestamps. They'll look ordered. They'll be lying to you. Watching that happen on purpose in a sandbox is a much cheaper way to learn this than debugging it in production.&lt;/p&gt;

&lt;p&gt;One caveat worth flagging on the way out: if what you actually need is a way to detect concurrent, conflicting writes rather than just some valid-looking order, a Lamport timestamp is the wrong tool. A vector clock trades the single integer for one counter per node, specifically so it can prove two events were concurrent instead of staying silent on the question. Different problem, different cost, and a decision I've gotten wrong in both directions at different points in my career.&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>computerscience</category>
      <category>programming</category>
      <category>backend</category>
    </item>
    <item>
      <title>How an Eventually Consistent DynamoDB GSI Let Mobile Retries Create 1,400 Duplicate Orders</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Fri, 30 Oct 2026 11:01:28 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-an-eventually-consistent-dynamodb-gsi-let-mobile-retries-create-1400-duplicate-orders-50lg</link>
      <guid>https://dev.to/darshan_turakhia/how-an-eventually-consistent-dynamodb-gsi-let-mobile-retries-create-1400-duplicate-orders-50lg</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;14:07 UTC. PagerDuty fires on &lt;code&gt;fulfillment-queue-depth&lt;/code&gt;, 340% of baseline and climbing. Checkout's on-call pulls up the dashboard expecting a stuck consumer or a slow downstream call. Instead she finds the queue isn't slow, it's just full of orders nobody placed twice on purpose.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory: the mobile app's retry logic, rewritten three weeks ago to handle flaky connections more aggressively, is double-submitting. Someone pulls the client-side request logs for a sample of the duplicate order IDs. Each one shows exactly one tap, one request, one &lt;code&gt;Idempotency-Key&lt;/code&gt; header generated client-side and reused correctly across any retries the client itself issued. The client is behaving. Ruled out in under five minutes.&lt;/p&gt;

&lt;p&gt;Second theory, and the one that eats fifteen minutes: API Gateway retrying a Lambda invocation that looked like it timed out, but actually succeeded after the gateway gave up and resent the request. This would produce exactly this symptom, two invocations for one logical request. CloudWatch says otherwise. Every invocation for the affected order IDs completed in 180-240ms, nowhere near the 29-second timeout, and API Gateway's own access logs show one request in, one request out, for every one of them. No retries at that layer either.&lt;/p&gt;

&lt;p&gt;Third, and this is the thread that actually goes somewhere: someone notices every duplicate pair shares the same &lt;code&gt;idempotencyKey&lt;/code&gt; attribute but has two different &lt;code&gt;order_id&lt;/code&gt; values. If the idempotency check were working at all, that shouldn't be possible. So the check itself becomes the suspect.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;The checkout Lambda's create-order path looks, on paper, like textbook idempotency handling: look up the key, bail out if it exists, write if it doesn't.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;

&lt;span class="nx"&gt;checkout&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;api&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;createOrder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;as &lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt; &lt;span class="nx"&gt;shipped&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;const existing = await ddb.send(new QueryCommand({&lt;br&gt;
  TableName: 'orders',&lt;br&gt;
  IndexName: 'idempotency-key-index', // GSI, partition key: idempotencyKey&lt;br&gt;
  KeyConditionExpression: 'idempotencyKey = :k',&lt;br&gt;
  ExpressionAttributeValues: { ':k': { S: idempotencyKey } },&lt;br&gt;
}));&lt;/p&gt;

&lt;p&gt;if (existing.Items &amp;amp;&amp;amp; existing.Items.length &amp;gt; 0) {&lt;br&gt;
  return { orderId: existing.Items[0].order_id.S, duplicate: true };&lt;br&gt;
}&lt;/p&gt;

&lt;p&gt;const orderId = crypto.randomUUID();&lt;br&gt;
await ddb.send(new PutItemCommand({&lt;br&gt;
  TableName: 'orders',&lt;br&gt;
  Item: { order_id: { S: orderId }, idempotencyKey: { S: idempotencyKey }, /* ... */ },&lt;br&gt;
}));&lt;br&gt;
return { orderId, duplicate: false };&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing here looks wrong until you ask one specific question: what consistency guarantee does that &lt;code&gt;Query&lt;/code&gt; against a Global Secondary Index actually carry? The answer, in DynamoDB, is none. GSIs are updated asynchronously from the base table and only support eventually consistent reads, full stop, there is no &lt;code&gt;ConsistentRead: true&lt;/code&gt; option for a GSI the way there is for the base table. AWS documents this plainly. Nobody on the team had needed to think about it before, because nothing had ever hit the index hard enough, fast enough, for the gap to matter.&lt;/p&gt;

&lt;p&gt;14:07 UTC lines up with a known mobile carrier blip in one metro area. Retry logs on the client are clean, as established, but the client's retry backoff for this release is 300ms, and CloudWatch Contributor Insights shows GSI propagation delay for the &lt;code&gt;orders&lt;/code&gt; table spiking to 400-600ms during the same window, well above its normal sub-100ms. Two legitimate-looking requests, 300ms apart, both querying a GSI that hadn't caught up from the first request yet.&lt;/p&gt;

&lt;p&gt;Reproducing it in staging confirms the mechanism: five identical create-order calls fired 250ms apart, same idempotency key, against a GSI primed to lag by artificially spiking write volume on the table beforehand.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

staging repro, 5 calls / 250ms apart / same key

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;$ for i in 1 2 3 4 5; do&lt;br&gt;
    aws lambda invoke --function-name checkout-create-order \&lt;br&gt;
      --payload '{"idempotencyKey":"repro-001"}' /tmp/out-$i.json &amp;amp;&lt;br&gt;
    sleep 0.25&lt;br&gt;
  done; wait&lt;/p&gt;

&lt;p&gt;$ grep -h order_id /tmp/out-*.json | sort -u&lt;br&gt;
"order_id":"a1c9..."&lt;br&gt;
"order_id":"a1c9..."&lt;br&gt;
"order_id":"f02e..."   # duplicate&lt;br&gt;
"order_id":"a1c9..."&lt;br&gt;
"order_id":"a1c9..."&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;Invocation 3 queried the GSI before invocation 1's write had propagated to it, saw zero matching items, and concluded, correctly given what it could see, that no order existed yet. It wasn't a bug in the retry logic, the Lambda runtime, or API Gateway. It was a correctness check built on a read path that DynamoDB never promised would be current. The code asked "does this exist yet?" of a view of the data that could lag the thing it was trying to protect against.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;The fix isn't a faster GSI, you can't request strong consistency on one at any price. It's moving the correctness check off the index entirely and onto the one place DynamoDB actually gives you atomicity: a conditional write against the base table (or, cleaner, a dedicated table keyed directly by the idempotency key).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;

&lt;span class="nx"&gt;checkout&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;api&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;createOrder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;after&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;const orderId = crypto.randomUUID();&lt;/p&gt;

&lt;p&gt;try {&lt;br&gt;
  await ddb.send(new PutItemCommand({&lt;br&gt;
    TableName: 'idempotency-locks',&lt;br&gt;
    Item: {&lt;br&gt;
      idempotencyKey: { S: idempotencyKey },&lt;br&gt;
      order_id: { S: orderId },&lt;br&gt;
      ttl: { N: String(Math.floor(Date.now() / 1000) + 86400) },&lt;br&gt;
    },&lt;br&gt;
    ConditionExpression: 'attribute_not_exists(idempotencyKey)',&lt;br&gt;
  }));&lt;br&gt;
} catch (err) {&lt;br&gt;
  if (err.name === 'ConditionalCheckFailedException') {&lt;br&gt;
    const existing = await ddb.send(new GetItemCommand({&lt;br&gt;
      TableName: 'idempotency-locks',&lt;br&gt;
      Key: { idempotencyKey: { S: idempotencyKey } },&lt;br&gt;
      ConsistentRead: true, // base table, strong reads are available here&lt;br&gt;
    }));&lt;br&gt;
    return { orderId: existing.Item!.order_id.S, duplicate: true };&lt;br&gt;
  }&lt;br&gt;
  throw err;&lt;br&gt;
}&lt;/p&gt;

&lt;p&gt;await ddb.send(new PutItemCommand({ TableName: 'orders', Item: { order_id: { S: orderId }, /* ... */ } }));&lt;br&gt;
return { orderId, duplicate: false };&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A conditional &lt;code&gt;PutItem&lt;/code&gt; on a base table is atomic: when two invocations race for the same key, exactly one succeeds and the other gets &lt;code&gt;ConditionalCheckFailedException&lt;/code&gt; back, deterministically, every time, regardless of replication lag anywhere else in the system. The loser reads the winner's &lt;code&gt;order_id&lt;/code&gt; off the same strongly-consistent base table and returns that instead of minting a new one. The TTL attribute lets DynamoDB expire old lock rows automatically after 24 hours instead of growing the table forever.&lt;/p&gt;

&lt;p&gt;Two more changes went in alongside it. A CloudWatch alarm on &lt;code&gt;ConditionalCheckFailedException&lt;/code&gt; rate for the &lt;code&gt;idempotency-locks&lt;/code&gt; table, so the next retry storm shows up as a metric spike within a minute instead of a downstream queue-depth alert twenty minutes later. And a reconciliation script to walk the affected window and merge or cancel the orders that had already been created in duplicate before the fix shipped.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;p&gt;1,400 duplicate orders created in the 22-minute window&lt;/p&gt;

&lt;p&gt;$61,800 fulfillment cost on duplicates not caught before pick-and-pack&lt;/p&gt;

&lt;p&gt;1,190 of the 1,400 duplicates were caught by the reconciliation script before fulfillment touched them. The remaining 210 had already been picked, packed, or shipped by the time the script ran, and became the bulk of the $61,800, split between return shipping and goodwill refunds for customers who'd received two of something they ordered once.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A Global Secondary Index in DynamoDB never supports strongly consistent reads, by design, and no configuration changes that. Querying one as a performance shortcut is fine. Querying one as a correctness check is a race condition waiting for enough concurrency to expose it.&lt;/li&gt;
&lt;li&gt;  Idempotency belongs on a conditional write against a base table, where DynamoDB actually gives you atomicity, not on a read-then-write pattern against any index, GSI or otherwise. The read and the write need to be the same operation, or there's a window between them for another invocation to run.&lt;/li&gt;
&lt;li&gt;  The race needed concurrency to show up, and the concurrency came from somewhere unrelated to the database: a retry policy tuned for a flaky network, three weeks earlier, by a team that had no reason to think about GSI propagation lag at all.&lt;/li&gt;
&lt;li&gt;  Alarming on the symptom (fulfillment queue depth) found this twenty minutes in. Alarming on the mechanism (&lt;code&gt;ConditionalCheckFailedException&lt;/code&gt; rate, which is now expected to show nonzero values during legitimate retries) would have shown the retry storm starting in under a minute.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nobody had written the original check carelessly. A query that filters on the exact key you're about to insert reads like a correctness guarantee in every language and every database that doesn't make an explicit distinction between "consistent" and "eventually consistent" read paths. DynamoDB does make that distinction, loudly, in its own documentation. The gap wasn't missing diligence, it was a guarantee that existed in the docs and not in the code.&lt;/p&gt;

</description>
      <category>database</category>
      <category>postgres</category>
      <category>backend</category>
      <category>programming</category>
    </item>
    <item>
      <title>How OpenTelemetry</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Tue, 27 Oct 2026 11:00:52 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-opentelemetry-npj</link>
      <guid>https://dev.to/darshan_turakhia/how-opentelemetry-npj</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;14:32 UTC. PagerDuty fires on 5xx rate for &lt;code&gt;/checkout/complete&lt;/code&gt;. The on-call engineer opens the trace dashboard to find the slow dependency. It's green. No red spans, no latency bump, nothing flagged in the service map. The load balancer says the checkout service is failing. Tracing says everything is fine.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory: a bad deploy. The deploy log shows nothing shipped in the last six hours. Second theory: the payment gateway is slow again, it's happened before. The gateway's own status page is green and our circuit breaker metrics show zero trips. Third theory, the one everyone wants to believe: it's a blip, give it two minutes and watch it clear.&lt;/p&gt;

&lt;p&gt;It doesn't clear. 5xx count on &lt;code&gt;/checkout/complete&lt;/code&gt; goes from a baseline of about two per minute to 1,900 over nine minutes. A marketing push notification had gone out to 400,000 users twelve minutes earlier, and checkout traffic was running at roughly 6x normal. Nobody in the incident channel is arguing it's noise anymore. The trace dashboard, the thing built specifically to answer "where is this slow," is still just sitting there green.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;With tracing giving nothing to go on, the hunt moves to raw signals. Application logs (not spans, actual log lines) show a climbing count of a specific exception on the checkout path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

checkout service logs, grepped for the error

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;asyncpg.exceptions.TooManyConnectionsError: too many connections for role "checkout_svc"&lt;br&gt;
  (occurred 340 times in the last 6 minutes)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A connection pool exhaustion. Someone runs a query directly against Postgres to see what's actually holding connections open.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

pg\_stat\_activity, filtered to the checkout role

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SELECT pid, state, now() - query_start AS duration, query&lt;br&gt;
FROM pg_stat_activity&lt;br&gt;
WHERE usename = 'checkout_svc' AND state != 'idle'&lt;br&gt;
ORDER BY duration DESC LIMIT 5;&lt;/p&gt;

&lt;p&gt;pid    | state               | duration | query&lt;br&gt;
  18422  | idle in transaction | 00:00:05 | SELECT ... FOR UPDATE&lt;br&gt;
  18390  | idle in transaction | 00:00:05 | SELECT ... FOR UPDATE&lt;br&gt;
  18401  | idle in transaction | 00:00:04 | SELECT ... FOR UPDATE&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Dozens of connections sitting &lt;code&gt;idle in transaction&lt;/code&gt; for four to five seconds each. The checkout handler wraps a row lock on the order table in the same transaction as a call to a third-party inventory and fraud-check API, a call that normally returns in 80ms and, under the spike's downstream load, was taking four to six seconds. Every slow inventory call held a database connection open and idle for the full duration. With a pool of 20 connections per pod across 14 pods, it didn't take many of these stacking up at once to starve everything else on the checkout path.&lt;/p&gt;

&lt;p&gt;That explains the outage. It still doesn't explain why tracing showed nothing for nine minutes of a real, 1,900-error spike. That question gets parked for the retro, because the immediate fix (move the external call outside the transaction, add a hard timeout) stops the bleeding and the team moves on to writing the postmortem.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;During the retro, someone pulls the raw request count from the load balancer access logs for the 14:32 to 14:55 window and compares it against the span count ingested by Tempo for the checkout service over the same window.&lt;/p&gt;

&lt;p&gt;94,612 Requests logged by the load balancer for the checkout service during the window&lt;/p&gt;

&lt;p&gt;11,480 Spans actually ingested by Tempo for the same service, same window&lt;/p&gt;

&lt;p&gt;Roughly 88% of the traffic that hit checkout during the outage has no trace at all. The gap starts almost exactly ninety seconds into the spike, right around when the queue behind the OpenTelemetry &lt;code&gt;BatchSpanProcessor&lt;/code&gt; would have filled on its default settings.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;

&lt;span class="n"&gt;tracing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;py &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;before&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;from opentelemetry.sdk.trace.export import BatchSpanProcessor&lt;/p&gt;

&lt;p&gt;processor = BatchSpanProcessor(&lt;br&gt;
    otlp_exporter,&lt;br&gt;
    max_queue_size=2048,&lt;br&gt;
    max_export_batch_size=512,&lt;br&gt;
    schedule_delay_millis=5000,&lt;br&gt;
)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At 6x traffic, span creation outran what the exporter could drain to the collector. The collector sat behind the same OTLP endpoint that was itself catching some of the spike's traffic, so it was draining slower right when it needed to drain faster. Once the queue hit 2048, the processor started refusing new spans outright. It logs a warning the first time this happens, at the Python &lt;code&gt;logging&lt;/code&gt; module's WARNING level. It does not log it again for every subsequent drop, because the service's log pipeline runs a dedup filter on repeated identical lines to cut noise. The one warning that would have told anyone tracing had gone blind fired once, got suppressed as a duplicate, and the drops kept happening silently for the rest of the incident.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;Two separate problems needed two separate fixes. The application bug: the inventory check moved outside the database transaction, with its own timeout, so a slow third-party call can no longer hold a connection hostage.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;

&lt;span class="n"&gt;checkout&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;async def complete_order(order_id: str) -&amp;gt; OrderResult:&lt;br&gt;
    inventory_result = await asyncio.wait_for(&lt;br&gt;
        check_inventory_and_fraud(order_id), timeout=1.5&lt;br&gt;
    )&lt;br&gt;
    async with db.transaction():&lt;br&gt;
        order = await lock_order_for_update(order_id)&lt;br&gt;
        apply_inventory_result(order, inventory_result)&lt;br&gt;
        await order.save()&lt;br&gt;
    return OrderResult(order)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The observability gap needed a fix that didn't depend on anyone noticing a suppressed warning line. Instead of trusting the SDK's internal counters, the team wired an alert directly off the signal that actually caught this: the gap between load balancer request count and ingested span count for the same service and window.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;

&lt;span class="n"&gt;alert&lt;/span&gt; &lt;span class="n"&gt;rule&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Prometheus&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;style&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;alert: TraceCoverageGap
expr: |
(sum(rate(alb_requests_total{service="checkout"}[5m]))

&lt;ul&gt;
&lt;li&gt;sum(rate(otel_spans_ingested_total{service="checkout"}[5m])))
/ sum(rate(alb_requests_total{service="checkout"}[5m])) &amp;gt; 0.15
for: 2m
annotations:
summary: "Checkout requests are outrunning ingested trace coverage"
&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The queue itself got widened too, &lt;code&gt;max_queue_size&lt;/code&gt; raised to 8192 and &lt;code&gt;schedule_delay_millis&lt;/code&gt; dropped to 2000 so the exporter drains more often under burst, but the team treated that as a buffer, not a fix. A bigger queue just moves the point where it fills. The log dedup filter was also changed to let repeated WARNING-level lines from the tracing SDK through after the third occurrence instead of suppressing indefinitely.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  A trace pipeline degrades under exactly the kind of load spike it exists to help diagnose, because span volume scales with request volume, and the exporter's throughput doesn't automatically scale with either.&lt;/li&gt;
&lt;li&gt;  A monitoring tool failing silent and green is more dangerous than one failing loud, because it actively points the hunt in the wrong direction instead of just leaving a blank.&lt;/li&gt;
&lt;li&gt;  Log dedup filters tuned to cut noise can suppress the exact one-time warning that would have shortened an incident, if that warning only fires once per condition instead of once per occurrence.&lt;/li&gt;
&lt;li&gt;  The most reliable way to catch a tracing pipeline lying isn't inside the tracing pipeline. It's comparing it against a source that doesn't share its failure mode, like raw load balancer request counts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The checkout fix was the kind of bug that's obvious once found, a slow call sitting inside a transaction it never needed to be in. The nine minutes it took to find it without any tracing to lean on is the part that changed how the team treats observability itself: not as a tool that's either working or broken, but as a system with its own capacity limits that can fail exactly when the traffic that needs it shows up.&lt;/p&gt;

</description>
      <category>python</category>
      <category>programming</category>
      <category>backend</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How an Unpinned Base Image Tag Let V8 Size Its Heap Off the Host, Not the Pod</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Fri, 23 Oct 2026 11:21:11 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-an-unpinned-base-image-tag-let-v8-size-its-heap-off-the-host-not-the-pod-5744</link>
      <guid>https://dev.to/darshan_turakhia/how-an-unpinned-base-image-tag-let-v8-size-its-heap-off-the-host-not-the-pod-5744</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;14:47 UTC, Tuesday. Pods in the &lt;code&gt;checkout-api&lt;/code&gt; deployment start OOMKilling in a loop, three or four at a time, in the middle of an ordinary afternoon. Traffic is flat. No deploy went out today. PagerDuty fires on &lt;code&gt;KubePodCrashLooping&lt;/code&gt;, not on anything tied to a release, because as far as the deploy history shows, nothing was released.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory: a memory leak shipped yesterday and took a day to build up enough pressure to matter. &lt;code&gt;git log --since=48h -- apps/checkout-api&lt;/code&gt; shows exactly one commit, a routine lockfile refresh from the nightly dependency job. No application code changed in either direction.&lt;/p&gt;

&lt;p&gt;Second theory: a traffic burst nobody's dashboard caught. Request rate for the service is within 3% of the same Tuesday last week. Whatever's killing these pods, it isn't load.&lt;/p&gt;

&lt;p&gt;Third theory: a slow leak finally crossing a threshold. The service exposes a &lt;code&gt;/debug/memory&lt;/code&gt; endpoint that dumps &lt;code&gt;process.memoryUsage()&lt;/code&gt; on demand. &lt;code&gt;heapUsed&lt;/code&gt; sits at a boring 140MB, same as it's sat for weeks. &lt;code&gt;rss&lt;/code&gt;, meanwhile, is climbing steadily past 480MB on a pod with a 512Mi limit and hasn't leveled off. Whatever's growing, V8's own heap accounting doesn't think it's growing.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;A gap between &lt;code&gt;heapUsed&lt;/code&gt; and &lt;code&gt;rss&lt;/code&gt; usually means something outside V8's managed heap: a native addon, an unreleased buffer, a leaking file descriptor. That's the next hour, chasing a native leak. &lt;code&gt;clinic doctor&lt;/code&gt; against a staging replay shows nothing unusual. No native module in the dependency tree changed in the lockfile refresh. Dead end.&lt;/p&gt;

&lt;p&gt;The clue that actually breaks it open is which pods die fastest. Traffic is round-robin across all pods, cgroup memory limits are identical at 512Mi, and yet pods scheduled onto the cluster's larger nodes survive noticeably longer before OOMKilling than pods on the smaller ones. That shouldn't matter. A container's enforced limit is supposed to be the same number regardless of what the underlying node has installed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

exec into two pods, same deployment, different nodes

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h1&gt;
  
  
  pod on a 16GB node
&lt;/h1&gt;

&lt;p&gt;$ kubectl exec checkout-api-7f9d-a1 -- node -e \&lt;br&gt;
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"&lt;br&gt;
2187.4&lt;/p&gt;
&lt;h1&gt;
  
  
  pod on a 64GB node
&lt;/h1&gt;

&lt;p&gt;$ kubectl exec checkout-api-7f9d-b2 -- node -e \&lt;br&gt;
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"&lt;br&gt;
8341.9&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Neither number is close to 512. Both are absurd for a container capped at 512Mi, and they scale with the node's physical memory, not with the pod's cgroup limit, which is the opposite of what should happen. A canary pod still running the previous image, from before last night's rebuild, gives a third data point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

same check against the pre-rebuild canary

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;$ kubectl exec checkout-api-canary-prev -- node -e \&lt;br&gt;
  "console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"&lt;br&gt;
462.8&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;462MB, correctly tracking the 512Mi limit regardless of which node it lands on. The canary and the failing pods are running the same application code. The only thing that differs is the image.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

diffing the two images

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;$ docker inspect checkout-api:canary-prev --format '{{.Config.Image}}'&lt;br&gt;
node:18.19.1-alpine&lt;/p&gt;

&lt;p&gt;$ docker inspect checkout-api:latest --format '{{.Config.Image}}'&lt;br&gt;
node:18.20.4-alpine&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Dockerfile's first line has always read &lt;code&gt;FROM node:18-alpine&lt;/code&gt;, a floating tag, never a pinned version. The nightly CI job that rebuilds this image runs &lt;code&gt;docker build --pull&lt;/code&gt;, which fetches whatever &lt;code&gt;node:18-alpine&lt;/code&gt; currently resolves to on every run. Somewhere between two nights ago and last night, that tag moved from 18.19.1 to 18.20.4. No commit, no changelog line, no PR. The base image changed underneath the Dockerfile without the Dockerfile changing at all.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;Root cause: Node determines its default heap ceiling by probing the container's cgroup memory limit through libuv, and falling back to the host's total memory if that probe fails. The 18.20.4 alpine layer repackaged its cgroup compatibility shims in a way that changed which path libuv checks first on this cluster's cgroup v2 nodes. On 18.19.1 the probe found &lt;code&gt;memory.max&lt;/code&gt; and sized the heap off 512Mi, as intended. On 18.20.4 the probe missed and silently fell back to &lt;code&gt;os.totalmem()&lt;/code&gt;, which reports whatever the underlying node actually has, 16GB on some nodes and 64GB on others.&lt;/p&gt;

&lt;p&gt;V8 isn't wrong about the number it was given. It's sizing its default old-space target off a number that has nothing to do with what the container is actually allowed to use. GC pressure heuristics only kick in as the heap approaches its perceived limit, so with a limit in the gigabytes, V8 kept committing more memory well past 512Mi, confident it had plenty of room left. The kernel's cgroup OOM killer doesn't consult V8's opinion. It watches RSS against the pod's actual 512Mi ceiling and sends SIGKILL the moment that's crossed, regardless of what &lt;code&gt;heapUsed&lt;/code&gt; reports.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;Pin the base image to a digest, not a tag, so a rebuild can never silently change what ships:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;

Dockerfile, before and after

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h1&gt;
  
  
  before
&lt;/h1&gt;

&lt;p&gt;FROM node:18-alpine&lt;/p&gt;

&lt;h1&gt;
  
  
  after
&lt;/h1&gt;

&lt;p&gt;FROM node:18.19.1-alpine@sha256:4a7a4e52c40e973c3e4b02f0c9d8d6c7f8e1a0b2c3d4e5f6a7b8c9d0e1f2a3b4&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second, stop letting V8 guess at a limit it can get wrong. The pod's memory limit is known at build time via Kubernetes resource specs, so the heap ceiling gets set explicitly instead of auto-detected:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;

&lt;span class="s"&gt;deployment.yaml&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;env:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;name: NODE_OPTIONS
value: "--max-old-space-size=384"
resources:
limits:
memory: "512Mi"
requests:
memory: "512Mi"
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;384MB leaves roughly 25% of the 512Mi limit for stack space, native allocations, and whatever else lives outside V8's managed heap, the same gap that let &lt;code&gt;rss&lt;/code&gt; outrun &lt;code&gt;heapUsed&lt;/code&gt; during the incident. With the ceiling fixed explicitly, a future base image change can't move it again.&lt;/p&gt;

&lt;p&gt;Third, the CI job that builds this image now fails loudly on an unpinned &lt;code&gt;FROM&lt;/code&gt; line instead of silently rebuilding against whatever a tag resolves to that day:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;

ci check, pinned-base-image.sh

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;if grep -qE '^FROM [a-z0-9./-]+:[a-zA-Z0-9._-]+$' Dockerfile &amp;amp;&amp;amp; \&lt;br&gt;
   ! grep -q '&lt;a class="mentioned-user" href="https://dev.to/sha256"&gt;@sha256&lt;/a&gt;:' Dockerfile; then&lt;br&gt;
  echo "FROM line is not pinned to a digest, failing build"&lt;br&gt;
  exit 1&lt;br&gt;
fi&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;p&gt;61 OOMKill restarts across 9 pods in the 2h10m window before mitigation&lt;/p&gt;

&lt;p&gt;18,400 Checkout requests that failed with 502 or 503 during the incident&lt;/p&gt;

&lt;p&gt;8.3GB V8's self-reported heap_size_limit on a pod capped at 512Mi&lt;/p&gt;

&lt;p&gt;0 App code commits in the 48 hours before the first OOMKill&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A floating base image tag is a deploy that never shows up in the deploy history. The Dockerfile didn't change, the git log didn't change, and the thing that actually shipped a behavioral difference into production left no trace anyone was watching.&lt;/li&gt;
&lt;li&gt;  &lt;code&gt;heap_size_limit&lt;/code&gt; from &lt;code&gt;v8.getHeapStatistics()&lt;/code&gt; should always track the container's actual memory limit. If it varies by node instead of by pod spec, Node's cgroup detection isn't seeing what you think it's seeing.&lt;/li&gt;
&lt;li&gt;  &lt;code&gt;heapUsed&lt;/code&gt; staying flat while &lt;code&gt;rss&lt;/code&gt; climbs doesn't always mean a native leak. It can mean V8 is committing memory toward a target that was never the real limit in the first place.&lt;/li&gt;
&lt;li&gt;  Pinning a base image by digest and setting &lt;code&gt;--max-old-space-size&lt;/code&gt; explicitly both remove the same failure mode: a runtime deciding its own resource ceiling from a number nobody in the deploy pipeline actually chose.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nothing in this incident involved a bug in the application. The code that ran at 14:47 UTC was identical to the code that had run cleanly for weeks. What changed was three layers down, in a base image tag nobody was watching, and it took exec-ing into pods on different physical nodes to even notice the one variable that should never have mattered was the variable that did.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>containers</category>
      <category>programming</category>
    </item>
    <item>
      <title>How a 30-Second Pod Grace Period Lost the Race Against a 300-Second ALB Deregistration Delay</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Tue, 20 Oct 2026 11:01:20 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-a-30-second-pod-grace-period-lost-the-race-against-a-300-second-alb-deregistration-delay-e7</link>
      <guid>https://dev.to/darshan_turakhia/how-a-30-second-pod-grace-period-lost-the-race-against-a-300-second-alb-deregistration-delay-e7</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;14:32 UTC. PagerDuty fires on the checkout service's error-rate alarm. Datadog shows the 502 rate on &lt;code&gt;POST /checkout&lt;/code&gt; jumping from a baseline of 0.02% to 4.8%, right in the middle of a rolling deploy that had shipped a dozen times before without anyone noticing it. By 14:36 the deploy finishes, the 502s stop, and the dashboard looks normal again. Nobody asked for this one to rotate twelve pods for a routine dependency bump, and yet here it was, the third deploy that week to show the exact same spike, each time clean before and after, dirty for about four minutes in the middle.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory in #incidents: the new image is bad. That's the obvious read when a deploy and an error spike land on top of each other. Someone pulls up the new revision's pod logs looking for stack traces, crash loops, anything. Nothing. Zero restarts across all twelve pods. Zero 5xx logged by the app itself. Whatever's producing these 502s, it isn't the checkout service throwing an error, because the checkout service never saw the request at all.&lt;/p&gt;

&lt;p&gt;Second theory: a health check regression is marking targets unhealthy mid-rollout and the ALB is handling the transition badly. Someone checks the target group in the AWS console. Zero unhealthy targets logged for the entire deploy window. By every signal AWS itself can show, nothing was ever wrong with any target. That's the part that stalls the room for a few minutes, because if the load balancer never thought a target was unhealthy, where are the 502s even coming from?&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;With both easy theories dead, someone pulls the ALB's own access logs from S3 and greps for the deploy window instead of trusting the target-group health view. ALB access logs record a &lt;code&gt;target_status_code&lt;/code&gt; of &lt;code&gt;-&lt;/code&gt; for exactly one case: it picked a target, tried to open the connection, and got nothing back, connection refused or reset. That's a different failure mode than "target failed its health check," and it doesn't show up on the target group's health dashboard at all.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

alb access logs, filtered to the deploy window

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;aws s3 cp s3://alb-logs/checkout/2026/10/20/ . --recursive&lt;br&gt;
zcat *.log.gz | awk '$8 == "-" &amp;amp;&amp;amp; $6 ~ /checkout/' | cut -d' ' -f4,8,14&lt;/p&gt;

&lt;p&gt;14:33:02.418481Z - 10.4.21.117:3000&lt;br&gt;
14:33:04.901223Z - 10.4.21.117:3000&lt;br&gt;
14:33:05.550112Z - 10.4.21.117:3000&lt;br&gt;
14:34:11.009887Z - 10.4.21.94:3000&lt;br&gt;
14:34:12.733410Z - 10.4.21.94:3000&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every failed request maps to a target IP, and those IPs belong to pods. Cross-reference that against &lt;code&gt;kubectl get events&lt;/code&gt; history for the same window: each burst of failures lines up within a handful of seconds of that exact pod's termination. &lt;code&gt;10.4.21.117&lt;/code&gt; got its &lt;code&gt;SIGTERM&lt;/code&gt; at 14:33:00. The last 502 mapped to it is at 14:33:05.55, five seconds after the process was told to shut down.&lt;/p&gt;

&lt;p&gt;That's the actual question now: why was the ALB still sending live traffic to a pod five seconds after Kubernetes told it to die? Checking the AWS Load Balancer Controller's own logs for the same window answers it. The controller doesn't poll for pod state, it watches EndpointSlice updates and reacts by calling &lt;code&gt;DeregisterTargets&lt;/code&gt; on the target group. The controller's reconcile loop timestamps show a four-to-seven-second gap between the pod going NotReady and the target group actually being told to drop that IP. The checkout app had no &lt;code&gt;SIGTERM&lt;/code&gt; handler, so Node's default behavior kicked in: no listener means the process exits almost immediately, well under a second, as soon as the signal arrives.&lt;/p&gt;

&lt;p&gt;The pod was dying in under a second. The signal that it was dying took four to seven seconds to reach the one system still routing traffic to it. The 300-second &lt;code&gt;deregistration_delay.timeout_seconds&lt;/code&gt; on the target group, set months earlier to drain long-lived connections gracefully once a target is actually marked for removal, was never the problem and never fired, because the controller hadn't called &lt;code&gt;DeregisterTargets&lt;/code&gt; yet when the pod went away. That setting governs what happens after deregistration starts. It has nothing to say about the gap before it starts.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;Root cause: the checkout app shut down faster than Kubernetes' own control plane could tell the load balancer it was going away. A bare &lt;code&gt;SIGTERM&lt;/code&gt; with no handler killed the process in under a second. In this cluster it took four to seven seconds for that pod's IP to drop from the EndpointSlice, for the AWS Load Balancer Controller to notice, and for &lt;code&gt;DeregisterTargets&lt;/code&gt; to actually fire on the ALB. Every request the ALB routed to that pod during that gap hit a socket nobody was listening on anymore.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;Two changes, working together. First, the app needed to survive &lt;code&gt;SIGTERM&lt;/code&gt; long enough to stop accepting new connections on its own terms instead of just dying:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

checkout service, graceful shutdown handler

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;const server = app.listen(3000);&lt;/p&gt;

&lt;p&gt;process.on('SIGTERM', () =&amp;gt; {&lt;br&gt;
  console.log('SIGTERM received, draining connections');&lt;br&gt;
  server.close(() =&amp;gt; {&lt;br&gt;
    console.log('server closed, no in-flight requests remain');&lt;br&gt;
    process.exit(0);&lt;br&gt;
  });&lt;/p&gt;

&lt;p&gt;// belt and suspenders: force exit if close() hangs on a stuck connection&lt;br&gt;
  setTimeout(() =&amp;gt; process.exit(1), 25_000).unref();&lt;br&gt;
});&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second, and this is the half that actually fixes the race: the pod needed to delay sending &lt;code&gt;SIGTERM&lt;/code&gt; in the first place, long enough for the EndpointSlice update and the ALB controller's deregister call to actually land before the process started shutting down.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;

&lt;span class="s"&gt;deployment.yaml, checkout container&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;spec:&lt;br&gt;
  terminationGracePeriodSeconds: 45&lt;br&gt;
  containers:&lt;br&gt;
    - name: checkout&lt;br&gt;
      lifecycle:&lt;br&gt;
        preStop:&lt;br&gt;
          exec:&lt;br&gt;
            command: ["sh", "-c", "sleep 10"]&lt;br&gt;
      readinessProbe:&lt;br&gt;
        httpGet:&lt;br&gt;
          path: /healthz&lt;br&gt;
          port: 3000&lt;br&gt;
        periodSeconds: 5&lt;br&gt;
        failureThreshold: 1&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;preStop&lt;/code&gt; hook runs before &lt;code&gt;SIGTERM&lt;/code&gt; is ever sent to the container, and the pod is marked &lt;code&gt;NotReady&lt;/code&gt; the moment termination begins, well before the hook finishes sleeping. Ten seconds was comfortably above the four-to-seven-second gap measured in the controller logs. &lt;code&gt;terminationGracePeriodSeconds&lt;/code&gt; went up to 45 so the sleep plus the app's own graceful shutdown both have room to finish before Kubernetes gives up and sends &lt;code&gt;SIGKILL&lt;/code&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;p&gt;4.8% Peak 502 rate on checkout during the deploy, against a 0.02% baseline&lt;/p&gt;

&lt;p&gt;4-7s Measured gap between a pod going NotReady and the ALB controller deregistering its target&lt;/p&gt;

&lt;p&gt;1,240 Failed checkout POSTs across the 14-minute rolling deploy before the fix&lt;/p&gt;

&lt;p&gt;0 Failed requests observed across the next 30 deploys after the preStop hook shipped&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A target group's &lt;code&gt;deregistration_delay&lt;/code&gt; only governs draining after a target has been told it's leaving. It does nothing about the time it takes the control plane to decide and propagate that a target is leaving in the first place, and that gap is where this bug lived.&lt;/li&gt;
&lt;li&gt;  A process with no &lt;code&gt;SIGTERM&lt;/code&gt; handler doesn't die gracefully, it just dies fast. During a rolling deploy that's the wrong kind of fast, fast enough to beat the signal telling everyone else it's on the way out.&lt;/li&gt;
&lt;li&gt;  ALB access logs record connection-level failures that never touch the target group's health dashboard. A target can pass every health check the dashboard tracks and still spend a few seconds quietly refusing live connections underneath it.&lt;/li&gt;
&lt;li&gt;  A synthetic check now hits &lt;code&gt;/checkout&lt;/code&gt; every two seconds during every deploy and pages if a single 502 shows up, specifically so this class of bug can't hide behind "the dashboards looked fine" again.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nothing about the checkout service's code was ever broken, and nothing about the ALB's configuration was wrong on its own terms. The bug lived entirely in the few seconds between two correct systems finding out about each other.&lt;/p&gt;

</description>
      <category>programming</category>
      <category>backend</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How a Bounce-Handling Bug Silently Suppressed Password Reset Emails for 1,800 Users</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Fri, 16 Oct 2026 11:13:27 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-a-bounce-handling-bug-silently-suppressed-password-reset-emails-for-1800-users-bh8</link>
      <guid>https://dev.to/darshan_turakhia/how-a-bounce-handling-bug-silently-suppressed-password-reset-emails-for-1800-users-bh8</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;09:14 UTC, day nine. A support ticket that's been bounced between two tiers finally lands on an engineer's desk: a customer says she's requested a password reset six times since Monday and gotten nothing, not even in spam. On-call pulls her account. Email domain: &lt;code&gt;contractorhq.biz&lt;/code&gt;. Two more tickets from the same domain, filed within the hour, same complaint.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory: SES is having a bad day. The SES console shows sending quota at 12% utilization and a bounce rate of 0.4%, nowhere near the 5% threshold that gets an account put under review. Not a sending problem.&lt;/p&gt;

&lt;p&gt;Second theory: deliverability, SPF or DKIM drifted out of alignment with the receiving domain's filters. Pulling the raw headers from a reset email sent to a different, working address shows both passing clean, DKIM signature valid, SPF aligned. Nothing about the message itself looks like spam.&lt;/p&gt;

&lt;p&gt;The pivot comes from checking the app's own send log instead of SES. The password reset endpoint didn't throw, didn't retry, didn't log an error for any of the three accounts. It logged &lt;code&gt;email_skipped: suppressed&lt;/code&gt;, cleanly, like that was the intended outcome. Nothing failed. The app chose not to send.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;Before calling SES for any transactional send, the app checks an internal &lt;code&gt;email_suppressions&lt;/code&gt; table, populated by a Lambda that consumes SES bounce and complaint notifications off an SNS topic. The three accounts are all in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

email\_suppressions, query against the three accounts

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SELECT email, reason, suppressed_at FROM email_suppressions&lt;br&gt;
WHERE email LIKE '%@contractorhq.biz';&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;email&lt;/th&gt;
&lt;th&gt;reason&lt;/th&gt;
&lt;th&gt;suppressed_at&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="mailto:j.ruiz@contractorhq.biz"&gt;j.ruiz@contractorhq.biz&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;bounce&lt;/td&gt;
&lt;td&gt;2026-10-07 03:12:41+00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="mailto:t.okafor@contractorhq.biz"&gt;t.okafor@contractorhq.biz&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;bounce&lt;/td&gt;
&lt;td&gt;2026-10-07 03:12:58+00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="mailto:m.lee@contractorhq.biz"&gt;m.lee@contractorhq.biz&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;bounce&lt;/td&gt;
&lt;td&gt;2026-10-07 03:13:04+00&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Three timestamps, nine days earlier, nineteen seconds apart. That's not three isolated bounces on three separate occasions, that's a batch. Something sent to all three at once and all three bounced the same way within the same window.&lt;/p&gt;

&lt;p&gt;The only thing that went out to a list that size that night was a product newsletter, sent to 40,000 subscribers at 03:09 UTC. &lt;code&gt;contractorhq.biz&lt;/code&gt; runs its own mail server, and the timing lines up with a documented, if rare, failure mode on their end: a burst of inbound mail from one sender hitting a mailbox quota ceiling on several accounts in the same minute. SES's raw SNS bounce payload for one of the three, pulled from the archived notification, backs it up:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight email"&gt;&lt;code&gt;&lt;span class="nt"&gt;

SES bounce notification, archived SNS message body

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;{&lt;br&gt;
  "notificationType": "Bounce",&lt;br&gt;
  "bounce": {&lt;br&gt;
    "bounceType": "Transient",&lt;br&gt;
    "bounceSubType": "General",&lt;br&gt;
    "bouncedRecipients": [&lt;br&gt;
      { "emailAddress": "&lt;a href="mailto:j.ruiz@contractorhq.biz"&gt;j.ruiz@contractorhq.biz&lt;/a&gt;",&lt;br&gt;
        "diagnosticCode": "smtp; 452 4.2.2 mailbox full" }&lt;br&gt;
    ],&lt;br&gt;
    "timestamp": "2026-10-07T03:12:41.000Z"&lt;br&gt;
  }&lt;br&gt;
}&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;bounceType: "Transient"&lt;/code&gt;. A full mailbox, a temporary condition that resolves itself once the recipient clears space, not a dead address. SES's own documentation is explicit that &lt;code&gt;Transient&lt;/code&gt; bounces should never result in permanent suppression. And yet here they are, suppressed indefinitely, nine days later.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;The bounce-handling Lambda's suppression logic, as deployed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;

&lt;span class="n"&gt;bounce&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;def handle_bounce(bounce: dict) -&amp;gt; None:&lt;br&gt;
    sub_type = bounce["bounceSubType"]&lt;br&gt;
    if sub_type == "General":&lt;br&gt;
        for r in bounce["bouncedRecipients"]:&lt;br&gt;
            suppress_email(r["emailAddress"], reason="bounce")&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;bounceSubType&lt;/code&gt; values aren't unique to one &lt;code&gt;bounceType&lt;/code&gt;. &lt;code&gt;"General"&lt;/code&gt; is a valid subtype under &lt;strong&gt;both&lt;/strong&gt; &lt;code&gt;Permanent&lt;/code&gt; (a truly dead address) and &lt;code&gt;Transient&lt;/code&gt; (a temporary delivery problem, mailbox full, message too large, content rejected). The handler branched on the subtype alone and never checked which top-level type it belonged to, so a wave of ordinary, self-resolving mailbox-full bounces from one slow mail server got routed through the exact same path as permanently dead addresses, and suppressed forever on the first occurrence.&lt;/p&gt;

&lt;p&gt;The newsletter send was the trigger, not the cause. It was simply the first send in months large enough to hit enough inboxes on one domain, at once, hard enough to push several of them over quota in the same minute. Any password reset or receipt email to those addresses afterward silently skipped SES entirely, logged as a clean, intentional-looking suppression rather than a failure, with no alert tied to it because nothing had technically gone wrong from the app's point of view.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;Immediate containment: identify every suppression entry whose archived bounce payload says &lt;code&gt;Transient&lt;/code&gt;, and release them.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

one-off reconciliation script

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;UPDATE email_suppressions s&lt;br&gt;
SET released_at = now()&lt;br&gt;
FROM bounce_events b&lt;br&gt;
WHERE b.email = s.email&lt;br&gt;
  AND b.bounce_type = 'Transient'&lt;br&gt;
  AND s.released_at IS NULL;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Lambda now gates on &lt;code&gt;bounceType&lt;/code&gt; first, and treats transient bounces as a backoff signal rather than a ban:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;

&lt;span class="n"&gt;bounce&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;def handle_bounce(bounce: dict) -&amp;gt; None:&lt;br&gt;
    bounce_type = bounce["bounceType"]&lt;br&gt;
    for r in bounce["bouncedRecipients"]:&lt;br&gt;
        email = r["emailAddress"]&lt;br&gt;
        if bounce_type == "Permanent":&lt;br&gt;
            suppress_email(email, reason="hard_bounce")&lt;br&gt;
        elif bounce_type == "Transient":&lt;br&gt;
            record_transient_bounce(email)&lt;br&gt;
            if transient_bounce_count(email, window_days=7) &amp;gt;= 5:&lt;br&gt;
                suppress_email(email, reason="repeated_transient")&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A single mailbox-full bounce no longer costs an address its ability to receive a password reset. Five in a week does, which still catches genuinely dead or abandoned inboxes without punishing a recipient for one bad night on their mail server.&lt;/p&gt;

&lt;p&gt;The other gap was visibility: &lt;code&gt;email_skipped: suppressed&lt;/code&gt; had been treated as a routine log line, not a signal worth watching. It now feeds a dashboard counter scoped to password-reset and receipt sends specifically, with an alert if skipped-suppressed count for either exceeds a handful per hour, since those two email types have no legitimate reason to be suppressed at any volume.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;p&gt;1,800 Addresses incorrectly suppressed from one newsletter send&lt;/p&gt;

&lt;p&gt;9 days From the bad bounce wave to the first escalated ticket&lt;/p&gt;

&lt;p&gt;212 Password reset and receipt sends silently skipped in that window&lt;/p&gt;

&lt;p&gt;0 Errors logged for any of the skipped sends&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;code&gt;bounceSubType&lt;/code&gt; is not a reliable signal on its own. The same subtype string can appear under more than one &lt;code&gt;bounceType&lt;/code&gt;, and any suppression logic that skips the type check is one coincidence away from treating a temporary problem as a permanent one.&lt;/li&gt;
&lt;li&gt;  A suppression system needs its own retry and backoff policy for transient failures, not a single binary gate. One bad delivery attempt and a pattern of bad delivery attempts are different signals and deserve different responses.&lt;/li&gt;
&lt;li&gt;  A clean, no-error log line can still be a failure. &lt;code&gt;skipped: suppressed&lt;/code&gt; looked identical whether it was protecting the system from a dead address or quietly breaking account recovery for a real user.&lt;/li&gt;
&lt;li&gt;  Transactional email categories that users can't route around, like password resets, need their own monitoring, separate from aggregate sending health. SES's dashboard was green the entire nine days.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nothing about the send pipeline ever broke. Every password reset email that got suppressed was, from SES's perspective and the app's own logs, working exactly as designed. The design just couldn't tell the difference between an inbox that was full for an hour and one that would never exist again.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>programming</category>
      <category>backend</category>
      <category>devops</category>
    </item>
    <item>
      <title>How Rotating S3 Presigned URLs Silently Defeated Next.js Image Optimization</title>
      <dc:creator>Darshan Turakhia</dc:creator>
      <pubDate>Sat, 10 Oct 2026 10:59:28 +0000</pubDate>
      <link>https://dev.to/darshan_turakhia/how-rotating-s3-presigned-urls-silently-defeated-nextjs-image-optimization-40ok</link>
      <guid>https://dev.to/darshan_turakhia/how-rotating-s3-presigned-urls-silently-defeated-nextjs-image-optimization-40ok</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;09:41 UTC, Tuesday. A billing alert from Vercel lands in the platform team's Slack channel: "Image Optimization usage is approaching your plan limit." The plan limit is 50,000 source images a month. The alert says the account used 2.3 million in the last 24 hours. Nobody had touched the image pipeline. Nobody had touched the plan. The only thing that had shipped recently was a new product gallery page, live since Monday morning.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  the scramble
&lt;/h2&gt;

&lt;p&gt;First theory: a scraper. The gallery page is public, product photos are easy to hot-link, and a bot crawling every product ID in sequence would explain a sudden jump in image requests. Pulling the Vercel function logs for &lt;code&gt;/_next/image&lt;/code&gt; rules that out inside ten minutes, the referer header on almost every request is the app's own domain, and the request rate tracks normal daytime traffic, not a crawler's steady drumbeat.&lt;/p&gt;

&lt;p&gt;Second theory: the gallery page itself was the problem, maybe it rendered more image variants per view than intended. The on-call engineer checks the diff for Monday's deploy. The &lt;code&gt;&amp;lt;Image&amp;gt;&lt;/code&gt; component's &lt;code&gt;sizes&lt;/code&gt; prop is unchanged, still three breakpoints, the same as every other image on the site. Forty products per gallery page, three sizes each, that's 120 image requests per page view at worst, nowhere near enough to explain 2.3 million unique source images in a day with the gallery's actual traffic.&lt;/p&gt;

&lt;p&gt;Both obvious explanations are gone within fifteen minutes, and the number on the Vercel dashboard keeps climbing. Whatever is driving this, it isn't more images and it isn't a bot. It's something about how the existing images are being counted.&lt;/p&gt;




&lt;h2&gt;
  
  
  the hunt
&lt;/h2&gt;

&lt;p&gt;Vercel's Image Optimization API doesn't bill per request. It bills per unique source image it has to fetch and transform. A healthy app re-fetches the same source rarely, the optimizer caches the transformed output at the edge and serves repeat requests for the same source URL, width, and quality straight from that cache. The volume alone doesn't explain the bill. What needs explaining is why almost none of these requests are hitting that cache.&lt;/p&gt;

&lt;p&gt;Pulling a sample of the &lt;code&gt;url=&lt;/code&gt; query parameters the optimizer is being asked to fetch for a single avatar, the same user's avatar, across five page loads inside two minutes, shows five different strings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

same avatar, five page loads, two minutes apart

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://assets-private.ourapp.com.s3.amazonaws.com/avatars/8841.jpg" rel="noopener noreferrer"&gt;https://assets-private.ourapp.com.s3.amazonaws.com/avatars/8841.jpg&lt;/a&gt;&lt;br&gt;
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256&lt;br&gt;
  &amp;amp;X-Amz-Credential=AKIA.../20261006/us-east-1/s3/aws4_request&lt;br&gt;
  &amp;amp;X-Amz-Date=20261006T094012Z&lt;br&gt;
  &amp;amp;X-Amz-Expires=900&lt;br&gt;
  &amp;amp;X-Amz-SignedHeaders=host&lt;br&gt;
  &amp;amp;X-Amz-Signature=7f2a9c... (load 1)&lt;/p&gt;

&lt;p&gt;...same path, X-Amz-Date=20261006T094047Z, X-Amz-Signature=b81e4d... (load 2)&lt;br&gt;
  ...same path, X-Amz-Date=20261006T094119Z, X-Amz-Signature=029cf7... (load 3)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same bucket, same key, same bytes on disk. Different query string every time. Dead end first: the team suspects an overeager cache-purge webhook, something firing on a schedule and evicting the edge cache before it can ever get used. Checking the optimizer's response headers for a repeat request to the exact same URL rules that out, when the query string is identical, the cache hits fine and returns in under 20ms. The problem isn't eviction. The problem is that the URL is never identical twice.&lt;/p&gt;

&lt;p&gt;That one line of logs is the whole bug. Next.js's image optimizer, and Vercel's hosted version of it, treats the full request URL, including every query parameter, as part of the cache key. A presigned S3 URL carries a fresh &lt;code&gt;X-Amz-Date&lt;/code&gt; and &lt;code&gt;X-Amz-Signature&lt;/code&gt; on every call, by design, that's what makes it a presigned URL instead of a permanent link. Which means every single page view of the same unchanged avatar was being treated as a request for a brand-new image the optimizer had never seen before.&lt;/p&gt;




&lt;h2&gt;
  
  
  the find
&lt;/h2&gt;

&lt;p&gt;Two weeks earlier, a SOC 2 audit finding flagged that user avatars and product photos sat in a publicly readable S3 bucket, no auth required if you had the URL. The fix, shipped quietly and correctly from a security standpoint, moved everything to a private bucket and started generating a presigned URL server-side for every image, with a 15-minute expiry, passed straight into the &lt;code&gt;src&lt;/code&gt; prop:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

image loader, as shipped two weeks earlier

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;// server component, product gallery&lt;br&gt;
const presignedUrl = await s3.getSignedUrl('getObject', {&lt;br&gt;
  Bucket: 'assets-private-ourapp',&lt;br&gt;
  Key: &lt;code&gt;products/\${product.imageId}.jpg\&lt;/code&gt;,&lt;br&gt;
  Expires: 900,&lt;br&gt;
});&lt;/p&gt;

&lt;p&gt;return ;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At the traffic the site was doing before Monday, this was a real inefficiency, every page view was already paying for a fresh optimization instead of a cached one, but the absolute numbers stayed small enough to hide inside normal usage. Nobody had a monitor on cache hit ratio for the image optimizer specifically, only a monthly usage total that nobody checks mid-cycle. The gallery launch didn't introduce the bug. It multiplied an existing one: forty product photos rendered per page view, each one a guaranteed cache miss, across a traffic volume high enough to turn a slow leak into something that blew through a 50,000-image monthly plan limit in under a day.&lt;/p&gt;




&lt;h2&gt;
  
  
  the fix
&lt;/h2&gt;

&lt;p&gt;The access control and the cache key needed to be separated. A presigned URL is the right tool for proving the request is authorized. It is the wrong tool for identifying the image to the optimizer, because its identity changes every time it's generated. The fix routes image requests through a stable, internal path, and does the S3 fetch and the authorization check server-side, where the signature never has to leave the backend:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;

&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;api&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;media&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;assetId&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;route&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ts&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;export async function GET(&lt;br&gt;
  req: NextRequest,&lt;br&gt;
  { params }: { params: { assetId: string } }&lt;br&gt;
) {&lt;br&gt;
  const authorized = await checkAccess(req, params.assetId);&lt;br&gt;
  if (!authorized) {&lt;br&gt;
    return new Response('Forbidden', { status: 403 });&lt;br&gt;
  }&lt;/p&gt;

&lt;p&gt;const object = await s3.getObject({&lt;br&gt;
    Bucket: 'assets-private-ourapp',&lt;br&gt;
    Key: &lt;code&gt;products/\${params.assetId}.jpg\&lt;/code&gt;,&lt;br&gt;
  });&lt;/p&gt;

&lt;p&gt;return new Response(object.Body as ReadableStream, {&lt;br&gt;
    headers: {&lt;br&gt;
      'Content-Type': 'image/jpeg',&lt;br&gt;
      'Cache-Control': 'public, max-age=31536000, immutable',&lt;br&gt;
    },&lt;br&gt;
  });&lt;br&gt;
}&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;&amp;lt;Image&amp;gt;&lt;/code&gt; call changes from a signed, expiring URL to a plain, stable path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

image loader, after

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;return (&lt;/p&gt;

&lt;p&gt;);&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;product.imageId&lt;/code&gt; never changes for a given photo, so the optimizer's cache key is stable now, no matter who's viewing it or when, until the underlying asset actually gets replaced. The route handler still checks authorization on every call, nothing about access control got weaker, it just stopped leaking into the one place that needed to stay constant. The &lt;code&gt;immutable&lt;/code&gt; cache directive is safe here specifically because asset IDs are never reused for a changed image, a new upload gets a new ID.&lt;/p&gt;




&lt;h2&gt;
  
  
  the aftermath
&lt;/h2&gt;

&lt;p&gt;2.3M → 44K Daily unique source images, before and after the fix deployed&lt;/p&gt;

&lt;p&gt;$1,840 Overage for the 24-hour window that triggered the alert&lt;/p&gt;

&lt;p&gt;14 days The underlying cache-miss bug ran before traffic made it visible&lt;/p&gt;

&lt;p&gt;1 Second instance of the same mistake found and fixed in an email template's inline images&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  A presigned URL and a cache key are different problems wearing the same string. Anything that intentionally changes on every call, a signature, a timestamp, a nonce, will defeat any cache that treats the full URL as its identity, no matter how good that cache otherwise is.&lt;/li&gt;
&lt;li&gt;  A security fix and a performance regression can ship in the same commit without anyone noticing, because nobody reviews a SOC 2 remediation for cache behavior. The review caught the access control problem it was meant to catch and missed the one it accidentally created.&lt;/li&gt;
&lt;li&gt;  Usage caps that only alert at a monthly total, not a daily rate of change, hide a slow leak until something unrelated, in this case a product launch, amplifies it past the threshold. A day-over-day anomaly alert would have caught this two weeks earlier, while the fix was still a one-line git blame away.&lt;/li&gt;
&lt;li&gt;  Keep authorization server-side and identity stable client-side. Any time a URL that gets cached, by a CDN, an image optimizer, a browser, also needs to prove the request is allowed, put the proof behind a stable endpoint instead of inside the URL itself.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The avatars never moved. The product photos never moved. For two weeks, the only thing wrong was a query string that looked identical in a browser's network tab and was, byte for byte, never the same request twice. The optimizer did exactly what it was built to do: cache what it could prove was the same image. It just never got the chance.&lt;/p&gt;

</description>
      <category>programming</category>
      <category>backend</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
