DEV Community

Cover image for How an emptyDir With medium: Memory Made Our Export Pods OOMKilled While the Heap Sat at 210 MB
Darshan Turakhia
Darshan Turakhia

Posted on Originally published at darshanturakhia.com

How an emptyDir With medium: Memory Made Our Export Pods OOMKilled While the Heap Sat at 210 MB

02:47 UTC, Wednesday. PagerDuty: reports-export restarts > 3 in 30m. I open the Kubernetes dashboard and the restart counter on the deployment reads 17. Last reason for each one: OOMKilled, exit code 137.


the scramble

Three of us end up in the channel. The export service builds CSV and XLSX files for enterprise customers, and the on-call for the finance team says three scheduled month-end exports are missing. Customers are asking where their files are.

First theory, from everyone at once: memory leak in the last release. It shipped on Monday and added a new XLSX writer. That fits the timeline, so I start there.

It stops fitting quickly. The container limit is 1 GiB. I pull the Node heap metrics we export from process.memoryUsage() and the heap never crosses 210 MB on any pod, including the ones that were killed. RSS is 340 MB at the last scrape before each death. A leak in the new writer would show up as a climbing heap, and it is a flat line.


the hunt

Dead end one: native memory. Streaming XLSX libraries sometimes buffer in Buffers outside the V8 heap. That would show in RSS, and RSS is 340 MB. Still nowhere near 1 GiB.

Dead end two: the kernel log. I read the node's dmesg for the OOM kill record, expecting to see a process using 1 GiB of anonymous memory. The record is there, and it is odd:

memory: usage 1048576kB, limit 1048576kB, failcnt 8841 Memory cgroup stats for /kubepods/burstable/pod4c1e.../a7f3...: anon 351068160 file 12582912 shmem 702545920 Killed process 29114 (node) total-vm:1204816kB, anon-rss:338120kB
Enter fullscreen mode Exit fullscreen mode

anon is 335 MB, which matches the Node process. shmem is 670 MB. Something holds 670 MB of shared memory inside this container's cgroup, and it is not our code's heap.

Shared memory in a container is usually /dev/shm or a tmpfs mount. I check the pod spec and find it. When the platform team enabled readOnlyRootFilesystem in September, the export service lost the ability to write to /tmp, so somebody added a scratch volume:

volumes: - name: scratch emptyDir: medium: Memory containers: - name: reports-export volumeMounts: - name: scratch mountPath: /tmp resources: limits: memory: 1Gi
Enter fullscreen mode Exit fullscreen mode

medium: Memory backs the volume with tmpfs. I exec into a live pod and run df -h /tmp. It reports 4.2 GB size (half the node's RAM, since no sizeLimit was set) and 640 MB used, with 22 leftover files from earlier exports.

The exporter writes each finished file to /tmp/exports/<job-id>.xlsx, uploads it to S3, and deletes it. Except the delete only runs on the success path. Any job that failed the upload, which happens a few times a day when the S3 call times out, leaves its file behind. Those files are in RAM, and tmpfs pages are charged to the memory cgroup of the container that wrote them.

So the heap graph was a decoy. Every dashboard we had measured the process. The kernel measures the container, and ours had a RAM disk in it.


the find

Month-end is when the large exports run. A single 180 MB XLSX plus the pod's baseline 340 MB is fine. But the S3 timeouts cluster at month-end too, because the same batch hammers the bucket. Each failed upload strands another 150 to 200 MB file in tmpfs. Four stranded files, one new export in progress, and the cgroup hits 1 GiB. The kernel cannot reclaim tmpfs pages (there is no swap), so it kills the process.

One more detail made the incident look like a leak: emptyDir lives as long as the pod, not the container. When the OOM killer restarted the container, the stranded files were still there, so the replacement started its life already carrying them. That is why restarts bunched up instead of clearing the problem.

To confirm, I delete the stranded files from one pod by hand and watch it. Its memory working set drops from 910 MB to 330 MB and it stops restarting. A second pod I leave alone is killed again at 03:31.


the fix

Two changes. First, the scratch volume goes back to node disk, with an explicit size cap and an ephemeral-storage limit so the kubelet evicts instead of the kernel OOM-killing a healthy process:

volumes: - name: scratch emptyDir: sizeLimit: 2Gi containers: - name: reports-export resources: requests: ephemeral-storage: 1Gi limits: memory: 1Gi ephemeral-storage: 2Gi
Enter fullscreen mode Exit fullscreen mode

Second, the code now cleans up on every path, and on startup it wipes anything left in the scratch directory from a previous container:

async function runExport(job) { const path = \\\`/tmp/exports/\\${job.id}.xlsx\\\`; try { await buildWorkbook(job, path); await uploadToS3(path, job.key); } finally { await fs.rm(path, { force: true }); } } // at boot await fs.rm('/tmp/exports', { recursive: true, force: true }); await fs.mkdir('/tmp/exports', { recursive: true });
Enter fullscreen mode Exit fullscreen mode

I also added a 30-second timeout and two retries on the S3 upload, which is what had been producing the stranded files in the first place.


the aftermath

  • Impact: 17 restarts across 5 pods between 22:10 Tuesday and 03:40 Wednesday, and 3 month-end exports delayed by about 4 hours. The jobs were re-run from the queue, so nothing was lost.
  • Alert added: container working_set_bytes divided by limit, plus a separate panel for container_memory_cache and shmem, so a gap between heap and container memory is visible in one place.
  • A policy check in CI now rejects medium: Memory on any emptyDir without a sizeLimit and a note about what the memory limit has to cover.
  • Runbook line: if OOMKilled and the process heap is flat, read shmem in the cgroup stats before looking for a leak.

A RAM-backed volume makes a file feel free, and it isn't: it draws from the same 1 GiB as the heap. We spent two hours hunting a leak in code that was fine, while the memory that mattered was sitting in a directory.

Top comments (0)