Our reporting database moved into Kubernetes in June: Postgres 15 in a StatefulSet we wrote ourselves, the same version, the same configuration file, more memory than the virtual machine it replaced. It ran for three weeks without a complaint. On the first working day of July the month end finance reports started failing, about one in three, with could not resize shared memory segment, No space left on device. The data volume was forty percent full. Small queries were fine.
The space it had run out of was /dev/shm. Postgres runs large queries in parallel, and its worker processes exchange data through dynamic shared memory, which on Linux is by default a file in /dev/shm. On the old virtual machine /dev/shm was a tmpfs sized at half the RAM, eight gigabytes. A container gets its own /dev/shm, and the container runtime makes it sixty four megabytes unless told otherwise. The pod spec has no field for that size. Daily queries used a few megabytes of it. Month end reports join three years of ledger lines with parallel hash joins, and each of them asked for hundreds of megabytes.
So it failed for the reports finance needed most and for nothing else, which is why our migration test, a week of normal daily queries, had passed.
That afternoon we switched off parallel query to get the reports out, slower but correct. The real fix is a memory backed emptyDir volume mounted at /dev/shm with a size limit of two gigabytes. Pages written to it count against the container's memory limit, so we raised the limit by the same two gigabytes; without that, a large report would have had the whole pod killed for memory instead of one query failing. The checklist for moving anything from a virtual machine into a container now has a line for every resource the old machine provided without being asked: /dev/shm, open file limits, huge pages, the process limit. And the database load test replays last month end's actual queries, not an average week.
A container gives you a filesystem that looks like a machine's, and some of its directories are much smaller than the machine's were. Ours had been sixty four megabytes from the first day, and it took a month end to ask for more.
– Sergey Shinder
Top comments (0)