Consolidation was the right call at the time. We had a dozen small clusters, each underused, each needing patching and monitoring and somebody who understood it, and the operational overhead was genuinely out of proportion to the workload. So we built one well-run shared platform and moved everybody onto it. The efficiency gains were real and I would make the same decision again. What I did not price in was that we had also merged twelve independent bad days into one shared one.
It took about seven months to find out. A batch job in a reporting namespace had no memory limit set, ran against an unusually large dataset, and grew until the nodes it was scheduled on started evicting other people's pods. None of the affected teams had done anything wrong. None of them had any visibility into what had hit them, because from inside their namespace everything looked healthy right up until their workloads vanished and reappeared somewhere else. My inbox that morning contained four separate incident reports describing the same event from four angles, and not one of them mentioned the actual cause.
The technical fixes are well known and we applied them: resource requests and limits enforced by policy rather than by convention, quotas per namespace, priority classes so that the things that matter survive pressure, and separate node pools for workloads whose failure profile is genuinely different. That work is not difficult. What is difficult is that it must be done before the incident, at a point where every one of those controls feels like unnecessary friction imposed on colleagues who have done nothing to deserve it.
The harder change was social. A shared platform needs someone who owns the contract between tenants, not just the machinery underneath them, and that role has to be able to say no to a team who wants an exception. Without it, the defaults erode one reasonable request at a time, and the isolation you designed exists only in the original architecture document.
Sharing infrastructure means sharing failure. That is an acceptable trade, often a good one, but it needs to be made deliberately rather than discovered on a Tuesday morning by four teams at once.
– Serguey Shinder
Top comments (0)