DEV Community: Sergey Shinder

Our deploys were also our restart policy

Sergey Shinder — Tue, 22 Sep 2026 07:33:18 +0000

At four on a Sunday morning an internal reconciliation service went down. All four pods were killed for memory inside twenty minutes of each other, came back, and were killed again a few hours later. The service had last been deployed in February. Seven months of continuous uptime, and it was the only thing in our estate that had any.

The cause was a leak of about forty megabytes per pod per day. We create an SDK client per request, and each client registers a listener on a static registry that nothing ever unregisters. A heap comparison showed one point two million listener objects, which at our request rate is exactly what you would expect after seven months. It is a beginner's mistake in a library that has had it since 2019.

The uncomfortable part came next. That library is in every service we run. I plotted resident memory against pod age across the whole fleet, and every service has the same slope. None of them had ever shown it, because the median age of a pod in our clusters is under thirty hours. We deploy several times a day, and a restart clears the leak, so no instance of anything had ever lived long enough to reach the ceiling. The one service nobody had a reason to change was the only one telling us the truth about our code.

We fixed the client, of course. The more useful changes were the other three. Memory growth is alerted on as a rate normalised by pod age, because an alert at ninety percent of the limit fires a few minutes before the kill and is not a warning, it is a commentary. One instance of each service now runs in a soak environment for thirty days against synthetic traffic with its memory graphed. And pod age distribution is on the platform dashboard, because a fleet in which nothing is older than two days cannot tell you anything about how it behaves on day thirty.

Deploying often had become a reliability control, and we had never decided to have it. Anything that hides a defect is load bearing. If the only reason a system stays up is that you keep restarting it, a quiet fortnight is a risk and not a rest.

– Sergey Shinder

We shipped the version that could not receive the next one

Sergey Shinder — Tue, 22 Sep 2026 07:28:07 +0000

We run our own till software on two thousand four hundred shop terminals. Version 4.2 rolled out over a fortnight without incident. Then 4.3 went out, carrying a fix we actually needed, and two days later the adoption dashboard still read zero percent. Not slow. Zero.

Nothing was wrong with 4.2 as a piece of software. The terminals ran it perfectly for eleven days. What 4.2 contained was a tidy up: the updater, which had been its own small service, was folded into the main application, because two processes where one would do had been on somebody's list for a year and the review agreed with them. Its check for a new version moved to application startup.

A terminal starts its application once, when the shop opens. On 4.2 the check ran about fifty seconds into startup, which is before the site's tunnel to us is established, so it got a connection error. Being a once per start check, it did not run again until the next morning, when it failed at the same point for the same reason. Nothing was flaky. Every terminal in the estate failed deterministically, every day, and reported nothing, because a failed update check was a debug line on a till in a shop.

Getting 4.3 out took five days through our remote support tool, which is licensed for sessions with an engineer on the other end and is not a distribution channel. Nineteen sites needed somebody to visit, because the terminal had been switched off during the window we were working through.

The updater is a separate package again with its own release train, changed rarely and never in the same release as the application it installs. It checks on a schedule through the day rather than once at start. Every terminal reports its version and last successful check hourly, and we alert on the proportion of the estate that has gone quiet, which is a fact about the channel rather than about any one rollout. And after every release the pipeline publishes a version that changes nothing and asserts that ninety five percent of terminals take it within the hour.

Everything you ship can be fixed by shipping again, except the part that does the shipping. Keep it apart from what it carries, and prove the road is still open before you need it.

– Sergey Shinder

We were the unreliable half of the integration

Sergey Shinder — Tue, 22 Sep 2026 07:22:56 +0000

On a Tuesday our carrier partner suspended our event subscription. Around fourteen thousand shipment updates over nine hours were never sent to us, and we learned about it from a customer asking why their tracking page had not moved since breakfast.

Their delivery platform grades the systems it delivers to. The rule is written in their documentation, in a section I had read four years ago as a paragraph about them: a receiver that takes longer than five seconds to answer is recorded as a failed delivery, and a subscription with more than a hundred consecutive failures is suspended until its owner switches it back on in their portal. All of that is sensible, and none of us had ever considered that our end of an integration carried a service level, measured by another company, against a number nobody here had looked up.

Our handler did everything inside the request. Check the event, load the shipment, update three rows, write an audit record. Two hundred milliseconds on an ordinary morning. That morning a long reporting query held a lock on one of those tables and our handler went to eleven seconds a call. We answered every single one of them successfully, just slowly, so our own instruments recorded nothing wrong at all. Zero errors. We were not failing, we were late, and late is a failure in somebody else's system rather than in ours.

The handler now does one thing. It confirms the event is a shape we expect, writes it to a durable queue and answers. Twelve milliseconds. The real work happens behind that, where it can take as long as it needs to without anyone else forming an opinion about us.

Two other things changed. Our receiver is monitored as a customer facing service with its own latency objective, because to them that is exactly what it is. And every hour we ask their API how many events they sent us in the previous hour and compare it with how many we stored, which would have told us in sixty minutes what a customer told us in nine hours.

Every integration makes you a supplier as well as a consumer. Somewhere in the other company's documentation is the standard you are held to and the penalty for missing it, and it is written as though it were about them.

– Sergey Shinder

Every question we asked our logs took six minutes to answer

Sergey Shinder — Tue, 22 Sep 2026 07:17:45 +0000

During a fifty minute incident in June, four of us sat in a call watching a spinner. Every search we ran against our log store took between six and eight minutes to come back. We were not short of data. We had the request ids, the error signatures and the customer references, and we could not ask about any of them faster than once every few minutes, which meant that in fifty minutes we asked our logs about seven questions between us.

Everything from forty services went into one index per day, thirty days of them kept on the same hot nodes, with no routing and no separation. A search for a request id with no service filter, over the default thirty day range that our saved view opened on, scanned roughly one point four terabytes. People widened ranges out of habit, because narrowing one means already knowing when the trouble started, which during an incident is precisely the thing you do not know.

We changed the storage shape and the questions. Logs route by service into per service indices, three days on fast local disk, twenty five days on cheaper nodes, then a searchable snapshot. The saved view for a service opens on one hour and carries the service filter already applied. And the five things we ask in every single incident, error counts by endpoint, failures for one customer id, slowest endpoints, the last ten deploys and a count per error signature, are now five buttons that run bounded queries rather than five expressions somebody types badly under pressure.

The request id lookup that took six minutes returns in about four seconds. The median query in our last three incidents was under ten seconds. We also record query latency now, and time to first answer goes into the incident review next to time to detect.

Two years of work had gone into making our telemetry complete and none into making it answerable. Your tooling has a latency and it is spent inside the outage, at the moment when a person is holding a hypothesis they cannot check. Correct and slow is a strange thing to be proud of.

– Sergey Shinder

Our cluster was full at nineteen percent CPU

Sergey Shinder — Tue, 22 Sep 2026 07:07:23 +0000

A routine deploy sat in Pending at five past nine on a Monday. The event on the replica set said zero of thirty eight nodes are available, insufficient cpu. On the same screen, the cluster's CPU utilisation for the past hour was nineteen percent and memory was thirty four.

Both were true. The scheduler does not look at what a container is using. It looks at what a container has asked for, and a request is a reservation held for as long as the pod exists, whether or not a single cycle is ever spent against it. Our Helm chart's default request was one CPU and two gibibytes. That number came from the first service we containerised in 2023, which genuinely needed it, and it had been copied into every chart since by people doing the sensible thing and starting from what already worked. Two hundred and forty pods, two hundred and forty reserved CPUs, and a median actual usage of about sixty millicores per container. Ninety two percent of allocatable CPU was spoken for and almost none of it was being used.

The cluster autoscaler had been doing its job throughout, adding nodes to satisfy claims, which is how thirty eight machines ended up idling.

We took thirty days of per container usage and set each request from the ninety fifth percentile with headroom, and deliberately left limits alone, because limits are a different argument and I did not want the two mixed up in one change. A weekly job now posts a recommendation per deployment from the same data. A policy rejects any pod requesting more than five hundred millicores without an annotation giving a reason. And the chart that matters most is new: reserved against used, per node pool, which is the only view in which any of this was ever visible. Node count went to fifteen. An ingestion service that had been starved for weeks got scheduled without anyone asking for more capacity.

Utilisation is what you are spending. Allocation is what you have promised. The scheduler reads the promises, and a cluster fills up with promises long before it fills up with work.

– Sergey Shinder

One workspace planned clean and eleven were never asked

Sergey Shinder — Tue, 22 Sep 2026 07:02:12 +0000

A change to our shared networking module moved a resource from a counted list to a keyed map. The author planned it against the workspace he works in, which showed no changes at all, two of us approved it, and it went out as a new module tag. Three weeks later a colleague bumping a different workspace to that tag read a plan that proposed to replace a NAT gateway, which is twenty minutes with no outbound connectivity for everything in that account. He stopped, which is the only reason this is a story about a plan and not about an outage.

The module has eleven callers. They pass different combinations of about fourteen variables, three of which switch whole resources on and off, and the state each of them holds was written at a different time by a different version. A module change is not one change. It is eleven changes that will happen on eleven different days, and the review we had performed covered exactly one of them, chosen because it was the one already open in somebody's terminal. Everything else we ran on the pull request, formatting, a linter, validate, checks the code as text. None of it instantiates the module against real state, so none of it can tell you what will be destroyed.

Module pull requests now run a plan against all eleven consuming workspaces, in parallel, with read only credentials, and post one table: workspace, resources to add, change, destroy and replace. Twelve minutes, and any destroy or replace anywhere fails the pull request until somebody writes down why in the description. Separately there is a job in a sandbox account that applies the module across four variable combinations, applies it a second time and asserts the plan is empty, then destroys everything. Four pounds a run, and in its first week it caught a naming expression that produced a different result on the second apply.

Shared infrastructure code has a fan out that the code in front of you does not show. Ours looked like one file with one reviewer and was in fact a change to eleven production environments, queued up to arrive whenever each of them next happened to run.

– Sergey Shinder

The cheapest check in our pipeline ran last

Sergey Shinder — Mon, 21 Sep 2026 07:22:36 +0000

At twenty past four on a Thursday a release failed thirty three minutes after it started, at the final stage, because a values file had a resource block indented one level too deep. The templating engine produced a manifest with a key the API server has never heard of, the apply was rejected, and the engineer fixed it in four seconds and started the whole thing again.

Our pipeline ran in the order the delivery happens. Lint and formatting, forty seconds. Unit tests, six minutes. Image build and push, five minutes. Integration tests against ephemeral dependencies, eighteen minutes. Then deploy, which is where the chart is rendered for the first time, because rendering needs the environment's values and the environment is a property of the deploy stage. Every mistake in the deployment description, and every mistake in the six values files that feed it, was therefore discovered at the far end of half an hour of work that had nothing to do with it.

I pulled eight weeks of pipeline history. A hundred and twelve runs had failed in the deploy stage. Sixty one of those were schema or syntax errors in a template or a values file, with a median of thirty four minutes burnt before the failure, and a person attached to most of them. That is a working week of waiting a month, to be told about mistakes that a parser can find instantly.

There is a preflight job now and it is the first thing in the pipeline. It renders the chart for all six environments, validates every resulting manifest against the schema for the API version our clusters actually run, and compares the key sets of the six values files against a reference so that a key added to staging and forgotten in production is a failure rather than a surprise. It takes eleven seconds, and in its first fortnight it caught the same class of mistake three times, each within a minute of the push instead of half an hour after it.

A pipeline is a list of questions, and ours had been ordered by how a deployment flows rather than by how quickly each question can answer. Put everything that can say no in the first minute. Waiting half an hour to learn about a space is a choice somebody made, even if nobody remembers making it.

– Sergey Shinder

Our rate limit punished everyone except the customer causing the problem

Sergey Shinder — Sun, 20 Sep 2026 09:45:46 +0000

At twenty past two on a Tuesday morning one customer began a historical backfill against our API. Within ten minutes a hundred and twelve other customers were being rejected, and the customer who started it was getting most of the throughput they had asked for.

Our edge holds a single token bucket for the whole service, two thousand requests a second, a number chosen years ago from what the service can actually serve. It is not per key. It never had been, and I had read the configuration many times without noticing that the absence of a key dimension was a decision about fairness rather than an implementation detail.

A shared bucket is drained by arrival. A client with nineteen hundred requests in flight takes tokens at nineteen hundred times the rate of a client with one, because every token that appears goes to whoever is asking at that instant, and asking is exactly what that client was doing. The limiter did its job perfectly. It kept the service standing, and it distributed the shortage in direct proportion to who was causing it, which is to say in inverse proportion to who deserved it.

That customer's steady rate is about forty a second. Our aggregate availability for the hour came out at ninety one percent, which reads as a moderate incident. For the hundred and twelve customers who were not doing anything unusual it was closer to thirty percent, and that number existed nowhere.

Each key now has its own bucket, sized from its own trailing thirty day peak with a multiplier, with the global bucket kept behind them as a backstop for the service rather than as the allocation mechanism. Requests carry a class, so an interactive call outranks a batch call from the same key. Responses say which limit was hit. And we record the throttled fraction per key, so shedding is something we can see happening to somebody specific.

The reporting changed too. We publish the worst tenant's success rate next to the aggregate, because the aggregate is an average over customers and had never once told us that a small number of them were having a completely different day.

A limit protects the service. It says nothing about who gets what, and where you have not said it, the answer is whoever pushes hardest.

– Sergey Shinder

Nine teams were taking turns on one staging environment

Sergey Shinder — Sun, 20 Sep 2026 09:35:25 +0000

We spent most of a morning investigating a staging failure in checkout, working through our own recent changes, before someone from another team mentioned they had run a migration the night before that renames a column we read. Their change, our failure, four engineers, one morning.

There was one staging environment. One namespace, one database, one set of partner stubs, shared by nine teams, booked first in a spreadsheet and later in a channel. A release soak took two hours in it, which meant a team got a slot roughly when the queue reached them.

The attribution problem was the expensive one and we had never measured it. When I went back through eight weeks of staging failures, thirty one out of seventy four had been caused by a change belonging to a different team than the one investigating. Every one of those was diagnosed eventually, by hand, by people reading a stack trace that pointed at their own service and was telling the truth about somebody else's change.

We already had the environment described in code, so building a per pull request environment was less work than the two years of process we had grown around the shared one. A namespace, a database restored from a sanitised nightly snapshot, partner stubs, a generated hostname. Six minutes to create, destroyed on merge or after twelve hours, and a little under three hundred pounds a month for all of them together.

The shared environment still exists, because a few things genuinely cannot be duplicated: one partner test account, one link to a mainframe. Those now sit behind a lock the pipeline takes and releases, rather than a person writing in a channel.

A failed soak names exactly one change now, which is what a soak was always supposed to do.

What I take from it is that we had spent two years making the automated part of our release faster, from twenty six minutes down to nineteen, and the automation had never been the constraint. The constraint was one scarce physical thing that everybody queued for, and it had been there so long that none of us counted it as part of the release process at all.

– Sergey Shinder

We never told the partner which version of their API we wanted

Sergey Shinder — Sun, 20 Sep 2026 09:30:14 +0000

Our nightly export stopped carrying customer names on a Wednesday. We had not deployed anything for nine days, their status page was green, and their support desk confirmed there had been no incident. Their API had moved us onto a newer version of itself, on a schedule they had published eighteen months earlier.

They version by date header. Send a date and you get that version's behaviour for as long as they support it. Send nothing and you get whatever is current. Our client sent nothing, because the integration was written in an afternoon against their quickstart, and their quickstart does not send the header either.

The change itself was sensible and well documented. A customer object that used to be returned inline on each order is now returned as an identifier string, to cut payload size, with an explicit parameter to expand it. Our mapper read the name field from inside that object. Given a string where it expected an object, the deserialiser produced a null instead of raising, because the field was declared optional in our model, which it had been for a legitimate reason nobody now remembers.

So sixty two thousand rows went into the CRM with an empty name column, overwriting values that had been correct, and the only reason anybody noticed within a day is that a salesperson opened an account and saw a blank.

The version is pinned now, in one constant, and the client checks it against a version endpoint at startup and refuses to run if the two disagree. Every partner we integrate with has an entry in a register recording how they version, what we are pinned to, what happens when you send nothing, and the date their support for our pin ends. Writing that register was the useful part: two of our six partners have no versioning mechanism at all, which is a thing worth knowing before it matters rather than afterwards. The mapper is strict as well, so an unexpected type is a failure rather than a null.

Sending no version is still a choice. It chooses whatever the other company thinks is best that morning, and it renews itself every day, and it is the one decision in the integration that appears nowhere in your own code.

– Sergey Shinder

We spent the first twenty three minutes of an outage reading staging's graphs

Sergey Shinder — Sun, 20 Sep 2026 09:25:03 +0000

Checkout was returning errors for about one request in three and the dashboard in front of four engineers was entirely healthy. Request rate normal, error rate flat, latency at a hundred and eighty milliseconds. Everything on the screen was accurate, current and about a different environment.

When we consolidated onto one Grafana with a datasource per environment, every dashboard gained a datasource template variable. A template variable has a default, and unless you set one the default is whichever option sorts first alphabetically. Ours sorted a staging datasource to the top. Grafana remembers your last selection in your own browser, so for most of us the dashboards opened where we left them and the variable was invisible. The link pinned in our incident channel had been copied out of somebody's browser two years earlier and carried no variable in its query string, so it opened at the default. The engineer who opened it that morning had never used that dashboard before, so there was nothing remembered to correct it.

What made it convincing rather than obviously wrong is that staging runs the same code with a synthetic load generator against it. The panels were full. The shapes were plausible. There was no error state to notice, because nothing had failed. The only evidence was a dropdown in the top left corner of the page, rendered in the same grey as everything else.

The alert that fired did contain a link with the variables set correctly. Nobody clicks the alert link. People go to the dashboard they know by heart.

Environment is no longer a variable. Production dashboards live in their own folder with the datasource fixed in the JSON, and a provisioning test fails the build if anything in that folder is templated. Every production dashboard now carries a row at the top that reads an environment label out of the data itself and prints it in large type, coloured red for anything that is not production. The pinned links are generated from the repository rather than pasted from a browser.

A tool that can show you two environments needs to make which one loudest, because the failure mode is not an error message. It is a completely convincing answer to a question you did not ask.

– Sergey Shinder

Our cluster had kept every job it ever finished

Sergey Shinder — Sun, 20 Sep 2026 09:19:51 +0000

Deploys started failing on a Thursday morning. All of them, in every namespace, with the same message from our rollout tool: timed out waiting for pods. The pods were healthy and running. The cluster simply could not answer a question about them inside thirty seconds.

Our import controller creates a Kubernetes Job for each file a customer sends us, roughly nine hundred a day since early 2024. It creates them through the API itself rather than through a CronJob, which matters more than it sounds. A CronJob keeps three successful runs by default and deletes the rest. A Job created directly keeps everything forever unless you set ttlSecondsAfterFinished, and we had not, because none of us knew it existed. Each finished Job also leaves its Pod object behind. We were carrying about three hundred and forty thousand Jobs and a similar number of Pods, none of which had done anything for months.

Every one of those is a row in etcd that the API server keeps in a watch cache and that every controller in the cluster lists on resync. Our etcd database had reached six point four gigabytes. List calls against large namespaces began paging for tens of seconds, the scheduler's resyncs took minutes, and our rollout tool, which lists pods before it watches them, gave up. Nothing was broken in a way that produced a single obvious error. Everything was slow in a way that expressed itself as unrelated timeouts.

Deleting them was its own operation, because three hundred and forty thousand deletions are three hundred and forty thousand writes into the database that is already struggling. We removed them oldest first in batches of five hundred with a pause between, over two days, then compacted and defragmented each etcd member in turn.

Now every Job the controller creates carries a one hour TTL, an admission policy rejects any Job submitted without one, and an alert fires when the object count for any resource type in a namespace passes five thousand.

A Kubernetes object is a database row that every component in the cluster reads. Anything in your system that creates objects at a rate needs a rule for removing them, written on the same day, because the platform will keep them faithfully until it cannot.

– Sergey Shinder