DEV Community: Sergey Shinder

Our lock expired while the first worker was still writing invoices

Sergey Shinder — Tue, 13 Oct 2026 06:02:27 +0000

On the second of June two hundred and twelve of our business customers received their May invoice twice. Nine of them received it three times.

Invoicing runs on eight workers that take customer accounts from a list. To stop two workers invoicing the same customer, each takes a lock in Redis first: a SET with NX and an expiry of sixty seconds, keyed on the customer. A minute was generous, because an invoice usually took about four seconds. When a worker finished, it deleted the key.

That month our largest customers had far more usage lines, and the tax service we call for every line was slow. The biggest invoices took between two and five minutes. After sixty seconds each lock simply expired. Redis has no idea whether the holder of a key is still working. Another worker found no lock on that customer, took one, and started the same invoice from the beginning. The first worker finished, saved the invoice, emailed it and deleted the key, which by then belonged to the second worker. So a third worker could start as well. That is where the triples came from.

A lock with an expiry is really a lease. It ends at a time, whether the work is finished or not, and the holder cannot tell that it has ended. A slow call or a long garbage collection pause looks the same from inside: the worker carries on, holding something it lost minutes ago.

We sent credit notes that afternoon and moved the guarantee out of the lock. Invoices have a unique constraint on customer and billing period, and the email goes out from an outbox only after that insert succeeds, so a second worker fails at the database instead of at the customer's inbox. The lock remains, to save wasted work, and each acquisition carries a fencing token, a number that only goes up, which the database checks on every write. A worker releases a lock with a script that deletes the key only if it still holds its own value. Workers renew their lease every twenty seconds while they work, and abandon the job if a renewal fails. And we alert on any lock that expires while its holder is still running.

Every expiry on a lock is a guess about how long the slowest job can take. Ours was a good guess for four years, and then the work grew.

– Sergey Shinder

I renamed an Argo CD application and it deleted everything it managed

Sergey Shinder — Tue, 13 Oct 2026 05:57:15 +0000

In September I renamed an Argo CD application and took our payments API down for six minutes.

We deploy with an app of apps. A parent Application watches a directory of child Application manifests, one per service, with automated sync and pruning turned on. Each child carries the resources finalizer, copied from Argo's own examples. That finalizer means: when this Application is deleted, delete everything it manages first.

We were preparing a second region, so I renamed the payments application to payments-eu1. One line of the manifest and one file name, approved by two people. To Kubernetes there is no such thing as a rename. There is a new object called payments-eu1 and a missing object called payments.

The parent synced a minute after the merge. It created the new Application, and because pruning was on and the old manifest had gone from git, it deleted the old one. The finalizer did what it was written to do. Argo deleted every resource the old application managed: the Deployment, the Services, the ConfigMaps, the autoscaler. The pods terminated. The new application synced, found nothing there and created all of it again from the same manifests, which took six minutes of image pulls, readiness checks and an autoscaler starting again from its minimum. The only things that survived were the database volumes, because claims created from StatefulSet templates were never in git and Argo had never managed them.

The diff had told the truth. It just did not say delete anywhere, because the deletion was written in a finalizer on another object, not in the change.

Child Application manifests now carry the Prune=false sync option, so the parent never deletes an application because its file disappeared. It marks it out of sync and waits for a person. Front door and stateful resources carry Delete=false, so even a cascade leaves them where they are. Renaming has a written procedure: create the new application, remove the finalizer from the old one, let the new one adopt the resources, and only then remove the old manifest. And a check comments on any pull request that removes or renames an Application file, and requires an approval from the platform team.

In a GitOps repository a file name is not a label. Removing a file is an instruction, and the system will carry it out with exactly the thoroughness you gave it.

– Sergey Shinder

One webhook in ten failed its signature check because of a trailing zero

Sergey Shinder — Tue, 13 Oct 2026 05:51:53 +0000

A marketplace that sells our products sends us a webhook for every order, signed with an HMAC SHA-256 of the request body, which we check before accepting anything. In April we moved the receiver out of an old service and into our Node gateway. Nine days later sellers asked why some orders had never reached their dashboards.

About one webhook in ten had been rejected with a 401 since the move. The marketplace retried each one for three days and then gave up on it, as their documentation says they will.

The gateway applies a JSON body parser to every route, so our handler received an object rather than bytes. To check the signature the new code turned the object back into text with JSON.stringify and computed the HMAC over that. For most payloads this gives back exactly what was sent: the marketplace writes compact JSON, and JavaScript keeps the keys in the order they arrived. Numbers are different. The marketplace writes amounts with two decimal places, so a total arrives as 24.50. Parsing turns that into the number 24.5, and stringify writes 24.5. One byte shorter, a different HMAC, and a perfectly genuine order refused as a forgery. Any amount ending in a zero failed, whole pounds included, since 30.00 comes back as 30. That is roughly one order in ten.

Our tests passed because their fixtures had been written with JSON.stringify. Text that our own code produced always survives a trip through our own code.

We replayed nine days of events through the marketplace's events API, eleven hundred and forty orders, and sellers had them by the evening. The signature is now computed over the raw bytes of the body, captured on that route before any parsing, and compared in constant time. The body is parsed only after it has been verified. Test fixtures are real payloads captured byte for byte from the marketplace's sandbox, and they include amounts ending in zero, escaped characters and a long description. And 401s on webhook routes are a metric with an alert above one percent, because a signature failure is either an attack or a bug of ours, and either one is worth knowing about the same hour.

A signature is computed over bytes, not over meaning. Our code had checked that the order meant what it said, and then hashed a different spelling of it.

– Sergey Shinder

Maintenance on our staging cluster muted the alert for our public website

Sergey Shinder — Tue, 13 Oct 2026 05:46:41 +0000

On a Tuesday night in July our public website returned errors for thirty five minutes. The external probe that checks it every minute fired the whole time, and nobody was paged. The alert was there in Alertmanager, active, and marked as inhibited.

Two years earlier we had added an inhibition rule: while a ClusterMaintenance alert is firing, mute warning and critical alerts that have the same value for the label cluster. The maintenance alert comes from an exporter that reads our change calendar. In June we replaced that exporter with one that reads the new calendar, and it labelled each maintenance window with target instead of cluster.

The rule compares the labels listed under equal, and Alertmanager treats a missing label as equal to another missing label. Its documentation says so: missing and empty are the same thing. A maintenance alert with no cluster label now matched every alert that also had no cluster label. Those were precisely the alerts that do not belong to any cluster: the external probes of the website, certificate expiry, the checks on our payment provider. Staging has a two hour maintenance window every Tuesday night. For five Tuesdays every global alert we had was muted for two hours, and on the fifth something actually broke inside the window.

Nobody saw it because an inhibited alert looks just like one that never fired. It does not reach the channel, the pager or the morning summary, and it shows up only to somebody who opens Alertmanager and asks for inhibited alerts.

The new exporter labels windows with cluster again, and the change calendar rejects a window without one. Every inhibition rule now requires each label in its equal list to be present and non empty on both sides, written as a matcher such as cluster=~".+", and a lint step in the repository fails any rule that leaves it out. Alerts that belong to no cluster carry scope="global", and no inhibition rule may target them. And a small job asks the Alertmanager API for inhibited alerts every five minutes and posts any page level alert that has been inhibited for more than half an hour, with the alert that is doing the inhibiting.

Two missing labels are equal in the same way that two empty envelopes hold the same letter. Our rule had been comparing envelopes for five weeks.

– Sergey Shinder

A service called redis gave every new pod a port it could not parse

Sergey Shinder — Tue, 13 Oct 2026 05:41:29 +0000

On a Wednesday in March new pods of our basket service began crashing at startup with invalid port: tcp://10.96.41.7:6379. Pods that were already running were fine. A deploy that morning stopped at two pods of twelve, so we rolled it back, and the pods from the rollback crashed in exactly the same way. That was the first useful clue, because it meant the code was not the problem.

The basket service reads REDIS_HOST and REDIS_PORT. Our manifest set only the host, because the code defaults the port to 6379 and that had always been right. That morning the platform team had given the basket team its own Redis, in our namespace, behind a Service called redis.

By default Kubernetes gives every container a set of environment variables for every Service in its namespace, as it stood when the pod started. For a Service called redis these include REDIS_SERVICE_HOST and REDIS_SERVICE_PORT, and also, for compatibility with the links of early Docker, REDIS_PORT, set to tcp:// followed by the address and port. A variable declared in the pod spec wins over these, so anything we set ourselves was safe. The one value we had left to a default in code was now set for us, in a format our parser rejected. Running pods kept the environment they had started with, which is why only new pods failed and why it looked so much like a bad release.

It cost us more than a stuck deploy. The autoscaler's new pods crashed too, so through the lunchtime peak ten pods did the work of fourteen.

We set REDIS_PORT explicitly within twenty minutes, then looked at what else we were being given. Our namespace had sixty three Services, which meant every pod in it carried over four hundred variables nobody had asked for. Our shared chart now sets enableServiceLinks to false, and an admission policy does the same for any pod that slips past the chart. Every setting our services read from the environment now carries the service's own prefix, BASKET_REDIS_PORT for example, so no Service name can ever collide with it. And at startup each service logs where every setting came from: the manifest, a file or a default.

An environment variable has no owner. The process that reads it cannot tell who wrote it, and in a shared namespace the writer can be anyone who creates a Service.

– Sergey Shinder

Our ninety day retention rule had not deleted a recording in nineteen months

Sergey Shinder — Tue, 13 Oct 2026 05:36:17 +0000

In August a customer asked for a copy of everything we held about them, and the export included phone call recordings from early 2024. Our privacy notice says recordings are deleted after ninety days. Our Terraform said the same thing, and it had said it correctly for two years.

The recordings bucket is created by our storage module, with a lifecycle rule that expires every object ninety days after it is written. We wrote it in 2022 and it worked. In early 2024 the security team rolled out a baseline module that turned on versioning for every bucket in the account, as protection against ransomware and careless deletes. It was a good change, applied to forty buckets, ours among them.

On a versioned bucket an expiration does not delete anything. S3 adds a delete marker on top of the object, the object disappears from normal listings and from the console, and the data stays underneath as a noncurrent version. A noncurrent version is kept until a rule for noncurrent versions says otherwise. Ours had no such rule, because when it was written the bucket had no versions to talk about.

So from the day versioning was turned on, every recording expired on schedule and none of them went anywhere. The console showed ninety days of files, exactly as promised. The bucket held forty one terabytes where we expected about six. The storage bill had been rising for nineteen months, and everyone had put that down to growing call volumes.

The rule now has a noncurrent version expiration of seven days, which keeps the week of protection versioning was turned on for, and a second rule cleans up the delete markers left behind. Lifecycle took about two days to work through the backlog, and our data protection officer handled the conversation with the customer. Every retention rule in the storage module now includes noncurrent versions, whether the bucket is versioned today or not, because that is decided in a different module by a different team. And a weekly job lists object versions in every bucket that declares a retention period and reports the oldest one it finds, taken from the API rather than from our code.

Two modules, each correct, written a year and a half apart. A retention rule promises that data is gone, and on a versioned bucket gone only means hidden until you also say for how long.

– Sergey Shinder

A hidden folder never reached our artifact and our app links opened in the browser

Sergey Shinder — Tue, 13 Oct 2026 05:31:05 +0000

In May our mobile team asked why links in our emails had started opening the website instead of the app. It had been going on for eleven days.

Our website is a static build. One job builds it and uploads the output folder with actions/upload-artifact, and a deploy job downloads that artifact and runs aws s3 sync with the delete option into the bucket behind our CDN. Two of the files in that folder matter only to phones: .well-known/apple-app-site-association and .well-known/assetlinks.json. iOS and Android read them to decide that a link to our domain may open our app.

Our workflows referred to the upload action as v4, a tag that moves. A minor release of that action, 4.4.0, changed its default so that hidden files and directories, anything whose name starts with a dot, are left out of the artifact unless include-hidden-files is set. It was a sensible security change, made after researchers found artifacts carrying .git folders with credentials in them. Our next build produced an artifact without .well-known, and the upload step said nothing worth reading. Then the sync did exactly what the delete option is for. It removed from the bucket every file the source no longer had, including the two that had sat there for four years.

Every page still returned 200, and every test we had passed. Phones do not fetch those files on each tap. Apple serves them from its own CDN and refreshes its copy now and then, and Android checks them when an app is installed or updated. So the breakage spread phone by phone, and the first reports, from some users, were put down to settings on their phones.

include-hidden-files is now set explicitly. Every third party action is pinned to a full commit SHA, and the update bot opens a pull request with the release notes, so a default changes when somebody has read about it. The build writes a manifest of every file with its hash, and the deploy refuses an artifact that does not match it. Deletions are listed before the sync and capped at twenty files without a label. And after each deploy a check fetches both files from the public domain and fails unless they return JSON that names our app.

By convention a dot hides a file from people. Ours were the only two files on the site that no person ever reads.

– Sergey Shinder

One transaction left open for three weeks stopped every write in our database

Sergey Shinder — Mon, 12 Oct 2026 07:13:41 +0000

At twenty past eleven on a Tuesday in June our order database stopped accepting writes. Reads worked. Every insert, update and delete failed with database is not accepting commands to avoid wraparound data loss.

Postgres numbers transactions with a 32 bit counter and compares them on a circle, about two billion either way. Old rows must be frozen by vacuum, marked as visible to everyone, before the counter comes round to them again. Autovacuum does this routinely, and urgently once a table's oldest unfrozen transaction is two hundred million old. If freezing still cannot keep up, Postgres refuses new transactions a few million short of the limit rather than risk corrupting anything.

Vacuum cannot freeze rows newer than the oldest transaction still open, because that transaction may still need to see them as they were. Ours had been open for twenty three days. A reporting service took a connection from its pool, began a transaction, hit an exception on a path that never rolled back, and returned the connection idle in transaction. The pool held twenty connections for a service that rarely used three, so that one sat at the back, kept alive by a health check every thirty seconds. At around a thousand transactions a second, twenty three days is close to two billion.

Autovacuum had been running anti wraparound vacuums on our largest tables for weeks, each one finishing and achieving nothing. Postgres had been logging warnings for eleven hours: database must be vacuumed within so many transactions. They went to the log, and nobody alerted on Postgres warnings.

We terminated the session, watched vacuum work through the four largest tables, and had writes back after two hours and ten minutes. Orders waited in the checkout's outbox in the meantime, so we lost none of them, only the time.

idle_in_transaction_session_timeout is now five minutes for every application role. The pool rolls back every connection before reuse. We alert when the oldest open transaction passes one hour, and when any database's transaction age passes six hundred million, which is weeks of warning instead of eleven hours. And Postgres log lines are counted by severity as a metric.

One idle connection used no CPU, no memory and no lock worth noticing. What it held was the database's ability to forget the past.

– Sergey Shinder

Our deploy script kept every value except the ones the new chart added

Sergey Shinder — Mon, 12 Oct 2026 07:08:29 +0000

In August a node in our production cluster came under memory pressure and the kubelet evicted every checkout pod on it, while pods of other services on the same node, using more memory, stayed put. Checkout lost a third of its capacity for six minutes. The events explained it in one word: the evicted pods were BestEffort. They had no resource requests at all.

They were supposed to. Since April version five of our shared service chart had set default requests and limits for every service, sized from a month of real usage. Every service repository bumped the chart version. In staging every pod had requests. In production twenty three services out of thirty one had none.

Our deploy script runs helm upgrade with --reuse-values and sets only the image tag. It was written that way three years ago, so that values set by hand at a service's first install would survive every later deploy without anyone repeating them. What --reuse-values actually does is take the previous release's values, including the defaults of the chart that release was built from, and use them in place of the new chart's defaults. A key that exists only in the new chart's defaults never reaches the templates. Our resources block was wrapped in a with statement, so an absent value rendered nothing: no error, a valid manifest, no requests.

Staging is rebuilt every week with fresh installs, so it read the new defaults. Production releases went back three years, each inheriting the values of the one before. Services installed after April were fine, which is why the eight with requests were precisely the eight newest.

We now deploy with an explicit values file per environment, kept in git and passed on every upgrade, and the script refuses --reuse-values. Getting there meant first collecting what each production release had been carrying invisibly, with helm get values, which turned up forty one settings nobody could explain and two that contradicted what git said. Every pull request now shows a helm diff against production, so a change in defaults appears as a change. And an admission policy rejects pods without memory requests in production namespaces.

A deploy that only changes the image tag is still deciding what to do with everything else. Ours had decided, every time, to keep things exactly as they were in 2023.

– Sergey Shinder

We replayed failed payouts after the provider had forgotten them

Sergey Shinder — Mon, 12 Oct 2026 07:03:18 +0000

We pay our suppliers through a payment provider's payouts API. Every payout request carries an idempotency key built from our payout id, so that if we send the same request twice the provider recognises it and returns the original result instead of paying again.

On a Friday evening in May the provider had a partial outage. For about forty minutes their API accepted payout requests, executed them, and failed to send us a response, so our client timed out after thirty seconds. We retried three times, timed out three times, and moved the requests to a dead letter queue for a person to look at, as designed.

A person looked on Monday morning. The provider's status page said the incident was resolved, so she replayed the queue, three hundred and eighty payouts, with the same keys. Every one was accepted and paid. Two hundred and twenty nine of them had already been paid on Friday.

The provider's documentation says it keeps idempotency keys for twenty four hours. After that a key is just a string it has never seen. Our replay came about sixty hours later. We had read that sentence when we built the integration and treated it as a detail, because our automatic retries all happen within minutes. The dead letter queue had no such limit, and nobody replaying it knew the keys had an expiry date.

The duplicates came to a hundred and ninety thousand pounds. Three weeks of emails to suppliers recovered all but about four thousand.

A request that timed out is now treated as unknown, not failed. Before anything sends it again, we look the payout up through the provider's API by our own reference, which every payout carries in its metadata, and resend only if the provider has no record of it. Every message in the dead letter queue shows its age against the twenty four hour window, and the replay tool refuses anything older than twenty hours that has not passed that lookup. And a daily reconciliation compares the provider's payout list with ours and flags any supplier paid twice for one invoice.

Idempotency is a promise with an expiry date, set by the side that keeps the keys. We had put a queue whose whole purpose was waiting in front of a promise that did not wait.

– Sergey Shinder

Every deploy told our autoscaler that traffic had gone up forty times

Sergey Shinder — Mon, 12 Oct 2026 06:58:07 +0000

For four months every deploy of our search API scaled it from fourteen pods to its maximum of sixty within a minute. It stayed there for five minutes, scaled back down, and the deploy itself usually finished late.

The autoscaler scaled on requests per second per pod, read from Prometheus through the metrics adapter. Its query used a recording rule somebody had written to make dashboards faster: the sum of the request counter across all pods, stored as a series of its own. The autoscaler took the rate of that sum.

rate is built for counters, and counters only go up, so it treats any decrease as a reset: the process restarted and the counter began again from zero, and the value after the drop is counted as new increase. For one pod's counter that is exactly right. For a sum across pods it is badly wrong. When a deploy replaced a pod that had served three million requests, the sum fell by three million, rate saw a reset, and counted the remaining forty million as requests that had arrived inside its two minute window. To the autoscaler that was traffic forty times our peak.

The rollout paid for it. The namespace has a resource quota sized for normal peak plus the surge of a rolling update. With sixty pods running, the quota refused the new version's surge pods, the rollout waited, and twice the deploy job reached its timeout and rolled back a perfectly good release.

The fix was to swap two words: rate first, then sum. Each pod's counter resets on its own and rate handles that, and a sum of per pod rates has no resets to misread. Across our recording rules we found nine more taking a rate or an increase of a sum, including the request panel in our capacity report. A check in the rules repository now rejects any rate, irate or increase whose argument is the output of another recording rule. The autoscaler may at most double the replica count per minute, so a wrong number can only be so wrong. And every deploy now annotates the request graphs, so a spike that lines up with a deploy is visible as exactly that.

Counter arithmetic has an order. Sum a set of counters first and every restart inside the set looks, to rate, like the whole fleet starting again from zero.

– Sergey Shinder

Our month end reports failed because the container had sixty four megabytes of shared memory

Sergey Shinder — Mon, 12 Oct 2026 06:52:57 +0000

Our reporting database moved into Kubernetes in June: Postgres 15 in a StatefulSet we wrote ourselves, the same version, the same configuration file, more memory than the virtual machine it replaced. It ran for three weeks without a complaint. On the first working day of July the month end finance reports started failing, about one in three, with could not resize shared memory segment, No space left on device. The data volume was forty percent full. Small queries were fine.

The space it had run out of was /dev/shm. Postgres runs large queries in parallel, and its worker processes exchange data through dynamic shared memory, which on Linux is by default a file in /dev/shm. On the old virtual machine /dev/shm was a tmpfs sized at half the RAM, eight gigabytes. A container gets its own /dev/shm, and the container runtime makes it sixty four megabytes unless told otherwise. The pod spec has no field for that size. Daily queries used a few megabytes of it. Month end reports join three years of ledger lines with parallel hash joins, and each of them asked for hundreds of megabytes.

So it failed for the reports finance needed most and for nothing else, which is why our migration test, a week of normal daily queries, had passed.

That afternoon we switched off parallel query to get the reports out, slower but correct. The real fix is a memory backed emptyDir volume mounted at /dev/shm with a size limit of two gigabytes. Pages written to it count against the container's memory limit, so we raised the limit by the same two gigabytes; without that, a large report would have had the whole pod killed for memory instead of one query failing. The checklist for moving anything from a virtual machine into a container now has a line for every resource the old machine provided without being asked: /dev/shm, open file limits, huge pages, the process limit. And the database load test replays last month end's actual queries, not an average week.

A container gives you a filesystem that looks like a machine's, and some of its directories are much smaller than the machine's were. Ours had been sixty four megabytes from the first day, and it took a month end to ask for more.

– Sergey Shinder