DEV Community: Sergey Shinder

Our label service ran out of local ports at four hundred and seventy requests a second

Sergey Shinder — Mon, 02 Nov 2026 08:06:41 +0000

On a Monday in October, the first day of a retailer's promotion, our label service began failing about one request in five from mid morning and kept doing it until the afternoon peak passed. CPU on its two virtual machines stayed under thirty percent. Every error in its log read: [Errno 99] Cannot assign requested address.

The service is a Python application that asks our rendering service for a label PDF for each parcel, through one internal load balancer address. The call was a bare requests.post inside a helper function. Called that way, requests creates a session for the single request and closes it afterwards, so every label meant a new TCP connection and a new TLS handshake, closed by our side as soon as the response arrived.

The side that closes a TCP connection keeps it in TIME_WAIT, and on Linux that lasts sixty seconds. Until it ends, the local port it used cannot carry another connection to the same address and port. Linux takes local ports from ip_local_port_range, which by default runs from 32768 to 60999, a little over twenty eight thousand ports. Divided by sixty seconds, that lets one machine open about four hundred and seventy new connections a second to one destination before it runs out. A normal day needs about two hundred and sixty per machine. That Monday needed more than five hundred. Once the range was used up, connect failed immediately, which is why nothing looked busy.

Our load tests had never found it because the test environment spread the same traffic across four machines.

The fix was one object. The helper now uses a single requests Session shared across the process, with a connection pool sized for the number of workers, so connections stay open and are reused. TIME_WAIT sockets on each machine fell from about twenty seven thousand to under a hundred. We did not widen the port range or enable reuse of TIME_WAIT sockets, because both would have moved the limit without removing the reason for it.

node_exporter's sockstat collector reports the TIME_WAIT count, and we now alert when any machine passes ten thousand. Our load tests run at production's rate per machine rather than its total.

A connection per request is paid for in ports as well as handshakes, and each port is held for a minute after you have finished with it.

– Sergey Shinder

Helm reported a successful release while its data job was still failing in the first region

Sergey Shinder — Mon, 02 Nov 2026 08:01:29 +0000

We release our pricing service one region at a time. The pipeline runs helm upgrade with --wait in the first region, and when Helm reports success it moves on to the next, on the reasoning that a release healthy in one place is safe for the others. In October a release reached all three regions in twenty minutes, and in all three our quotes went out with the previous year's fuel surcharge.

The chart contains a Job that loads the current tariff tables into the service's database after each upgrade. It is an ordinary resource, not a Helm hook, and its name carries the release revision, because a Job's pod template cannot be changed once the Job exists and Helm would otherwise fail trying to update it. In this release the job could not read the bucket holding the tariff files, because a bucket policy change had gone out the day before. It failed, retried and kept failing.

Helm said success anyway. In Helm 3, --wait waits for the pods, claims and services in the release to be ready, and for Deployments and StatefulSets to reach their minimum number of ready pods. It does not wait for Jobs. That takes a second flag, --wait-for-jobs, which makes Helm wait for every Job in the release to complete within the same timeout. Our new pods were ready in two minutes, serving whatever tables the last successful load had left behind, and Helm marked the release deployed while the job was on its second retry.

The gate between regions checked the signals that could not see the failure. Quotes were still being returned, error rates were flat, and a stale surcharge is not an error on anybody's dashboard.

The pipeline now passes --wait-for-jobs, so a failing load fails the release in the first region and stops it there. The tariff job records the version of the tables it loaded, the service exposes that version, and the gate compares it with the version the release was meant to carry. The release script also checks that every Job the release created has succeeded before anything is promoted, because we would rather not depend on a single flag again.

Success from a deployment tool is a statement about the things it was told to watch. Ours was watching the pods and had never been asked about the job that mattered.

– Sergey Shinder

One apostrophe in a delivery address ended our despatch advice in the middle of a segment

Sergey Shinder — Mon, 02 Nov 2026 07:56:17 +0000

We send a despatch advice to a supermarket chain for every lorry we load for them. Each one is an EDIFACT DESADV message listing the pallets, the cases on them and where they are going, and the chain's distribution centres will not book in a delivery that arrives without one. On a Friday in October every message we sent them was rejected, thirty eight lorries arrived unannounced, and each was booked in by hand while the drivers waited.

EDIFACT is built from separators. By default a segment ends with an apostrophe, data elements are separated by a plus sign and the components inside them by a colon. A question mark is the release character: written before any of those characters, including itself, it means take the next one literally. A name stored as O'Neill has to go out as O?'Neill.

That week the chain had opened a store at St John's Retail Park, and the address went into the NAD segment for the delivery party exactly as it was typed. Our generator built segments by joining strings with separators, and nothing in it knew about the release character. The apostrophe ended the segment after St John. What followed it, s Retail Park and the rest of the line, was read as the start of a new segment with a tag nobody had heard of.

The chain's translator reported the error in a CONTRL message, the syntax report EDIFACT partners send back, and rejected the whole interchange. We put each day's messages for the chain into one interchange, so a single address took every lorry down with it. The CONTRL arrived within ten minutes. It went into a folder that our integration platform archives without reading, because in four years every CONTRL we had received had said accepted.

The generator now writes every value through an encoder that applies the release character, and its tests include an address with an apostrophe, a plus sign, a colon and a question mark. Every CONTRL that comes back is parsed, and anything other than accepted raises an alert. Each despatch advice now travels in its own interchange, so one bad value can delay only one lorry.

A format built from separators needs an encoder in the same way SQL needs parameters. Our template had worked for four years because nobody had put a separator in a street name.

– Sergey Shinder

Our disk alert waited for ninety seven percent and Postgres stopped at ninety five

Sergey Shinder — Mon, 02 Nov 2026 07:51:05 +0000

On a Sunday night in October the database behind our returns service shut itself down with a PANIC in its log: it could not write to a file in pg_wal because there was no space left on the device. Nobody was paged. Our disk alert pages at ninety seven percent full, and the dashboard for that volume, when we opened it on Monday morning, said ninety five.

The volume is two terabytes of ext4, and for weeks it had been filling slowly with a table of return photographs nobody had archived. The warning at ninety percent had raised a ticket on the Tuesday, and the ticket was planned for the following week, because at that rate the page would arrive with days to spare.

Both numbers on our dashboards came from node_exporter, and both were correct. It exports two figures for free space. node_filesystem_free_bytes counts every free block. node_filesystem_avail_bytes counts the free blocks an ordinary user is allowed to write to. The difference is the reserve ext4 keeps for root, which by default is five percent of the filesystem, set when it is created. On our volume that is about a hundred gigabytes that Postgres, running as the postgres user, can never touch.

Our alert and our panel were both built on free_bytes. When Postgres wrote the last block it was allowed, the expression still saw a hundred gigabytes free and reported ninety five percent. It could never have reached ninety seven. df, which bases its percentage on the available figure, would have said one hundred.

We had Postgres running again within twenty minutes of looking, by lowering the reserve to one percent with tune2fs -m 1, which takes effect at once on a mounted filesystem. The photographs were archived properly that afternoon.

Every disk alert and panel now uses avail_bytes, the figure our applications actually live within. Volumes written only by a service account are created with a one percent reserve, because the reserve exists to keep a system disk usable for root and does nothing for a database volume. And before a new disk alert is trusted, we fill a scratch volume as an ordinary user until writes fail, and check what the alert said at that moment.

On Linux, how much space is left has two answers. Our alert had been asking on behalf of root.

– Sergey Shinder

Our third search replica waited six hours for a node in a zone our node group did not cover

Sergey Shinder — Mon, 02 Nov 2026 07:45:53 +0000

We run OpenSearch for parcel search as a StatefulSet on EKS, on a node group of memory optimised instances. In October, ahead of the peak season, we scaled it from two replicas to three. The third pod sat in Pending for six hours, through the busiest evening of the month, while two nodes carried the load we had sized for three.

The event on the pod said that a node had a volume node affinity conflict. The cluster autoscaler had not added anything. It had considered the pod and concluded that no node it was able to create would help.

The memory optimised node group spans only two of our three availability zones, eu-west-1a and eu-west-1b, because the instance type was not offered in the third zone when we built it. Each replica keeps its data on an EBS volume, and an EBS volume lives in exactly one zone and can only be attached to an instance in that zone. The new replica's volume had been created in eu-west-1c.

That came from our StorageClass. A storage class has a volumeBindingMode, and when it is not set the default is Immediate: the volume is provisioned as soon as the claim exists, before the scheduler has chosen a node for the pod. The provisioner picked a zone with no idea where the pod could run, and from then on the pod was tied to eu-west-1c, where we had no node of the kind it needed. The first two replicas had landed in a and b by luck.

The default class that EKS creates uses WaitForFirstConsumer. Ours was a gp3 class someone wrote two years ago without the field. With WaitForFirstConsumer the claim waits until a pod using it has been scheduled, and the volume is created in that node's zone.

The fields of a storage class cannot be changed once it exists, so we created a new class with the mode set and made it the default. Our claim template names no class, so we deleted the stuck pod and its empty claim and let the StatefulSet create both again. A policy check now rejects any StorageClass without WaitForFirstConsumer, and any pod Pending for more than ten minutes raises an alert.

Immediate binding decides where the data will live before anyone has decided where the work is able to run.

– Sergey Shinder

We changed the user data on our integration server and Terraform restarted it without running a line

Sergey Shinder — Mon, 02 Nov 2026 07:40:42 +0000

Our integration hub is one EC2 instance in a private subnet. It runs the small services that exchange files and API calls with our carriers, and Terraform builds it with a user data script that installs packages, fetches certificates and writes the configuration for each service.

In September one carrier moved its API behind a certificate issued by a new private authority and sent us the root to trust. The engineer handling it did the reasonable thing for a server defined in code. He added two lines to the user data script, one to download the new root and one to run update-ca-certificates, and opened a pull request. The plan showed a single change, user_data updated in place. It was approved and applied at eleven on a Tuesday morning.

Updated in place did not mean what either of us read it to mean. AWS only lets you change an instance's user data while the instance is stopped, so for this change the AWS provider stops the instance, writes the new user data and starts it again. Every carrier integration on the hub was down for about four minutes, in the middle of the morning collection run.

Then the instance came back and nothing in the new script ran. Cloud-init runs user data scripts once per instance, and it recognises an instance by its ID. A stop and start keeps the ID, so cloud-init saw a machine it had already set up and skipped the scripts entirely. The new root was never installed. Calls to that carrier kept failing for two more days, while the ticket said fixed, until someone ran the two lines by hand.

So the change caused an outage, fixed nothing, and was described by the plan as a minor update.

We decided what user data is for. On the hub it now does only what has to happen at first boot, and everything after that, certificates included, is applied by Ansible, which we can run again at any time and which reports what it changed. Instances we treat as immutable set user_data_replace_on_change to true, so a change builds a new instance that really does run the script. Any plan that touches user_data on anything else needs a second reviewer.

User data runs when a machine is first built. Changing it afterwards costs a restart and does nothing to the machine you already have.

– Sergey Shinder

Our deploy workflow waited for CI to finish and never asked whether it had passed

Sergey Shinder — Mon, 02 Nov 2026 07:35:31 +0000

In September we split the pipeline for our parcel tracking API in two. One workflow, CI, ran the tests on every push to main. A second, Deploy, built the image and released it to production, and it was triggered by workflow_run: when CI completes on main, run Deploy. The split let us rerun a deploy without rerunning forty minutes of tests, and for three weeks it looked like an improvement.

On a Thursday in October a change to how we parse carrier scan codes reached production, and for an hour and a half every tracking page for one carrier showed its parcels as unknown. CI had failed on that commit. Two tests had caught the bug exactly, the commit was marked red, and the deploy had gone out eleven minutes later anyway.

The trigger did what it said. A workflow_run event of type completed fires when the other workflow finishes, whatever its conclusion. Success, failure and cancelled are all completed. GitHub's documentation shows the check you are meant to add, a condition on github.event.workflow_run.conclusion, and our Deploy workflow did not have one. For three weeks every commit on main had been deployed whether its tests passed or not. We had simply not merged anything broken until that Thursday.

Reading the workflow again, we found a second assumption. A workflow started by workflow_run runs in the context of the default branch, and a plain checkout step fetches the latest commit on main, not the commit CI tested. Two merges close together could have shipped a commit whose tests had not even finished.

Deploy now has a condition on its first job requiring the conclusion to be success, and it checks out github.event.workflow_run.head_sha, so the commit it builds is the commit that was tested. That sha is written into the release record, and the job fails before building anything if it cannot find a successful CI run for it. Once a quarter we push a deliberately failing commit to a copy of the repository with the same workflows, to prove the deploy refuses it.

An event that tells you something has finished is not telling you how it finished. We had connected a release to a signal and never read the one field that carried the answer.

– Sergey Shinder

Our Redis replica spent three days restarting the same sync

Sergey Shinder — Sun, 01 Nov 2026 13:22:09 +0000

In October we patched the host running the primary of our delivery tracking Redis, expecting Sentinel to promote the replica as it always had. It did not. The driver app could not sync for six minutes while the host rebooted, and when we looked at the replica it had not held a current copy of the data for three days.

Three days earlier there had been a forty second network interruption during switch maintenance. When a replica reconnects, the primary tries a partial resync from its replication backlog, a ring buffer of recent writes. Ours was at the default size of one megabyte. We write about fifteen megabytes a second at peak, so the backlog covered well under a second, and a forty second gap meant a full resync: the primary forks, produces a snapshot of the whole dataset and streams it to the replica.

While it does that, every new write for the replica is held in that replica's output buffer on the primary. The default limit for replica connections is 256 megabytes outright, or 64 megabytes held for sixty seconds. When the replica was first set up the dataset was four gigabytes and a full sync took under a minute. It was now twenty two gigabytes and took about four minutes, which at our write rate is far past both limits. The primary closed the connection, the replica reconnected, could not resync partially, and the whole thing began again. Roughly every four minutes for three days, each time with a fork of a twenty two gigabyte process.

Sentinel was right not to promote it. A replica whose link to the primary had been down that long had nothing current to offer.

The backlog is now 512 megabytes, enough for about thirty seconds at peak, so short interruptions resync partially. The replica output buffer limit is sized from the measured duration of a full sync multiplied by our peak write rate, with room to spare. We alert when master_link_status on a replica is down for more than two minutes and whenever the sync_full counter on the primary increases. And every quarter we force a full sync in staging with production's dataset size and time it.

Those limits were set for the database we had in 2023. The data kept growing and the limits stayed where they were.

– Sergey Shinder

We moved one Go variable and every copy of our new CLI was told to upgrade

Sergey Shinder — Sun, 01 Nov 2026 13:16:58 +0000

We publish a command line tool that our partners use to book collections and download labels from our API. Version 2.7.0 went out on a Tuesday in September. By Wednesday afternoon support had forty tickets, all saying the same thing: every command answered with a message telling them their client was too old and to upgrade. They had all just upgraded.

The tool learns its own version at build time. The release workflow builds it with go build and an ldflags option, -X main.version followed by the tag, which tells the Go linker to set the string variable version in package main to that value. The variable defaults to dev, so a laptop build says so.

In 2.7.0 a tidy up moved build information out of main into a package of its own, internal/buildinfo, and the variable went with it. The release workflow still said main.version. The Go linker does not treat an -X naming a symbol that does not exist as an error, or even a warning. It ignores it. The build was green, and every binary we signed and published believed it was version dev.

The tool sends its version in a header on every request, and since the spring our API gateway has refused clients older than 2.4.0, because we removed an endpoint those versions depend on. The check parsed the header as a semantic version, and anything it could not parse it treated as 0.0.0. So anyone who stayed on 2.6 was fine, and everyone who did exactly what we asked was refused.

None of our tests could see it. The end to end suite runs the tool from source with go run, and no linker flag is involved there.

The release workflow now runs the built binary with its version command and fails unless the output matches the tag, character for character, before anything is signed. The ldflags line uses the full import path of the new package. The gateway no longer turns an unreadable version into an old one. It lets the request through and counts it, and a version of dev in production pages the release owner.

A default that looks like a real value will travel all the way to production without anyone noticing it. The build should have to prove what it stamped.

– Sergey Shinder

The schema registry approved a change that only worked if we deployed first

Sergey Shinder — Sun, 01 Nov 2026 13:11:47 +0000

On a Tuesday in October the carrier collection for our Leeds warehouse left half empty, and about six thousand picked orders sat on the dock overnight. They had been picked, packed and announced on Kafka exactly as normal. Our carrier integration had read every one of those announcements and parked each of them on its dead letter topic as unreadable.

The topic belongs to the warehouse team. Every message is Avro, registered in a Confluent Schema Registry, and our consumer uses classes generated from the schema as it stood when we last built, so it reads each record through that schema. That afternoon the warehouse team released a new version that dropped depot_code. In our generated schema depot_code had no default. Avro can resolve a record that is missing a field only if the reader's schema says what to fill in, and ours did not, so every message written after two o'clock failed to deserialise.

The warehouse team had done what they were supposed to do. The registry checks every new schema before accepting it, and it accepted this one. The subject used the registry's default compatibility mode, BACKWARD, and backward compatibility asks one specific question: can a consumer using the new schema read data written with the old one? Removing a field passes that easily. What it assumes is that consumers move to the new schema before producers start writing it. Their own consumers had. We were in another team, on another release cycle, and nothing in the check knew we existed.

The question we needed answered was the opposite one, whether a consumer still on the old schema can read what the new one writes. That is FORWARD compatibility, and under it the registry would have rejected removing a field with no default. Every subject with a consumer outside the team that owns it is now set to FULL_TRANSITIVE, which checks both directions against every earlier version. The warehouse team put depot_code back with a default and we replayed the dead letter topic that evening. Our generated schemas now give every field we read a default, and the dead letter topic alerts on its first message rather than its thousandth.

A compatibility check is only as useful as the question it asks. Ours was asking about deploy order, and assumed we would go first.

– Sergey Shinder

One slow count query took every metric our dispatch service had off the dashboard

Sergey Shinder — Sun, 01 Nov 2026 13:06:36 +0000

During a database slowdown in September our dispatch service failed about one request in six for twenty minutes, and our error rate alert never fired. When we opened the dashboard afterwards, it had a gap over exactly those twenty minutes. No data at all, on every panel the service had.

The service exposes its metrics for Prometheus to scrape every fifteen seconds, with the default scrape timeout of ten. Two years earlier we had added a gauge for the number of dispatches waiting to go out, implemented as a custom collector that ran a count query against the database each time the endpoint was read. Every one of eight pods ran it, and both of our Prometheus replicas scraped every pod, so the database answered that same question about once a second. On a normal day it took a third of a second.

When the database slowed down that query took around fourteen seconds. Prometheus gave up at ten, and a scrape that times out is a failed scrape, all of it. The target's up series went to zero, and Prometheus marked every series it had collected from that target as stale, the request counters and latency histograms as well as the one gauge that was slow. Our error rate alert divided one rate by another, and with no samples on either side there was nothing to compare with the threshold. The alert that did fire was TargetDown, five minutes in, routed as a ticket because a missing target is usually a deploy.

Meanwhile sixteen count queries every fifteen seconds were adding load to a database that was already struggling.

The pending count now comes from one place. A small background task in a single instance runs the query every thirty seconds with its own timeout and publishes the result, and the service's metrics endpoint serves only values it already holds in memory. We alert when scrape_duration_seconds for any target passes half its timeout. And TargetDown for a service carrying live traffic now pages, because we learned it can be the only thing still speaking.

A metrics endpoint that depends on the system it measures will go quiet exactly when that system is in trouble, and it can take everything else it reports down with it.

– Sergey Shinder

Our statement job ran out of memory at seventy percent and Kubernetes started it again from the first customer

Sergey Shinder — Sun, 01 Nov 2026 13:01:23 +0000

On the first working day of October about forty thousand of our customers received their monthly statement email twice, and a few thousand received it three times. The statement job had not been changed in a year. Our customer base had grown by a quarter since the spring.

The job runs once a month as a Kubernetes Job. It reads every account, renders a PDF, and calls our email provider for each customer in turn. Its pod had a memory limit of two gigabytes, which had been plenty when the account list was smaller. It also kept a summary of every statement sent in memory for a report at the end, and that grew with the customer count. In October it reached the limit about seventy percent of the way through. The kernel killed the container, the pod exited with code 137 and was marked failed.

A Job exists to reach a number of successful completions, and failure is something it is built to recover from. backoffLimit defaults to six, so when a pod fails the Job controller creates a new one, waiting ten seconds before the first retry and doubling the wait each time. Our new pod had no idea what the old one had done. It started at the first account, sent seventy percent of the statements again and was killed at the same point. The third attempt was most of the way through the same customers when somebody in support asked why their own statement had arrived twice, and we deleted the Job.

Nothing in that was a fault in Kubernetes. A Job's promise is that the work will finish, not that it will run once, and we had written something that was only safe to run once.

The job now records each sent statement in a table with a unique key on account and month, and checks it before sending, so a second attempt resumes where the first stopped. backoffLimit is set to zero on every Job in our cluster that talks to customers, until someone can show that running it twice is harmless. Batch jobs now alert on memory against their limit, as services do.

Retry is the default for a Job. If the work cannot be repeated, the code has to know where it stopped or the Job has to be told not to try.

– Sergey Shinder