DEV Community: Serguey Shinder

Replacing an Algorithm Is a Decade of Inventory Work

Serguey Shinder — Thu, 17 Sep 2026 05:05:55 +0000

The cryptography we rely on is going to be replaced. Standards bodies have published the successors, suppliers have started shipping them, and regulators in several sectors have begun attaching dates. The mathematics is not our problem and will arrive more or less on time.

Our problem is that almost no organisation can say where its cryptography currently is, and I include my own.

We tried to produce the list. The certificates we issue were straightforward, because we already run an inventory for expiry reasons. Everything else was archaeology. A payment file signed daily with a key generated in 2014 by a process nobody documented and a person who has retired. A partner integration where the algorithm is agreed in both directions, so we cannot change ours without their supplier changing theirs, on a timescale neither of us controls. Appliances with trust stores that can only be updated by a firmware release, from two vendors whose support for those models ends before the deadlines do. Archived data encrypted with keys that must outlive the data they protect, which turns a retention policy into a key management commitment nobody has costed. And a quantity of code, written by us, in which an algorithm name is a string literal.

There is also a timing argument that makes this present rather than future work. Anything confidential with a long life can be captured now and read later, so the material that matters most is the material we are protecting today.

The work that pays is not cryptographic. It is an inventory, an owner for every key, a supplier question asked at contract rather than at renewal, and the ability to change an algorithm through configuration rather than through a project. We have started by adding cryptographic use to the asset record, and by deliberately shortening certificate lifetimes, because frequent renewal exercises exactly the muscle a migration will require.

Two suppliers have given us a plan and a date. Four could not say who in their organisation owns the question.

The change itself will be handed to us. Finding every place it has to be applied is the decade of work, and it belongs entirely to us.

– Serguey Shinder

Approving a Project Cost Nothing, So We Approved Everything

Serguey Shinder — Thu, 17 Sep 2026 05:00:44 +0000

We had forty-one live projects at the start of this year. A realistic assessment of what my department can run concurrently, with the people we actually employ, is about twelve.

Nobody had ever decided to overcommit by that margin. The investment committee assesses each case on its merits, one at a time, against a budget. It has never assessed a case against capacity, because no capacity figure existed in a form the committee would recognise, and a business case that stands up on its own is very difficult to refuse on the grounds that we are busy.

What overcommitment produces is not a queue. It is a uniform slowing of everything. Six named specialists appeared in nineteen separate plans, so every one of those plans was built on a person who could give it a fraction of a week. Elapsed times roughly doubled, which meant the benefit assumptions in the cases were decaying while the work was in progress. And because attention was allocated by whoever asked most persistently, the order in which things finished had very little to do with their value.

The part that took me longest to see is that nothing was ever stopped. Pausing a project requires no decision, no meeting and no admission; it happens by itself when the people are needed elsewhere. Cancelling one requires somebody to say out loud that an approved case was wrong, and no organisation makes that easy. So the portfolio only ever grew, and a third of it consisted of work that had been effectively dormant for months while still counting as live.

We now publish a capacity statement in the same units as the plan, by team and by month, and a project is either staffed or it has not started. There is a visible queue with an order in it. Starting something requires naming what finishes or stops. And we report work started against work finished, rather than percentage complete, which is a number that can rise forever without anything being delivered.

In the first quarter we stopped four, paused six deliberately and in writing, and completed nine. Three of the four we stopped turned out to have no sponsor left at all.

Prioritisation is not deciding what matters. Nearly all of it mattered. It is deciding what waits, and somebody has to be permitted to say so.

– Serguey Shinder

The Only Thing Our Records Kept Was the Latest Answer

Serguey Shinder — Thu, 17 Sep 2026 04:55:32 +0000

A customer disputed the discount applied to an order in March. It took us three weeks to establish that we could not answer him. The record showed the rate as it stands today, and the last person to save it was a nightly integration account, in August.

Not what the rate had been in March. Not who changed it. Not on whose authority, or whether it had been changed once or four times.

Our core systems store the present. That is what they are built to do, and for the work they were bought for it is entirely sufficient. History exists in our estate in three accidental places, and each one is an accident. One application keeps a change log, retained for thirty days because that was the default. The data warehouse takes a nightly snapshot, which is why we can sometimes answer a question to the day and never to the hour. And backups hold everything, at the cost of a restored environment, a week of effort and a specialist, which is a price nobody will pay to settle a dispute about a discount.

The same gap cost us four times in eighteen months. A regulator asked when a consent flag had been set. A supplier disagreed with us about a contracted price and had better records than we did. A payroll correction could be demonstrated as wrong and not as when it became wrong. And an internal investigation ended without a conclusion because the evidence available was a current value and a last-modified date.

This was never a technology limitation. Every one of those platforms supports change tracking. It was switched off because nobody specified it, and nobody specified it because history costs storage and requires somebody to decide a retention period.

We stopped trying to keep everything. About forty fields across six systems were agreed as consequential, meaning money, entitlements, consent, prices and identity, and change capture was enabled for those alone, recording the old value, the new value, the actor and the source, kept for seven years beside the record itself. The warehouse tracks the same set. New software is now asked to demonstrate it during evaluation, and two products could not.

A system that holds only the present can tell you what is true. It can never tell you what you did.

– Serguey Shinder

If the Job Cannot Show What It Did, Somebody Will Do It Again

Serguey Shinder — Thu, 17 Sep 2026 04:50:21 +0000

Our provisioning job has worked reliably for years. It also gets checked by hand, every single time, by the engineer who triggered it, who opens the console afterwards and confirms each setting. Twelve minutes of somebody's attention, roughly ninety runs a month, on top of an automation that already did the work.

I assumed this was habit and asked the team to stop. They declined, and they were right.

Two years ago the same job reported success while quietly skipping a step. An interface accepted a request, returned a perfectly cheerful response and applied nothing, and because the script was checking whether its call had been accepted rather than whether the world had changed, it wrote a green tick and moved on. The gap was found several weeks later by somebody who could not understand why a group had no members. Since that morning nobody on the team has believed a green tick, and no amount of encouragement from me was going to alter that.

The defect was in the reporting rather than in the logic. Our automation told us that it had run. It did not tell us what it had changed, and an exit code of zero is a statement about a script, not about a system.

So every job now finishes by reading back the state it was asked to create and writing a record of it: what existed before, what exists now, what it changed, and what it deliberately skipped and why. The last of those was the surprise. About four percent of runs were skipping something, silently and by design, in branches written years ago by people who had reasonable intentions and no way to report a partial outcome.

A daily reconciliation compares intent against reality for the whole estate, so drift is reported rather than discovered. And we set a date, with a name against it, for removing the manual check, because a habit formed by one bad morning does not dissolve on its own and somebody has to be accountable for deciding it is safe.

Automation is not trusted because it works. It is trusted because it can be checked, and the check has to be cheaper than doing the job again.

– Serguey Shinder

The Data Stayed in the Region and Almost Nothing Else Did

Serguey Shinder — Thu, 17 Sep 2026 04:45:10 +0000

A customer asked us, reasonably, to describe every country in which their data is processed. I expected the answer to take a day, because our contracts specify storage in our own region and I had checked that personally when we signed them.

The contracts were accurate. Data at rest stayed where it was promised. Everything else moved.

Diagnostic telemetry from the platform, which includes record identifiers and whatever free text appears in an error message, went to the supplier's global logging estate. Backups replicated to a paired region in a neighbouring jurisdiction, which is a documented default on that service and one we had accepted at provisioning without noticing. Support operates around the clock from three countries, which is the entire reason it can answer at three in the morning, and any file we attach to a ticket is readable by whoever picks it up. A recently enabled assistant feature processed its requests somewhere else again, and had been switched on by the supplier as part of a release. A sub-processor handled outbound email.

None of this was concealed. Every item was described in documentation we had accepted, in the pages nobody reads after the commercial terms are agreed.

What I took from it is that residency is a property of flows rather than a property of storage, and the storage is the easy half. Data leaves a region for operational reasons, continuously, in small quantities, for logging, for support, for analytics and now for features that did not exist when the contract was signed.

We produce a processing map for each significant service now. One page: where it runs, where backups go, where telemetry goes, who can read a support ticket, which sub-processors exist, and what happens when the supplier enables something new. It is refreshed annually and at every renewal, and it is the document we hand a customer rather than the contract clause.

Where it mattered, we bought commitments on backup region and on support access location. One supplier charged for it, one refused outright, and the refusal was the most useful thing we learned that quarter.

We had bought a promise about where data sits. We were being asked a question about where it goes.

– Serguey Shinder

Break Glass Had Become the Front Door

Serguey Shinder — Thu, 17 Sep 2026 04:39:59 +0000

We built the privileged access arrangement properly. A vault, credentials checked out against a stated reason, time-limited sessions, recording, and an approval for anything at the highest tier. It was a year of work and it was the right design.

Then I asked how often it was used. Four thousand one hundred check-outs in twelve months, by twelve people, with a median session length of most of a working day.

That is not an emergency control. That is how the team works, described flatteringly. When we sat down with the engineers and went through a fortnight of check-outs line by line, the reason was neither laziness nor bravado. Role-based permissions had been completed for the platforms that were easy and abandoned on two of the ones that mattered, so the only route to perform an ordinary daily task on those systems was to take the emergency credential. Everybody knew this. It had simply never been written down anywhere that a manager would read.

Underneath that sat the things the vault had never covered. The storage array, the hypervisor consoles and the building management system each had a single shared administrator login. An application whose supplier requires one named support account, shared among their staff. Four suppliers logging in through a single credential each, so that the person on the other end of any given session was somebody we could not name. When an auditor asked who had performed a specific action, the honest answer was a group of eleven.

The fix that worked was not tighter control of the emergency path. It was making the ordinary path sufficient, because the emergency path is only abused when the ordinary one is inadequate. We finished the role definitions we had abandoned, and check-outs fell by about eighty percent within a quarter. What remains is genuinely rare, reviewed weekly with a name against it, and it now raises a page, because if something is an emergency then somebody ought to be told it is happening.

Supplier staff have individual accounts in our directory or they do not get in. Two objected and both complied.

A control invoked four thousand times a year has stopped being a control. It has become a process, and nobody decided to make it one.

– Serguey Shinder

Half of Our Successful Changes Were Never Verified

Serguey Shinder — Thu, 17 Sep 2026 04:34:48 +0000

Our change success rate last year was ninety-eight point six percent, and I have quoted that figure in three governance meetings. It took an afternoon of reading to work out that it does not mean what everybody in those rooms assumed.

A change is marked successful in our process when the implementer records that it completed and no incident is raised in the hour afterwards. That is a statement about whether anything broke. It says nothing whatsoever about whether the change achieved the thing it was raised to achieve, and when we took a sample of two hundred standard changes and asked what evidence existed for that, roughly half had none beyond the service continuing to run.

The examples were not dramatic, which is the point. A firewall rule added to the correct device and the wrong group, so it matched nothing and blocked nothing. A backup schedule amended on a policy that no longer had any systems attached to it. A monitoring threshold updated on the template, which applied to new hosts and left the existing three hundred as they were. A security patch installed correctly on a server where nobody restarted the service, so the process kept the old library loaded for four months. Every one of those was implemented by a competent person, closed the same day, and counted towards the figure I was quoting.

The cause is structural rather than careless. The person who raises a change describes an outcome, the person who implements it performs a task, and only the first of them can tell whether the outcome occurred. Our form had a field for implementation notes and nowhere at all to say what would be observably different afterwards.

So the change record now carries a verification statement, written by the requester before approval, in the form of something that can be looked at. Evidence is attached before closure. A change that is implemented but not confirmed goes into a state of its own rather than counting as either outcome, and one in ten is checked by somebody who was not involved.

The rate fell to ninety-one percent immediately, and the number is now worth something. We had spent years measuring whether the work happened, and no time at all measuring whether it worked.

– Serguey Shinder

Nothing Was Late for a Technical Reason

Serguey Shinder — Wed, 16 Sep 2026 08:09:27 +0000

I went back over twelve completed projects and compared two numbers for each: the effort recorded against them, and the elapsed time from approval to go live. The gap was large and consistent, so I asked the project managers to account for it week by week.

The median project spent forty-one percent of its elapsed time waiting for a decision that nobody in my department was entitled to make.

They are not dramatic decisions. Which department owns a process that crosses three of them. Whether a data owner will sign off a field being shared. A legal opinion on one indemnity clause. Confirmation that a budget line released in principle in March is available in fact. Agreement on what should happen to the four hundred records that do not fit the new model. Each of these took between three and nine weeks, and almost none of the time was spent thinking. It was spent finding a forum, getting on its agenda, discovering the right person was absent, and going round again.

Our reporting hid this perfectly. A project sitting in amber with the note awaiting business input looks like a project being managed. It is a project that has stopped, and because the status stays the same for six weeks, nobody escalates, and the eventual conversation is about a missed date rather than about the eleven weeks of stillness that produced it.

Three changes, and none of them cost anything. Every project keeps a decision log alongside its milestones, with a question, a named decider and a date, reported at the same level as delivery progress. There is a standing forty-five minute slot each week attended by people with delegated authority from each function, which exists solely to answer questions that have got stuck. And decisions have a default: if no answer arrives within ten working days, the recommended option proceeds and is recorded as having been taken by the clock. That last rule has been invoked twice, and its real effect is that it almost never needs to be.

One uncomfortable finding stayed with me. A third of the slow decisions were slow because we had framed the question badly, offering the business a technical choice instead of a business one.

– Serguey Shinder

The Next Wave of Shadow IT Will Be Built, Not Bought

Serguey Shinder — Wed, 16 Sep 2026 08:09:12 +0000

For the last decade, unsanctioned technology arrived by credit card. Somebody signed up for a service, it appeared on an expense claim eventually, and the way we found it was to read the card statements. Imperfect, but it worked, because the act of acquiring the thing left a financial trace outside the technology estate.

That trace is disappearing. What people are building now costs nothing to start, happens inside platforms we already own and pay for, and generates no purchase order at all.

The example that changed my thinking was modest. An analyst in finance built a scheduled flow inside our own productivity suite that pulls a report from one system, reshapes it, and writes it into another, replacing about two days of monthly manual work. It is good. It runs under her personal account, it moves customer data between two systems that have never been assessed as connected, four people now depend on its output for a regulatory return, and it exists nowhere in our architecture, our recovery plan or our access reviews. No procurement process was avoided, because none applied. Nobody did anything wrong.

What I expect over the next few years is that the volume of this rises sharply, for a plain reason: the skill required to build a working automation is falling fast, and the skill required to judge whether one is safe to depend on is not falling at all. Assistants inside the tools will write the script, connect the systems and schedule the job for anybody who can describe the outcome in a sentence. That is genuinely good. It also means the gap between what our organisation can build and what it can operate responsibly gets wider every year.

I do not think prohibition is available, and the departments that try it will simply be told less. The work that looks useful is duller. Find out what your existing platforms can already report about who is building what, because most of them know. Define the point at which a personal automation becomes something the organisation depends on, and make a supported route for it to graduate into. Put the guardrails in the platform rather than in a policy document, particularly on which connectors may touch which data.

The scarce skill will not be building things. It will be deciding what is safe to leave running.

– Serguey Shinder

Staff Labelled Everything Confidential and They Were Right To

Serguey Shinder — Wed, 16 Sep 2026 07:48:29 +0000

We introduced a four-level classification scheme with the usual care. Definitions agreed with legal, a training module, a mandatory prompt on save, examples for each level. Eighteen months later, eighty-two percent of everything created carried the second-highest label, including lunch menus, meeting agendas and a document containing the office wifi password for guests.

The instinct in my department was to treat this as a training failure and run the module again. It was not a training failure. It was people responding sensibly to the incentives we had built.

Consider what we asked of them. Several times a day, with no context about who might need the document later, choose a label whose definitions run to a page and turn on words like material and adverse. Choose too high and nothing whatsoever happens to you. Choose too low and, in the worst case, you are the subject of an investigation. Every rational person in that position picks the safe answer, and the safe answer is always up.

The consequences arrived slowly and all pointed the same way. Our data loss rules keyed off the label, so ordinary documents could not be sent to the customers they were written for, and people started using personal mail to get work done. External sharing needed an exception, so exceptions became routine and stopped being read. Encryption applied at the top two levels broke search, so nobody could find anything and duplicated it instead. And retention schedules were keyed to classification, which meant the office lunch menus are on a seven year hold.

What we do now is put the sensitivity on the container rather than on the document. A system, a site or a folder is classified once, by somebody who genuinely understands what is in it, and everything inside inherits that. Individuals choose a label only when they move something out of its home, which is the moment the decision actually matters and the only moment they have the context to make it. Known patterns are detected automatically. There are two levels and a safe default, because the difference between the middle two never once changed what a system did.

We had asked nine hundred people to make a legal judgement every afternoon, at speed, and then blamed them for being cautious.

– Serguey Shinder

Somebody Has to Absorb Every Change We Ship

Serguey Shinder — Wed, 16 Sep 2026 07:43:18 +0000

Over three years we took our release cadence from quarterly to weekly, and by every measure my department reports, that was a success. Smaller changes, fewer failures, faster fixes, a rollback that is genuinely routine. I presented the figures with some pride.

Then the head of operations asked us, in front of other people, to slow down, and my first reaction was to hear it as resistance to change. It was not. It was a capacity statement about her department, and she had the numbers I did not.

Each release with any visible effect produced a burst of contacts to the service desk: on average forty-one calls from branch staff, concentrated on the two days after. Her team of fourteen hundred people includes a large number who use our systems for one narrow task, occasionally, and who had learned that task by being shown it once. The training team was rewriting the same three process documents every few weeks and had stopped being able to keep the published version accurate. Two of her supervisors had started keeping their own crib sheets, which were wrong within a fortnight.

We had optimised our own throughput and pushed the constraint downstream, into a part of the organisation that has no pipeline, no automation and no slack, and then measured only our half.

The change was not to slow the pipeline down. Technical changes still go out continuously, because the cost of holding them is real. What we separated was deployment from exposure. Anything that alters what a person sees or does is held behind a flag and released on a published monthly date, batched, with one set of notes written for the people who do the work rather than for us. The training team gets the batch two weeks ahead. The service desk gets the same notes on the same day, which they previously did not.

Support contacts per release are down by roughly two thirds, and the flow of work through my department has not changed at all.

Automation raises how fast we can produce change. It does nothing for how fast anybody can absorb it, and that second number now sets the pace of everything we do.

– Serguey Shinder

Both of Our Circuits Ran Through the Same Duct

Serguey Shinder — Wed, 16 Sep 2026 07:32:57 +0000

We bought two internet circuits from two different carriers, at some expense, specifically so that one failure could not take the site off the network. An excavator working on a drainage scheme four hundred metres away cut both of them at eleven in the morning, and we were dark for five hours.

The contracts both said diverse routing. They were not lying, in the sense that the diversity was real somewhere further up. The last mile was not. Both fibres entered our building through the same duct, crossed the same field, and terminated in the same exchange three miles away. Carrier B, it turned out under questioning, was reselling capacity on carrier A's physical infrastructure for the segment that mattered, which is an entirely normal commercial arrangement that nobody had thought to ask about because the logos on the two invoices were different.

Then we looked at the rest of it, which was worse. The link to our cloud environment and the remote access concentrator both sat behind the same pair of circuits. The disaster recovery site's connectivity had been ordered by the same account manager at the same time, from the same carrier, and followed a route nobody had ever requested a diagram for. And the building has one point of entry for communications because that is how it was built in 1998.

What I would tell anyone buying resilience at this layer is that diversity is a property of the ground, not of the contract. The questions that matter are physical and slightly tedious. Which duct, which entry point, which exchange, which power feed, and who actually owns the fibre rather than who sends the invoice. Carriers will answer these, in writing, if you ask before you sign.

We now hold a register of circuits with their routes and their real underlying providers, refreshed annually. A second entry point into the building was expensive and is done. Card payments and telephony fail over to a wireless service, which is slower and uses a different medium entirely, which is the whole point. And the recovery test includes physically unplugging a circuit, because everything else we were told about it turned out to be a description rather than a fact.

– Serguey Shinder