DEV Community: Serguey Shinder

Our Next Handsets Have No SIM Tray and Our Depots Still Swap SIM Cards From a Drawer

Serguey Shinder — Tue, 03 Nov 2026 07:47:33 +0000

Each of our depots keeps a small drawer of spare SIM cards. When a driver's phone breaks, a supervisor takes the SIM out of the broken phone, puts it into a spare and hands it back, and the driver is on the road again in ten minutes with the same number. Nobody in IT is involved, which is exactly why it works at six in the morning.

This spring our mobile supplier offered the next handset model for our drivers, and it has no SIM tray. The number lives on an eSIM, a profile downloaded onto a chip inside the phone. We ran a pilot with forty handsets at two depots for three months to see what would change.

The handsets were fine. The drawer was not. Four phones broke during the pilot, and in each case the number could not simply move to another phone. Someone with rights to our account on the network's business portal had to retire the old profile and issue a new one, which the driver then scanned onto the replacement from a code. Only two people had those rights, both in my team at head office and both working office hours. One driver waited a whole morning. On the other three occasions the depot sent the lorry out with a phone borrowed from the yard, and customers spent the day ringing a number the driver did not have.

There is a security side too. Moving a number becomes something anyone with portal access can do remotely, without touching the phone. That is convenient for us, and equally convenient for anyone who takes over that portal account, so the access has to be guarded like any other system that could redirect our sign in codes.

We have about a year before the older models stop being offered, and most of the work is about people rather than technology. Each depot will have two named supervisors with limited rights to reissue a profile, signing in with security keys, and every reissue will alert my team. The whole process will be rehearsed at every depot before the first new handset arrives.

I expect more of the small physical jobs that depots do for themselves to turn quietly into permissions in somebody's portal. When the card in the drawer disappears, decide who holds the right that replaces it before the first phone breaks.

– Serguey Shinder

Three Business Cases Claimed the Same Four Planners and Only One of Them Could Be Right

Serguey Shinder — Tue, 03 Nov 2026 07:42:22 +0000

In last year's budget round three projects came to our investment committee with my department named as the delivery team. Transport wanted route planning software. The Midlands distribution centre wanted a tool that books delivery slots with customers automatically. Customer service wanted a portal where trade customers could see and move their own deliveries. All three were sensible, and all three were approved.

Each business case was written honestly by the team that wanted it, and each included the same saving. The Midlands depot uses four agency planners who build the next day's loads and ring customers to confirm them, at a cost of about one hundred and forty eight thousand pounds a year. Each case said its system would remove most of that work, and each counted the four planners as its saving. The committee looked at the cases one at a time, at different meetings, so nobody saw the same number three times.

We found out a year later, when finance compared the savings promised with the budget. Between them the three projects had promised to save four hundred and forty four thousand pounds from a cost of one hundred and forty eight thousand. The planners were still there, now working mostly on the exceptions all three systems passed to them. The agency bill had fallen by about forty thousand pounds.

Nobody had lied. Each team answered the question it was asked, which was what its own project would save, and none was asked what another project had already claimed. As the department delivering all three, mine was the only one in a position to notice, and we did not look, because benefits were not our part of the form.

Every business case that claims a saving now has to name the budget line it will come from, and finance keeps a register of the savings already claimed against each line. A second project can still claim the same line, but it has to say what is left after the first. The first time we ran it, it found two more pairs of cases counting the same people.

A saving can only be made once. If projects are approved one at a time, keep one list of what each has claimed, or the business will approve the same benefit as many times as people ask for it.

– Serguey Shinder

Our Telematics Said Our Crane Lorries Idled Two Hours a Day and the Engine Was Working the Crane

Serguey Shinder — Tue, 03 Nov 2026 07:37:10 +0000

We run forty eight lorries with a crane mounted behind the cab, which is how pallets of blocks and bundles of timber get lifted onto a building site. Every lorry carries a telematics unit that reports its position, its speed and whether the engine is running.

Last year our fleet manager started a fuel saving campaign built on one figure from that data: idle time, meaning minutes with the engine on and the lorry not moving. The crane lorries averaged two hours and ten minutes a day, against twenty five minutes for our vans. The figure went onto a ranking pinned up in each depot's drivers' room, and the twelve drivers at the top were invited to a conversation about leaving engines running.

The first of those conversations ended it. The driver pointed out that his crane is driven by the lorry's engine, through a power take off. When he is lifting twenty pallets onto a site, the engine is running and the lorry is still. That is not idling. It is the job he is paid to do.

We checked, and he was right across the fleet. The power take off has its own signal that the telematics unit could have read, but the units had never been wired to read it, and the idle report had never been asked to leave it out. When we connected it on six lorries for a month, real idle time came to about thirty five minutes a day, close to the vans. The drivers at the top of the ranking were mostly the ones making the most deliveries to sites without a forklift.

We apologised to the twelve drivers in writing, and the ranking came down. All forty eight crane lorries now report crane use, which cost about ninety pounds a vehicle to connect. Idle time is calculated with crane use taken out, and crane use has become a figure of its own, which has turned out to be useful for deciding which sites need a crane lorry at all.

A measurement can only tell apart what its signal can tell apart. Before you publish a figure about people's work, ask someone who does that work what the figure would look like on a day they did it well.

– Serguey Shinder

Our Statement Run Matched Files to Customers by Their Place in the List and Two Hundred Got Someone Else's

Serguey Shinder — Tue, 03 Nov 2026 07:31:59 +0000

Every month our credit control system produces a PDF statement for each of our six thousand trade account customers, and a script emails them out. The system writes the PDFs into one folder. A separate export gives the script the list of customers and their email addresses. For four years the script sent the first file to the first customer, the second to the second, and so on down both lists.

That worked because both lists came out of the system in the same order, by account number. In September a supplier patch changed how the PDFs were named, adding the branch code to the front of each file name. The folder now sorted by branch first. The customer list did not change.

The run started at nine. The first complaint arrived at twenty past. A roofing firm in Derby had received the statement of a plumber in Leicester, with his address, every invoice from the past three months and what he owed us. By the time credit control rang me at ten, the script had sent two hundred and twelve statements, and all but four had gone to the wrong customer. It stopped there only because our mail server limits it to two hundred an hour, which is the only reason it was not six thousand.

We stopped the job, wrote to every customer on both sides, and asked those who had received someone else's statement to delete it. Many of our trade customers are sole traders, so their statements are personal data, and we reported the incident to the Information Commissioner's Office within seventy two hours. The supplier's patch notes had mentioned the new file names. Nobody reading them knew that anything depended on the old ones.

The script now reads the account number printed inside each PDF and looks up the customer by that number. If the number in the file and the address it is going to do not belong to the same account, that statement is held back and the run carries on without it. Before each monthly run, credit control opens ten statements at random and checks them against the customer list.

Never pair two lists by position. Pair them by a value both records share, and check that it matches before anything leaves the building.

– Serguey Shinder

Our Storage Promised Our Servers Thirty One Terabytes and Owned Eighteen

Serguey Shinder — Tue, 03 Nov 2026 07:26:49 +0000

Most of our head office systems still run as virtual machines on a cluster in our own server room, with a single storage array behind them. The array has eighteen terabytes of usable space.

When we create a virtual machine, we give it disks of the size it might one day need, and the array only uses real space as data is written. This is called thin provisioning, and it is normal and sensible. A finance server given two terabytes that holds four hundred gigabytes uses four hundred gigabytes. Over six years we had created disks totalling thirty one terabytes on an array of eighteen. Each one had been reasonable on its own. Nobody had ever added them up.

Our monitoring watched each virtual machine's disks, and every one showed plenty of free space, because each was measured against the size it had been promised. The array's own alert was set at ninety five percent full and went to the mailbox of an engineer who had moved to the network team.

On the third working day of January, in the middle of month end, the finance system wrote its year end archive while the warehouse system rebuilt its indexes, and the array filled. The hypervisor did what it is designed to do when a thin disk cannot grow. It paused every virtual machine that tried to write. Twenty six stopped at once, including payroll, the finance system and depot label printing. We freed space by deleting old copies and had them running again in about three hours. Nothing was lost, but month end slipped by two days.

We now report two numbers for the array every week: what is physically used and what has been promised. The ratio between them is a decision, written down, and we have set ours at one and a half. Any new virtual machine that would take us past it has to come with a smaller disk or more storage. The array alert now warns at seventy five percent, goes to the infrastructure team's queue rather than to a person, and at our current growth gives us about four months' notice.

Thin provisioning lets you promise more than you own, and that is fine for as long as someone is counting the promises. If nobody is, the shortfall arrives all at once, and usually on your busiest day.

– Serguey Shinder

A Counter Assistant Signed the Branch Browser Into Her Own Account and Took Our Saved Passwords Home

Serguey Shinder — Tue, 03 Nov 2026 07:21:37 +0000

In July a trade counter assistant at one of our branches left to work for a competitor. A month later the branch manager noticed that somebody had signed in to a supplier's ordering portal with the branch's shared login and looked at our buying prices three days running. The sign in came from a home broadband connection.

It took us a day to work out how. The counter PC at that branch, like most of ours, is used by four or five people in a shift. Staff had saved the passwords for supplier portals, the carrier booking site and our own trade website in the browser, because typing them at a busy counter is slow. Two years earlier the assistant had signed the browser into her personal account to get her own bookmarks. The browser did what it is designed to do. It synchronised everything saved on that PC to her account, and from there to her phone and her laptop at home. When she left, the passwords left with her, and nothing on our side recorded it.

We then checked the rest of the estate. Of the three hundred and ten shared PCs at our branches and depots, sixty four had a browser signed in to a personal account. Between them they had synchronised four hundred and thirty saved passwords to accounts we did not control, belonging to thirty nine people, eleven of whom no longer worked for us. We had no reason to think the others had misused anything, but we could not show that they had not.

We changed every one of those passwords within a fortnight, starting with anything that could place an order or move money. Our device policy now allows the browser to sign in only with a work account, and saving passwords in the browser is switched off on shared PCs. The shared logins the counters genuinely need are kept in a password manager with access by role, so when someone leaves, their access goes when their account does. We told the supplier whose portal had been used, and it reset our account.

We had thought of the browser as a window onto websites. On a company PC it is also a copying tool, and it copies to whichever account is signed in to it. Decide which accounts it is allowed to copy to.

– Serguey Shinder

Our Network Supplier and Our Warehouse System Supplier Both Changed Something on the Same Sunday Night

Serguey Shinder — Tue, 03 Nov 2026 07:16:25 +0000

Our main distribution centre runs its warehouse system on servers hosted by the software supplier, reached over a private network line that a second supplier manages for us. Both have maintenance windows written into their contracts, and both use them.

In March, on the same Sunday night, the warehouse supplier upgraded the database underneath our system and the network supplier replaced the router at their end of our line. Each had told us. The warehouse supplier emailed our application team three weeks earlier. The network supplier posted a notice on its customer portal, which our infrastructure team only opened when something went wrong. Neither notice reached our change calendar, because the calendar only held changes my own team was making.

At six on Monday morning the day shift took over and the handhelds could not stay connected. The warehouse supplier's engineers saw a healthy database and a network that dropped connections after a minute. The network supplier saw a working line and an application that kept opening connections and abandoning them. For five hours each sent us evidence that the fault was the other's. It was a setting on the new router that closed idle connections sooner than the upgraded database expected, and once we put both suppliers on one call, chaired by us, they found it in forty minutes. By then the day shift had picked about a third of its normal volume, and ninety one customer deliveries went out late.

Nobody had done anything wrong on their own terms. Each change was planned, approved and announced. What was missing was one place where someone could see two changes landing on the same service on the same night and ask whether both should.

Our change calendar now holds every change that touches our services, whoever makes it. Each supplier contract names who in my team receives their notices, and that person enters them. When two changes affect one service in the same window, one of them moves, and the supplier who moves is told why. Since April the calendar has caught six clashes, two of them between suppliers who did not know the other existed.

A change calendar that only lists your own work shows you the changes you control. The changes your suppliers make to your services can break them just as easily, and they belong on the same page.

– Serguey Shinder

Our Trade Customers Want to Pay by Bank App at the Counter and Our Tills Can Only Refund a Card

Serguey Shinder — Mon, 02 Nov 2026 08:06:24 +0000

Earlier this year several of our larger trade customers asked whether they could pay at the counter by approving a payment in their banking app instead of using a card. For some, their card limits were lower than a big order. Others simply preferred it, because the money left the business account directly and their bookkeeping was simpler.

We ran a pilot at four branches through our payment provider. The till shows a code, the customer scans it with their phone, approves the payment in their banking app, and the till receives confirmation within seconds. Over three months the four branches took one thousand one hundred and eighty payments this way, with fees well below what we pay on cards. On those numbers alone it was a success.

Then the refunds arrived. Thirty seven customers brought goods back from orders paid by bank. Our till knows two ways to refund: back to the card that paid, or in cash. Neither fitted. Some branches gave cash, which drained the till float. Others sent the refund to finance, where somebody looked up the original payment, found the account it came from and arranged a transfer by hand. Each one took about twenty five minutes and a second person to approve it, and customers waited up to five days for money a card refund would have returned straight away.

There was a reconciliation gap as well. The bank payments settled in a separate report from the provider, and the branch end of day totals did not include them, so for the whole pilot finance matched them by hand every morning.

None of this was the payment method's fault. The till, the refunds process and the end of day reports were all built on the assumption that money arrives by card or by cash, and that assumption lives in places nobody thinks of as payment systems.

We have paused the rollout until refunds can go back to the paying account from the till and bank payments appear in the branch totals. The provider can do the first. The second is our own work.

I expect the ways customers pay us to keep changing over the next few years, and each new method will reach the counter long before the back office is ready for it. Design the refund and the reconciliation before you switch the payment on.

– Serguey Shinder

We Asked Our Yard Staff to Use Their Own Phones to Sign In and Forty Three Said No

Serguey Shinder — Mon, 02 Nov 2026 08:01:13 +0000

Last year we extended second factor sign in to everyone, including the yard and warehouse staff who use our systems only for payslips, holiday requests and the daily vehicle and equipment checks. There were three hundred and eighty of them. The project plan assumed each would install an authenticator app on their own phone, because that is what the office staff had done without complaint.

Forty three said no. Six did not own a smartphone. Others did not want a work app on a phone their children used, did not want to be reachable by work outside their shift, or asked who would pay if the app needed a newer phone than the one they had. The staff forum representative put the questions to the operations director, who forwarded them to me, because it was an IT project.

I had treated it as a technical choice. It was not. Whether a job requires people to provide their own equipment is a question about the terms of the job, and it belongs to the people who set those terms. IT can say what each option costs and how secure it is. It should not be deciding, by default, that staff will supply the device.

So we took it back to the people who own it. Human resources and operations agreed a position with the forum: using a personal phone is optional, and anyone who prefers not to gets a hardware security key at the company's cost. The keys cost about twenty two pounds each, so the whole question came to under two thousand pounds, less than the management time already spent arguing about it. The vehicle checks moved to shared tablets at each depot, so nobody needs a personal device to do the safety part of their job.

Seventy one people ended up choosing a key, far more than had refused. The forty three had not been the whole of the concern, only the people willing to say so out loud.

When a system needs staff to bring something of their own, that decision has a cost and an owner outside IT. Put it in front of that owner before the rollout, not after the complaints.

– Serguey Shinder

We Stopped Selling Three Hundred and Forty Products and Our Returns Screen Stopped Recognising Them

Serguey Shinder — Mon, 02 Nov 2026 07:56:02 +0000

Builders over-order. It is part of the trade, because running out of blocks on a Friday afternoon costs more than taking a few back, and our branches accept unused goods back within twenty eight days of the sale.

Last autumn our buying team ran a range review and retired about three hundred and forty products, mostly slow selling sizes and finishes. In the product system that is a single flag. The record is marked inactive, and the product disappears from the till search, the website and the replenishment run, which is exactly what the buyers wanted.

The returns screen used the same search. For the next month, whenever a customer brought back something from that list, the counter could not find it. Staff did what they could. Some turned customers away, and we had complaints. Most keyed the item under a miscellaneous code we keep for oddities, typed a description and refunded the price on the receipt.

Six weeks later finance asked why the miscellaneous code had taken six hundred and ten returns worth about thirty eight thousand pounds, against a normal month of a dozen. The goods had physically gone back on the shelves, but the system did not know, so they were never counted and could not be sold off at a clearance price. None of the refunds were linked to the original sale, so we could not check whether any had been refunded twice.

The cause was not the flag itself. It was that one flag had to answer several different questions. Can we buy it? Can we sell it? Can a customer return it? Should it appear in reports? The buyers had answered the first two, and every other system inherited their answer.

A retired product now has dates rather than a switch. It stops being bought on one date, stops being sold on another, and stays returnable for sixty days after its last sale, which covers our returns window with room for receipts that turn up late. Reports keep it for as long as it has history. We went through the miscellaneous returns by hand, matched about five hundred of them back to products, and put that stock back on the system.

Whenever a record is switched off, ask each process that reads it what switched off should mean there. The answers are rarely the same.

– Serguey Shinder

Every Update Reached Our Aberdeen Branch First Because the Rollout Ran in Alphabetical Order

Serguey Shinder — Mon, 02 Nov 2026 07:50:51 +0000

We push software to the counter PCs and tills in our forty six branches in waves. The first wave gets an update on Monday, and if nothing breaks the remaining waves follow through the week. It is a sensible design. The person who built it generated the waves by sorting the branch list by name and cutting it into groups.

That meant the first wave was Aberdeen, Ashford, Barnsley, Bedford and Bolton, every time, for two years.

I found out when the Aberdeen branch manager rang me to ask, politely, why his branch was always the one that broke. He had kept a list. Of nineteen faults we had shipped to the tills in that period, his counter had met fourteen first, and his staff had spent Monday mornings on the phone to our service desk while builders queued. Nobody had told him his branch was a test site. Nobody had asked.

The ordering also made the test weaker. The five first wave branches happened to be among our smallest, and none of them ran a timber cutting service or a large trade accounts desk. In April an update that slowed the till badly on any order with more than forty lines went through the first wave without a single complaint and reached our three busiest branches on the Wednesday.

We rebuilt the waves on purpose. The first wave is now chosen to cover the variety of the estate: one large branch, one small, one with a cutting service, one in Scotland and one in Wales, and the branches rotate every quarter so no single counter carries the risk for long. Each first wave manager is told on the Friday before, gets a direct number for that week, and has agreed to ring it rather than wait. The wave list lives in a file that a person reviews, not in a sort order.

In the two quarters since, first waves have caught four faults, including one that only appeared at branches with a cutting service, and none of them reached all forty six branches.

A pilot group only tells you about the sites in it. If you let a sort order choose them, you test on the same unrepresentative sites every time, and the people working there are the last to know they are part of the test.

– Serguey Shinder

We Upgraded Ten Access Points at a Depot and Fourteen Desk Phones Lost Power

Serguey Shinder — Mon, 02 Nov 2026 07:45:39 +0000

The offices and goods in area at one of our depots run from a single forty eight port switch in the comms room. It powers everything plugged into it over the network cable: the wireless access points, the desk phones and a couple of door entry panels.

In August we replaced the eight wireless access points with newer models and added two more to cover a mezzanine. It was a routine change, done on a Tuesday evening and tested by walking a laptop round the building. Everything worked.

At seven the next morning the transport office rang to say their phones were dead. So were the phones at goods in and on the planners' desks, fourteen in all. Ten others on the same switch were fine.

The switch can supply three hundred and seventy watts in total. The old access points each asked it for about fifteen. The new ones reserve thirty, because they have more radios to run. Ten of them took three hundred watts before anything else was counted. When the switch runs short it protects its lowest numbered ports and cuts power to the highest, and the access points were on ports one to ten. The seventy watts left over kept ten phones alive. The other fourteen went dark at about half past seven on the Tuesday evening, when the last access point came up and the offices were empty, with their port lights still on and nothing on the screen to say why.

We moved the phones to a second switch that morning. The lasting change was to start treating power the way we already treat bandwidth. Every switch that supplies power now reports its budget and what is reserved against it, our monitoring warns at eighty percent, and any change that adds or replaces a powered device has to show the figures before and after. When we ran the numbers across all our depots, nine more switches would have failed the same way at their next access point refresh.

We had planned the upgrade by counting ports, and there were plenty spare. Network equipment that powers the devices on it has a second capacity, measured in watts, and it fills up long before the ports do.

– Serguey Shinder