We had just started supporting a new database technology, and we hired an engineer who already had experience with it. He became the person managing it. The product using it was not in production yet. It was still in pre-production, where testing was going on, and it was running into some issues.
Then he went on personal leave for a wedding.
The other engineers on the team had a basic idea of the technology, and they tried to answer the questions that came up during testing. They could not solve them, so they had to call him. He stepped out of the wedding and helped them over the phone for a while.
Nobody did anything wrong here. The team tried their best, and he helped because he cared. But it showed me a problem we had built ourselves. Only one person on the team knew this system well enough, so being the expert meant he could never really be on leave. It was not even production yet, and we still needed him.
Managers sometimes talk about people like him as "critical resources," and they say it with pride. I do not think it is something to be proud of. A person who cannot take leave is not a strength for the team. He is a single point of failure, and he usually knows it, which makes it worse.
The rule
Every system gets a primary owner and a backup owner. That is not a new idea, most teams already have a "backup owner" column somewhere in a wiki. The difference is that the backup actually does the work.
Having your name in a wiki does not make you able to run a system. So in my teams, the backup does the real work while the primary watches. The backup runs the version upgrade, does the failover drill and runs the maintenance window. The primary sits with them and only steps in if something is about to go wrong. Next time, they swap.
We still write documentation, and we keep it up to date. But a runbook only describes the system the way someone once wrote it down. Working on it with the primary watching teaches you how the system actually behaves, like the node that always lags after a restart, the alert that is not reliable, or the step everyone skips because they "just know." That gap between what is written and how the system really behaves is exactly what leads to a call during someone's leave.
What you actually get from it
It looks like a training plan, but it does three other things at the same time.
It protects leave. If someone can be called at any time, they are not really on leave, they are just working from somewhere else. Once the backup can genuinely run the system, the primary's phone stays quiet and the leave becomes real rest.
It keeps people in the team. People rarely leave because of one bad quarter. They leave after years of never being fully off. Protecting leave is one of the cheapest ways I have found to keep good people.
It helps during incidents. On the night something big goes down, "only one person knows this system" is a real risk to the business. A team where everyone has worked on everything can handle it even when someone is away.
There is one more benefit that is easy to miss. Everyone depends on everyone, and they can see it. A senior engineer watches a junior run the upgrade on their system, and the junior gets to work on systems that would normally take years to get access to. In one of the teams I ran a few years ago, this is how two engineers I had hired as interns went on to lead teams of their own.
The one number I watch for all of this is how many times someone on leave had to take a work call. The target is zero.
What it costs
Cross training slows the team down in the short term. The backup takes twice as long to run a change as the expert would, and the expert is sitting right next to them, wanting to just do it. If your team measures everything by how fast work gets done, this looks like waste, and you will have to defend it more than once.
The harder cost is less visible. Your most experienced engineer might see this as taking away what makes them special. Being the only person who knows a system can feel like job security, and you are deliberately removing that. The real management work is the conversation where you help them see that nobody gets promoted while they are the only one who can run a system. Setting up the rota is easy compared to that conversation.
When it works, nobody notices
When this works, nobody talks about it. People plan weddings and festivals and just go, without checking their phone every hour, the way people in normal jobs do.
Top comments (0)