My phone went off at 03:07 and I lay there for a few seconds deciding whether it was worth getting up. The alert said the destination volume was filling fast, which is the kind of message that gets you out of bed because it never means anything good.
By 03:20 I was at the kitchen table with a laptop, watching a folder of about nine hundred gigabytes copy itself for the second time in a week. Nothing was on fire. Nothing was even broken, as far as I could tell, and that turned out to be the whole problem.
What the job was actually doing
The sync ran on a schedule somebody set up during a storage migration two years earlier. It compared source and destination, decided a handful of files had changed because their timestamps had moved, and then helpfully re-copied the entire tree around them.
I’ve seen this pattern with a few replication tools. They are honest about what they do, but they are not clever about why, and a nightly window that used to take forty minutes quietly grew into something that overlapped the morning shift.
The alerting was worse than the job. We monitored free space on the destination, so we learned about the problem only once it was halfway to being a real one. Nobody had ever set a threshold on how long the job should take, which would have told us six months earlier.
We also found two other sync jobs on the same host that nobody could explain. One copied a folder that no longer existed, and one had been failing since spring with retries every ten minutes, quietly filling a log file that nobody read.
Then there is the opposite failure, which scares me more. A sync that stops without saying anything looks exactly like a sync with nothing to do, and I have watched a team discover eleven days of missing copies during a restore test. The silent stop is the one that gets you.
There were three of us on a bridge call by four, one of them in a taxi on the way to the office and one of them pretending to be awake. The first fifteen minutes went into working out who had changed what, and the answer was nobody, which is its own kind of uncomfortable.
Why it stays set up and forgotten
Sync jobs are set up in a hurry during a migration, and the day they start working is the day everyone stops thinking about them. They do not ask for attention, they do not appear in a dashboard anybody checks, and they sit in that category of infrastructure that is only discussed when it misbehaves.
Every one of these jobs was a sensible answer to a real deadline. The problem is that nobody ever went back and asked whether the answer was still needed, or whether it should run at 2am on a Tuesday forever.
Ownership is the other gap. The person who built it has moved teams, the wiki page is dated, and the only remaining documentation is a line in a crontab with a comment that says "temp."
Defaults did not help either. The tool shipped with a nightly schedule and no concurrency limit, so we accepted the suggestion because it looked reasonable at install time. Nobody questions a default until it wakes them up.
What I have seen other teams do
Some teams try consolidated platforms like Ping64 to reduce the daily busywork of multi-tool security stacks. That is not quite our situation, but the underlying idea lands, since a lot of late-night surprises come from small jobs that each made sense and were never looked at together.
Small things that helped us
Alert on duration as well as failure. If a job normally takes forty minutes, a run that passes ninety deserves a look, and you will hear about it long before the disk fills.
Put a kill switch somewhere you can reach from a phone. Ours is a single command in a runbook, and testing it once a quarter means I trust it at three in the morning.
Cap the job’s throughput so a catch-up run cannot eat the whole pipe. A slower copy that finishes by breakfast beats a fast one that saturates the link everyone else needs.
Check the age of the last successful run, not only the current status. That single number would have caught our eleven-day gap in about a second.
Where I landed
I got back to bed around five, and the job finished at 06:40 with no drama. The only lasting change is a duration alert and a note in the runbook that says, in plain words, what to do if it happens again.
That is most of my job, honestly. Not preventing the strange night, just making the second one shorter.
Top comments (0)