DEV Community

Cover image for What "IT admin" actually means at a company like mine
Mario Fang
Mario Fang

Posted on Originally published at mariosfong.hashnode.dev

What "IT admin" actually means at a company like mine

The phone rings on a Tuesday morning and someone says the invoicing system is "slow", which after four questions turns out to mean they cannot save anything at all and have been retyping the same form for twenty minutes. The job is not running machines. The job is absorbing whatever breaks today and making sure the business never has to think about it.

People outside IT hear the title and picture someone who installs software and keeps the server room cold. That is the smallest part of it. Most days I am a translator between a complaint and a cause, and the machines are just where the evidence happens to live.

Most of the day is intake, not engineering

"Slow" is not a symptom, it's a feeling. "The system is down" can mean a network cable, an expired password, a full disk, a database lock, or one person's browser zoomed to 400%.

So the first skill is not technical. It is asking four questions in the right order without making the person feel interrogated: what were you doing, what did you expect, what actually happened, when did it last work. Half the time the answer already contains the fix. The other half, those four answers take me from a vague complaint to a candidate cause in under a minute, which is the only reason I get to fix anything before lunch.

The urgent thing is almost never the important thing

The loudest problem is the one with a person standing next to it. The important work — patching, checking that the backup actually completed, retiring the account of someone who left last month — has no person standing next to it and no deadline except the one I invent.

That asymmetry is the whole difficulty of the role. If I spend the morning on prevention, nothing visibly happens and someone wonders what I do all day. If I spend it firefighting, everyone sees me working hard while the thing that will page me next month quietly gets worse.

I stopped treating this as a scheduling problem and started treating it as a policy problem. Prevention gets a slot on the calendar before the week begins, not the leftovers of it. Anything that must happen monthly lives in a checklist with a date attached, because I have learned I will not remember it during a bad week.

Half the knowledge is knowing the shape of the business

The same server behaves very differently on the last working day of the month than it does on a quiet Wednesday. Reports that nobody mentions for weeks all get run at eight in the morning on the same day.

A generic administrator can keep a machine healthy. Keeping a company running requires knowing which hour of which day actually matters, whose spreadsheet the finance team cannot work without, and which of the "temporary" workarounds from last year are now load-bearing. None of that is in the documentation. It comes from paying attention while fixing other things.

Nobody notices the days that went right

The measure of this job is an absence. No lost invoices, no restore at midnight, no explaining to a customer why their order vanished.

That makes it a strange thing to be good at, because the better it goes the less visible it is. The reward for a well-run month is the vague impression that nothing needed doing.

The part that actually keeps you out of trouble

You are the single point of failure long before any server is. Every undocumented setup, every credential that only lives in your head, every procedure nobody else has run, is a future outage with your name on it.

So the work I count as real includes the unglamorous parts: writing down what I just fixed, keeping an inventory that matches reality, making sure a second person can restore something without calling me, and refusing to be the only one who knows how a thing works. Not because documentation is virtuous, but because the alternative is a company that stops when I take a week off.

Three things I'd tell anyone who just took this job:

  1. Write the fix down the same day you find it. You will not remember it in six months, and the version of you who does remember will be on holiday.
  2. Put prevention on the calendar and defend it like a meeting with your boss, because that is what it is.
  3. Assume anything that depends on you remembering will eventually not happen. Automate it or accept that it's already broken.

That last one is why I care more about a backup job that runs and disconnects itself than about one I have to supervise. A backup that needs me to remember it is a backup that will stop the week I'm busy — and "I was busy" is not a recovery plan.

— Mario

I write about backups, self-hosting and keeping systems alive. I also build RansomShield (getransomshield.com), the air-gapped backup tool described above.

Top comments (2)

Collapse
 
aliah_meddev profile image
Ali Ahmed •

Really relatable read. Wearing all those hats, especially keeping backups, security, and self-hosted infrastructure running smoothly without constant firefighting, is the real day-to-day reality of sysadmin work.

Collapse
 
mariosfong profile image
Mario Fang •

Thanks Ali, and yeah — the "without constant firefighting" part is the bit nobody budgets for. The change that helped me most was unglamorous: I stopped treating every alert as equally urgent and wrote down, in advance, which fires are allowed to burn until Monday. About a third of them are. Deciding that on a quiet Friday is much cheaper than deciding it at 9am on the day the month-end reports run. How does it work on your side — do you triage by severity, or by whose work is actually blocked?