90 Days of DevOps, Day 4-6: Turning Linux Practice Into Real Troubleshooting

Part of #90DaysOfDevOps - a 90-day challenge to build Linux and DevOps fundamentals, one small hands-on task a day. This post covers Day 4 (process & service checks), Day 5 (a troubleshooting drill), and Day 6 (basic file read/write) as one story, because they turned into one.

---

The setup

Day 4's task was simple on paper: run a few basic commands, check what's running, inspect one service. I started with the most basic one there is.

ps aux | head -5

Then I sorted by memory to see the bigger picture.

top -l 1 -o mem -n 8

727 processes. On a machine where I thought I had ten things open. Most of it was normal background stuff — Chrome runs a separate process per tab, so five "Chrome Helper" entries isn't strange. But two entries stood out: mysqld at 515MB, and node at 348MB, both idling, both things I hadn't opened today.

I checked what the node one actually was:

ps -p 745 -o pid,ppid,command

It was OpenClaw — one of the AI automation agent tools I'd installed myself a while back, because I genuinely like building with it. Nothing sinister. But its parent process ID was 1, meaning it hadn't been started by me clicking anything today — it had been started by launchd, the thing that boots every service on a Mac, at login.

So I checked what else was set up the same way:

ls ~/Library/LaunchAgents

Five files. Three belonged to Chrome's updater. Two were mine — OpenClaw and Hermes, another agent tool I'd set up. I'd installed both on purpose. What I hadn't clocked was that they were both registered as LaunchAgents with RunAtLoad: true — meaning launchd fires them up automatically every single time I log in, forever, whether I open them that day or not. And both had KeepAlive: true, meaning even if I killed the process by hand, launchd would just restart it. I'd never actually stopped either one — I'd just closed the terminal window I originally started them from, and they kept quietly running underneath that.

I decided I didn't want them running 24/7 anymore, so I stopped them the right way — through launchd, not with kill (which would have just triggered a restart) — removed their startup files, and uninstalled the leftover pieces, including a launcher for Hermes still sitting on my PATH pointing at a folder I'd already deleted.

One thing was still unexplained, though: mysqld.

Day 5: actually finding out why MySQL was running

Day 5's task was a proper troubleshooting drill — pick a service, check its health, read its logs, write it up as a runbook. MySQL was the obvious target.

Health-wise it was fine — 0.2% CPU, idle:

ps -o pid,pcpu,pmem,etime,comm -p $(pgrep -x mysqld)

Then I checked what was listening on its port:

lsof -i :3306

Empty. But when I actually tried connecting:

nc -vz localhost 3306

"Connection succeeded." Contradiction. Turned out lsof only shows processes you own unless you run it with sudo, and mysqld runs as a different system user. Empty didn't mean nothing was there — it meant I wasn't allowed to see it yet.

sudo lsof -i :3306

That's when I found the actual issue: it wasn't just listening on my machine, it was listening on *:3306 — every network interface, not just localhost. Same on a second port, 33060. A database on my personal laptop had been reachable from any network I connected to, not just from me.

I checked when it had even arrived:

pkgutil --pkg-info com.mysql.mysql

November 2025. Running at every boot since, and inside it, one database called employee_db — almost certainly leftover from some SQL course I'd followed along with last winter. Healthy, clean logs, nothing wrong with it — it just had no reason to still be running, and definitely no reason to be network-reachable. Backed up the data folder, then removed it the same way: stopped through launchd, registration removed, software uninstalled.

Day 6: back to basics

After two days of chasing down services, Day 6 was refreshingly small: create a text file, write into it, read it back. Three ways to put text in a file, and the difference between them turned out to matter a lot:

echo "Line 1" > notes.txt      # replace the whole file
echo "Line 2" >> notes.txt     # add to the end
echo "Line 3" | tee -a notes.txt  # add to the end AND show it on screen

Small lesson, but it's the same instinct as the last two days: read what's already there before deciding whether to overwrite it or add to it.

What actually happened this week

I went in expecting three small tasks. What I actually did was realize two tools I'd installed on purpose had quietly been running non-stop since I set them up, find an unrelated database that had been exposed to the network for eight months, and clean up all three properly instead of just force-quitting and hoping.

Biggest lesson: an empty result from a command doesn't always mean "nothing's here." lsof without sudo told me nothing was on port 3306. It was wrong — or more accurately, it just wasn't allowed to tell me. One flag fixed it.

---

Command reference — for future me

The commands below, grouped by what question they answer, with the Linux equivalent next to the macOS one I actually used. macOS uses launchd where Linux uses systemd — different tool, same job: start services, keep them alive, own their logs.

"What's running right now?"

What I want to know	macOS	Linux
List all processes	ps aux	ps aux
Live view, sorted by memory	top -l 1 -o mem -n 8	top -b -n 1
Find a process by name	pgrep -l <name>	pgrep -l <name>
Full detail on one PID	ps -p <pid> -o pid,ppid,command	same

PPID 1 means launchd/systemd started it, not a person clicking something. That's the tell for "this is an auto-starting service."

"What auto-starts, and how do I see its config?"

What I want to know	macOS	Linux
Things that start when I log in	ls ~/Library/LaunchAgents	user systemd units, systemctl --user list-units
Things that start at boot, system-wide	ls /Library/LaunchDaemons	systemctl list-units --type=service
Is this service running, and its config	`launchctl list \	grep then read its .plist`
System-owned services (needs sudo)	`sudo launchctl list \	grep `

Key config flags to look for in a .plist (or a systemd unit): RunAtLoad/WantedBy = starts automatically. KeepAlive/Restart= = comes back if killed. If both are set, kill will not work — you have to stop it through the supervisor.

"How do I actually stop something for good?"

Step	macOS	Linux
1. Stop it via the supervisor	launchctl bootout gui/$(id -u) <path-to-plist> (user) or sudo launchctl bootout system <path> (system-wide)	systemctl stop <name>
2. Stop it from coming back	rm <path-to-plist>	systemctl disable <name>
3. Remove the software	rm -rf, npm uninstall -g, or the app's uninstaller	apt remove / yum remove etc
4. Verify it's gone	`launchctl list \	grep and pgrep -fl ` — both should be empty

Never just kill something with KeepAlive/Restart set. Always go through the supervisor first.

"Is anything listening on a port, and who?"

What I want to know	macOS	Linux
What's listening on a port	lsof -i :<port>	ss -tulpn or lsof -i :<port>
See services owned by other users too	add sudo	usually no sudo needed
Can I actually reach it	nc -vz localhost <port> or curl -I	same

If a plain lsof/ss shows nothing but you know something's there — re-run with sudo. Empty output can mean "wrong permissions," not "nothing's here."

Also check what it's bound to: localhost / 127.0.0.1 = only reachable from this machine. * or :: = reachable from the whole network. The second one is usually a mistake on a laptop.

"Where are its logs?"

What I want to know	macOS	Linux
System-wide unified log	log show --last 5m --predicate 'process == "<name>"'	journalctl -u <name> -n 50
A specific log file	tail -n 50 <path>	tail -n 50 <path>

Not every service uses the system log — some (like MySQL) write their own log file. Check the service's own directory or its config for where it actually logs.

"Is the machine healthy?"

What I want to know	macOS	Linux
Disk space free	df -h /	df -h
What's using the disk	du -sh <folder>	same
Memory	vm_stat	free -h
Load average	iostat (bottom right) or uptime	uptime, vmstat

Reading and writing files — the Day 6 basics

What I want	Command	Notes
Create an empty file	touch file.txt	does nothing if it already exists
Write, replacing the file	echo "text" > file.txt	wipes anything already in the file
Write, adding to the end	echo "text" >> file.txt	safe, doesn't touch what's there
Write and see it at once	`echo "text" \	tee -a file.txt`
Read the whole file	cat file.txt
Read just the top	head -n <N> file.txt
Read just the end	tail -n <N> file.txt, or tail -f to watch it live	-f is what you use to watch a log grow

The one habit worth keeping from all three days

Snapshot before you touch anything. Interpret before you act. If something needs to stop, stop it through whatever's supervising it, never with a bare kill. And treat "no output" as a question, not an answer — check if you're even allowed to see the whole picture before you trust it.