My sudo Logs Hit 10 GB. Session Count Told Me Nothing.

A disk-usage alert sent me looking at /var/log/sudo-io on a Chatwoot production box. It held 5.9 GB and was adding roughly 654 MB every week. Nothing on that server had changed. No new services, no new admins, no new cron jobs.

My first instinct was the wrong one: I assumed the server was simply busier than the others. It isn't. It runs the fewest sudo sessions of any box in the fleet that logs them.

Here is what the numbers look like, measured live across the whole fleet this morning.

The measurement

Two commands. The first counts sessions, the second measures bytes:

# how many sudo sessions were recorded
sudo find /var/log/sudo-io -name log -type f | wc -l

# how much disk those sessions occupy
sudo du -sk /var/log/sudo-io

Ten hosts in my SSH config. Only four of them have /var/log/sudo-io at all. The other five never had log_output enabled, and one was unreachable during the run. That inconsistency is its own finding, and I'll come back to it.

Host	Log volume	Sessions	KB per session
chatwoot	5.48 GB	16,168	356
supabase	2.33 GB	56,794	43
website	1.71 GB	14,289	126
chatwoot_admon	0.49 GB	8,788	59
Total	10.02 GB	96,039	109

Read the last two columns together. supabase recorded 3.5x more sudo sessions than chatwoot and produced less than half the volume. Per session, chatwoot is 8.3x heavier.

Session count and log size are not measuring the same thing, and I had been reading the wrong one.

Why bytes and sessions diverge

log_output in sudoers does not record which command ran. It records everything the command writes to its pty. A short admin command that prints two lines costs a few hundred bytes. A command that streams a file costs the size of that file.

So this:

sudo systemctl restart nginx

costs almost nothing, while this:

sudo cat /opt/chatwoot-backups/dump-2026-09-14.sql

costs 60 MB. The dump gets written to disk a second time, into the audit log, carrying no information the original didn't already have.

That was the leak. The offsite backup job pulls a database dump every night by reading it through sudo. supabase and website already had those reads exempted in /etc/sudoers.d/iolog-exemptions. chatwoot did not. chatwoot_admon had no exemptions file at all. Same fleet, same backup design, three different configurations — because the exemption file was written by hand on each host, one host at a time, and nothing ever checked that they matched.

The fix is an exemption, not a shutdown

The tempting fix is to turn log_output off. Don't. The recording is the point; you want to know what a human typed at 2am on a box holding customer conversations. What you don't want is a byte-for-byte copy of files you already have.

sudoers lets you carve out specific commands:

Cmnd_Alias IOLOG_EXEMPT = /usr/bin/cat /opt/chatwoot-backups/*, \
                          /usr/bin/stat /opt/chatwoot-backups/*, \
                          /usr/bin/zcat /var/log/caddy/*, \
                          /usr/bin/tail /var/log/caddy/*

Defaults!IOLOG_EXEMPT !log_output

The session still gets logged: who ran what, when. Only the byte stream is dropped.

Two things worth saying out loud about this file:

Validate before you apply it. visudo -cf /etc/sudoers.d/iolog-exemptions parses the file without installing it. A syntax error in sudoers.d can lock you out of sudo on a machine you may only reach through sudo. Keep a timestamped backup next to it.

The paths matter more than the commands. Exempting /usr/bin/cat outright would let anyone pipe any file past the audit log. Exempting cat restricted to the backup directory keeps the hole the size of the actual problem.

What ten days later looks like

I applied the exemptions across all four hosts on September 14 and measured again today, September 24.

At 654 MB/week, chatwoot should have been near 6.8 GB by now. It reads 5.48 GB. The curve turned over. Two things are doing that work together. New writes stopped, and the 60-day retention cron keeps deleting old sessions, so I'd expect the number to keep falling into mid-October as the pre-fix backlog ages out. I'm not claiming the exemption alone recovered 1.3 GB. I'm claiming the growth stopped, and that's the part I was chasing.

The part I'd actually change

The exemption fixed one symptom. The real defect is that five of ten hosts don't record sudo I/O, two of the four that do had drifted apart, and I only noticed because a disk filled up. A config that exists on some machines and not others isn't a policy, it's a habit that happened to repeat.

So the check I added isn't "is the log too big." It's this, run across the fleet:

# does this host record sudo I/O at all, and is the exemption file present?
grep -rl log_output /etc/sudoers /etc/sudoers.d/ 2>/dev/null | head -1
test -f /etc/sudoers.d/iolog-exemptions && echo "exemptions: yes" || echo "exemptions: MISSING"

Two lines, and it answers the question the disk alert was too late to ask. This is the same reasoning behind most of the fleet automation I build at achiya-automation.com — the alert that fires when a threshold breaks is always a worse version of the check that confirms a configuration is still what you think it is.

The metric to steal

If you have log_output enabled anywhere, don't look at total size and don't look at session count. Look at bytes per session:

sudo bash -c 'kb=$(du -sk /var/log/sudo-io | cut -f1); \
  n=$(find /var/log/sudo-io -name log -type f | wc -l); \
  echo "$((kb / (n>0?n:1))) KB per session across $n sessions"'

Across my fleet the number is 109 KB. Anything in that range is ordinary admin work. A host sitting at 350+ has a command streaming a file through sudo, and you can find it in one pass. A host at 43 is doing plenty of work and costing you nothing.

---

A question for anyone running log_output in production: what's your bytes-per-session number, and did the outlier turn out to be a backup job like mine, or something you'd never have guessed? I'm especially curious whether anyone exempts by path the way I did here, or whether you found a cleaner pattern than hand-maintained Cmnd_Alias blocks that drift between hosts.

Comments

[אחיה כהן]

One follow-up I owe this post, because I ran out of runway on it today.

Bytes-per-session tells you a host has a streamer on it. It does not tell you which command. The obvious next pass is to attribute volume back to the command that produced it — walk each session directory, take its size, read the cmd= line out of the session's log file, and sum by command basename:

sudo find /var/log/sudo-io -name log -type f -mtime -30 | while read -r f; do
  kb=$(sudo du -sk "$(dirname "$f")" | cut -f1)
  cmd=$(sudo sed -n 's/^cmd=//p' "$f" | head -1 | awk '{print $1}')
  echo "$kb $(basename "${cmd:-unknown}")"
done | awk '{s[$2]+=$1} END {for (k in s) printf "%10d KB  %s\n", s[k], k}' | sort -rn

Two things I'd flag before anyone runs that on a real box:

1. It reads session metadata, not session content. Keep it that way. The cmd= line can still carry arguments, so if you print it in full you may pull a secret that was passed on a command line straight into your terminal scrollback. Cutting to the basename with awk '{print $1}' is deliberate.
2. On a host with tens of thousands of sessions this is one du per session and it is slow. Bound it with -mtime first.

I'd also split volume by age window (-mtime -7, -7..-14, and so on) rather than trusting a single total, because retention deletion and new writes move the number in opposite directions at the same time — which is exactly why I wouldn't attribute the 1.3 GB in the post to the exemption alone.

If anyone has already got a cleaner attribution than a du-per-session loop, I'd rather steal yours than keep tuning mine.