so our mysql healthcheck is mysqladmin ping -uroot -p$MYSQL_ROOT_PASSWORD. i figured since we're passing the password it's actually checking auth too, right?
wrong lol. turns out mysqladmin ping returns exit 0 even when it gets Access Denied back. apparently that's just how it works, the server responded so technically it's "alive", doesn't matter if it refused you. container's been sitting there green this whole time.
meanwhile mysqldump is out here failing with Access Denied every single night and just writing a basically empty .gz file. our backup script does something like docker exec ... | gzip > "$FILE" and with set -eu the script only cares about gzip's exit code, not mysqldump's. so gzip succeeds (it did its job, compressing basically nothing), script exits 0, everyone's happy except there's no actual backup.
only found out because i needed to restore something for a totally unrelated reason and opened the file and it was tiny
the part that actually got me a little freaked out: we prune old backups and only keep the last 7, and that runs right after every dump. so this whole time it was broken, every night it's writing one more empty file AND deleting one more good one to make room. if nobody had noticed for a week straight it would've just... quietly deleted every real backup we had, on its own, no crash no error nothing
Top comments (0)