Found that all the data accumulated since day one was sitting on a single disk with no backup, and rebuilt off-site backup and version control from scratch
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Discovering that everything I'd accumulated had been sitting without backup
Today wasn't about new features — all the energy went into infrastructure. A recent system review turned up a critical problem.
All the data accumulated since the project started — the incident-free ledger, market records, execution logs, analysis archives — turned out to be sitting on a single disk, with zero backups and zero version control.
Public exchanges don't provide free intraday data, which means the data I generate myself is the only original copy. If that disk fails once, everything up to that point goes to zero.
Building off-site backup on Google Drive
To fix this, I set up a daily automated backup system using Google Drive as off-site storage.
Deciding what to back up and what to exclude
Scoped the backup to four projects: the trading system and three related data/infrastructure projects. Left out the personal hobby blog and other unrelated side projects.
Decided to exclude sensitive items like tokens and passwords from the backup entirely. Those can be reissued after they expire, but data that's gone for good once lost absolutely has to be backed up.
Everything that passed the security screen — logs and record data — got included in the backup.
Backup runs right after market close, every day
Set the backup to run daily at market-close wrap-up time, before the evening compute-heavy jobs start, so there's no resource contention.
A successful backup is silent. Only a failure sends an alert.
Silence is normal, but if the monitoring system itself dies, that also means success notifications stop arriving — silently. So I added a weekly liveness ping from the monitor, sent once every Monday. If that message doesn't arrive, the monitor itself is down.
Rebuilding version control from scratch
Bundled the whole project into a single git repository and built a pre-commit hook that runs automatic checks.
It automatically blocks staging of sensitive filenames, personally identifying data, oversized files, and hardcoded secret values. Tested it with a fake sensitive file, and it caught it correctly.
A couple of small incidents after reboot
Hit two small issues after a recent reboot.
One was the system clock briefly drifting and throwing off scheduled automated jobs; the other was that the backup timer had a startup delay configured but the monitoring timer didn't, causing a false alarm right after boot. Fixed both.
Also changed remote-lookup failures to fall back to local cache instead of alerting immediately, added a weekly integrity check on the backups themselves, and limited heavy cleanup jobs to run only on Sundays.
No new features today. No lines of code added, no new model run.
But if the data I've built up goes to sleep every night without a backup, everything I analyze today is a castle built on nothing.
Foundations come first. That's the kind of thing that keeps needing to be relearned.
Top comments (0)