What we shipped on 2026-07-19
We almost wiped our primary stack based on a rehearsal backup that looked healthy but was fundamentally incomplete. In fix(migration): close a silent data-loss gap in the Pop!_OS cutover + correct backup-precious.sh against the live stack (PR #2712), we found a critical flaw: pg_dump snapshots at the moment it starts, but our writers--the brain daemon and scheduled agents--keep pushing rows to the database via asyncpg even while the migration process is running.
The gap was silent and dangerous. When we measured this against the live stack, we saw the audit_log gain 86 rows in just a few minutes between the dump and verification (`98dc177`). Over a full cutover weekend, that's thousands of posts, pipeline tasks, and embeddings that would have been erased without warning. We fixed this by inserting Task 3.0 at the top of Phase 3: stop the stack, disable host-side writers, prove the DB is quiet via pg_stat_activity and audit_log sampling, and perform a final dump to a dedicated -FINAL directory (PR #2712).
This scare prompted a larger strategic pivot. We replaced our full-wipe strategy with a reversible dual-boot evaluation (PR #2713). The problem with the original plan was that it forced an irreversible step before we could actually prove the core hypothesis: that bare-metal Linux kills the wedge class under real workload over several days.
By installing Pop!_OS to the MP600 alongside Windows--utilizing the 1.7 TB of free space--the wipe becomes the conclusion of the experiment rather than the starting line. We've rewritten the runbook to enforce a strict "prove BOTH OSes still boot" gate and a Windows pre-flight to ensure Fast Startup is off, preventing NTFS corruption during read-write mounts (PR #2714).
We also refined our partition layout as a bet on future hardware moves. Root is now capped at 150 GB, with everything heavy shifted to a separate /data volume (PR #2714). This ensures that when we eventually move to the MP700, we only have to reinstall root and remount /data, rather than relocating hundreds of gigabytes of Docker data.
The migration is no longer a leap of faith; it's a falsifiable experiment. We can now run the real workload and count wedge events before we commit to the wipe.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)