InfraGuard Engineering Devlog, part 2 of 6.
On August 18, a weekly backup was repeatedly creating a roughly 5 GB full archive. The cause was a chain of process, reporting and scheduling failures.
Five links in the failure chain
- PM2 restarted the backup runner every minute through
cron_restart. - When the runner died, its child backup shell lost its output pipe.
- A subsequent log write could terminate the shell with
SIGPIPE, before it reported a result. - With no report, the application did not advance the next-run timestamp.
- The next runner tick still saw an overdue job and started it again.
Redirecting errors or adding || true did not solve signal-driven termination. Swallowing an exit status and surviving a signal are different things.
Change the scheduling trade-off explicitly
We changed eligibility to a phase-based minute slot derived from the start time and interval, rather than an indefinitely overdue next-run value. A runner missing its eligible minute skips that occurrence instead of repeatedly catching up.
That trades automatic recovery of a missed slot for avoiding this runaway replay path. Slot eligibility is not a general exactly-once guarantee across multiple runners; concurrency control remains separate.
The scripts ignore SIGPIPE and attempt failure reports for handled termination signals such as TERM, INT and HUP.
Successful work, rejected report
A second incident produced a 2.1 GB incremental archive but zero execution records. A preview script had been rendered before the job existed, then saved with JOB_ID=0. Its report was rejected with HTTP 422.
Without --fail, curl can exit successfully after an HTTP error response. We had confused transport completion with application acceptance.
The fix was to render with the real ID before saving, use curl --fail, and log rejected reports. Retrying an unchanged invalid payload would not make it valid.
Tool-specific semantics matter
GNU tar's exit code 1 can mean files changed during archiving. We treated that documented case as a warning, while retaining failure handling for a nonzero database-dump result. A single generic exit-code policy was wrong.
The manifest workflow also moved preparation ahead of the expensive transfer and used a rename to finalize the validated result. Previously, completed archives had failed to establish their expected incremental baseline.
Takeaway: A backup is a schedule, a process tree, a baseline and an accepted result record, not just an archive. Test the boundaries between them.
About this series: Adapted from InfraGuard engineering records. Measurements describe the recorded environment, not product-wide benchmarks. AI was used to draft and translate this article.
Disclosure: Solution Market develops and sells InfraGuard. This is our engineering devlog, not an independent review. InfraGuard is a self-hosted infrastructure operations and security platform. Project · Documentation · Solution Market
Top comments (0)