Any tool that breaks a file on purpose and puts it back (a mutation harness, a codemod, a benchmark swapping a config) has four ways to leave the tree wrong. The standard try/finally covers exactly one of them. First, an exception mid-run, which is the covered case. Second, a signal: finally does not run on SIGTERM, so a plain kill leaves the file broken. Third, the restore itself being wrong: a restore that ran is not a restore that worked. And fourth, SIGKILL, which nothing inside the process can cover.
restore-verified is those three uncovered rows as a package, in Python and JavaScript, and using it is one context manager:
from restore_verified import guarded
with guarded("src/parser.py") as g:
g.write(g.read().replace("<=", "<"))
run_the_suite()
# restored here, and the restore is checked byte for byte
pip install restore-verified and npm install restore-verified carry the same API and the same CLI, with a callback in place of with on the JavaScript side. The second row is measured rather than asserted. The suite runs the same mutation three ways in a real subprocess and really kills it. The control is test_SIGTERM_leaves_a_try_finally_harness_broken: an ordinary try/finally harness, SIGTERMed mid-run, must leave the mutated file on disk. If try/finally ever survives SIGTERM, the premise of the package is wrong and that test fails saying so. Beside it, the guarded version of the same harness must come back. A third test asserts the signal is re-delivered afterwards, so the process still dies with status -15 and a kill is never converted into "nothing happened".
The third row is where the name comes from. The npm atomic-write packages make a write all-or-nothing, and PyPI's in-place restores on an exception. Nothing I found re-reads what it put back. A restore can run perfectly and still be wrong: a buffer captured after mutating instead of before, a different encoding on the way out, one of the two files you touched. All three leave the restore path looking healthy, and every run after them scores code nobody wrote. So the guard hashes before and compares after, and g.read() hands back the snapshot rather than re-reading the possibly-mutated file. The snapshot lives in a temp directory instead of a foo.py.bak beside the code, because a scratch file in the directory being measured changes what a file walker collects.
The fourth row cannot be fixed and can only be caught. The ordinary way to be SIGKILLed is not an impatient person, it is a timeout. subprocess.run(..., timeout=...) calls kill() when the deadline passes, and so does a CI runner that has waited long enough. A harness carrying a perfect in-process guard, invoked under a timeout it exceeds, leaves the tree exactly as broken as one carrying no guard at all. So the check lives in whatever invoked the harness. A Sentinel records per-file digests into a manifest (one JSON document either language half can read) before the run, verify compares after, and the CLI form is one line: restore-verified run --paths src/ --timeout 600 --restore -- ./harness.sh, which exits 3 when the tree did not come back, its own code, never the command's. Even --restore still exits 3, because a run this thing had to repair is recoverable, not trustworthy. What Mutation Testing Frameworks Do When a Timeout Kills Them is this argument run against four real frameworks. The short version is that frameworks which mutate a copy never need any of this, and the README says so plainly: avoiding in-place mutation is a better answer than guarding it, and the package is for the tools that cannot.
On a clean git checkout you should also just use git, and the README opens with that. git diff --quiet catches an unrestored change for free. The cases that defeat it are asserted in a test class called TheGitControl. On a developer's dirty checkout, a failed restore is indistinguishable from the uncommitted work, and git checkout -- FILE destroys that work rather than restoring it. A snapshot here is per-file and taken when you start; git's is repo-wide and taken at the last commit, and those are the same thing only on a clean tree.
The JavaScript half is not a translation, and its differences were found by failing tests rather than by reading. A Node signal handler cannot unwind an awaited body the way a Python handler raises into a with block. So the JS handler performs the restore itself, synchronously. And a registered signal listener is also a handle keeping the event loop alive. Removing the last one before re-raising the signal can leave the loop empty, so Node exits before the re-raised signal lands; a timer held across the re-raise closes that hole. The parity suite asserts the manifest in both directions: Python restores from a JavaScript manifest, and the two halves produce byte-identical manifests for the same tree.
One design choice cuts against a sibling package and both are right. The guard restores mtime on purpose, so a guarded edit does not trigger a rebuild in anything keyed on file times. canfail opts out of that deliberately, for reasons its README records and a later post here will walk through.
Five mutations were applied to the source, under this package's own guard, and all five were caught: removing the signal installation, removing the verification, dropping the re-delivery, keeping the snapshot beside the code, and making verify always report clean.
Top comments (0)