Dakota Huang

Posted on Aug 15

Free Server, Untrusted Model: Diff the Filesystem Before You Trust the Exit Code

#testing #ai #codequality #devops

Exit code 0 is not a clean run. It is only the last number the process printed.

The failure mode

A model-generated helper script can:

write a new config file
replace a lockfile
start a background process
change file permissions

The script can still return 0. A later job running on the same host inherits that state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Assume MonkeyCode offers free model access and a free server option. This gate does not rely on a specific model name, quota, or uptime promise. It only needs:

a script generated by the model
a place to execute the script
permission to run two filesystem snapshots

The goal is not to prove the model is correct. The goal is to reduce the free server run to a smaller, reviewable statement: only the files you allowed changed.

The gate

Use two filesystem snapshots:

Snapshot the working directory before the run.
Run the generated script.
Snapshot the same directory after the run.
Compare path sets, sizes, and SHA-256 hashes.
Fail on unexpected additions, removals, or content changes.

The snapshot script records each regular file as a tuple of relative path, byte size, and SHA-256 hash.

import hashlib, json, os, sys

root, out = sys.argv[1], sys.argv[2]

def snapshot(root):
    items = []
    for dirpath, _, names in os.walk(root):
        for name in names:
            p = os.path.join(dirpath, name)
            rel = os.path.relpath(p, root)
            h = hashlib.sha256()
            with open(p, 'rb') as f:
                for chunk in iter(lambda: f.read(65536), b''):
                    h.update(chunk)
            st = os.stat(p)
            items.append((rel, st.st_size, h.hexdigest()))
    items.sort()
    return items

json.dump(snapshot(root), open(out, 'w'), indent=2)

The diff script compares the two snapshots and applies an allowlist. A path can match exactly or sit under an allowed directory.

import json, sys

before = {p: (s, h) for p, s, h in json.load(open(sys.argv[1]))}
after = {p: (s, h) for p, s, h in json.load(open(sys.argv[2]))}
allow = {a for a in sys.argv[3].split(',') if a}

def ok(p):
    return any(p == a or p.startswith(a.rstrip('/') + '/') for a in allow)

failed = False
for p in sorted(set(before) | set(after)):
    if ok(p):
        continue
    if p not in before:
        print('ADDED', p)
        failed = True
    elif p not in after:
        print('REMOVED', p)
        failed = True
    elif before[p] != after[p]:
        print('CHANGED', p)
        failed = True
sys.exit(1 if failed else 0)

Wrap the generated script like this:

python snapshot.py /srv/workspace before.json
./generated_script.sh
python snapshot.py /srv/workspace after.json
python diff.py before.json after.json '.cache/,logs/run.log'

This example is a proposal, not a finished production gate. Paths, ignore rules, and timeout handling still need adjustment.

Tuning for free hosts

keep the scanned tree small
write snapshot files outside the scanned root
add a timeout around the generated script
start with an empty allowlist, then add known paths only after review
remove snapshot files after the job ends

Decision table

Finding	Meaning	Action
`ADDED` unexpected	script wrote a new file outside the allowlist	fail the job
`REMOVED` unexpected	script deleted a file outside the allowlist	fail the job
`CHANGED` unexpected	file content or size changed outside the allowlist	fail the job
`ADDED` or `CHANGED` inside allowlist	expected log or generated artifact	pass

The allowlist is the review layer. If a generated script is allowed to write only .cache/ and logs/run.log, a surprise write to config.json becomes a hard stop.

What this gate does not catch

in-memory state changes
background processes that keep running after the exit code
environment variable changes
network egress or data exfiltration
nondeterministic behavior that produces identical files
symlink or device changes outside the walked tree

A large tree also makes the snapshot expensive. Use a small, dedicated working directory for each free server job.

Who should skip this approach

Skip the gate when:

the runner is already an immutable container or clean checkout
the team needs process or network monitoring, not file state checks
the repository is too large for two full snapshots
the generated script is adversarial and can modify the gate itself

Do not treat this as a security boundary. It is a consistency check for ordinary model-generated scripts on a shared free host.

Bottom line

A green exit code is cheap. A filesystem diff is also cheap, but it changes the claim from 'the script ran' to 'the script ran and touched only these files'.

If a free model writes the script and a free server runs it, put the two snapshots around the run. Then delete the snapshot files when the job ends.

DEV Community