AI Shell Scripts Fail in the Environment, Not the Syntax: A Pre-Merge Gate That Shows You Why

The most dangerous AI-generated change is often a shell script that looks completely ordinary. It passes shellcheck, it has exit codes, and it reads like a competent engineer wrote it. The failure usually appears later, not because the model made a syntax mistake, but because the script was written against an imagined directory layout, an environment variable that does not exist in your CI worker, or a symlink that points somewhere else on the machine where it finally runs. Reviewers miss this because a diff cannot show the mismatch between the command and the environment. The fix is therefore not a stricter prompt or a longer review checklist. It is a small pre-merge gate that replays the generated script against a copy of your actual directory state and shows you the resulting file differences before a human ever opens the pull request.

The workflow has two parts. First, use a free model endpoint to turn a large generated script into a short ordered step list, so you can see what the model believes the script does before you run anything. Second, use a disposable server to execute the original script inside a copied fixture, then compare the resulting file state with a checked-in manifest. MonkeyCode's free model access and free server option fit this workflow because the gate only needs a small amount of model time to summarize the script, and the execution machine only needs to last long enough to run one throwaway test. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are operator-supplied; this article does not assume any hidden quota, runtime limit, or model name beyond what the operator has stated.

The gate is intentionally boring. You do not need a fancy sandbox or a security scanner, because you are not trying to prove the script is safe. You are trying to prove the script does what your manifest expected. Start with a fixture directory that mirrors the layout the script will touch. For a log-rotation script, the fixture might contain deploy/app.log, etc/rotation.conf, and an empty archive/ directory. You run the script in a temporary copy of that fixture, pinned to the environment variables the real worker will have, and then hash every file that remains. The hashes are compared against a manifest that you reviewed and committed with the pull request.

That manifest is the key artifact. It is not generated by the model. You generate it before the change with a command like this, then update it by hand after you inspect the expected side effects:

find fixtures/rotation -type f -print0 | sort -z | xargs -0 sha256sum > manifests/rotation.sha256

When the gate runs, it does the same thing on the throwaway copy. A small runner script keeps the comparison deterministic, because it captures the exact files and state:

#!/usr/bin/env bash
set -euo pipefail

script="$1"
fixture="$2"
manifest="$3"
workdir="$(mktemp -d)"
trap 'rm -rf "$workdir"' EXIT

cp -a -- "$fixture"/. "$workdir"/
cd "$workdir" || exit 1

export RELEASE_DIR="$workdir/deploy"
export CONFIG_ROOT="$workdir/etc"
export PATH="/usr/local/bin:/usr/bin:/bin"

runner=(bash -euo pipefail)
if command -v timeout >/dev/null 2>&1; then
  runner=(timeout 120 bash -euo pipefail)
fi

"${runner[@]}" "$script" >gate.out 2>&1 || {
  status=$?
  echo "script exited with $status" >&2
  cat gate.out >&2
  exit "$status"
}

find . -type f ! -name gate.out -print0 | sort -z | xargs -0 sha256sum > observed.sha256

if diff -u "$manifest" observed.sha256 > state.diff; then
  echo "gate passed: file state matches expected manifest"
else
  echo "gate failed: unexpected file state" >&2
  cat state.diff >&2
  exit 2
fi

This harness is deliberately narrow. It does not inspect network calls, process exits alone, or whether the script will behave differently on a machine with permissions you did not model. It answers one question well: given this folder and these environment variables, did the script leave the file system in the state I predicted? The answer is frequently no, and that is exactly why the gate is useful before review.

When the diff fails, you start debugging from the changed paths rather than from the model's explanation. A changed line such as archive/app.log in the observed file list but not the manifest usually means the script moved the file to a directory you forgot to include in the fixture. A modified etc/rotation.conf suggests an sed -i or tee that surprised you. Once you can see the concrete path mismatch, you can go back to the free model and ask a much better question: not "is this script safe," but "why does this step assume archive/ exists before the rotation command?" The resulting step list is then more useful, because it is tied to a reproduced failure rather than to a hypothetical.

The main limitation is that this gate is not a sandbox. If the generated script contains an absolute destructive command, you must run it on a disposable machine, never on your laptop or inside a container with access to shared credentials. A free server suits this because it can be discarded afterward. The harness also depends on a fixture that honestly represents the target environment, so it will miss failures caused by network conditions, clock skew, file locks, or external services. For those, you still need a broader integration test, but this gate can remove the most common and least visible class of AI shell failure: a valid script that runs against the wrong picture of your system.

This is not a replacement for reading the diff or reviewing the generated step list. It is a pre-merge filter that turns "looks fine" into "reproduced the expected state." If you already have a spare Linux machine, the whole setup costs less than an afternoon. If you do not, the free server option removes the main practical excuse for skipping it.