Static Review Is Not a Runtime Guarantee: Diff the Shape of an AI-Managed Server Before You Accept It

#security #devops #ai #systemd

The position I want to argue is unambiguous: when an AI model writes or edits a systemd service on a Linux server, reviewing the generated diff before merge is not enough. A unit file is a set of promises about runtime behavior, and a runtime shape diff is what verifies those promises after the process starts.

I used MonkeyCode's free model access to generate the baseline service and its free server option to run the before and after states, so the artifact below is reproducible rather than hypothetical. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The promise gap

Systemd service properties such as ProtectSystem=full, PrivateTmp=true, NoNewPrivileges=true, and CapabilityBoundingSet= describe the environment that systemd should create. Those properties are enforced by the service manager, but they do not capture everything a running process can touch through its own code or through other enabled services. An AI-generated unit can declare a strict sandbox and still bind an unexpected localhost port, create a writable socket in a private namespace, or interact with a process outside the unit's control group. Static review sees the declaration; a runtime shape diff sees the resulting process.

This matters because AI-generated changes often arrive as small patches against a working service. A reviewer may approve a three-line edit that looks harmless, while the actual runtime footprint grows in a way the diff does not reveal. The more useful gate is a structured comparison between two observed states: before the change and after the change.

A runtime shape fingerprint

The following script captures a minimal runtime fingerprint for a running systemd unit. It records the unit's declared properties, the main process's effective capabilities, whether NoNewPrivileges is active, listening sockets associated with that process, and the mount points visible to it.

#!/usr/bin/env bash
set -euo pipefail

unit=${1:?unit name required}
main_pid=$(systemctl show -p MainPID --value "$unit")

if [ -z "$main_pid" ] || [ "$main_pid" -le 0 ]; then
  jq -n --arg unit "$unit" '{unit: $unit, state: "not-running"}'
  exit 1
fi

properties=$(systemctl show "$unit" -p CapabilityBoundingSet,NoNewPrivileges,ProtectSystem,PrivateTmp,RestrictAddressFamilies,SystemCallFilter)

cap_eff=$(awk '/^CapEff:/ {print $2}' "/proc/$main_pid/status")
no_new_privs=$(awk '/^NoNewPrivs:/ {print $2}' "/proc/$main_pid/status")
sockets=$(ss -lntup | grep "pid=$main_pid" || true)
relevant_mounts=$(grep -E ' /( |$)' "/proc/$main_pid/mounts" | awk '{print $2}')

jq -n \
  --arg unit "$unit" \
  --arg main_pid "$main_pid" \
  --arg properties "$properties" \
  --arg cap_eff "$cap_eff" \
  --arg no_new_privs "$no_new_privs" \
  --arg sockets "$sockets" \
  --arg relevant_mounts "$relevant_mounts" \
  '{unit: $unit, main_pid: $main_pid, unit_properties: $properties, runtime_cap_eff: $cap_eff, runtime_no_new_privs: $no_new_privs, listening_sockets: $sockets, relevant_mounts: $relevant_mounts}'

Save this as runtime-shape.sh, make it executable, and run it with the unit name as the only argument. The output is a JSON object that can be stored in a before.json file and later compared with an after.json file.

A diff that rejects silent drift

The comparison script below reads two shape files and flags any field that changed. It does not decide whether a change is safe; it forces a human or a policy gate to explain each change.

#!/usr/bin/env bash
set -euo pipefail

before=${1:?before json required}
after=${2:?after json required}

jq -n --slurpfile before "$before" --slurpfile after "$after" '
  ($before[0] // {}) as $b |
  ($after[0] // {}) as $a |
  {
    unit: $a.unit,
    cap_eff_changed: ($a.runtime_cap_eff != $b.runtime_cap_eff),
    no_new_privs_changed: ($a.runtime_no_new_privs != $b.runtime_no_new_privs),
    sockets_changed: ($a.listening_sockets != $b.listening_sockets),
    mounts_changed: ($a.relevant_mounts != $b.relevant_mounts),
    before: $b,
    after: $a
  }
'

When both scripts are run inside a CI job or a post-deploy hook, the boolean fields become the actual review gate. A one-line change that flips mounts_changed from false to true cannot pass silently, even if the unit file diff looked acceptable.

A five-step workflow

Generate a baseline service. Use the free model access in MonkeyCode to create a minimal Python HTTP service that listens on 127.0.0.1:8080 and runs as nobody. Ask for a unit file with ProtectSystem=full, PrivateTmp=true, NoNewPrivileges=true, and an empty CapabilityBoundingSet.
Run the baseline on the free server option. Start the service, wait for the listening socket to appear, and capture before.json with runtime-shape.sh shape-demo.service > before.json.
Ask the model to add a feature. For example, request a periodic cleanup task that writes operational logs to /var/tmp/shape-demo/. This is the kind of small AI edit that frequently expands runtime scope.
Run the changed service in the same environment. Stop the baseline, start the modified unit, and capture after.json.
Diff the shapes and reject unexplained drift. Run runtime-diff.sh before.json after.json and inspect every boolean field that changed. The expected change may be a new writable mount, but it must be intentional and visible.

The procedure turns an AI-generated patch into a testable runtime assertion. The generated code is not trusted; the observed process state is compared against the previous observed state, and any unseen delta fails the gate.

A triage table for changed fields

Changed field	Likely meaning	Required action
`cap_eff_changed`	The process gained or lost kernel capabilities	Reject unless a policy owner explicitly approved the change
`no_new_privs_changed`	The privilege model changed	Reject and ask the model to state why it removed the restriction
`sockets_changed`	A new listener or bind address appeared	Review the socket endpoint and firewall rule before continuing
`mounts_changed`	A different filesystem path became visible or writable	Diff `/proc/<pid>/mounts` and confirm the path is necessary

This table keeps the review objective. The scripts do not give an AI model the benefit of the doubt; they give the reviewing engineer a concrete list of runtime deltas to approve or reject.

Limitations and who should not use this

This approach is not a security boundary. It observes a subset of process state and cannot prove that a service is safe, that data is not exfiltrated over an allowed socket, or that the code contains no logic errors. It also requires systemd on a Linux host and enough privilege to inspect the target process. The ss command may not associate a socket with a process on every distribution without additional options, and the mount filter only includes top-level paths that match a simple pattern.

Do not use this as a substitute for a sandbox, a firewall, or a code review. Teams running non-systemd platforms, Windows services, containers without a visible PID namespace, or services that legitimately create many short-lived sockets will see too much noise or no useful signal. The workflow is best for small, stable systemd services where an AI-generated change should produce a narrow and explainable runtime delta.

If you already have access to a free model and a free server for this kind of experiment, the two scripts above are the entire gate. The harder part is not building the tool; it is refusing to trust a clean unit-file diff when the runtime shape has not been compared.

DEV Community