Letter to Last-Tuesday-Me: The Baseline Probe I Skipped on a Free Cloud Box

I lost a day to a sandbox, not to a model.
The patch was fine on my laptop and wrong on the remote box.
Fixing the box took twenty minutes. Finding out it was the box took eight hours.

That is the whole letter. Here is the version with receipts.

The Story in Four Lines

An agent drafted a small refactor. I ran it locally: green. I copied it to a free cloud server and ran the same suite: red.

I assumed the model was wrong. I rewrote the prompt three times. I read the diff four times. The diff was never the problem.

The remote box had a lower open-file limit and a slower disk. My test wrote hundreds of temp files. Locally it survived. On the box it hit the limit partway through, and the suite reported the failure as a logic error.

Nothing in that story is exotic. Everything in it is preventable with one script and one habit.

Mistake 1: I Measured the Code Before the Box

I had a benchmark for the patch and no baseline for the host.
A benchmark without a baseline is a rumor.

Fix: capture a baseline before the first agent call. That is step 2 of my harness.

#!/usr/bin/env bash
# probe.sh — record the box before any patch lands
set -euo pipefail
out="${1:-baseline.json}"
args=(
  --arg captured_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  --arg kernel "$(uname -sr)"
  --arg python "$(python3 -V 2>&1)"
  --arg umask "$(umask)"
  --argjson cpu_count "$(nproc)"
  --argjson mem_mib "$(awk '/MemTotal/ {print int($2/1024)}' /proc/meminfo)"
  --argjson disk_free_mib "$(df -Pm / | awk 'NR==2 {print $4}')"
  --argjson tmp_free_mib "$(df -Pm /tmp | awk 'NR==2 {print $4}')"
  --argjson open_files "$(ulimit -n)"
  --argjson user_procs "$(ulimit -u)"
)
jq -n "${args[@]}" '{
  captured_at: $captured_at,
  kernel: $kernel,
  python: $python,
  umask: $umask,
  cpu_count: $cpu_count,
  mem_mib: $mem_mib,
  disk_free_mib: $disk_free_mib,
  tmp_free_mib: $tmp_free_mib,
  open_files: $open_files,
  user_procs: $user_procs
}' > "$out"
echo "wrote $out"

Run it twice: once right after provisioning, once before the failing run.
Then diff the two files instead of diffing your opinions.

diff <(jq -S . baseline.before.json) <(jq -S . baseline.after.json)

If open_files or tmp_free_mib moved, you have a resource story, not a logic story.

Next, add a reference workload so timings mean something. Two runs of a fixed, boring task.

run_ref() {
  local label="$1" start end
  start=$(date +%s%N)
  for i in $(seq 1 200); do sha256sum /etc/hostname > /dev/null; done
  end=$(date +%s%N)
  echo "$label $(( (end - start) / 1000000 )) ms"
}
run_ref ref-a
run_ref ref-b

Fill those milliseconds from your own box. I am deliberately not publishing mine, because a number from my machine would only mislead yours.

Mistake 2: I Let the Model Write Straight into the Shell

Step 4 of my harness stages model output as a file, never as a command.
The model is an untrusted text generator. Treat it that way.

# ask, but do not execute
agent_draft() {
  local prompt_file="$1" out_file="$2"
  mkdir -p "$(dirname "$out_file")"
  # placeholder client — substitute your own model CLI
  your-model-cli --prompt "$(cat "$prompt_file")" > "$out_file"
  sha256sum "$out_file" | tee "$out_file.sha256"
}

I keep a short path allowlist in front of git apply. If a patch touches anything outside src/ and test/, the run stops before the shell sees it.

The gate matters less than the habit. The habit is: no model bytes reach bash directly.

This is the part where MonkeyCode participates. I draft patches against its free model access, then run the suite on its free server option, so early iteration does not require a billing decision. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both of those are availability claims, not performance claims. Check the current limits on the project page before you build a habit on them, because free tiers change.

Mistake 3: I Kept No Receipt, So I Blamed the Wrong Layer

I had no record of what ran. So every failure looked like a model failure.
A receipt turns an argument into a lookup.

run_receipt() {
  local run_id="$1" patch="$2" start_ms end_ms code
  start_ms=$(date +%s%N)
  if git apply --check "$patch" && git apply "$patch"; then
    code=0
  else
    code=1
  fi
  end_ms=$(date +%s%N)
  local args=(
    --arg run_id "$run_id"
    --arg commit "$(git rev-parse HEAD)"
    --arg patch_sha256 "$(sha256sum "$patch" | cut -d' ' -f1)"
    --arg baseline_sha256 "$(sha256sum baseline.json | cut -d' ' -f1)"
    --argjson exit_code "$code"
    --argjson wall_ms "$(( (end_ms - start_ms) / 1000000 ))"
  )
  jq -n "${args[@]}" '{
    run_id: $run_id,
    commit: $commit,
    patch_sha256: $patch_sha256,
    baseline_sha256: $baseline_sha256,
    exit_code: $exit_code,
    wall_ms: $wall_ms
  }' > "runs/$run_id.json"
}

Add model_id and tokens_used by hand from the provider console. Do not guess them, and do not infer them from wall time.

Now a failure has a shape. Match the shape, skip the guesswork.

Symptom	Likely layer	First check
Passes locally, dies at file ~100	Host limits	ulimit -n, baseline diff
Same prompt, different patch twice	Model nondeterminism	patch_sha256 in two receipts
Job stops mid-run with no stack trace	Quota or rate limit	Provider console, not the code
Permission denied after a copy	umask or mount flags	umask in baseline.json
Tests slow only on the box	Disk or CPU class	run_ref timings
Patch applies, tests unchanged	Stale commit	commit field in the receipt

Seven rows, one hour of measurement. That table is the day I got back.

The Harness, Numbered

1. Freeze the target: git rev-parse HEAD > runs/commit.txt.
2. Capture the box: ./probe.sh baseline.json.
3. Time a reference workload twice with run_ref.
4. Draft the patch with the model; keep raw output in out/patch.raw.
5. Gate the patch with the path allowlist; stop on any miss.
6. Apply inside run_receipt; write the receipt before running tests.
7. Run the suite. Attach the receipt to any failure report.

Seven steps, roughly ten minutes to assemble once.
After that, every failure names its own layer without a rewrite of the prompt.

Limitations and Who Should Skip This

• Do not put secrets, customer data, or private keys on a shared free box.
• Do not use this loop for GPU training, long-lived daemons, or licensed datasets.
• Do not treat free-tier limits as stable; rerun probe.sh after every reprovision.
• Do not run model output with sudo. Not once, not in a container, not with a note.
• If your team already has pinned CI runners with reproducible images, you may not need a second box. You still need the receipts.

The Letter, Short Version

Dear last-Tuesday-me: the diff was fine.
You skipped the baseline, you piped the model, and you kept no receipt.
Next time, capture the box first and let the failure name itself.

If you want to try that loop without a billing decision, start with the free model access and the free server option, run probe.sh once, and write one receipt before you trust anything the agent produces.