DEV Community: Quinn Li

Don't Run AI-Generated Code on Your Laptop: Free Sandbox Harness

Quinn Li — Fri, 07 Aug 2026 08:06:20 +0000

AI-generated code should be treated as untrusted input: never execute it on the machine that holds your SSH keys. This post gives you a ~60-line JavaScript harness that runs model output on a disposable remote server with a scrubbed environment, hard timeouts, and pre-written pass/fail assertions — so evaluation becomes reproducible instead of "looks right to me."

Last week I asked a model to write a small utility that "cleans up temp files older than 7 days." The output looked fine — until I read it line by line and found it happily recursing through $HOME because I had forgotten to pin a working directory in my prompt. Nothing ran. Nothing broke. But it was a useful reminder of a rule I keep violating: AI-generated code is untrusted input, and I keep executing it on my development laptop.

There's been a lot of discussion lately about what happens when agent tool boundaries fail — the OWASP Top 10 for LLM Applications lists excessive agency and insecure output handling as first-class risks. I don't have a grand theory of agent safety. What I have is a boring, reproducible workflow: generate on one side, execute on a disposable server, evaluate with a tiny harness. This post is that harness, plus an honest list of where it falls apart.

The setup: three constraints that shape everything

The workflow has three constraints I care about:

Generated code never executes on my development machine.
Every evaluation run is reproducible — same prompt, same seed of test cases, same pass/fail criteria.
It costs nothing to tear down and redo. If I can't afford to run the experiment ten times, I won't trust the result.

For constraints 1 and 3 I use a free remote server as the execution sandbox. I've been running this with MonkeyCode, which currently offers free model access for the generation side and a free server option for the execution side, so the whole loop stays at zero cost.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Any equivalent setup works — a throwaway VPS, a cloud free tier, a container host. The harness below doesn't depend on any specific provider; it depends on the separation between where code is generated and where it runs.

The harness: subprocess isolation plus fixed assertions

The core idea: model output goes into a file, the file runs in a subprocess with a timeout, no network-dependent behavior, a scrubbed environment, and a scratch working directory. Then a set of assertions decides pass/fail — not my eyes scanning the diff. The mechanics lean entirely on Node's built-in child_process module, so there are zero dependencies to audit.

// sandbox-run.mjs — executes one generated snippet with hard boundaries.
// Run this ON the sandbox server, never on your dev machine.
import { spawn } from "node:child_process";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";

const TIMEOUT_MS = 5_000;

export async function runInSandbox(code) {
  const dir = await mkdtemp(join(tmpdir(), "aigen-"));
  const entry = join(dir, "snippet.mjs");
  await writeFile(entry, code);

  const result = await new Promise((resolve) => {
    const child = spawn("node", [entry], {
      cwd: dir,                        // confined working directory
      env: { PATH: process.env.PATH }, // scrubbed env: no tokens, no keys
      stdio: ["ignore", "pipe", "pipe"],
    });

    let stdout = "", stderr = "";
    child.stdout.on("data", (d) => (stdout += d));
    child.stderr.on("data", (d) => (stderr += d));

    const killer = setTimeout(() => {
      child.kill("SIGKILL");
      resolve({ status: "timeout", stdout, stderr });
    }, TIMEOUT_MS);

    child.on("close", (code) => {
      clearTimeout(killer);
      resolve({ status: code === 0 ? "ok" : "error", exitCode: code, stdout, stderr });
    });
  });

  await rm(dir, { recursive: true, force: true });
  return result;
}

And the evaluation side — a fixed table of test cases that I write before looking at any model output, so I don't bend the criteria to fit what the model produced:

// eval.mjs — one prompt, many runs, fixed assertions.
import { runInSandbox } from "./sandbox-run.mjs";

const cases = [
  { name: "sorts numbers ascending", input: [3, 1, 2], expect: [1, 2, 3] },
  { name: "handles empty array",     input: [],         expect: [] },
  { name: "does not mutate input",   input: [2, 1],     expect: "no-mutation" },
];

async function evaluate(generatedCode) {
  const wrapped = `
    import fs from "node:fs";
    const input = JSON.parse(fs.readFileSync(0, "utf8"));
    ${generatedCode}
    console.log(JSON.stringify(solution(input)));
  `;
  const results = [];
  for (const c of cases) {
    const r = await runInSandbox(wrapped);
    results.push({ case: c.name, ...r });
  }
  return results;
}

// Feed in code produced by whatever model you're evaluating.
const codeFromModel = process.argv[2]
  ? await (await import("node:fs/promises")).readFile(process.argv[2], "utf8")
  : `const solution = (a) => [...a].sort((x, y) => x - y);`;

console.table(await evaluate(codeFromModel));

Two deliberate design choices

Assertions before generation. I write the test table first, paste the task into the model (via MonkeyCode's free model access in my runs, but any model works), save the raw output to a file, and only then run node eval.mjs output-attempt-1.js. This turns "looks right to me" into "passed 3/3 with zero mutations."
The temp dir is the blast radius. The scrubbed env means even deliberately hostile output can't read my credentials, because there are none in scope on the sandbox box. Combined with cwd pinning, the $HOME recursion bug from my opening anecdote physically cannot escape the scratch directory.

When this harness is worth it — and when it isn't

Situation	Use the sandbox harness?	Why
Evaluating a model's output quality across many runs	Yes	Reproducible pass/fail beats vibes
Generated code touches the filesystem, shell, or network	Yes	This is exactly the failure mode that hurts
Quick syntax question, code never executed	No	Just read it; a sandbox adds nothing
Output needs real secrets or production data to be meaningful	No	Scrubbed env makes the test unrealistic — test manually in staging instead
Code needs OS-level isolation (untrusted dependencies, install scripts)	Not enough	A subprocess is not a security boundary; add a real container/VM layer

Limitations, stated plainly

A child process is not a security sandbox. It stops accidents, not adversaries. If you're running code from an agent loop that fetched instructions from the internet, add container isolation, a read-only filesystem, and network egress rules. My harness is seatbelts, not an airbag system.
The free tier is a convenience, not a guarantee. Free model access and free server options change over time; treat cost as zero today, design so you can swap providers tomorrow. That's why the harness has no provider-specific code in it.
Pass/fail tables can be gamed by the model. If the task leaks into a model's training data, high scores may reflect memorization. Rotate in novel test cases — contamination is a documented problem in benchmark evaluation, as covered in NIST's guidance on AI risk management.
This measures one narrow thing — whether output runs correctly under constraints. It says nothing about readability, maintainability, or whether you should ship it.

Who should skip this — and what to do tonight

If you review every generated line before running anything and your tasks are low-risk, the overhead isn't justified. If you already have CI-based eval infrastructure, you don't need my 60-line version. This workflow is for the gap in between: people pasting model output straight into a terminal on their main machine and hoping.

If you're in that gap, the smallest useful step isn't adopting any particular platform — it's moving execution off your laptop tonight and writing three assertions before your next prompt. Copy the two files above onto any throwaway box, run node eval.mjs against your next model output, and see what the pass/fail table tells you that your eyes didn't. If you want a zero-cost place to try that loop, the free model access and free server from MonkeyCode are what I used for the runs above; the harness itself will follow you to whatever you use next.

Deciding If AI Code Review Fits Your Repo: A Throwaway Experiment That Costs Nothing

Quinn Li — Wed, 05 Aug 2026 07:07:29 +0000

Buying an AI review tool to find out whether AI review works on your codebase is backwards. The sensible order is: define what "works" means, run a cheap experiment against code you already understand, then spend money only if the numbers justify it.

This article is that experiment. It's a deliberately disposable setup — a Node script, a handful of your own historical pull requests, and a spreadsheet-style labeling pass — hosted on free infrastructure so the total cost of answering the question stays at zero.

Start with a claim you can disprove

"Does AI code review help?" is not testable. This is:

For our typical PRs, does a general-purpose model produce at least one finding a human reviewer would act on, per PR, without flooding us with comments we'd immediately dismiss?

Two numbers fall out of that framing: actionable findings per PR and noise ratio. Everything below exists to measure those two things and nothing else. If you skip this step, you'll finish the experiment with vibes instead of a decision.

Why free infrastructure is enough for this

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The experiment needs two things: model access and a place to run the script. MonkeyCode's free model access and free server option cover both, which is why I'm using them here — but the script talks to any OpenAI-compatible endpoint, so nothing below is locked to one provider. Point the base URL elsewhere and it runs unchanged.

Free tiers are the right tool for an evaluation specifically because the experiment is short-lived. You're not building production capacity; you're buying information.

The review runner

// pr-probe.mjs — usage: node pr-probe.mjs <path-to-diff>
import { readFileSync, writeFileSync, existsSync } from 'node:fs';

const target = process.argv[2];
if (!target) {
  console.error('need a diff file');
  process.exit(1);
}

const patch = readFileSync(target, 'utf8');
const changedLines = patch.split('\n').filter(l => /^[+-]/.test(l) && !/^[+-]{3}/.test(l)).length;

if (changedLines > 250) {
  console.error(`skipping: ${changedLines} changed lines is out of scope for this probe`);
  process.exit(1);
}

const system = `You review code diffs for a senior engineer.
Respond with JSON only — an array where each element is:
{"path": string, "approx_line": number, "kind": "defect"|"hazard"|"style", "note": string}
Rules: one sentence per note. Flag only things that would change behavior,
corrupt data, leak secrets, or break under load. Never repeat a linter.
Empty array if clean.`;

const response = await fetch(`${process.env.PROBE_BASE_URL}/v1/chat/completions`, {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.PROBE_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: process.env.PROBE_MODEL,
    temperature: 0,
    messages: [
      { role: 'system', content: system },
      { role: 'user', content: patch },
    ],
  }),
});

if (!response.ok) {
  console.error(`API error ${response.status}: ${await response.text()}`);
  process.exit(1);
}

const payload = await response.json();
const raw = payload.choices?.[0]?.message?.content ?? '[]';

const record = {
  source: target,
  changed_lines: changedLines,
  model_reply: raw,
  captured_at: new Date().toISOString(),
};

const logFile = 'probe-log.jsonl';
writeFileSync(logFile, (existsSync(logFile) ? readFileSync(logFile, 'utf8') : '') + JSON.stringify(record) + '\n');
console.log(raw);

Two deliberate choices worth noting:

Temperature 0. You want run-to-run consistency so disagreements between runs mean something. A creative reviewer is the last thing you need when measuring.
Raw output is logged verbatim. You label it by hand afterward. Automating the judgment step at this stage would just move the unverified assumption somewhere harder to see.

Generating ground truth from your own history

The clever part of this experiment is that you don't need new code. Your merged PRs from last month already contain the answer key: whatever human reviewers caught, plus whatever slipped into production and got fixed later.

mkdir -p patches
git log --since="6 weeks ago" --merges --format=%H -12 | while read sha; do
  git diff "${sha}^1" "$sha" > "patches/${sha:0:10}.patch"
done

for p in patches/*.patch; do
  node pr-probe.mjs "$p"
  sleep 3   # stay polite to free-tier rate limits
done

For each PR, jot down before you look at the model output: what did the human reviewer flag, and did anything from this PR cause a follow-up fix? That's your baseline. Looking at model output first will contaminate your memory of what humans actually said.

Labeling and the decision rule

Open probe-log.jsonl and tag every model finding:

Tag	Definition	Effect on verdict
`hit`	A human would have acted on this	Counts as value
`noise`	Wrong, unverifiable, or pure taste	Counts against
`echo`	Duplicate of another finding in the same PR	Counts against
`gap`	Something the follow-up fix proved was real, that the model missed	Weakens the case

Set your thresholds now, before labeling. Mine were: proceed to a paid pilot only if hit / total PRs >= 0.5 and noise <= hits. Yours should reflect how patient your reviewers are — a team of two tolerates more noise than a team of twenty. Writing thresholds down first is the whole trick; otherwise you'll grade on a curve and "validate" whatever happened.

Running it on the free server

For a batch job over twelve historical PRs, a laptop is fine. Where the free server option earns its place is the next step: a tiny webhook listener that probes each newly opened PR for a week, so you're evaluating live diffs rather than archaeology.

Keep the deployment stateless — append results to a gist or a remote JSONL file — because the honest end state of this experiment is often "delete everything." If teardown takes more than a minute, you've accidentally built infrastructure instead of running an experiment.

What this probe cannot tell you

Nothing about latency or throughput. Free-tier performance is not a preview of a paid plan. Don't time these requests and extrapolate.
Nothing about large changes. The 250-line cutoff is a scope guard, not a suggestion. Big refactors need chunked context and per-file passes, which this script deliberately doesn't attempt.
Nothing about model choice. One prompt, one model, temperature zero. You're testing the workflow hypothesis on your diffs, not shopping for the best reviewer.
Whether your org allows it. Check code-handling policy before sending real diffs anywhere. If the answer is murky, run the probe against a toy repo and accept the weaker signal.

When to skip this entirely

If AI review already runs in your CI and you're tuning comment volume, this is a step backward — you have production data, use that instead. Also skip it if your diffs are dominated by generated code, lockfiles, or migrations; diff-only review degrades badly there and the experiment will tell you something false.

The actual point

The deliverable of this exercise isn't a configured tool. It's a sentence you can say in a planning meeting: "On our last twelve PRs, a model found N things humans would have acted on, at a noise level we'd tolerate" — or the opposite, which is equally valuable and much cheaper than learning it from an invoice. If you want to run that loop without opening a billing page, MonkeyCode's free model and server tier is one convenient substrate; the probe itself doesn't care where it points.