DEV Community: Emery Lin

The Release-Day Reality Check: A Small Model Evaluation You Can Rerun

Emery Lin — Mon, 10 Aug 2026 11:00:22 +0000

A new model announcement can create useful evidence very quickly, but it rarely creates evidence about your work. Release posts usually answer broad questions; a developer evaluating a coding assistant needs narrower ones:

Can it repair the kind of defect I encounter?
Does it respect the libraries and style already in the project?
Can it follow a small interface contract?
Is it responsive enough for the way I work?

Instead of assembling a fresh experiment after every launch, I use a reusable release scorecard. It is small, deliberately personal, and designed to run against any OpenAI-compatible chat endpoint. It will not produce a leaderboard ranking, but it can tell you whether a new option deserves a deeper trial.

Start with a question, not a benchmark number

A useful personal scorecard compares the candidate against a task you already understand. I group mine into four lanes:

Lane	Number of cases	What it exposes
Fault isolation	4	Whether the model finds the likely cause instead of restating symptoms
Controlled edits	4	Whether it changes only the requested behavior
Contract work	3	Whether types, validation, errors, and compatibility survive
Project explanation	3	Whether it can reason from unfamiliar context without inventing files

Fourteen prompts are manageable to review carefully. More importantly, every prompt has an expected outcome because it comes from a repository, ticket, or refactoring exercise I already know.

A case should contain three things: the prompt, the constraints, and a review note describing what a good answer must preserve.

Create cases.json:

{
  "suite": "release-reality-check-v1",
  "tests": [
    {
      "name": "session-cookie-regression",
      "lane": "fault-isolation",
      "prompt": "Users remain authenticated for only one request after login. Here is the middleware and cookie configuration. Find the most probable defect and propose the smallest safe correction.\n\n<redacted source goes here>",
      "constraints": [
        "Do not replace the authentication framework",
        "Explain the failure before editing",
        "Call out any security side effect"
      ],
      "expected": "Identifies the cookie attribute or proxy mismatch, preserves the current session flow, and suggests a focused regression test."
    },
    {
      "name": "price-rounding-contract",
      "lane": "contract-work",
      "prompt": "Update this checkout helper so totals are rounded only at the final display boundary. Preserve the public function signature.\n\n<redacted source goes here>",
      "constraints": [
        "No floating-point money totals",
        "No breaking change to returned fields",
        "Include boundary examples"
      ],
      "expected": "Uses integer minor units or an equivalent decimal representation, keeps the existing API, and covers half-cent boundaries."
    }
  ]
}

The examples above are placeholders rather than executed results. Replace them with cases from your own codebase after removing credentials, customer information, and anything else you cannot send to an inference service.

A portable runner

This Node.js runner avoids a vendor-specific SDK. It records the exact suite, model identifier, response, status, and elapsed time for every attempt.

Save it as run-scorecard.mjs:

import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";

const baseUrl = process.env.MODEL_BASE_URL;
const model = process.env.MODEL_ID;
const apiKey = process.env.MODEL_API_KEY;

if (!baseUrl || !model) {
  throw new Error("Set MODEL_BASE_URL and MODEL_ID before running.");
}

const source = await readFile("cases.json", "utf8");
const suite = JSON.parse(source);
const suiteHash = createHash("sha256").update(source).digest("hex");

const report = {
  suite: suite.suite,
  suiteHash,
  model,
  startedAt: new Date().toISOString(),
  attempts: []
};

for (const test of suite.tests) {
  const prompt = [
    test.prompt,
    "",
    "Constraints:",
    ...test.constraints.map((item) => `- ${item}`)
  ].join("\n");

  const started = performance.now();

  const response = await fetch(`${baseUrl.replace(/\/$/, "")}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      ...(apiKey ? { Authorization: `Bearer ${apiKey}` } : {})
    },
    body: JSON.stringify({
      model,
      temperature: 0,
      messages: [
        {
          role: "user",
          content: prompt
        }
      ]
    })
  });

  const elapsedMs = Math.round(performance.now() - started);
  const text = await response.text();

  let answer = "";
  let parseError = null;

  try {
    const payload = JSON.parse(text);
    answer = payload.choices?.[0]?.message?.content ?? "";
  } catch (error) {
    parseError = error.message;
  }

  report.attempts.push({
    name: test.name,
    lane: test.lane,
    httpStatus: response.status,
    elapsedMs,
    expected: test.expected,
    answer,
    rawResponse: answer ? undefined : text,
    parseError
  });
}

const outputName = `report-${Date.now()}.json`;
await writeFile(outputName, `${JSON.stringify(report, null, 2)}\n`);
console.log(`Wrote ${outputName}`);

Run it with environment variables rather than placing credentials in the repository:

export MODEL_BASE_URL="https://your-compatible-host/v1"
export MODEL_ID="model-under-test"
export MODEL_API_KEY="your-token"

node run-scorecard.mjs

A few details are intentional:

The case-file hash makes it obvious whether two reports used the same prompts.
temperature: 0 reduces avoidable variation, although it should not be treated as a proof of deterministic serving.
Raw error responses are preserved when an endpoint fails or returns an unexpected body.
Timing is stored per case, but it should be read as a rough usability signal—not a controlled benchmark.

Grade against consequences

After the run, score each response without looking at the model name:

Score	Meaning
2	The answer satisfies the expected note and could plausibly be applied after normal review
1	It has the right direction but misses an important constraint or needs substantial repair
0	It misunderstands the problem, invents project details, or creates a risky change

I also mark any response with one or more failure labels:

constraint-ignored
fabricated-context
unsafe-edit
scope-expansion
unverifiable-claim
timeout-or-endpoint-error

The labels are more useful than the total. Two candidates can receive similar scores while failing for completely different reasons. One may be too cautious; another may confidently invent missing APIs. Those behaviors lead to different adoption decisions.

For a cleaner comparison, grade the current assistant first, put the notes away, and then grade the release candidate. If possible, shuffle the reports so the newest model does not automatically receive the benefit of the doubt.

Using a free evaluation environment

Hosted inference is the part that often stops a personal evaluation before it starts. One practical option is MonkeyCode, which currently provides free model access along with a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The runner does not need product-specific code: set its endpoint and model identifier through the environment variables shown above. That separation matters because hosted offerings, available models, limits, and operating conditions can change. Keep the scorecard independent so you can move it to another compatible endpoint or a local server without editing the evaluation logic.

Free access is especially useful for the first pass: deciding whether a model is worth a paid, private, or longer evaluation. It should not be treated as evidence that the same environment is suitable for production traffic.

A compact report format

Once grading is complete, write a short summary that another developer could inspect later:

# Model release check

- Date:
- Candidate identifier:
- Scorecard version/hash:
- Endpoint type:
- Scores by lane:
  - Fault isolation: x/8
  - Controlled edits: x/8
  - Contract work: x/6
  - Project explanation: x/6
- Median observed response time:
- Repeated failure labels:
- Most concerning response:
- Best response:
- Decision:
  - [ ] Ignore for now
  - [ ] Watch for the next revision
  - [ ] Test in a noncritical workflow
  - [ ] Run a larger evaluation

The decision is the important output. A score without an action tends to become another benchmark screenshot.

What this cannot prove

This method is intentionally narrow. It has several limitations:

Small samples have weak statistical power. Fourteen cases can reveal obvious fit problems, but they cannot establish a small percentage improvement.
Personal cases introduce personal bias. Write the expected result and constraints before viewing the response.
Observed latency depends on the hosting environment. Queueing, geography, shared capacity, and request size can dominate the measurement.
A single run is not a regression suite. Rerun the same scorecard when the model, endpoint, prompt, or serving configuration changes.
A hosted endpoint may be inappropriate for sensitive material. If source code cannot leave your environment, connect the same runner to local inference instead.

Skip this approach when making a high-stakes procurement decision, validating production service levels, testing regulated data, or publishing a comparative benchmark. Those situations need representative datasets, controlled infrastructure, documented privacy terms, and proper statistical analysis.

For launch-week triage, however, a stable case pack and a portable runner are usually enough to separate "interesting announcement" from "worth trying in my workflow." If you need somewhere to run that first pass, MonkeyCode is one compatible option; the same scorecard can also be pointed at any other endpoint you trust.

Your AI Coding Agent Has Tools. Here's a Test Suite for the Sandbox You Hope It Respects

Emery Lin — Mon, 10 Aug 2026 08:33:31 +0000

There's a growing unease in the dev community right now: we keep handing AI agents more tools — shell access, file writes, HTTP clients, git push — and mostly we check the boundaries by vibes. The agent didn't delete anything today, so the sandbox works, right?

I wanted something more defensible than vibes. So this article is a small, runnable red-team suite you can point at any environment where you let an agent execute commands. It doesn't assume any particular agent, model, or vendor. It tests the sandbox, not the model — which is where most real failures live.

Everything below is reproducible. The test suite is code you can run. The failure modes are described, not dramatized.

The four boundaries that actually matter

When people say "sandbox," they usually mean one blurry thing. In practice an agent execution environment has at least four separate boundaries, and each fails differently:

Boundary	Question	Typical failure
Filesystem	Can the agent read/write outside its workspace?	`../` traversal, symlink escape, bind mounts
Secrets	Can it read env vars or files you didn't intend?	`env` dump, `~/.ssh`, `~/.aws`, `.env` in parent dirs
Network	Can it make outbound calls you didn't authorize?	Direct egress, DNS exfiltration, metadata endpoints
Persistence	Does anything it changes survive past the session?	Leftover processes, cron, shell rc files, git hooks

A container answers some of these. A VM answers more. "The agent is polite" answers none.

A reproducible boundary test suite

The trick: don't test whether the agent behaves. Test whether the environment refuses misbehavior. Run these commands yourself (or have the agent run them — same thing, since the agent runs commands as some user somewhere) and check the results.

Save as boundary_check.sh:

#!/usr/bin/env bash
# Boundary checks for an agent execution environment.
# Run INSIDE the sandbox, as the same user the agent runs as.
# Every line should print BLOCKED. Anything else is a finding.

check() {
  local name="$1"; shift
  if "$@" >/dev/null 2>&1; then
    echo "OPEN:    $name"
  else
    echo "BLOCKED: $name"
  fi
}

# 1. Filesystem: can we read outside the workspace?
check "read /etc/shadow"        head -n1 /etc/shadow
check "read parent dir"         ls "$HOME/../../"
check "write outside workspace" touch /tmp/outside_workspace_probe

# 2. Secrets: are credentials reachable?
check "read ~/.ssh"             ls "$HOME/.ssh"
check "read ~/.aws"             ls "$HOME/.aws"
check "env contains secrets"    sh -c 'env | grep -qiE "(KEY|TOKEN|SECRET|PASSWORD)"'

# 3. Network: is outbound egress open?
check "outbound HTTPS"          curl -s --max-time 3 https://example.com
check "cloud metadata endpoint" curl -s --max-time 2 http://169.254.169.254/latest/meta-data/

# 4. Persistence: can we install hooks that outlive the session?
check "write shell rc"          sh -c 'echo true >> "$HOME/.bashrc"'
check "install cron job"        sh -c 'crontab -l 2>/dev/null; crontab - >/dev/null 2>&1 <<< ""'

Run it, and you get a table of OPEN / BLOCKED per boundary. That table is your actual security posture — not the diagram in someone's README.

Two important details:

Run it as the agent's user, not as root. Root in a container can often do things the agent's UID can't, and vice versa for misconfigured mounts. Test the identity that matters.
OPEN isn't always wrong — it's always a decision. Maybe you want outbound HTTPS so the agent can fetch docs. Fine. But then it's a choice you made, not an assumption you inherited.

What each finding means in practice

read /etc/shadow OPEN — usually means the agent runs as root in a container. Root-in-container plus a writable Docker socket or CAP_SYS_ADMIN is a classic container-escape setup. Run the agent as a non-root UID; it costs you nothing.
env contains secrets OPEN — the most common real-world leak. If you inject API keys as environment variables into the same process space where an agent executes arbitrary commands, assume the agent's context window eventually contains those keys. Prefer short-lived, scoped tokens fetched by a broker process the agent can call but not read.
cloud metadata endpoint OPEN — on AWS/GCP/Azure this can hand the agent instance credentials. Block 169.254.169.254 at the network layer or use IMDSv2 with hop limits.
outbound HTTPS OPEN — exfiltration path. If you need egress, an allowlist proxy beats a firewall rule you'll forget about.
Persistence probes OPEN — an agent that can append to .bashrc or install a git hook can arrange for code to run later, possibly by you, outside the sandbox. Ephemeral environments (fresh instance per session) neutralize this whole category.

That last point is why I now prefer disposable environments for agent experiments over my dev box. For this kind of boundary testing, I've been spinning up sessions on MonkeyCode's free server option — it gives you a remote execution environment with free model access, which is convenient here precisely because the environment isn't my machine: if a persistence probe succeeds, I throw the instance away instead of auditing my dotfiles.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same logic applies wherever you run agents, though — a throwaway VPS, a CI runner, a Firecracker microVM. The test suite doesn't care.

A minimal isolation checklist

If you'd rather fix things than just measure them, this is the order I'd work in:

Non-root user for the agent's process. (useradd, not --privileged.)
Workspace-only filesystem: mount the project dir, read-only-mount or omit everything else, no Docker socket.
Secret broker: the agent calls a helper that holds credentials; the agent never sees them.
Egress allowlist (or no egress) at the network layer, not in a prompt.
Ephemeral lifetime: the environment is destroyed after the session. Persistence of artifacts (code, PRs) goes through git, not through the machine.

None of this is exotic. It's the same hardening you'd apply to a CI runner executing untrusted pull requests — because that's essentially what an agent session is.

Limitations, honestly

This tests configuration, not cleverness. A determined exploit (kernel bug, container runtime CVE) won't show up in a shell script. This suite catches misconfiguration, which is the common case, not zero-days.
A passing suite doesn't make the agent's output safe. Code the agent writes still needs review before it runs anywhere privileged. Boundary tests and code review are different layers.
Free tiers have limits. Whatever hosted environment you use for this — including the one mentioned above — check current quotas, session duration, and region before building a workflow on it; free offerings change and I can't promise what yours includes. For heavy CI-scale testing, you'll want your own infrastructure anyway.
Prompt-level guardrails are not a boundary. "I instructed the agent not to delete files" is a hope. The tests above measure mechanisms. Keep the distinction clear in your threat model.

Who this is not for

If your agent never executes commands — it only suggests code in a chat window — you don't need any of this; your boundary is your own code review. And if you're running agents against production data or regulated workloads, a shell script and a free sandbox are a starting point for thinking, not a compliance posture. Get a real security review.

For everyone in between — experimenting with agents that can run shell commands, and quietly wondering what "sandboxed" actually guarantees in your setup — run the script. Fifteen minutes, and you'll know which of your four boundaries are real and which are decorative.

If you try it and find something surprising in your own setup, I'd genuinely like to hear which check caught it — the env leak is my bet for the most common one.