DEV Community: Jordan Huang

A Personal Scorecard for Evaluating New Coding Models (With the Scripts I Use)

Jordan Huang — Mon, 10 Aug 2026 11:18:35 +0000

Every model launch triggers the same ritual on my timeline: screenshots of impressive one-liners on one side, screenshots of embarrassing failures on the other, and a hundred hot takes in between. After my third cycle of switching daily-driver models based on screenshots, I admitted the uncomfortable part — I wasn't evaluating anything. I was adopting other people's anecdotes.

So I built myself a scorecard: a small, frozen set of tasks, a clean room to run them in, and a written grading policy. New release lands, I spend a couple of hours running it through the scorecard, and I walk away with notes instead of feelings. Here's the full setup, the reasoning behind it, and the honest list of ways it can lie to you.

The two ways casual testing fooled me

Before the scorecard, my evaluation process was "open a chat, try a few prompts, form an opinion." That process has structural defects I couldn't prompt my way around.

You're sampling blind. Three prompts is three draws from a distribution you can't see. Pick tasks the model has effectively memorized and it looks superhuman. Pick tasks outside its comfort zone and it looks useless. Both impressions are artifacts of your prompt choice, not properties of the model.

Your environment contaminates the result. When I tested inside my working editor, the model silently benefited from my open files, my earlier corrections, and half-finished edits. I wasn't scoring the model — I was scoring the model wearing my project as a life-support system. Great for shipping, worthless for comparison.

The fix isn't clever. It's the boring discipline of any experiment: same stimulus, isolated environment, more than one repetition.

Cost and safety were the real blockers

Two practical problems kept me from running proper evaluations for months.

First, cost discipline fights measurement discipline. A serious run means repeating each task several times, and on a metered API every repetition is a small financial decision. I kept cutting trials to save money, which destroyed exactly the statistical confidence I was paying for.

Second, I didn't want an agent executing generated commands on the machine that holds my SSH keys, cloud credentials, and browser sessions.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What unblocked me was MonkeyCode, an open-source coding agent. Its free model access lets me point a fresh release at my scorecard without setting up billing, and its free server option means the agent loop runs in a throwaway remote environment instead of my laptop. Since the agent is open source, I can also confirm that trial runs don't share memory or leftovers — something I'd have to take on faith with a closed runner.

None of that is load-bearing, though. Any agent harness you can inspect, plus any model endpoint you don't have to ration, works. The scorecard below doesn't care what executes it.

Scorecard structure: eight tasks, repeated four times

The scorecard has three pieces. Swap my placeholder tasks for sanitized fragments of your own work — that's the version that actually tells you something.

tasks.yml — the frozen task list. Eight tasks split across three skill areas, each one sized to be small enough that a pass/fail verdict is unambiguous:

- id: fix-01
  area: bugfix
  sandbox: ./sandboxes/order-pipeline
  prompt: |
    Orders placed between 23:59 and 00:01 are assigned to the wrong
    fulfillment date. Find the defect, repair it, and add a regression
    test that fails on the old code and passes on yours.
  verify: npm test -- --grep boundary-date

- id: feat-03
  area: feature
  sandbox: ./sandboxes/pastebin-lite
  prompt: |
    Add an expiry feature: pastes accept a TTL in minutes and return
    HTTP 410 after expiry. Test the exact expiry boundary.
  verify: cargo test expiry

- id: explain-02
  area: analysis
  sandbox: ./sandboxes/worker-queue
  prompt: |
    Under burst load, queue latency spikes then never recovers.
    Explain the mechanism in writing. Do not modify any file.
  verify: manual  # graded against rubric.md

score.py — the harness. Only two design decisions really matter: every repetition starts from a byte-identical copy of the sandbox, and the full conversation transcript gets saved so I can re-read failures months later.

import shutil, subprocess, time
from pathlib import Path

def repetition(task, rep, model):
    workdir = Path(f"results/{task['id']}/rep{rep}")
    if workdir.exists():
        shutil.rmtree(workdir)
    shutil.copytree(task["sandbox"], workdir)

    started = time.time()
    transcript = run_agent(model=model, prompt=task["prompt"], cwd=workdir)
    elapsed = round(time.time() - started, 1)

    if task["verify"] == "manual":
        outcome = None  # graded by hand against the rubric
    else:
        result = subprocess.run(task["verify"], shell=True, cwd=workdir,
                                capture_output=True)
        outcome = result.returncode == 0

    return {"task": task["id"], "rep": rep, "model": model,
            "outcome": outcome, "elapsed_s": elapsed,
            "transcript": transcript}

# 8 tasks x 4 reps = 32 runs per model. Not science. Enough for a decision.

rubric.md — grading rules for the analysis tasks, since they have no automated check. Each explanation gets 0–2 points on three dimensions: identifies the true mechanism, proposes a workable mitigation, and invents nothing. Five of six points is a pass, and any fabricated log output or nonexistent config option zeroes that dimension instantly. A model that hallucinates evidence is more dangerous than one that admits confusion.

Read the failure pattern, not the percentage

Thirty-two runs give you a grid of outcomes. The aggregate pass rate is the single least useful cell in that grid. What changes my behavior is the geometry of the failures:

Pattern in the grid	Probable cause	How I respond
Strong on fix/feature, weak on analysis	Retrieval strength, weak causal modeling	Use for implementation, keep architecture decisions human
Rep 1 passes, reps 2–4 wobble	Sampling instability at default settings	Distrust any single demo of this model; add reps
Passes my public sandboxes, fails my private ones	Has seen the public code before	Discount heavily for proprietary codebases
Verify step fails, transcript looks right	My sandbox or check script is broken	Fix the scorecard, not the model
Passes everything, takes forever	Capability without latency	Fine for overnight batch jobs, wrong for pairing

The fourth row is the one I re-learn constantly. "The model broke" and "my test harness broke" produce identical output until you open the transcript. Archive everything.

Why reproducibility beats verdicts

There's a reason I lean on open components end to end. Open weights can't be silently swapped between my repetitions. An open agent loop lets me prove exactly what context the model received. And an open scorecard means anyone who doubts my conclusion can rerun it instead of arguing with it. When the next open-weight model drops, the most valuable community contribution isn't another hot take — it's a task suite with transcripts attached.

Opinions expire with the next release. A frozen task list doesn't, because the next model walks into the same room and faces the same eight tasks.

Where the scorecard fails

Thirty-two runs is not a study. It ranks models for my own workflow. Procurement or platform decisions need far more data plus human review of every failure.
Small sandboxes generalize poorly. A model that shines on a 1,500-line fixture may drown in a 500k-line monorepo. If that's your reality, build sandboxes from your actual code (scrubbed of secrets).
Free tiers are borrowed, not owned. Free model access and free server capacity can be throttled, re-priced, or withdrawn at any time. Use them for weekend evaluations, never wire them into CI, and read the current terms before assuming anything.
Don't bother with this at all if stakeholders need an audit-grade benchmark, if your code legally cannot leave your network and you have no local harness, or if your real question is answerable by simply using the tool for a week. Daily lived experience is a legitimate instrument — it's just not a calibrated one.

The actual takeaway

Launch-week discourse is a machine for manufacturing confidence from other people's single prompts. A scorecard converts "who should I trust?" into "what did I measure?" — eight tasks, four repetitions, one written rubric. If the two blockers I described (metered trials, unsafe execution) are what's stopping you, MonkeyCode's free model access and free server option are one workable way to start this weekend. But the scorecard is the asset that outlives any vendor, and it plugs into whatever tooling you already have.

If you maintain your own task list, I'd genuinely like to know what skill areas you test that mine misses — especially anything involving large multi-file refactors, which I still haven't found a good way to grade.

Giving an AI Coding Agent a Job Without Giving It Your Credentials

Jordan Huang — Mon, 10 Aug 2026 08:56:46 +0000

There's a conversation happening on DEV right now about what happens when AI agents get more tools and the boundaries around those tools fail. Most of the discussion is philosophical. I want to make it concrete: if you're going to let an AI coding agent run inside your CI pipeline — even on your own infrastructure — what does the actual sandbox look like, and how do you prove it holds?

This article walks through a repeatable harness: a decision table for what the agent is allowed to touch, a runnable sandbox script, and a canary test that fails loudly the moment a boundary leaks. Everything here runs on a plain Linux box.

The problem in one sentence

An agent that can read your repo and execute shell commands is, from a security standpoint, an unprivileged remote user who happens to be very fast — so treat its environment like you'd treat an untrusted contributor's laptop.

Concretely, the three failure modes I care about:

Credential exfiltration — the agent (or a prompt injected via a file/issue it reads) prints an env var into a place that gets committed or POSTed somewhere.
Network egress — the agent downloads or uploads something you never approved.
Scope creep in the repo itself — the agent edits files outside the task you gave it, quietly.

Decision table: what goes where

Before any code, decide which capabilities the agent actually needs. This is the table I use as a starting point — adjust for your own tasks:

Capability	Code-fix task	Doc-generation task	Dependency-upgrade task
Read repo files	✅	✅	✅
Write repo files	✅ (scoped paths)	✅ (docs/ only)	✅ (lockfiles, manifests)
Execute tests/build	✅	❌	✅
Network egress	❌	❌	✅ (package registry only)
Secrets in env	❌	❌	❌ (use a short-lived token if truly needed)
Git push	❌ (open MR instead)	❌	❌

The pattern: network and secrets default to denied, and writing is always path-scoped. The agent proposes, CI disposes.

Where the agent actually runs

You need a machine to host the agent loop and a model endpoint. For experimentation, I used MonkeyCode here — it offers free access to coding models and a free server option, which made it cheap to iterate on the harness without burning a budget on my own mistakes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the current product documentation for exactly which models and server limits apply, since availability details change; the sandboxing below is provider-agnostic anyway.

The important part isn't where the model lives — it's that the execution environment is locked down regardless. A generous free tier doesn't change the threat model.

The sandbox script

Below is a minimal, reproducible wrapper using only standard tooling. It runs the agent's working directory read-only-except-scratch, strips the environment, and blocks network with unshare (Linux namespaces — no Docker required for the demo, though Docker works too):

#!/usr/bin/env bash
# agent-sandbox.sh — run a command against a repo with minimal privileges.
# Usage: ./agent-sandbox.sh /path/to/repo "your-agent-command --flag"
set -euo pipefail

REPO="$(realpath "$1")"
CMD="$2"
SCRATCH="$(mktemp -d)"
trap 'rm -rf "$SCRATCH"' EXIT

# 1. Strip environment: no inherited secrets, no CI tokens.
# 2. Drop network entirely with a private net namespace.
# 3. Bind-mount the repo read-only; only $SCRATCH is writable.
env -i PATH=/usr/bin:/bin HOME="$SCRATCH" \
  unshare --net --mount --map-root-user \
  bash -c "
    mount --bind '$SCRATCH' /tmp 2>/dev/null || true
    cd '$REPO'
    $CMD
  "

Notes:

env -i is the single highest-value line. Most leaks I've seen discussed are just inherited environment variables.
unshare --net removes networking for the whole process tree. If your task legitimately needs a registry (the dependency-upgrade row above), replace this with an egress proxy allowlist, not open internet.
For real CI, run this inside an ephemeral job container/VM as well — defense in depth. The script is a second wall, not the only wall.

Prove it: the canary test

A sandbox you haven't attacked is a rumor. Plant canaries and assert they never escape:

#!/usr/bin/env bash
# canary-test.sh — boundary checks that must all pass before trusting the harness.
set -euo pipefail

REPO="$(mktemp -d)"
echo 'console.log("hello")' > "$REPO/app.js"

fail=0

# Test 1: a fake secret in the environment must not be readable.
export AWS_SECRET_ACCESS_KEY="CANARY-7f3d-not-a-real-key"
if ./agent-sandbox.sh "$REPO" 'env' | grep -q "CANARY-7f3d"; then
  echo "FAIL: secret leaked into sandbox environment"; fail=1
else
  echo "PASS: environment stripped"
fi

# Test 2: network must be unreachable.
if ./agent-sandbox.sh "$REPO" 'curl -sS --max-time 3 https://example.com' 2>/dev/null; then
  echo "FAIL: network egress succeeded"; fail=1
else
  echo "PASS: network blocked"
fi

# Test 3: repo must be unchanged after a hostile command.
BEFORE=$(sha256sum "$REPO/app.js" | cut -d' ' -f1)
./agent-sandbox.sh "$REPO" 'echo pwned >> app.js; git init -q . 2>/dev/null || true' || true
AFTER=$(sha256sum "$REPO/app.js" | cut -d' ' -f1)
if [ "$BEFORE" != "$AFTER" ]; then
  echo "FAIL: repo was modified"; fail=1
else
  echo "PASS: repo intact (modifications confined to scratch)"
fi

rm -rf "$REPO"
exit $fail

Run this in CI before any agent job. If any check fails, the agent doesn't run. That ordering matters — most setups test the agent's output but never test the cage.

One more canary worth adding once you allow limited egress for package installs: embed a unique fake token in a file the agent reads, then alert if that string ever appears in outbound requests or in the diff the agent produces. Cheap to build, catches both naive leaks and injection-driven ones.

Limitations and who shouldn't do this

Namespace-based sandboxing is not a hard security boundary against a determined adversary with a kernel exploit. For genuinely hostile input, use a separate VM per job (most CI platforms already give you this if you don't cache runners).
Prompt injection is not solved by sandboxing. Sandbox limits blast radius; it doesn't stop the agent from being manipulated into writing bad code within its allowed scope. Human review of the diff is still mandatory — the table above says "open MR instead of push" for exactly this reason.
Don't use this pattern at all if your task requires the agent to touch production secrets, customer data, or signed release artifacts. Get a scoped, short-lived credential from your secrets manager and audit it, or keep that step manual.
The free-tier setup I mentioned is fine for prototyping the harness; I haven't load-tested it against a large monorepo pipeline, and quota/availability details are documented by the provider, not guaranteed by this article.

Takeaway

The current debate about agent tool boundaries gets a lot more tractable once you write the boundaries down as a table, enforce them with a hundred lines of shell, and attack your own enforcement with canaries before trusting it. The agent platform matters less than the cage. If you want a zero-cost sandbox to try this harness yourself, MonkeyCode's free models and server are one way to get an agent loop running — then point the canary tests at it and see what holds.