DEV Community: Harper Xu

Judge New Models With the Bugs That Already Burned You

Harper Xu — Mon, 10 Aug 2026 10:57:03 +0000

Every model launch arrives with confetti: a chart, a cherry-picked demo, a thread full of people declaring the old stack dead. I still read those threads, but I no longer let them choose my tools. A public score can tell me which system wins across a crowd of strangers. It cannot tell me whether a model will respect the weird constraints that keep my own projects upright: the retry rule that only fails during deploys, the ORM migration that is fine until it meets a legacy table, the timezone edge case that appears every fiscal rollover.

So I stopped asking, “Is this model good?” and started asking, “Does it step on the same landmines I already paid to learn?” The raw material for that answer is not on a leaderboard. It is in old incident notes, reverted commits, and issue threads with titles nobody should ever have to write twice.

A scoreboard cannot see your blast radius

General evals are useful for broad comparison, but they sample a public universe. My expensive failures are local. One came from a payment exporter that rounded correctly in tests and still produced a one-cent mismatch when a customer’s daylight-saving boundary landed inside a settlement window. Another was a websocket client that looked stable in staging, then amplified reconnects after a proxy timeout and turned a small network blip into a queue backup. A third was a CLI that worked everywhere except CI images with a different $HOME layout.

None of those are famous benchmark problems. All of them are exactly the kind of context-shaped failure that makes a “better” model feel worse in practice. If a coding assistant rewrites the retry loop and removes jitter because it looks untidy, aggregate accuracy will not save my evening.

That is why my evaluation unit is not a prompt. It is a scar: a small, runnable package built from something that once hurt, with the embarrassing details preserved enough to matter.

Build a scar suite, not a demo reel

The process starts after the fix, when the bruise is still visible.

Pick 5–8 closed incidents with teeth. Favor bugs where the wrong answer looked plausible: off-by-one retries, narrowing types under a strict config, path handling on another OS, cache invalidation around partial writes. Skip pure typos unless they reveal a real misunderstanding.
Cut each case down until it runs alone. Copy the smallest module that still shows the behavior. Replace private services with fakes, but keep the constraint that made the bug real. If the case needs your whole monorepo, it is archaeology, not an eval.
Write the oracle from the incident, not from taste. The check should encode the failure mode: “must include bounded backoff,” “must not emit naive UTC midnight,” “must return exit code 2 on unreadable config.” A prettier solution that breaks the constraint fails.
Date and rotate the suite. Models change, stacks change, and yesterday’s sharp edge becomes today’s routine. I keep a retired/ folder so old cases stop pretending they represent current risk.
Keep secrets out. Scrub tokens, customer names, hostnames, and internal URLs. A personal benchmark is still a liability if it leaks the map to your house.

A case directory is deliberately plain:

scars/
  2026-04_dst-cent-drift/
    prompt.md
    starter/
    verify.sh
    origin.md
  2026-06_ws-reconnect-storm/
    prompt.md
    starter/
    verify.sh
    origin.md

origin.md is short and human: what paged you, what the fix taught you, why the verifier is strict. That note is what turns a pass/fail count into engineering memory.

A small runner that only cares about evidence

I want the harness to be boring enough that I trust it when I am tired. This version uses only the Python standard library, talks to any OpenAI-compatible chat endpoint, and records JSONL so results are easy to diff later:

#!/usr/bin/env python3
# Replay a local scar suite against one candidate model.
# EVAL_BASE_URL=https://host/v1 EVAL_API_KEY=... python scar_run.py MODEL scars/
import json, os, shutil, subprocess, sys, tempfile, time, urllib.request
from pathlib import Path

BASE = os.environ.get('EVAL_BASE_URL', '').rstrip('/')
KEY = os.environ.get('EVAL_API_KEY', '')

if not BASE or not KEY:
    raise SystemExit('set EVAL_BASE_URL and EVAL_API_KEY')

def chat(model, prompt):
    body = json.dumps({
        'model': model,
        'temperature': 0,
        'messages': [{'role': 'user', 'content': prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE + '/chat/completions',
        data=body,
        headers={'authorization': 'Bearer ' + KEY, 'content-type': 'application/json'},
        method='POST',
    )
    with urllib.request.urlopen(req, timeout=120) as r:
        data = json.loads(r.read().decode())
    return data['choices'][0]['message']['content']

def verify(case_dir, candidate_text):
    with tempfile.TemporaryDirectory() as tmp:
        work = Path(tmp) / 'work'
        shutil.copytree(case_dir / 'starter', work)
        (work / 'candidate.patch').write_text(candidate_text)
        probe = subprocess.run(
            ['bash', str(Path(case_dir / 'verify.sh').resolve())],
            cwd=work, capture_output=True, text=True, timeout=60,
        )
        return probe.returncode == 0, (probe.stdout + probe.stderr)[-500:]

def main(model, suite):
    out = Path('scar_results.jsonl').open('a')
    for case in sorted(Path(suite).iterdir()):
        if not (case / 'prompt.md').exists():
            continue
        prompt = (case / 'prompt.md').read_text() + '\nReturn only a unified diff for starter/.'
        began = time.time()
        try:
            text = chat(model, prompt)
            ok, log = verify(case, text)
        except Exception as e:
            ok, log = False, 'harness-error: ' + repr(e)
        rec = {'model': model, 'case': case.name, 'ok': ok, 'sec': round(time.time() - began, 2), 'log': log}
        out.write(json.dumps(rec) + '\n')
        print(('PASS' if ok else 'FAIL'), case.name, rec['sec'], 's')
    out.close()

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else 'scars')

The important choices are not the HTTP calls. First, the model is asked for a diff, so accidental prose usually breaks application and gets counted as a failure instead of being politely ignored. Second, every record keeps a verifier log tail, because “missed an import” and “removed rate limiting under load” both score zero while demanding different reactions. Third, latency lives next to correctness; a model that is right after a long think may be perfect for nightly migration review and miserable in an editor.

Turn counts into decisions, not mythology

The output is only useful if it changes a small, concrete choice. I read a fresh run like this:

Signal in the JSONL	Likely meaning	Next move
One old case fails, everything else clean	Possible task quirk, possible real weakness	Open `origin.md` and hand-check before caring
Failures cluster around one incident family	The model shares my blind spot	Treat as workload-specific evidence
All green on old cases	Suite may be stale or contaminated	Mine newer bugs and retire solved shapes
Correct but consistently slow	Fine for batch review, risky for interactive help	Route by task, not by hype
Same case flips between runs	Unstable on my constraints	Do not trust a single score

This is deliberately unheroic. I am not crowning a champion. I am deciding whether a candidate deserves a longer trial in the places where I already know the floor is thin.

Cost is the difference between ritual and aspiration

The reason many personal eval plans die is not philosophy; it is friction. A serious pass means several models, several seeds, and enough repetition that a lucky run does not become religion. On a metered bill, that can turn a healthy habit into a quarterly exception.

Open-weight releases help because the same artifact can be reached through more than one host, which weakens the idea that evaluation belongs to whichever endpoint sent the most polished announcement. I also run these replays through MonkeyCode where its current free model access and free server option make the sweep cheap enough to do after a release instead of merely intending to. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That matters here for one practical reason: the harness above is just files and environment variables, so if an offer changes, EVAL_BASE_URL changes and the methodology survives. I would not build a durable habit around any plan, free or paid, that required rewriting the tests when billing changed.

There is also a credibility benefit. When the model is portable and the eval cases are inspectable, another engineer can rerun the claim and argue with the logs instead of the vibe.

Where this can mislead you

Small suites lie by omission. Eight scars can say, “this trips on my known traps,” not “this is universally better.” The moment you hear yourself making a market ranking from six folders, stop.
Yesterday’s bug becomes today’s memorized answer. If a case is based on a public repo or a famous postmortem, assume leakage is possible. Prefer private incidents and keep mining.
A verifier can be too strict. If the check demands one exact implementation, you may reject a safer design. Review failures before converting them into dogma.
Free access is an availability statement, not a vow. Do not hard-code quotas, model lifetimes, or permanence into your team process. Keep endpoints swappable and archive results.
This is not compliance. Security reviews, regulated releases, procurement, and customer-facing guarantees need broader coverage, threat modeling, and human sign-off. A scar suite can narrow the field; it cannot absolve the decision.

Skip it if your risk is already formal

If you are a solo builder or a small team drowning in release notes, this gives you a repeatable Monday move: take the announcement, run the suite against the bugs that taught you something, keep the log, and get back to work. If you are choosing a platform for hundreds of developers, use this only as an early filter before a real evaluation program with representative tasks, privacy review, and rollout criteria.

The deeper shift is that model choice stops being a spectator sport. Your git history is a private benchmark shaped by the exact ways your system fails. Freeze a few of those lessons while they are still sharp, replay them when the next model arrives, and let the most painful old bug cast the first vote. If you try it, I would rather hear which resurrected incident broke the new hotness than which badge it earned somewhere else.

Give Your AI Coding Agent a Sandbox Before You Give It Your Shell

Harper Xu — Mon, 10 Aug 2026 08:28:41 +0000

A discussion thread that got a lot of attention on DEV this week asked a pointed question: we're handing AI agents more and more tools — what happens when the boundaries fail?

It's a fair thing to worry about, and it's not abstract. Any workflow where a model can write code and trigger execution is a workflow where a bad suggestion, a hallucinated flag, or a poisoned dependency becomes a real command on a real machine. The fix isn't to stop using agents. It's to stop running them in an environment where a mistake is expensive.

This post lays out a sandboxing pattern I think every developer using an AI coding assistant should adopt, plus a concrete container setup and a boundary test plan you can run in under an hour. The examples are illustrative — adapt them to your stack and verify them before trusting them.

The mental model: blast radius first, capability second

Most people evaluate an agentic coding tool by asking "what can it do?" The better first question is "what's the worst it can do from where it runs?" Concretely, an agent that can execute code should be assumed capable of:

Reading any file the runtime user can read (including ~/.ssh, .env, cloud credentials).
Writing anywhere the runtime user can write (including your git config and shell rc files).
Making network calls to anywhere the machine can reach.
Installing and executing arbitrary packages.

None of this requires a malicious model. It just requires a model that's confidently wrong one time out of a hundred. So the goal of the sandbox is simple: make all four of those abilities boring.

A minimal sandbox with Docker

Here is a starting Dockerfile for an isolated agent workspace. It's deliberately restrictive and intentionally unglamorous:

FROM node:20-slim

# Non-root user; the agent never runs as root.
RUN useradd -m agent

# Only the project directory is writable.
WORKDIR /workspace
RUN chown agent:agent /workspace

USER agent

# No network by default — enable explicitly per-run if needed.
# Run with: docker run --network none -v ./project:/workspace agent-sandbox

Key properties and why they matter:

Non-root user. A surprising amount of damage requires root. Don't grant it.
Mounted project only. The agent sees the repo, not your home directory. Credentials stay outside the mount.
--network none by default. If the agent needs to npm install, run a separate, explicit "dependency step" with network on, then run the agent's edit/test loop with network off. This single habit blocks entire categories of supply-chain and exfiltration surprises.
Ephemeral containers. docker run --rm means every session starts clean. State is an attack surface too.

Where free compute actually helps this pattern

The obvious objection: spinning up isolated environments per task sounds expensive, or at least annoying, if you're paying per hour for a dev machine.

This is where I've found it handy that MonkeyCode — an AI coding assistant — offers free access to its models and a free server option to run on. That combination maps neatly onto the pattern above: the model side costs nothing to experiment with, and the server side can act as the sacrificial environment — a machine that isn't your laptop and doesn't hold your personal credentials — where the agent's commands execute.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

To be clear about what I'm not claiming: I haven't benchmarked MonkeyCode's models against anything else, and I can't speak to quotas, hardware specs, or how long the free tier lasts. The point of this article stands with any tool — if your agent runtime is free or cheap, use that fact to make the runtime disposable. If it's your daily-driver laptop, containerize it yourself with the Dockerfile above.

A boundary test plan (run this before trusting the setup)

Don't assume your sandbox works. Test the boundaries the same way you'd test auth on an API. Here's a checklist you can execute in one sitting:

#	Test	Pass condition
1	Agent asked to read `/etc/shadow` or host home dir	Fails with permission error
2	Agent asked to `curl` an external URL	Fails when network is disabled
3	Agent asked to write outside `/workspace`	Fails; host filesystem untouched
4	Agent asked to install a package at runtime	Blocked or confined to the ephemeral container
5	Agent runs `rm -rf` on its own workspace	Host unaffected; container discarded
6	Secrets scan of the container image	No tokens, keys, or `.env` files present

Tests 1–3 catch misconfiguration. Tests 4–5 catch overconfidence — yours, not the model's. Test 6 is the one everyone skips: bake no credentials into the image, and pass any needed secrets in explicitly, per-run, with the narrowest scope possible.

A quick automated version of test 2, for example:

docker run --rm --network none agent-sandbox \
  sh -c "curl -sS --max-time 3 https://example.com" \
  && echo "FAIL: network is open" || echo "PASS: network blocked"

Add equivalents for the other rows to CI or a make check-sandbox target, and re-run them whenever the image or the tool config changes.

Limitations, honestly

Sandboxing is not alignment. A sandbox limits damage; it doesn't make the agent's code correct. You still review diffs before merging. Always.
Network-off breaks real workflows. Package installs, fetching docs, hitting test APIs — you'll need a deliberate, audited network-on step, which is friction. That friction is the price of the boundary.
A free server is still someone else's machine. Don't point an agent at proprietary code or customer data on any third-party environment without understanding the provider's data handling. Read the terms; when in doubt, sandbox locally.
Container escapes exist. For ordinary dev work, a non-root container with no network is a strong boundary. For hostile-input scenarios, look at stronger isolation (microVMs like Firecracker, gVisor, separate hardware).

Who should skip this

If your agent usage is purely "suggest a snippet in my editor that I paste myself," you already have a human sandbox — you. This pattern is for workflows where the tool executes code, runs tests, or touches a shell. And if you're working with regulated data, treat this article as a starting point for a conversation with your security team, not a compliance plan.

Takeaway

The answer to "what happens when the boundaries fail?" is: whatever your environment lets happen. So make the environment boring. Non-root, no network, one mounted directory, ephemeral everything, and a six-row test plan you actually run. If you want a low-cost place to try the pattern, MonkeyCode's free model access and free server option are an easy on-ramp for a disposable agent workspace — but the pattern itself belongs in your workflow no matter which assistant you use.