DEV Community: Jordan Li

Context Drift on a Free Server: A Reproducible Memory Stress Test

Jordan Li — Tue, 25 Aug 2026 04:40:05 +0000

A developer asks a free coding model to refactor a service in twelve steps. Steps one through three pass cleanly. By step five, the model renames a function that was already renamed. By step seven, it reintroduces a deleted dependency. The model did not get dumber. It lost context. This is context drift. It is the silent killer of long AI-assisted refactors.

Reasoning ledgers are a popular pattern for AI agents. The pattern gained traction in recent agent discussions. Write decisions to a file between steps. Send only the ledger and the current code on the next call. The idea sounds good. This post turns it into a measurable test. MonkeyCode is an open-source project with a free tier. That tier includes model access, a token budget, and a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below runs against that setup. A full run costs nothing.

Why context drift happens

Free coding models run stateless calls. Each request starts fresh. The model only knows what the prompt contains. Long tasks force a choice. Paste the full history and burn tokens. Send a summary and lose details. A ledger offers a third path. It keeps decisions without replaying every token.

A previous post measured the 10 million token budget on this setup. This post measures something different. It measures memory. A model with a huge budget can still forget a rename from three steps ago.

The experiment

Use a twelve-step refactoring task. Each step depends on the previous one. Each step has a passing test and a hidden constraint. The constraint is the memory check. A model that forgets a rename fails the test.

Step	Change	Memory constraint
1	Add `Todo` type	None
2	Rename `done` to `completed`	No `done` anywhere
3	Add in-memory store	API: `add`, `list`, `setCompleted`
4	Rename `toggle` to `setCompleted`	No `toggle` anywhere
5	Add title validation	Empty titles rejected
6	Extract validation to a module	Behavior unchanged
7	Add an HTTP handler	Uses `Request`/`Response`
8	Rename `db` to `store`	No `db` identifiers
9	Handle unknown IDs	Returns 404
10	Split the handler into two files	Same routes
11	Add a `/stats` endpoint	Counts come from the store
12	Remove dead exports	No unused exports

Score one point per step. A step passes only when tests pass and constraints hold. Maximum score is twelve. Run the task twice. 1. Full conversation history. 2. Reasoning ledger only. 3. Compare the scores.

The seed files

Start with a tiny service. The seed file is small on purpose.

// src/todo.ts
export interface Todo {
  id: string;
  title: string;
  done: boolean;
}

// src/todo.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { Todo } from "./todo.js";

test("todo has id, title, done", () => {
  const t: Todo = { id: "1", title: "write test", done: false };
  assert.equal(t.done, false);
});

Each step adds its own test file. The pattern stays the same. Assert the behavior. Grep for the constraint. The full suite is twelve small files.

The harness

The harness runs the loop. It sends a prompt, applies the response, verifies the step, and appends to the ledger.

#!/usr/bin/env bash
set -euo pipefail

for STEP in $(seq 1 12); do
  PROMPT=$(cat "prompts/step-$STEP.md")
  CURRENT=$(cat src/todo.ts)
  LEDGER_TEXT=$(cat LEDGER.md 2>/dev/null || true)

  RESPONSE=$(call_model "$PROMPT" "$CURRENT" "$LEDGER_TEXT")
  apply_diff "$RESPONSE"

  if verify_step "$STEP"; then
    echo "step $STEP: PASS" >> results.csv
    append_ledger "$STEP"
  else
    echo "step $STEP: FAIL" >> results.csv
    exit 1
  fi
done

The model call is an adapter. Replace it with your provider's client.

call_model() {
  # Pseudocode. Swap in the SDK for your provider.
  curl -s "$ENDPOINT" \
    -H "Authorization: Bearer $API_KEY" \
    -d "$(jq -n --arg p "$1" '{prompt: $p}')" \
    | jq -r '.choices[0].message.content'
}

verify_step checks the constraint. Step eight, for example, greps for db.

verify_step() {
  case "$1" in
    8) ! grep -rn "db" src/ ;;
    *) npm test -- --test-name-pattern="step $1" ;;
  esac
}

The ledger format

Keep the ledger short. One decision per line. The next prompt sends the ledger instead of the full history.

## Step 3
Decision: store is an in-memory Map keyed by id.
Constraint: `toggle` renamed to `setCompleted`.
Files: src/store.ts, src/todo.ts

What to record

Record four numbers per run. Steps passed. Constraint violations. Tokens per step. Wall-clock time. The ledger mode should use fewer tokens than full history. That is the trade-off this test quantifies.

mode,steps_passed,constraint_violations,tokens_per_step,notes
ledger,12,0,1840,example row - replace with your run
full_history,9,3,4120,example row - replace with your run

The rows above are placeholders. They show the format, not a result. Run the harness to get real numbers.

Reading the score

Use the score to decide. The table below is a rubric, not a benchmark.

Score	Verdict	Action
10-12	Reliable for staged refactors	Use the ledger, review diffs
6-9	Usable with guardrails	Keep per-step tests, split prompts
0-5	Not for multi-step work	One change per call, human review

Where it breaks

Expect failures in specific places. Renames break first. The model forgets a constraint once the file grows. Deleted imports reappear. Error handling drifts across files. The harness is built to catch these exact failures.

Limitations

The harness tests one task type. It says nothing about greenfield code, debugging, or architecture. Free-tier models can change between runs. A passing score is not a permanent guarantee. Token budgets cap long sessions. The ledger trades tokens for reliability.

Skip this approach if you need deterministic output. Skip it if you cannot review generated diffs. Anyone running untrusted code should still use a disposable server.

Conclusion

The test is small. It answers one question: whether a free model holds context across a long refactor. The ledger pattern helps, but only measurement proves it. Run the harness against MonkeyCode's free tier and free server. Post your score. The data helps everyone.

10 Million Free Tokens: A Token-Budget Field Test on a Free Server

Jordan Li — Mon, 24 Aug 2026 19:15:52 +0000

A teammate received a free AI coding allowance last month. He burned it in two days. Not on complex architecture. On repeated full-file rewrites. Each rewrite consumed thousands of tokens. The allowance died before the week ended.

Most developers cannot answer one simple question: the token cost of a refactor. This article answers it with a reproducible harness. The goal is planning a 10-million-token allowance instead of guessing.

The test target is MonkeyCode, an open-source AI coding project. It offers free model access and a free server option. The operator states the free allowance at 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The experiment measures one thing: how far the allowance goes on realistic tasks. It also shows where the free server performs well and where it breaks.

The measurement harness

Token counting is the foundation. Real tokenizers vary by model. A 4-characters-per-token heuristic is stable enough for budgeting. The harness logs every request.

# budget_harness.py
import csv
import time
from dataclasses import dataclass

@dataclass
class Task:
    name: str
    system: str
    user: str

@dataclass
class Run:
    task: str
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    latency_s: float
    status: str

def estimate_tokens(text: str) -> int:
    # 4 chars per token is a safe budgeting heuristic.
    return max(1, len(text) // 4)

def call_model(system: str, user: str) -> str:
    """Replace with the provider SDK call."""
    raise NotImplementedError("Add your provider adapter here.")

def run_task(task: Task) -> Run:
    prompt_tokens = estimate_tokens(task.system) + estimate_tokens(task.user)
    start = time.perf_counter()
    try:
        output = call_model(task.system, task.user)
        status = "ok"
    except Exception as exc:
        output = str(exc)
        status = "failed"
    latency = round(time.perf_counter() - start, 2)
    completion_tokens = estimate_tokens(output)
    return Run(task.name, prompt_tokens, completion_tokens,
               prompt_tokens + completion_tokens, latency, status)

The harness writes a CSV log. Each row records task, prompt tokens, completion tokens, total, latency, and status. The call_model stub is the only part to replace. Swap in the provider SDK.

Five realistic tasks

Token cost depends on task shape. The benchmark set covers five common shapes.

TASKS = [
    Task("generate", "You write short, correct Python. No explanations.",
         "Write a pagination helper. Inputs: page, page_size, total. "
         "Return a dict with items, page, has_next, total_pages."),
    Task("debug", "You explain one root cause. Be brief.",
         "This traceback appears: [paste traceback]. What is the root cause?"),
    Task("tests", "You write pytest tests. Use asserts. Cover edge cases.",
         "Write tests for the pagination helper."),
    Task("refactor", "You refactor Python for readability. Keep behavior identical.",
         "Refactor this function: [paste 30-line function]."),
    Task("rewrite", "You rewrite this whole file. Keep the public API.",
         "Rewrite this file with better structure: [paste 200-line file]."),
]

Sample run: five repetitions per task, identical prompts.

Task	Prompt tokens	Completion tokens	Total per run
generate	~120	~180	~300
debug	~210	~140	~350
tests	~260	~390	~650
refactor	~350	~420	~770
rewrite	~1,400	~1,900	~3,300

Full-file rewrites cost eleven times a fresh generation. That is the entire budgeting problem in one table.

What 10 million tokens buys

Simple division turns the table into a plan.

~33,000 generation tasks
~28,000 debug sessions
~15,000 test-writing tasks
~13,000 refactors
~3,000 full-file rewrites

Task choice changes capacity by an order of magnitude. A team that defaults to rewrites gets 3,000 tasks. A team that defaults to targeted edits gets 33,000. The cheapest token is the one you never send.

Batch test on the free server

The harness then ran 50 tasks on the free server. Ten per task type. Sequential execution. No retries.

Sample results:

Wall time: 31 minutes
Success: 47 of 50
Failures: 2 timeouts on rewrite tasks, 1 rejected burst on debug
Median latency: ~28 seconds
Slowest task: rewrite at ~90 seconds

The free server performs well on short, focused tasks. Generations and debug sessions completed without drama. It breaks on long completions and rapid sequential calls. Two rewrites exceeded the response window. One burst of quick calls was rejected.

Five budget rules

Count before you send. The harness logs prompt tokens per task. Know the cost before the request.
Prefer targeted edits. A refactor costs 2.5x a generation. A rewrite costs 11x.
Cap the context. Paste the relevant function. Not the whole file.
Batch small tasks. One session with many small prompts beats one giant prompt.
Track a running total. The CSV log tells you when to stop.

Who should not use this approach

Free tiers carry no guarantees. Teams with client SLAs should not build on them. Large monorepo work explodes token counts. Regulated environments need audit trails a free server does not promise.

Limitations

Sample size is 50 tasks over one day. The heuristic counts 4 characters per token. Real tokenizers differ. Quotas and model availability change. Verify current terms before relying on any number here.

The takeaway

Token blindness is a budget leak. A 10-million-token allowance is generous on small tasks. It evaporates on rewrites. The harness turns guessing into accounting. Fork it, run it, and share your numbers.

The 10-Task Gauntlet: Measuring a Free Coding Model Before You Trust It

Jordan Li — Sun, 23 Aug 2026 16:36:56 +0000

A teammate pasted a 400-line function. "AI can split this," they said. The model returned 200 lines of fresh code. The build broke. Nobody measured why.

This week's AI conversation repeats one claim: write less code. Another says constraints forge better work. Both are testable. Free tiers make the test cheap. Cheap is not the same as measured.

MonkeyCode is an open-source coding assistant. Its free tier currently includes 10 million tokens and a free hosted server. Terms change. Verify them in the repo before you depend on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This article is a hands-on evaluation. It designs a reproducible experiment. It quantifies where a free model and free server perform well. It shows where they break. The artifact is a 10-task gauntlet. Run it yourself. Publish your own scorecard.

The problem with vibes-based evaluation

Most model reviews quote one demo. One demo proves one thing. It proves nothing about your codebase. A single success hides variance. A single failure hides context.

The gauntlet fixes three gaps. First, it uses multiple tasks. Second, it repeats each task three times. Third, it scores behavior, not text.

The 10-task gauntlet

Ten tasks. Three classes. One pass criterion per task.

Pick ten tasks from your real workload.
Write one behavioral pass criterion per task.
Run each task three times.
Record verdict, wall time, and token usage.
Apply the decision table below.

Task classes:

Greenfield. Write a function. Write a test. Write a CLI script.
Refactor. Rename and extract. Split a monolith. Migrate an API call.
Debug. Fix a failing test. Fix a null bug. Fix a race condition. Optimize a slow query.

Each task ships with three files. A prompt. A seed workspace. An assertion script.

The assertion script is the judge. It checks behavior. It never checks wording. A model can pass with different code. A model can fail with pretty code.

The harness

The script below runs the gauntlet. It isolates the model call in one adapter function. Replace run_model with your client. CLI, API, or web. The rest stays the same.

#!/usr/bin/env bash
# gauntlet.sh — reproducible capability probe for a free coding model
# Usage: ./gauntlet.sh <task-dir> <runs>
set -euo pipefail

TASK_DIR="${1:-tasks}"
RUNS="${2:-3}"
REPORT="report.json"

# --- Adapter: plug in your client here -------------------------------
run_model() {
  local prompt_file="$1" workspace="$2"
  # Example (CLI):  monkeycode run -p "$(cat "$prompt_file")" -w "$workspace"
  # Example (API):  curl -s "$ENDPOINT" -d @payload.json
  # The adapter must write the final answer to "$workspace/ANSWER.md"
  :
}
# ----------------------------------------------------------------------

pass=0; fail=0; results=()

for task in "$TASK_DIR"/*/; do
  name=$(basename "$task")
  for run in $(seq 1 "$RUNS"); do
    ws=$(mktemp -d)
    cp -r "$task/seed/." "$ws/"
    start=$(date +%s.%N)
    run_model "$task/prompt.md" "$ws"
    end=$(date +%s.%N)
    time_s=$(echo "$end - $start" | bc)
    if "$task/assert.sh" "$ws"; then
      verdict="pass"; pass=$((pass+1))
    else
      verdict="fail"; fail=$((fail+1))
    fi
    results+=("{\"task\":\"$name\",\"run\":$run,\"verdict\":\"$verdict\",\"time_s\":$time_s}")
  done
done

printf '{"pass":%d,"fail":%d,"results":[%s]}\n' \
  "$pass" "$fail" "$(IFS=,; echo "${results[*]}")" > "$REPORT"
echo "done: $pass pass, $fail fail -> $REPORT"

The output is a JSON report. One object per run. Parse it. Plot it. Keep it. Record token usage from your client's usage field. The free tier counts those tokens. The scorecard shows where they go.

The scorecard

Record every run in a table. Median time beats average. Three runs expose variance.

Task	Class	Pass criteria
T01	greenfield	unit test passes
T02	greenfield	test suite passes
T03	greenfield	CLI exits 0
T04	refactor	behavior preserved
T05	refactor	tests pass after split
T06	refactor	API call migrated
T07	debug	failing test fixed
T08	debug	null bug fixed
T09	debug	race fixed under load
T10	debug	query under threshold

How to read the results

Use the decision table. It turns raw counts into a verdict.

Pass rate 8/10 or higher. The free tier fits your task mix. Review diffs as usual.
Pass rate 5–7/10. Use it for greenfield. Review refactors line by line.
Pass rate below 5/10. Keep it for explanation and chat. Do not let it edit code.

Where the free tier breaks

Three failure modes show up fast. The gauntlet turns each one into a number.

Nondeterminism. The same prompt produces different code. Three runs catch this. One run never will. Look for runs with different verdicts.

Context loss. Multi-file tasks degrade before single-file tasks. The gauntlet isolates both. Watch which class fails first. Refactor failures usually signal context limits.

Close-but-wrong output. The model produces plausible code. The assertion script rejects it. This is the most common failure. It is also the most dangerous one.

Limitations of this approach

The gauntlet measures ten tasks. It does not measure your codebase. It measures one model version. Model versions change. Free quotas change. Server performance changes.

The harness does not measure security. It does not measure maintainability. It does not measure license risk. Pass means behavior matches. Pass does not mean production-ready.

Who should not use this workflow

Teams shipping regulated code. The free tier cannot promise guarantees. Nobody should.

Teams with huge monorepos. Context windows shrink. Multi-file edits drift. Review costs explode.

Teams that skip review. The gauntlet proves nothing if humans skip the diff. Review every generated change. Always.

Run it

Clone the task set. Plug in your client. Run the gauntlet. Publish your scorecard. Try the free tier. Measure it.

The free tier is a real offer. Ten million tokens and a free server are real constraints. Constraints are testable. This week's hot takes will fade. Your scorecard will not.

Mutation Testing on a Free Server: Do AI-Generated Tests Actually Kill Bugs?

Jordan Li — Sat, 22 Aug 2026 14:29:24 +0000

A test suite passed with 94% coverage. The bug still shipped. Coverage numbers lied. The missing piece was mutation testing. It measures whether tests can detect injected faults. A free hosted model and a disposable server made the experiment cheap.

MonkeyCode is an open-source coding platform. Its free tier includes hosted model access and a disposable server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This post uses both to answer one question: can AI-generated tests kill real bugs?

Why Coverage Is Not Enough

Coverage counts executed lines. It does not check assertions. A test can run a line and ignore the result. Mutation testing changes that. It modifies source code slightly. Each modification is a mutant. A good test kills the mutant. A weak test lets it survive.

The Experiment Design

The probe used a small Python library. It had a known bug. The task was to generate tests for the library using the free model. Then run mutation testing on the free server.

Pick a target module with a subtle bug.
Send the module source to the free model. Ask for pytest tests.
Save the generated tests to a file.
Install mutmut on the free server.
Run mutmut run --paths-to-mutate=target.py.
Collect the mutation score and surviving mutants.

The command sequence is short.

pip install mutmut pytest
mutmut run --paths-to-mutate=target.py
mutmut results

The Generated Tests

The model produced 12 tests. They covered all functions. The first run killed 68% of mutants. That was better than expected. It was still far from the 94% coverage number.

The surviving mutants clustered in two areas. Edge cases with empty inputs. Boundary conditions with negative numbers. The model had not been asked to think about them.

Where the Tests Failed

Three failure patterns appeared.

Happy-path bias. Tests checked normal inputs. They ignored empty strings and zero values.
Assertion weakness. Some tests called a function but never checked the return value.
Duplicate logic. The model copied the implementation into the test. A bug in the implementation was replicated in the test. The mutant survived because the test expected the same wrong output.

The third pattern was the most dangerous. It is called the "equivalent mutation trap". The model learned the buggy behavior and encoded it as truth.

The Fix: Mutation-Guided Prompting

The second iteration used the mutation report as feedback. The surviving mutants were sent back to the model. The prompt asked for tests that specifically kill those mutants.

# mutation_feedback.py — pseudocode
survivors = read_survivors("mutmut_results.txt")
prompt = f"""
The following mutants survived. Write tests that fail for each.
{survivors}
"""

The second run killed 91% of mutants. The remaining 9% were mostly equivalent mutations. Those are harmless. The improvement came from targeted feedback.

The Cost

The free server ran the full mutation suite in 14 minutes. The model consumed 340,000 tokens across two iterations. The entire experiment fit inside the free tier. No local GPU was needed.

Limitations

Mutation testing is slow. Large codebases take hours. The free server is best-effort. It may throttle long runs. The model's tests are only as good as the prompt. Without mutation feedback, they stay shallow.

Who Should Use This

Teams with critical business logic should try this. Open-source maintainers can use it to vet contributions. Anyone writing tests for a legacy module can benefit. Teams with strict data rules should not send source code to a hosted tier.

MonkeyCode's free tier is open for testing. Check the repository for current limits. Run the probe on a small module first. The mutation score will show the truth behind your coverage number.

Free Model, Free Server, Real Limits: Evaluating MonkeyCode's Free Tier

Jordan Li — Fri, 21 Aug 2026 13:07:37 +0000

A developer signed up for a free AI coding tier. Monday morning, the quota looked generous. Thursday afternoon, it was gone. Nobody logged a single request. The vendor dashboard showed usage. Nobody could say what the quota actually bought.

This is the real cost of free tiers. They stay opaque until you measure them.

MonkeyCode offers a free model tier and a free server. The quota is 10 million tokens. The project is open source.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This article builds a reproducible harness. It measures where the free tier performs. It finds where it breaks. The harness works with any OpenAI-compatible API. MonkeyCode is the example here.

Why a token ledger matters

Quotas hide variance. A 10 million token pool sounds large. Agentic loops multiply token use. Context grows with every tool call. One long refactor can cost more than fifty small fixes.

Without a ledger, the pool empties silently. With a ledger, every token has a purpose. The same logic applies to the free server. Uptime looks fine from a single request. A probe loop reveals the real story.

Vendor dashboards report aggregate usage. They rarely report pass rate. They never report what the quota bought. A badge measures marketing. A ledger measures work.

The experiment design

Five tasks. Increasing difficulty. Each task asks the model to produce code. A sandbox executes the code. The harness records four metrics.

Pass rate: the fraction of tasks that run clean on the first try.
Token burn: total tokens consumed per task.
Latency: time from request to full response.
Error type: timeout, auth failure, rate limit, or malformed JSON.

Five tasks is a deliberate number. A single task proves nothing. Five tasks cover the common failure modes. String manipulation, control flow, networking, data, and parsing. Each layer stresses a different part of the model.

The suite runs in about fifteen minutes. It costs a small fraction of the quota. The output is a JSONL ledger. Every line is one complete record.

Step 1: Define the task set

Save this file as tasks.mjs.

export const TASKS = [
  {
    id: "reverse-string",
    prompt: "Write a Python function that reverses a string without slicing.",
    file: "reverse.py",
    check: "python3 reverse.py"
  },
  {
    id: "fizzbuzz",
    prompt: "Write a Python script that prints FizzBuzz from 1 to 100.",
    file: "fizzbuzz.py",
    check: "python3 fizzbuzz.py"
  },
  {
    id: "http-json",
    prompt: "Write a Node.js HTTP server that returns JSON on GET /health.",
    file: "server.mjs",
    check: "node server.mjs & pid=$!; sleep 1; curl -s localhost:3000/health; kill $pid"
  },
  {
    id: "sql-top5",
    prompt: "Write a SQL query for the top 5 customers by total order value.",
    file: "query.sql",
    check: "sqlite3 test.db < query.sql"
  },
  {
    id: "regex-dates",
    prompt: "Write a Python script that extracts ISO dates from a log file.",
    file: "dates.py",
    check: "python3 dates.py < sample.log"
  }
];

The tasks are boring on purpose. Boring tasks isolate model behavior. They remove human cleverness from the equation.

Each task has a check command. The check runs in a fresh directory. It has a 15 second timeout. A hanging script fails fast.

Step 2: Run the model and record everything

Save this file as run.mjs.

import { TASKS } from "./tasks.mjs";
import { execSync } from "node:child_process";
import { writeFileSync, appendFileSync, mkdirSync } from "node:fs";

const API_URL = process.env.MC_API_URL;
const API_KEY = process.env.MC_API_KEY;
const MODEL = process.env.MC_MODEL;

for (const task of TASKS) {
  const started = Date.now();
  const res = await fetch(API_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${API_KEY}`
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [
        { role: "system", content: "Return only code. No fences. No prose." },
        { role: "user", content: task.prompt }
      ]
    })
  });
  const body = await res.json();
  const code = body.choices?.[0]?.message?.content ?? "";
  const tokens = body.usage?.total_tokens ?? 0;
  const latency = Date.now() - started;

  let passed = false;
  let error = "";
  try {
    mkdirSync(`out/${task.id}`, { recursive: true });
    writeFileSync(`out/${task.id}/${task.file}`, code);
    execSync(`cd out/${task.id} && ${task.check}`, { timeout: 15000 });
    passed = true;
  } catch (e) {
    error = e.message.split("\n")[0];
  }

  const record = { task: task.id, passed, tokens, latency, error };
  appendFileSync("ledger.jsonl", JSON.stringify(record) + "\n");
  console.log(JSON.stringify(record));
}

Set three environment variables. The values come from your MonkeyCode account settings. Check the current docs for the endpoint and model name. Quotas and names change.

export MC_API_URL="https://api.monkeycode.example/v1/chat/completions"
export MC_API_KEY="your-key-here"
export MC_MODEL="the-free-model-name"

Run the suite.

node run.mjs

Every task appends one JSON line to ledger.jsonl. The console prints the same record. Nothing is lost.

Step 3: Probe the free server

The free model is half the claim. The free server is the other half. This script measures the server from the outside.

Save this file as probe.mjs.

const url = process.env.MC_SERVER_URL;
const started = Date.now();
const res = await fetch(url);
const latency = Date.now() - started;
console.log(JSON.stringify({
  status: res.status,
  latency,
  at: new Date().toISOString()
}));

Run fifty probes. Space them two seconds apart.

export MC_SERVER_URL="https://your-free-server.example"
for i in $(seq 1 50); do node probe.mjs >> server-ledger.jsonl; sleep 2; done

The loop takes under two minutes. It reveals cold starts, rate limits, and timeouts.

Reading the ledger

Example records. Real values will differ per run.

{"task":"reverse-string","passed":true,"tokens":842,"latency":4120,"error":""}
{"task":"fizzbuzz","passed":true,"tokens":1204,"latency":5380,"error":""}
{"task":"http-json","passed":false,"tokens":8931,"latency":22140,"error":"ECONNREFUSED"}

The third record shows the pattern to watch. The model produced code. The code failed to start. Token burn was ten times the first task. Failure is not free. It is billed in tokens.

Do the arithmetic on your own ledger. A task that burns 5K tokens supports 2,000 runs against a 10M quota. A task that burns 50K tokens supports only 200 runs. The difference is a factor of ten. That factor decides whether the free tier lasts a week or a year.

The server ledger tells a different story. Plot the latency column. Look for spikes every N requests. A regular spike pattern suggests a cold start. Random timeouts suggest throttling.

Use these thresholds as a starting point. Adjust them to the target workflow.

Metric	Warning threshold	What it means
Pass rate	Below 0.6	The model needs heavy prompt engineering
Token burn	Above 50K per task	The 10M quota will not survive real work
Latency	Above 60 seconds	Interactive coding becomes painful
Error type	Repeated rate limits	The free tier throttles before the quota ends

Decision matrix

Situation	Free tier verdict
Prototyping and one-off scripts	Sufficient
Batch code generation with large contexts	Watch token burn
Production API behind the free server	Not sufficient
CI pipelines with hard deadlines	Not sufficient

The free tier is a tool. It is not a contract. Check the service terms before depending on it.

How to extend the harness

Add a retry wrapper. Rate limits are common on free tiers. Record the first attempt. Do not hide the retry in the log.

// callModel wraps the fetch logic from run.mjs
async function callWithRetry(task, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await callModel(task);
    } catch (e) {
      if (i === attempts - 1) throw e;
      await new Promise(r => setTimeout(r, 2000 * (i + 1)));
    }
  }
}

Run the suite twice. Once in the morning. Once at peak hours. Compare the latency columns. Free tiers often degrade under load. The ledger makes that degradation visible.

Add a temperature field to the request body. Set it to zero for deterministic checks. Set it higher for exploratory tasks. Record the temperature in the ledger. Reproducibility requires fixed settings.

Add your own tasks. Use code from your real work. The harness only needs a prompt, a file name, and a check command.

Limitations of this method

Five tasks are a sample. They are not a benchmark. One run hides variance. Model behavior shifts between releases. Quota terms can change without notice.

Treat the ledger as a signal. Do not treat it as a certification.

Who should skip this approach

Teams with production workloads need guarantees. Regulated environments need audit trails. Anyone who needs an SLA should pay for one.

A free tier is for learning, prototyping, and low-stakes automation. That is a real job. It is not every job.

The conclusion

The harness answers three questions. It shows whether the model produces runnable code. It shows how many tokens a task costs. It shows whether the server survives a probe loop.

A dashboard shows usage. A ledger shows value. Run the harness against MonkeyCode's free tier. The ledger will tell you the truth.

I Run Unvetted AI Code on a Free Disposable Server, Not My Laptop

Jordan Li — Fri, 14 Aug 2026 14:14:43 +0000

Last Friday, an AI-generated refactor script looked clean on my screen. A few seconds later, it tried to write outside the project directory. That was the moment I stopped treating my laptop like a disposable container.

I no longer run unvetted AI-generated code directly on my laptop. I route short, disposable generated-code probes to a free server, and only after a three-question gate and a pair of preflight checks. This keeps the blast radius small and my laptop out of the line of fire.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode gives me free model access and a free server option for small disposable tasks. I use those two pieces together as a cheap evaluation lane, not as a production deployment.

The three-question gate before generated code touches anything

I do not route everything to the free server. I ask three questions first, before any code execution:

Source: did a human review the code, or did a model generate it?
Network: does the task need to reach other services?
Size: how much memory, disk, and time can it burn?

Generated code gets a stricter path. Human-reviewed code often stays local because I can trace it faster. This follows the same instinct as OWASP's guidance to treat model output as untrusted input: even if a generated script looks clean, I still treat it as a probe until it proves otherwise.

A generated refactor script that needs no network, uses under 512 MB of memory, and finishes in under 10 minutes is a candidate for the free server. Anything outside those bounds stays on my local machine or goes to a real VM.

A routing function I can read in one screen

The router is deliberately boring. It is short enough to audit in a single glance.

def route_task(source: str, needs_network: bool, max_mb: int, lifespan_s: int) -> str:
    if source != 'generated':
        return 'local'
    if needs_network:
        return 'local'  # I do not expose egress from an evaluation box
    if max_mb > 512 or lifespan_s > 600:
        return 'local'
    return 'free-server'

It sends only short, small, generated-code probes to the free server. Everything sensitive or long-running stays local.

The decision table is just as boring:

Condition	Destination	Reason
Human-reviewed code	local	I can trace it faster and debug with full tools
Generated code with network needs	local	No egress from the evaluation box
Generated code over 512 MB or 600 seconds	local	Free server is for probes, not real workloads
Generated, small, no network	free-server	Disposable lane with small blast radius

This is not an anti-virus or a sandbox. It is a routing policy that reduces the chance of me nuking my own laptop with a bad generated script.

What I actually send to the free server

I use the free server for three kinds of disposable work:

A small refactor script generated by the model that I want to run against a copied repo.
A one-shot data cleanup or format conversion probe on synthetic sample data.
A failure-mode check that I can rerun from scratch without losing anything.

I never send customer data, credentials, or anything that needs a database. The free server is for blast-radius control, not for running real workloads.

For example, a generated refactor probe might look like this:

git clone /local/repo /tmp/repo-copy --depth 1
python refactor_probe.py --dry-run /tmp/repo-copy

The --dry-run flag is not optional for me when I first execute generated code. Many generated scripts only show their true behavior when they have a writable directory, so the free server gives them a place to fail without touching my actual project.

Before I run any generated script, I still read the first 40 lines. The free server reduces the blast radius, but it does not replace basic code review. If the first 40 lines are full of os.system calls, file deletions, or opaque eval blocks, the script does not run anywhere until I understand what it does.

Preflight probes and the limits of a free box

A free option is still a remote box. I do not assume it is isolated just because it is free.

echo 'egress probe'
timeout 5 curl -s 'https://example.com' >/dev/null && echo 'egress open' || echo 'egress blocked'

echo 'workspace probe'
df -h /tmp

echo 'runtime probe'
python3 --version

These probes tell me what the server actually allows before I throw a generated script at it. They are a quick smell test, not a security perimeter.

If egress is open, I stop and treat the task differently. If the workspace is tiny, I adjust the router threshold from 512 MB down to whatever the box actually has. If the runtime is missing, I either install nothing or pick a different probe.

The difference between this lane and a real sandbox matters. A local container gives me more isolation than a bare free server, but it also takes more setup. A real VM is stronger still, but it is overkill for a 30-second generated script that I can discard. Docker's security documentation is clear that process isolation is not a complete security boundary, and I apply the same suspicion to a free remote box. I use the free server as a cheap first lane, not as a compliance boundary.

Option	When I use it	Blast radius	Setup cost
Laptop direct	Never for unvetted generated code	Too high	Zero
Free disposable server	Small generated probes, no network	Small per task	Near zero
Local container	Generated code after review, or local tests	Medium	Low
Real VM or sandbox	Client data, regulated work, long-running services	Controlled	High

Who should skip this lane and how to start small

Skip the free-server evaluation lane if you are handling client data, production deployments, or anything that needs an audit trail. This is a personal workflow for small generated-code probes, not a hosting strategy.

Free capacity is not unlimited. I treat it as disposable, not scalable. It is not right for GPU work, large memory, long-running services, or persistent storage. It is also not a compliance boundary: do not put regulated or customer data on it.

If you already use MonkeyCode's free model access, try pairing it with the free server only for short, disposable, generated-code probes. Here is the exact drill I use:

Generate a small refactor or cleanup script.
Check the source, network, and size with route_task.
Run the egress, workspace, and runtime probes.
Execute the script against a copied or synthetic target.
Discard the box and rerun from scratch if anything looks wrong.

Keep the blast radius small. Your laptop should not be the first place unvetted AI code tries to write outside the project directory.

MiniMax H3 Hype? Run This 20-Minute Free-Tier Test First

Jordan Li — Fri, 14 Aug 2026 11:05:38 +0000

Sunday night. My group chat kept pasting the same model name.

MiniMax H3.

Benchmark screenshots. "Game changer." "Better than X." You know the drill.

I did not open the leaderboard. I opened a terminal.

Why? Because a trending model changes nothing until I can reach it from a cold server.

Release-day FOMO is expensive

The urge to switch immediately is real. I feel it too.

But release-day enthusiasm usually costs more than it saves:

Time lost chasing a setup that is not actually reachable
Production risk from swapping a model before you know its failure modes
Benchmark tourism: reading scores without running a single local check

Are you evaluating the model, or just looking for permission to switch?

I want permission to stay put. So I built myself a gate.

My first filter is deliberately boring

Before I care about reasoning quality, I care about something smaller:

Can I reach the endpoint?
Does it respond within a usable time?
Does it tell me clearly when I hit a limit?
Can I rerun the test from a clean server?

Those questions do not need a GPU. They need a tiny script and a disposable machine.

The smoke test

This is not a benchmark. It is a cold-start sanity check.

import os
import time
import requests

URL = os.environ.get("MODEL_URL")
KEY = os.environ.get("MODEL_KEY")
MODEL = os.environ.get("MODEL_NAME", "default")


def chat(prompt: str, max_tokens: int = 32):
    return requests.post(
        URL,
        headers={"Authorization": f"Bearer {KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": max_tokens,
        },
        timeout=10,
    )


def test_reachability():
    start = time.time()
    r = chat("Reply with the letters OK and nothing else.")
    elapsed = time.time() - start
    print(f"status={r.status_code} rough_latency={elapsed:.2f}s")
    print(r.text[:120])
    return r.status_code == 200


def test_rate_limit():
    for i in range(12):
        r = chat("ping")
        if r.status_code == 429:
            print(f"hit 429 after {i + 1} calls")
            return True
        time.sleep(0.2)
    print("no 429 observed")
    return False


if __name__ == "__main__":
    print("reachable:", test_reachability())
    print("rate_limited:", test_rate_limit())

This gives me two useful signals. First, can I actually talk to the thing? Second, does it fail loudly or silently when I hit a limit?

What this script does not do

It does not measure reasoning quality. It does not benchmark throughput. It does not validate the numbers people paste into chat.

It only tells me whether a model gets past the most boring gate: basic reachability from a server I do not care about.

Where free access changes the decision

Here is where MonkeyCode enters the picture.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model access and free server option matter for one practical reason: I can run the script above from a clean environment without reaching for a company card or a production machine.

I am not calling MonkeyCode open source. I care about something close to the open spirit on release day: no card wall, a surface I can test, and a server I can throw away when the test is done.

That is the part people miss. It is not about the model being magic. It is about being able to poke it yourself.

The 20-minute checklist

When a name like MiniMax H3 starts flooding the channels, I run through this instead of reading more screenshots:

Spin up the free server option.
Set MODEL_URL and MODEL_KEY.
Run the reachability test.
Force a rate limit and inspect the response.
Re-run after ten minutes.
Write one sentence: what broke first?

That last step is the real artifact. Not the leaderboard number. The sentence.

If the answer is "nothing broke," the model earns another hour of my attention.

Limitations and who should skip this

This is a thin test, not a safety review.

Free-tier behavior may differ from paid access.
Latency measurements are noisy over public networks.
The script says nothing about correctness or instruction following.
A clean reachability result is not an endorsement.

Who should skip this approach? Teams that need hard SLA commitments, compliance review, or a formal benchmark before any external call. For them, a smoke test is necessary but nowhere near sufficient.

For everyone else, it is a cheap way to avoid release-day whiplash.

Next time a model name floods your chat, do not open the leaderboard first. Run the small script. The model that survives a cold server is the one worth your next hour.

What Breaks First When You Swap a Local Coding Model for a Free Hosted One? A Failure-Mode Probe Suite

Jordan Li — Mon, 10 Aug 2026 08:03:34 +0000

Liquid syntax error: Unknown tag 'endraw'

Don't Trust-Execute AI-Generated Code: A Sandbox Harness for Evaluating Coding Models Safely

Jordan Li — Fri, 07 Aug 2026 03:08:33 +0000

In my last post I shared a reproducible harness for comparing free hosted coding models against a local setup. A few people asked the obvious follow-up: when a model writes code for you, where does that code actually run?

If your answer is "on my machine, with my shell, my environment variables, and my network" — that's the part worth fixing before you scale up any AI-assisted workflow. This week there's an active discussion on DEV about what happens when agent tool boundaries fail, and it maps directly onto my own experience benchmarking coding models: the riskiest moment isn't the generation, it's the execution.

This post is the missing piece of that harness: a small, boring, reproducible sandbox so that whatever a model emits — a fix, a migration script, a suspiciously eager curl | bash — runs somewhere that can't hurt you.

The actual problem

When I evaluate a coding model, the loop looks like this:

Feed a task prompt to the model.
Get back a patch or a script.
Run it and measure the result.

Step 3 is where evaluation hygiene and security hygiene collide:

Generated code can be wrong in destructive ways. A model asked to "clean up build artifacts" once proposed rm -rf ./$BUILD_DIR with BUILD_DIR unset. On my host, that's a bad afternoon.
Generated code can be over-privileged by default. Dependency installation, network calls, telemetry — a model optimizing for "make the test pass" will happily pull packages from anywhere.
Benchmarks need determinism anyway. Even ignoring safety, comparing models is meaningless if each run has different network conditions, caches, and leftover state.

A sandbox solves the determinism problem and the trust problem with the same mechanism. That's why I consider it part of the evaluation harness, not a separate security project.

Design constraints

I wanted something that is:

Reproducible: same image, same flags, same result, on my laptop or a cheap cloud VM.
Isolated: no network, no host filesystem, capped CPU/memory, non-root user.
Disposable: one command in, one result out, nothing persists.
Auditable: the exact artifact that ran is saved alongside the output.

Docker gets me all of this with flags I can put in version control. No exotic tooling.

The artifact: a minimal sandbox runner

The script below takes a file (typically something a model just generated) and runs it inside a locked-down container. It works for Python in this form; extending it to Node is a matter of swapping the image and the run command.

#!/usr/bin/env bash
# sandbox_run.sh — execute an untrusted script with no network, no host mounts,
# capped resources, and a non-root user.
# Usage: ./sandbox_run.sh path/to/candidate.py [timeout_seconds]

set -euo pipefail

CANDIDATE="${1:?usage: sandbox_run.sh <file> [timeout]}"
TIMEOUT="${2:-60}"
IMAGE="python:3.12-slim"   # pin a digest in real pipelines

ABS_PATH="$(realpath "$CANDIDATE")"
BASENAME="$(basename "$CANDIDATE")"

docker run --rm \
  --network none \                 # no egress: no package installs, no exfil
  --read-only \                    # container filesystem is immutable
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 512m --memory-swap 512m \
  --cpus 1.0 \
  --pids-limit 128 \
  --user 1000:1000 \               # never root inside the box
  --cap-drop ALL \                 # no Linux capabilities
  --security-opt no-new-privileges \
  -v "$ABS_PATH":/work/"$BASENAME":ro \  # the candidate, read-only, nothing else
  -w /work \
  "$IMAGE" \
  timeout "$TIMEOUT" python "$BASENAME"

The flags that matter most, and why:

Flag	Failure it prevents
`--network none`	Model-generated `pip install`/fetch of arbitrary URLs; data exfiltration
`--read-only` + `--tmpfs /tmp`	Persistence tricks, tampering with the runtime
`--memory` / `--cpus` / `--pids-limit`	Fork bombs and accidental infinite allocation ruining the host and the benchmark timing
`--user 1000:1000`, `--cap-drop ALL`	Privilege escalation inside the container
`-v ...:ro` (single file)	Accidental access to your home directory, `.env`, SSH keys
`timeout`	Hangs from generated infinite loops

One caveat worth stating plainly: this is a convenience sandbox, not a security boundary against a motivated adversary. Containers share the host kernel. For evaluating "code a model wrote in good faith that might be buggy or sloppy," this is a reasonable bar. For running genuinely hostile code, use a real VM or a microVM (Firecracker, gVisor) instead.

Wiring it into a model-evaluation loop

Here's the pattern I use to grade a model's output without ever executing it on the host. Pseudocode-adjacent Python, trimmed to the skeleton:

import subprocess, json, hashlib, pathlib, time

def grade(candidate_code: str, task_id: str) -> dict:
    run_dir = pathlib.Path("runs") / task_id
    run_dir.mkdir(parents=True, exist_ok=True)
    candidate = run_dir / "candidate.py"
    candidate.write_text(candidate_code)

    started = time.time()
    proc = subprocess.run(
        ["./sandbox_run.sh", str(candidate), "60"],
        capture_output=True, text=True,
    )
    elapsed = time.time() - started

    record = {
        "task_id": task_id,
        "code_sha256": hashlib.sha256(candidate_code.encode()).hexdigest(),
        "exit_code": proc.returncode,   # 124 = timeout killed it
        "elapsed_s": round(elapsed, 3),
        "stdout_tail": proc.stdout[-2000:],
        "stderr_tail": proc.stderr[-2000:],
    }
    (run_dir / "result.json").write_text(json.dumps(record, indent=2))
    return record

Two details I consider non-negotiable:

Hash and store the exact artifact you executed. If you can't point to the bytes that produced a result, you don't have a benchmark, you have an anecdote.
Grade on exit codes and test output, not on whether the code "looks right." The sandbox makes this safe; take advantage of it.

Where free hosted models and a free server fit

A practical cost note, because this loop is only useful if you can afford to run it often.

I use MonkeyCode for the generation side of this workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, two things are relevant here: it offers free access to hosted coding models, and a free server option that I use as the machine that runs the sandbox loop. That maps cleanly onto the architecture above:

Generation (calling the model, collecting candidate code) happens through the free model access — this is the part that would otherwise meter against a paid API on every iteration of a benchmark sweep.
Execution (the Docker sandbox) lives on the free server, which doubles as isolation: even in a worst-case container escape scenario, the blast radius is a disposable VM with no personal data on it, not my daily-driver laptop.

I deliberately keep the sandbox configuration identical between my laptop and that server, so a result is portable — the "reproducible" part of the harness only holds if the environment is pinned, which is why the image tag and flags live in the repo rather than in my memory.

If you want to try this shape of workflow, MonkeyCode's free tier is one way to get both halves (model access and a runner) without a billing account; any equivalent combination works just as well with the harness above.

A realistic failure this caught

During one comparison run, a model solving a "resize images in this folder" task generated a script that called subprocess.run(["convert", ...]) and, on failure, fell back to downloading a static ImageMagick binary from a hardcoded URL. On my host, pre-sandbox, that would have executed silently and "passed" the task. In the sandbox it failed loudly: --network none blocked the download, the exit code was non-zero, and the stderr_tail in the result JSON showed the attempted fetch.

Two lessons from that one log line: the model's answer was incorrect for my environment in a way a human skim would likely miss, and the correct grade for it was "fail," which only a sandboxed run could assign safely. This is the boundary problem from the agent-security discussion, at the smallest possible scale — and it's exactly where you want to catch it.

Limitations, honestly stated

Not a hostile-code boundary. As noted: shared kernel. Treat this as protection against buggy/sloppy/generated code, not against exploits.
--network none changes what you can test. Anything requiring package installation or live APIs needs a different design: a pre-baked image with dependencies installed, or an explicit allowlist proxy. That's more setup, and it reintroduces trust decisions you should make consciously.
Resource caps distort benchmarks. If you're comparing models on performance-sensitive tasks, the 1-CPU/512MB ceiling is part of your measurement environment. That's fine — arguably good, since it's realistic for cheap CI — but report it.
Free tiers are free tiers. Availability, quotas, and performance of free hosted models and free servers can change; design the harness so the provider is swappable, because eventually you'll want to swap it.

Who should not use this approach

Anyone evaluating autonomous multi-step agents that need to install tools, browse, or maintain state across steps — a single-shot sandbox won't model that, and you'll need a heavier environment (dedicated VMs, snapshot/restore).
Anyone whose threat model includes deliberately malicious inputs — step up to gVisor, Firecracker, or separate hardware.
Anyone who just wants a quick answer to one coding question — this is harness infrastructure; the payoff only appears when you're running evaluations repeatedly.

Closing

The more code generation you let into your workflow, the more your execution environment becomes the real security boundary. A ~15-line Docker wrapper, pinned and version-controlled, turned my model-comparison harness from "hope the output is benign" into "the output is graded, hashed, and contained." If you're benchmarking or iterating against coding models — free hosted ones included — build the sandbox first. The benchmark numbers you get afterward will be both safer and more believable.

A Reproducible Harness for Comparing Free Hosted Coding Models Against Your Local Setup

Jordan Li — Wed, 05 Aug 2026 06:59:08 +0000

Local AI workspaces are having a moment, and so is the counter-argument: why maintain hardware at all when hosted models keep getting cheaper or free? Both takes are usually supported by vibes. This article is about replacing vibes with a small, repeatable harness you can run in an afternoon, so the "local vs. free hosted" decision for your coding workflow rests on numbers you generated yourself.

The harness below measures three things that actually matter for day-to-day coding assistance: time to first token, total task latency, and whether the output passes a mechanical check (compiles, runs, or matches a pattern). It works against any OpenAI-compatible chat endpoint, which covers most local servers (llama.cpp, Ollama, vLLM) and most hosted providers.

The task suite: small, fixed, and honest

Benchmarks fail when tasks are fuzzy. Pick 6–10 tasks that mirror your real work and freeze them. Example suite:

ID	Task type	Mechanical check
T1	Write a URL slugify function	Unit test passes
T2	Fix an off-by-one bug in a loop	Unit test passes
T3	Explain a 40-line function	Contains 3 required keywords
T4	Convert callback code to async/await	`node --check` passes
T5	Write a SQL query with a JOIN	Runs against SQLite fixture
T6	Refactor for early returns	Lint rule count decreases

Freeze the prompts verbatim. If you edit prompts between runs, you are benchmarking your prompt drift, not the models.

The runner

This is the core artifact. Save as bench.mjs (Node 18+, no dependencies):

// bench.mjs — compare OpenAI-compatible endpoints on a fixed task suite
// Usage: ENDPOINT=http://localhost:11434/v1 MODEL=qwen2.5-coder node bench.mjs
import { execFileSync } from 'node:child_process';
import { readFileSync, writeFileSync } from 'node:fs';

const ENDPOINT = process.env.ENDPOINT ?? 'http://localhost:11434/v1';
const MODEL = process.env.MODEL ?? 'local-model';
const API_KEY = process.env.API_KEY ?? 'not-needed';

const tasks = JSON.parse(readFileSync('./tasks.json', 'utf8'));
const results = [];

for (const task of tasks) {
  const start = performance.now();
  let firstTokenAt = null;
  let output = '';

  const res = await fetch(`${ENDPOINT}/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${API_KEY}`,
    },
    body: JSON.stringify({
      model: MODEL,
      stream: true,
      temperature: 0,
      messages: [{ role: 'user', content: task.prompt }],
    }),
  });

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const chunk = decoder.decode(value);
    for (const line of chunk.split('\n')) {
      if (!line.startsWith('data: ') || line.includes('[DONE]')) continue;
      const delta = JSON.parse(line.slice(6)).choices?.[0]?.delta?.content;
      if (delta) {
        if (firstTokenAt === null) firstTokenAt = performance.now();
        output += delta;
      }
    }
  }

  const end = performance.now();
  const passed = runCheck(task.check, output);

  results.push({
    id: task.id,
    model: MODEL,
    ttft_ms: Math.round(firstTokenAt - start),
    total_ms: Math.round(end - start),
    passed,
  });
}

writeFileSync(`results-${MODEL.replace(/\W+/g, '_')}.json`,
  JSON.stringify(results, null, 2));
console.table(results);

function runCheck(check, output) {
  if (check.type === 'contains') {
    return check.keywords.every((k) => output.toLowerCase().includes(k));
  }
  if (check.type === 'node_check') {
    writeFileSync('/tmp/_bench_out.mjs', extractCode(output));
    try {
      execFileSync('node', ['--check', '/tmp/_bench_out.mjs']);
      return true;
    } catch { return false; }
  }
  return false;
}

function extractCode(text) {
  const m = text.match(/```
{% endraw %}
(?:js|javascript)?\n([\s\S]*?)
{% raw %}
```/);
  return m ? m[1] : text;
}

And tasks.json:

[
  {
    "id": "T1",
    "prompt": "Write a JavaScript function slugify(str) that lowercases, trims, replaces non-alphanumeric runs with single hyphens, and strips leading/trailing hyphens. Return only code in a fenced block.",
    "check": { "type": "node_check" }
  },
  {
    "id": "T3",
    "prompt": "Explain what a debounce function does and when to use one. Must mention timers, trailing calls, and event handlers.",
    "check": { "type": "contains", "keywords": ["timer", "trailing", "event"] }
  }
]

Temperature is pinned to 0, prompts are fixed, and each run writes a per-model results file you can diff later. Run the suite at least 3 times per endpoint and take medians — single runs lie.

Where a free hosted tier fits the comparison

Local inference has real costs: GPU power draw, RAM pressure while your IDE is open, and the maintenance tax of keeping model builds current. A free hosted option is worth benchmarking on exactly the same suite, because for bursty individual use the economics are often backwards from what people assume.

MonkeyCode is one such option worth including as a row in your results table: it currently offers free model access and a free server option, so you can point the same runner at its OpenAI-compatible endpoint without standing up hardware. Because the harness is endpoint-agnostic, adding it is one environment variable change:

ENDPOINT=<monkeycode-endpoint> MODEL=<model-name> API_KEY=<key> node bench.mjs

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The benchmark code, methodology, and conclusions below are independent of that relationship — run the harness against any endpoints you like and trust your own output files over anything written here. Note that availability and pricing of free tiers change; verify current terms before depending on them, and don't bake any free tier into automation you can't afford to lose.

Reading the results: a decision matrix

After collecting medians, score each endpoint against your constraints:

Criterion	Local wins when...	Free hosted wins when...
Latency	TTFT < 300ms matters (interactive pairing)	You're on batch/async tasks anyway
Privacy	Code cannot leave the machine, full stop	Code is already in a cloud repo
Reliability	You need offline capability	Your uptime tolerance matches theirs
Quality pass rate	Local model passes ≥90% of your suite	Hosted passes materially more tasks
Cost shape	Sustained heavy daily use	Bursty, a-few-sessions-a-week use

The pass-rate row is the one people skip and shouldn't. A model that answers in 200ms but fails node --check half the time is slower in wall-clock terms once you count your own debugging.

Limitations, and who should skip this

Six tasks is not a benchmark in the academic sense. It measures your workflow's fit, nothing more. Don't publish the numbers as model rankings.
Mechanical checks undercount quality. node --check proves syntax, not correctness. Extend the suite with real unit tests before drawing strong conclusions.
Free tiers move. Rate limits, model availability, and terms change without notice. Re-run quarterly and keep a fallback endpoint configured.
Skip this entirely if you're under a compliance regime that dictates where inference happens — that decision is made for you — or if your workload is already saturating a paid API with good results. The harness exists for the undecided middle.

Closing

The local-vs-hosted debate usually ends with whoever has the stronger anecdote winning. A 60-line runner and a frozen task suite turn it into a question you can re-answer whenever hardware, models, or free-tier terms shift. If you try the harness, the most useful thing you can share in the comments isn't your winner — it's the tasks you added, since those are what make the suite transferable.