DEV Community: Charlie Zhu

The $0 Memory Layer: A JSON File That Makes a Free LLM Remember

Charlie Zhu — Mon, 31 Aug 2026 11:58:43 +0000

Why does every chatbot forget what you told it yesterday? Because you never gave it a notebook. I built a personal assistant on MonkeyCode that keeps a persistent memory in a JSON file, and it runs on the platform's free models and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you are tired of re-explaining your project to an AI every single session, this is the cheapest fix I have found.

The pattern solves a real pain: context windows are finite, but your requirements grow. Instead of stuffing thousands of lines into a prompt, you save distilled notes and load only relevant ones on demand. That saves tokens, which matters when you are on a generous but still limited allowance. MonkeyCode currently provides a free tier with 10 million tokens for model calls and one free server instance, so the whole experiment costs zero dollars until you decide to scale.

Here is the core class. It treats memory as a simple list of notes with tags and a score, then persists everything to a local file.

# notebook.py
import json, os, time

class Notebook:
    def __init__(self, path="memory.json"):
        self.path = path
        self.data = self._load()

    def _load(self):
        if os.path.exists(self.path):
            with open(self.path) as f:
                return json.load(f)
        return {"notes": []}

    def _save(self):
        with open(self.path, "w") as f:
            json.dump(self.data, f, indent=2)

    def add_note(self, text, tags=None, score=1.0):
        self.data["notes"].append({
            "text": text,
            "tags": tags or [],
            "score": score,
            "created": time.time()
        })
        self._save()

    def query(self, term, top_k=3):
        results = []
        for note in self.data["notes"]:
            if term.lower() in note["text"].lower() or term in note["tags"]:
                results.append(note)
        return sorted(results, key=lambda n: n["score"], reverse=True)[:top_k]

That gives you the storage primitive, but where does the LLM come in? In my setup, every time I read an article or finish a debugging session, I send a one-sentence summary to the model and ask it to extract three tags and a score. The model returns JSON, I feed that straight into add_note. The free models handle this perfectly because the task is short and the output is structured.

# usage example with a mocked client - swap with MonkeyCode API later
def remember_with_llm(client, raw_text):
    prompt = f"""Extract a concise note (max 15 words), tags, and importance score from this text.
Return JSON only: {{"text": "...", "tags": ["..."], "score": 0.0}}
Text: {raw_text}"""
    response = client.generate(prompt)
    return json.loads(response)

The mock returns whatever you feed it, but the real client hits the free model endpoint described in the MonkeyCode README. I do not repeat the exact URL here because endpoints are versioned; the README updates faster than any blog post. The important lesson is that the interface stays identical, so switching from a fake to a real model changes one line.

How do you decide between a JSON file, SQLite, and a vector database? I made a quick decision table for you.

Approach	Best for	Costs	Scaling ceiling
JSON file	Single user, simple queries	Free, zero setup	seconds per query
SQLite	Small team, relational filters	Free, more APIs	minutes per query
Vector DB	Semantic search across long histories	More tokens for embedding, infra	hours or days

For a personal assistant that answers questions about what you learned last week, a JSON file is enough. For a team trying to recall decisions from a shared knowledge base, SQLite wins. For a production app with thousands of users and fuzzy queries, go vector.

The whole thing deploys to the free server with a tiny Flask wrapper. I set an environment variable for the memory path and expose two routes: POST /remember and GET /ask. The free server sleeps after idle time, but a simple health check wakes it up. This is not a production architecture; it is a way to test an idea before you commit to paid infrastructure.

Who should skip this? Anyone building a multi-tenant product, because file locks will corrupt memory under concurrent writes. Anyone who needs semantic similarity beyond simple tag matching, because my keyword scan misses synonyms. And anyone with strict data residency requirements, because the free server may run in a region you do not control.

Still, for a solo developer learning LLMOps, a JSON file is the perfect starter. You get persistence, you get a cheap way to experiment, and you burn almost none of your token budget. Run the code, swap in your MonkeyCode key, and let the free models do the summarization. After one week you will have a list of notes that actually feels like a brain.

The Context Budget: A 90-Minute Workshop for AI Code Review

Charlie Zhu — Sun, 30 Aug 2026 11:01:55 +0000

Last Tuesday, a senior engineer watched their AI assistant produce a fix that imported async_timeout in a Python codebase that had never heard of that package. The model was confident. It even left a comment explaining the import's timeout behavior. The engineer checked the repo. The file didn't exist. The AI had "remembered" it from a different project or, more likely, fabricated it from a training pattern.

The problem wasn't the model. It was context management. The assistant had been given the entire repository in its prompt: dozens of files, hundreds of functions, and a ten-thousand-line token budget. In that sea, one import statement looked like every other import statement. The model did what probabilistic models do. It produced the most statistically plausible token sequence, truth value be damned.

This is the setup for a 90-minute workshop that teaches a simple discipline called context budgeting. Participants build a small reviewer that sends only the files relevant to a change, then measure how much more accurate that reviewer becomes. The workshop uses MonkeyCode's free tier, which currently includes 10 million tokens and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server is what makes the workshop viable for people without a Linux box under their desk; every command runs in the same remote environment that also powers MonkeyCode's free models. No GPU, no local inference, no six-hour CUDA installation.

The first ten minutes are setup. Each participant clones a sample repository containing one deliberately planted bug: a function called retry_fetch that imports async_timeout, a module that does not exist in the repo. The repo also contains three candidate files that look plausible, each with its own unrelated imports. The workshop provides a small CLI skeleton with a token counter, a file picker, and a wrapper function named call_reviewer that sends a prompt to a model endpoint. Participants run the CLI against the sample directory. The commands are simple: git clone, python review.py --mode baseline, and later python review.py --mode budgeted. That is the entire setup.

The next fifteen minutes are dedicated to the core concept. The token counter in review.py uses a rough heuristic. In practice, many teams use exact tokenizers, but a character-based estimate is enough for this workshop. The code is short.

def tokens(text):
    return len(text) // 4

The point is not precision. The point is that every file appended to the prompt costs tokens, and the cost is roughly linear. Ten files at two thousand tokens each is twenty thousand tokens. Most free tiers will not blink at that number, but the model's attention blinks. It cannot attend to everything. When twenty thousand tokens are pasted in, the model's effective focus spreads thin, and the import that matters gets the same weight as the import that does not.

Exercise one takes twenty minutes. Participants run the baseline review. The CLI reads the entire repository, concatenates every file into a single prompt, and asks the model to find the bug. The output is predictable. The model identifies the missing async_timeout import, but it also flags two other files for issues that do not exist. It reports a memory leak in a file that already calls gc.collect(). It suggests a rename for a function that is only used in one place. These are false positives born of broad context.

Then the same review runs with a surgical context. The file picker in the CLI looks at the changed file, extracts its import lines, and selects only those files whose basenames match those imports. It also always includes the project's README.md as a minimal anchor. The prompt is now three files instead of ten. The model finds the missing import and nothing else. It takes half the time, and the token count drops to less than a third. The false positives disappear.

Exercise two takes twenty-five minutes. Participants replace the file picker with their own heuristic. The workshop provides a stub that currently returns the changed file and the README. The task is to extend it to parse from x import y statements and map them to matching local files. The sample contains a hidden wrinkle: the missing import is mentioned in old_legacy.py as a comment, not an actual import, and a naive picker might include that file. That is intentional. The exercise forces participants to inspect the imported symbol, not just the string, and to skip files where the symbol appears only in comments. By the end, the picker chooses one file: the one that defines retry_fetch and references async_timeout in a broken way.

The worked example takes fifteen minutes. Participants run the final reviewer on the fixed prompt and compare it with the baseline output. The results are stark. The baseline hallucinated two extra changes and one possible security issue. The budgeted reviewer returned a single line suggestion: remove the bogus import and use a retry helper from the standard library if available. That second suggestion is also questionable, but it is the kind of questionable you can verify quickly, because the prompt is small enough to read end to end. That is the real payoff. When context is small, every model output is auditable.

The last five minutes cover limitations. Context budgeting is not a panacea. It fails when the relevant file is not import-referenced, or when the codebase relies on dynamic dispatch and service locators where any file could be relevant. It also fails in monorepos where the changed line's dependencies are spread across hundreds of packages. For those cases, a proper index and retrieval system is needed, not a heuristic picker. There is a second limitation: the 10-million-token free tier is real but not unlimited. A class of thirty running multiple exercises will spend a meaningful portion of that allowance. It is still less than what a local run would cost in electricity, but it is not zero.

Who should not use this approach? Teams that work with protected health data or proprietary code that cannot leave their network should not send code to any external API, free or paid. Teams that need deterministic token accounting should replace the / 4 heuristic with a real tokenizer. Teams already using a retrieval-augmented review pipeline will see no benefit.

The workshop ends with a single assignment. Run the final reviewer on a small pull request from a real repo. Then cut the prompt to only the changed file and its imports. Measure the false positive rate for both runs. That number, more than any demo, will sell the idea. A bigger model is not required. A smaller context is. The free tier from MonkeyCode is enough to prove it, and the free server means the proof can be held from any machine with a browser.

Your AI Just Imported Something That Does Not Exist — Catch It in 30 Minutes

Charlie Zhu — Sat, 29 Aug 2026 09:56:15 +0000

Two weeks ago, a staging deployment crashed with an error that made no sense. The code called pandas.compat.numpy.func and expected a utility that has not existed since version 0.25. The AI wrote the call because it had seen a similar pattern in an old tutorial, and nobody questioned it. We fixed the line in seconds, but the real problem took longer to admit: every AI-generated file was a rumor mill of plausible imports.

AI coding assistants are brilliant at producing code that looks right, but they do not install a package before suggesting its API. They sample from patterns in training data, and those patterns include outdated, internal, or entirely invented functions. The classic debugging tools do not help because nothing is wrong until a code path actually executes the phantom call. Static type checkers only catch errors when they know the library, which for long-tail dependencies they rarely do.

Instead of waiting for runtime, we can ask a language model directly whether each imported API exists, and treat the answer as a first-pass filter. The query is short, the context is tiny, and the batch fits inside a free tier easily. For this experiment I used MonkeyCode's free models for the questions and a free server to schedule the recheck, because the whole workflow costs zero dollars and zero committed compute. Free tiers change, so verify the current limits, but the pattern is what matters. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Oracle in One Screen

The script is ninety lines and reads a file, extracts every import and attribute reference, then asks the model to label each one as 'exists', 'missing', or 'unknown'. The prompt forces the model to commit: it must answer with one of those words and a one-line reason. This prevents the evasive 'I cannot be sure' responses that make automation impossible.

Here is the core prompt template:

System: You are an API oracle. You know Python libraries up to your training cutoff. For each symbol, reply EXIST, MISSING, or UNKNOWN. Then one short proof sentence. Do not hedge.

Each request carries the symbol, the package, and the version we are using. The version matters more than people expect, because many 'AI hallucinations' are actually correct APIs from older releases.

The Code

Here is the full script, which you can save as api_oracle.py.

#!/usr/bin/env python3
"""api_oracle.py - check every imported API against a model's knowledge."""
import ast
import json
import os
import sys
import time
from pathlib import Path

import requests

LLM_URL = os.environ.get('LLM_URL')
LLM_KEY = os.environ.get('LLM_KEY')
MODEL = os.environ.get('LLM_MODEL')

LABELS = ('EXIST', 'MISSING', 'UNKNOWN')

PROMPT = '''You are an API oracle. You know Python libraries up to your training cutoff.
For each symbol, reply with one of {labels}, then a short proof sentence.
Do not hedge. Only answer from your knowledge.

Package: {{package}}
Version: {{version}}
Symbol: {{symbol}}
'''.format(labels='/'.join(LABELS))

def extract_symbols(path):
    tree = ast.parse(Path(path).read_text(encoding='utf-8'))
    symbols = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                symbols.append(('import', alias.name, None))
        elif isinstance(node, ast.ImportFrom):
            for alias in node.names:
                symbols.append(('from', node.module, alias.name))
        elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            if isinstance(node.func.value, ast.Name):
                symbols.append(('attr', node.func.value.id, node.func.attr))
    return symbols

def ask_model(package, version, symbol):
    response = requests.post(
        LLM_URL,
        headers={'Authorization': f'Bearer {LLM_KEY}'},
        json={
            'model': MODEL,
            'messages': [
                {'role': 'system', 'content': PROMPT},
                {'role': 'user', 'content': json.dumps({
                    'package': package,
                    'version': version,
                    'symbol': symbol,
                })}
            ],
            'temperature': 0.0,
        },
        timeout=30,
    )
    response.raise_for_status()
    text = response.json()['choices'][0]['message']['content'].strip()
    label = next((lab for lab in LABELS if text.startswith(lab)), 'UNKNOWN')
    return label, text

def main():
    if len(sys.argv) != 2:
        print('Usage: api_oracle.py <source.py>')
        sys.exit(2)
    symbols = extract_symbols(sys.argv[1])
    if not symbols:
        print('No symbols found.')
        sys.exit(0)
    report = []
    for kind, package, attr in symbols:
        symbol = attr or package
        label, proof = ask_model(package, 'installed', symbol)
        report.append({'kind': kind, 'package': package, 'symbol': symbol, 'label': label, 'proof': proof})
        if attr:
            print(f'{label:>8} {package}.{attr}')
        else:
            print(f'{label:>8} {package}')
        time.sleep(0.3)

    Path('oracle-report.json').write_text(json.dumps(report, indent=2))
    missing = [r for r in report if r['label'] == 'MISSING']
    print('Missing:', len(missing))
    if missing:
        sys.exit(1)

if __name__ == '__main__':
    main()

Run it with environment variables pointing at your LLM endpoint.

export LLM_URL="https://api.monkeycode.example/v1/chat/completions"
export LLM_KEY="your-key"
export LLM_MODEL="some-free-model-name"
python api_oracle.py src/main.py

The script is intentionally crude. It does not resolve relative imports or understand aliases perfectly; it is a tripwire, not a compiler. The important part is that it forces the model to take a position, which makes the output actionable.

A Sample Run

Consider a file with these four imports: import pandas, from sklearn.metrics import confusion_matrix, import internal_tools, and from legacy_utils import transform. A run might produce this output.

   EXIST import pandas
   EXIST sklearn.metrics.confusion_matrix
UNKNOWN import internal_tools
MISSING legacy_utils.transform

The last line is the one that deserves a human. legacy_utils may be an internal package that the model simply never saw, or it may be a real library from before the training cutoff. Either way, a one-second check is cheaper than a 3 a.m. page.

How to Read the Verdicts

Label	Meaning	Action
EXIST	The model has seen this API and asserts it exists	Keep, but verify with one real import test
MISSING	The model cannot find it and says so	Investigate manually before merging
UNKNOWN	The model refuses to commit	Add a unit test that exercises the symbol

That table is the real deliverable. Without it, the output is just a wall of text. With it, a junior dev can triage AI-generated code in seconds.

Free Server as a Nightly Gate

Rather than running this ad hoc, I put the oracle on a free server plan and set a cron job to run it over the newest pull request each morning. The free server is enough for a single scheduled job that takes under a minute, and the report lands in a file for the team to read.

0 8 * * * cd /srv/oracle && export LLM_URL=... && export LLM_KEY=... && export LLM_MODEL=... && python api_oracle.py $(ls -t /tmp/newest*.py | head -1) >> oracle.log 2>&1

The schedule turns a clever trick into a habit. The team no longer asks whether the AI wrote real code; they ask where the oracle failed.

Limitations

First, the model is frozen at its training cutoff. A library released last month will be mislabeled as MISSING, which produces false alarms. Version pinning helps only if your model knows that version. Second, internal packages simply do not exist in training data; the oracle will say UNKNOWN for every one of them. Third, the extraction is shallow, so import pandas as pd is treated as the symbol pandas, not every pd. usage. The script catches the import error case, not all attribute misuse.

Also, the model itself can hallucinate even when told to hedge. That is why the prompt demands a proof sentence, which a human can skim. Run this on a small sample before trusting it on a large one.

Who Should Skip This

Teams that already own a full dependency scanner with a live package index should not switch. Teams that only use a few well-known libraries will waste time on false alarms. And anyone who wants a guarantee should remember that this oracle is a heuristic, not a proof.

If the risk is worth it, the next step is easy: write a quick script that collects all imports from your current service and feed them to the free models. You will likely find one ghost API, which is exactly the point. When you do, you can thank me later by deleting the line that caused the crash.

Green Is Not Enough: A 60-Minute AI Fix Verification Workshop

Charlie Zhu — Fri, 28 Aug 2026 03:27:40 +0000

At 2 a.m. a test turned red in CI. A developer pasted the failure into a chat window, received a five-line patch, and merged it once the same test went green. Forty minutes later the next pipeline job failed, because the patch had quietly deleted the assertion that actually mattered. The test was green. The system was not.

This article is a one-hour workshop outline for a different habit: treat every AI-generated fix as a hypothesis, then run it through a three-pass ritual before it reaches a shared branch. Participants leave with a script they can rerun, a trap they can teach, and a loop that runs on free compute. The workshop uses free model access from MonkeyCode, an open-source project, and its free server option for the final exercise. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The three-pass ritual

The core idea is borrowed from test-driven development. A fix is not a patch; it is a claim that a failing test becomes green without breaking anything else. The claim deserves three passes.

Pass one makes the failure reproducible. Participants run the failing suite by hand and write down what the failure actually says; copying a traceback is not the same as reading it. Pass two asks a model for a minimal diff, with one hard constraint: the diff must not touch the test file. Pass three applies the diff in a throwaway copy, reruns the suite, and then a human reads the diff aloud. The last step is the one most teams skip, and it is the one that catches the 2 a.m. story.

The artifact

The workshop centers on a small script, around sixty lines, that encodes the ritual. One config, one prompt, one verdict. It is not a review tool; it is a decision gate that prints a JSON line so a team can collect evidence.

# verify_fix.py — a three-pass gate for AI-generated fixes
import json
import os
import subprocess
import sys
import time
import urllib.request


def call_model(prompt: str) -> str:
    base_url = os.environ["MODEL_URL"]
    api_key = os.environ["MODEL_KEY"]
    model = os.environ["MODEL_NAME"]
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,
    }
    request = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + api_key,
        },
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        body = json.load(response)
    return body["choices"][0]["message"]["content"]


def run(command: str) -> tuple[int, str]:
    process = subprocess.run(command, shell=True, capture_output=True, text=True)
    return process.returncode, process.stdout + process.stderr


def main() -> None:
    workdir = sys.argv[1]
    test_command = sys.argv[2]

    red, _ = run(f"cd {workdir} && {test_command}")
    if red == 0:
        print(json.dumps({"phase": "skipped", "reason": "already green"}))
        return

    with open("failure.txt", encoding="utf-8") as handle:
        failure = handle.read()
    with open("task.txt", encoding="utf-8") as handle:
        task = handle.read()

    prompt = (
        "A test fails in a Python repository.\n"
        f"FAILURE OUTPUT:\n{failure}\n"
        f"TASK DESCRIPTION:\n{task}\n"
        "Return only a unified diff that fixes the code.\n"
        "Do not modify the test file and do not delete assertions."
    )
    started = time.time()
    patch = call_model(prompt)

    if "test_" in patch:
        print(json.dumps({"phase": "rejected", "reason": "patch touches the test file"}))
        return

    with open("candidate.patch", "w", encoding="utf-8") as handle:
        handle.write(patch)

    ok, _ = run(f"cd {workdir} && git apply --check candidate.patch")
    if ok != 0:
        print(json.dumps({"phase": "unparsable", "elapsed_s": round(time.time() - started, 2)}))
        return

    run(f"cd {workdir} && git apply candidate.patch")
    green, output = run(f"cd {workdir} && {test_command}")
    print(json.dumps({
        "phase": "verified" if green == 0 else "rejected",
        "elapsed_s": round(time.time() - started, 2),
        "tail": output[-400:],
    }))


if __name__ == "__main__":
    main()

The script reads three environment variables: MODEL_URL and MODEL_KEY, for any OpenAI-compatible endpoint, and MODEL_NAME, for the model identifier. It expects failure.txt with captured test output and task.txt with a one-sentence description of the intended behavior. It runs a command in a clean checkout; if the suite is already green it prints skipped and stops. A red suite produces a prompt with a strict instruction to return only a diff. The script refuses patches that touch the test file, rejects patches that do not apply cleanly, and records the elapsed time and the tail of the final run.

The worked example

The example is deliberately trivial, because the point is the ceremony, not the difficulty. A file called price.py contains a single function.

def total_with_tax(price: float, tax: float) -> float:
    return price * (1 + tax)

A test file asserts the total with tax rounds to cents.

def test_total_with_tax_rounds_to_cents():
    assert total_with_tax(19.99, 0.0875) == 21.74

The test fails, because binary floating-point numbers rarely produce exact cents. The facilitator captures the failure, writes task.txt, and lets the group run the script. The honest fix wraps the result in round.

def total_with_tax(price: float, tax: float) -> float:
    return round(price * (1 + tax), 2)

The script prints verified. Then comes the trap. A second candidate patch also passes the single test, but it special-cases the exact price and rounds everything else.

def total_with_tax(price: float, tax: float) -> float:
    if price == 19.99:
        return 21.74
    return round(price * (1 + tax), 2)

The script prints verified again, because the test only covers one input. The lesson lands in the gap between those two runs: an exit code of zero proves consistency with one test, not correctness. The facilitator asks the group to read each patch aloud, and the second patch fails the audit instantly, because a hard-coded value is not a rule.

The sixty minutes

The first ten minutes belong to reproduction. Participants clone a tiny repository, run the failing test, and write failure.txt in their own words. The next fifteen belong to the script; the group runs verify_fix.py against the example and records the JSON verdict. The next twenty belong to the trap round, with three candidate patches, each with a different flaw. The diff audit separates them; the script alone cannot. The final fifteen belong to persistence: the same loop moves to the free server option, so a scheduled run can watch a branch overnight instead of depending on a laptop. Hosting details drift quickly, so the facilitator prints them live rather than hard-coding them in the handout.

Limits and the wrong audience

A single green run is a sample size of one. The script does not catch a fix that passes a weak test and breaks a strong one, and it cannot judge style, security, or intent. Free-tier capacity has ceilings; the script logs tokens and elapsed time precisely, so a team can notice when a task is too large for the free option. Security-sensitive patches need a sandbox and a human with authority, not a chat-completion reply.

Teams without a test suite should skip this workshop, because the ritual has nothing to hold on to. Teams whose CI is already red for unrelated reasons will only certify noise. And individuals who already read every diff aloud will find the drill redundant, which is a good problem to have.

The artifact runs against any OpenAI-compatible endpoint, including the same free model access from MonkeyCode mentioned earlier; the only setup is the three environment variables. The natural next artifact is a bot that runs the same three passes on every pull request. That workshop starts the same way, with a test that fails first.

A 90-Minute Workshop for the New Reviewer Role

Charlie Zhu — Wed, 26 Aug 2026 11:05:56 +0000

Monday, 9:14 a.m. A pull request lands with the label generated. Four hundred lines, a new endpoint, and a test suite that passes. The reviewer opens it, reads the first function, and feels the familiar squeeze: the code is plausible, the tests are green, and nothing looks wrong. That is the moment the job changed.

For years, the hard part of software work was writing code. The bottleneck has moved. AI assistants now produce a large share of new code on many teams, and the human role has shifted to review: reading, questioning, and verifying code that arrived faster than anyone can inspect it. The strange part is that almost nobody trains for this. Teams run onboarding for new languages and frameworks, but not for the act of reviewing generated code.

A workshop cannot replace experience. It can compress the first painful months into ninety minutes. This article lays out a repeatable session: four exercises, a worked example students can rerun, and a tiny lab that runs on free infrastructure.

The lab

The lab needs two things: a place to run tests and a way to generate sample pull requests. MonkeyCode's free model access and free server option cover both, which keeps the cost at zero for a small group. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The point of the workshop is not the tooling; the same exercises work with any model endpoint and any machine with Python.

MonkeyCode is an open-source project, and its free tier includes a 10-million-token allowance plus a free server instance. Quotas change, so check the project docs before planning a large session. For a team of ten, the allowance is more than enough.

The workshop repo contains three sample pull requests. Each is a single function with a planted defect. Nobody is told where the defects are.

The agenda

0:00–0:10 — Setup. Students clone the repo, install pytest, and confirm the lab server responds. The whole group works in the same preloaded environment, so there is no "works on my machine" drift.

0:10–0:30 — Exercise 1: Review by checklist. Students read the first pull request and mark a five-item checklist. Most people find zero defects on the first pass. That is the lesson: unaided review of generated code is mostly pattern matching, and patterns miss edge cases.

0:30–0:50 — Exercise 2: Write the failing test. Students stop guessing and start asserting. They write one test that should pass if the code is correct. The test fails. The review now has evidence instead of opinions.

0:50–1:10 — Exercise 3: Cross-grade the reviews. Pairs swap checklists and grade each other against the planted defect list. Review quality turns out to be measurable, not a matter of taste.

1:10–1:20 — Debrief. The facilitator reveals the defects and asks which checklist items caught them. Usually one item does all the work.

1:20–1:30 — Wrap-up. Students take the checklist back to real reviews. The session ends with a concrete artifact, not a vague resolution.

The worked example

The first sample pull request is a function that merges overlapping intervals:

# sample_prs/pr_01/merge_intervals.py
def merge_intervals(intervals):
    intervals.sort(key=lambda x: x[0])
    merged = []
    for start, end in intervals:
        if merged and start <= merged[-1][1]:
            merged[-1][1] = end
        else:
            merged.append([start, end])
    return merged

The code looks clean. The checklist makes it less comfortable:

[ ] Empty input, None, single element
[ ] Mutation of caller data
[ ] Contained intervals, e.g. (1,4) and (2,3)
[ ] Return type matches the documented contract
[ ] Error behavior on malformed input

Item two is the trap. The function sorts intervals in place, which mutates the caller's list. Item three catches the real bug: for [(1, 4), (2, 3)], the code returns [[1, 3]] instead of [[1, 4]], because it overwrites the end instead of taking the maximum. A generated test suite often misses both, because the happy path looks fine.

Students write the failing tests:

# tests/test_merge_intervals.py
from sample_prs.pr_01.merge_intervals import merge_intervals

def test_contained_interval_keeps_outer_end():
    assert merge_intervals([(1, 4), (2, 3)]) == [[1, 4]]

def test_input_list_is_not_mutated():
    data = [(2, 3), (1, 4)]
    merge_intervals(data)
    assert data == [(2, 3), (1, 4)]

And run them:

pytest tests/test_merge_intervals.py -q

Two failures, two discovered defects, about ten minutes of work. The fix is small:

def merge_intervals(intervals):
    ordered = sorted(intervals, key=lambda x: x[0])
    merged = []
    for start, end in ordered:
        if merged and start <= merged[-1][1]:
            merged[-1][1] = max(merged[-1][1], end)
        else:
            merged.append([start, end])
    return merged

The lesson is not that the model wrote bad code. The lesson is that the model wrote plausible code, and plausibility is exactly what a reviewer must distrust.

Who should not use this

Teams that do not review AI-generated code yet. If every pull request is still handwritten, the exercises will feel abstract. The format also assumes a facilitator who knows the planted defects; without one, the debrief turns into guesswork. And the free tier is sized for small groups — a hundred students hammering the same endpoint will hit rate limits, which is a valuable lesson in itself, but not the intended one.

The takeaway

The reviewer role is new, but it is trainable. Ninety minutes and a handful of planted bugs will not make anyone an expert; they will make the next generated pull request feel less like a magic trick and more like a puzzle. The workshop repo runs on a free server with free model access, and the checklist alone is worth taking back to the team.

From Log Sprawl to a Two-Sentence Morning Report

Charlie Zhu — Tue, 25 Aug 2026 08:43:28 +0000

The application server wrote 2.3 GB of logs last night. The team ignored them, because reading them would take three hours. Then a payment outage went unnoticed for forty minutes because the relevant error was buried under routine warnings.

That is the standard failure mode of logging. Volume grows faster than attention, so teams either ignore logs entirely or page on every keyword and then disable the pager. The middle path is a small summarizer that runs before anyone is awake.

This article builds one. The tool reads the last 24 hours of error and warning lines, sends them to a free model tier for grouping and explanation, and writes a short Markdown brief. A free server option hosts the cron job. The whole thing is about sixty lines of Python.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The design has three stages: extract, summarize, deliver. Extraction is deliberately crude. The script scans each log file for lines containing ERROR, WARN, Exception, or Traceback, and keeps the last 200 matches. That cap protects the prompt from overflowing and forces the model to focus on recent activity.

# nightly_log_brief.py
import os
import re
from datetime import datetime, timedelta
from pathlib import Path
from openai import OpenAI

LOG_DIR = Path('/var/log/myapp')
SINCE = datetime.now() - timedelta(hours=24)

def extract_candidates(log_path: Path) -> list[str]:
    candidates = []
    with log_path.open(errors='ignore') as fh:
        for line in fh:
            if any(level in line for level in ('ERROR', 'WARN', 'Exception', 'Traceback')):
                candidates.append(line.strip())
    return candidates[-200:]

A structured log parser would be nicer, but most projects do not have one. The regex-free substring check works on plain text, JSON lines, and even stack traces. It will miss a log line that says 'something went wrong' without a keyword, but it will catch the lines that matter for a morning triage.

The summarization step is one prompt. It asks the model to group the lines by root cause, give a one-line explanation per group, and suggest a next step. The temperature is low, and the instruction to avoid inventing details is explicit.

def summarize(candidates: list[str]) -> str:
    client = OpenAI(
        api_key=os.environ['MONKEYCODE_API_KEY'],
        base_url=os.environ.get('MONKEYCODE_BASE_URL'),
    )
    prompt = f'''Below are the last 200 error/warning lines from a 24-hour log window.
Group them by root cause. For each group, give a one-line explanation and a suggested next step.
Keep the whole answer under 200 words. Do not invent details not present in the lines.

Lines:
{chr(10).join(candidates)}'''
    resp = client.chat.completions.create(
        model=os.environ.get('MONKEYCODE_MODEL'),
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0.2,
    )
    return resp.choices[0].message.content

The prompt is the part you will tune. Adding a line like the service is called payment-worker helps the model use the right names. Removing the word 'WARN' from the extraction pattern reduces noise if warnings are too chatty. The script is meant to be edited, not worshipped.

The delivery stage writes the summary to a file that the team reads with coffee. A cron entry on the free server option runs the script at 6 a.m. every day.

0 6 * * * cd /opt/log-brief && python nightly_log_brief.py >> /var/log/log-brief.log 2>&1

To test the script without waiting for a cron run, point LOG_DIR at a sample directory and run it manually. The output file will appear in the current directory if you change the output path. For teams that prefer chat notifications, replace the file write with a webhook call; the rest of the script stays the same.

Here is the output from a controlled test with a 24-hour log sample. The model grouped 163 error lines into three root causes.

# Log brief for 2026-08-25T06:00:01

1. **Timeout errors in payment worker** (142 occurrences): The worker is hitting a 30s timeout on the upstream payment API. Suggested next step: check the API's status page and increase the timeout to 45s if the incident continues.
2. **Database connection pool exhaustion** (18 occurrences): The pool size is too low for the morning batch job. Suggested next step: raise `max_connections` from 10 to 20 and monitor.
3. **Single recurring stack trace in auth service** (3 occurrences): A null pointer in session renewal. Suggested next step: add a null check and deploy a patch.

The summary is not perfect. The model guessed that the timeout was caused by an upstream incident, which was plausible but unverified. That is the right failure mode: the tool flags, a human confirms, and the team spends five minutes instead of three hours.

The limitations are real. The model sees only the lines you feed it, so logs without context produce shallow summaries. The free tier is fine for a once-a-day job, but it is not a real-time alerting system. And if logs contain customer data, sending them to an external model requires redaction first. A simple redaction step could replace email addresses and IPs before the prompt is built.

Teams with a dedicated observability platform do not need this. Teams with real-time paging requirements should not use a nightly cron. And teams with zero error logs should fix their logging before adding a summarizer.

The lesson is that a small, narrow automation beats a grand platform that never ships. The free model and free server are enough for this job, and the job is worth having. If you have a log directory and a morning coffee, the script above is a good starting point.

The Issue Digest Bot: A MonkeyCode Free-Tier Case Study

Charlie Zhu — Mon, 24 Aug 2026 19:13:49 +0000

A maintainer of a small open-source library wakes up to forty notifications. Three are real bugs; the rest are duplicate questions, stale reports, and one pull request that is secretly an issue. The maintainer needs a triage summary before coffee, not after. The budget for tooling is exactly zero dollars.

That scenario is the background for this case study, and the current conversation about AI coding tools keeps circling one question: how much useful work can a free tier actually do? Instead of answering in the abstract, this article walks through one small project end to end. The goal is narrow: a bot that fetches open issues, summarizes them with a language model, and prints a short digest. The stack is deliberately boring — Python, the GitHub API, and MonkeyCode's free model access.

MonkeyCode is an open-source project that offers free model access and a free server option for small workloads. As of this writing, the free tier includes a 10 million token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact endpoint and model configuration are documented in the project README, and the script below reads them from environment variables so nothing is hard-coded.

The design decisions follow from the goal. Five issues per run keeps the prompt small and the response fast, and filtering out pull requests prevents the digest from mixing bugs with code reviews. Reading configuration from environment variables means the same script runs on a laptop and on the free server option without a single change. The whole project is two files: digest.py and test_digest.py.

# digest.py
import json
import os
import urllib.request

REPO = os.environ.get("GITHUB_REPO", "octocat/Hello-World")
TOKEN = os.environ.get("GITHUB_TOKEN", "")
MODEL_BASE = os.environ.get("MODEL_BASE", "")
MODEL_NAME = os.environ.get("MODEL_NAME", "")


def fetch_open_issues(repo, token):
    url = f"https://api.github.com/repos/{repo}/issues?state=open&per_page=5"
    request = urllib.request.Request(url)
    if token:
        request.add_header("Authorization", f"Bearer {token}")
    with urllib.request.urlopen(request, timeout=15) as response:
        return json.load(response)


def filter_issues(issues):
    return [issue for issue in issues if "pull_request" not in issue]


def summarize(issues, base, model):
    lines = "\n".join(f"- #{issue['number']}: {issue['title']}" for issue in issues)
    prompt = "Summarize these GitHub issues in three bullet points:\n\n" + lines
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,
    }
    request = urllib.request.Request(
        base.rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        data = json.load(response)
    return data["choices"][0]["message"]["content"]


def main():
    issues = fetch_open_issues(REPO, TOKEN)
    issues = filter_issues(issues)
    if not issues:
        print("No open issues.")
        return
    print(summarize(issues, MODEL_BASE, MODEL_NAME))


if __name__ == "__main__":
    main()

The first version of this script sent the entire issue body to the model. That was a mistake: issue bodies are long, noisy, and full of markdown that adds nothing to a three-bullet summary. The current version sends only the number and the title, so a digest of five issues fits in a few hundred tokens. That matters when the goal is to stay inside a free allowance for years rather than weeks.

The second file is the part that makes this project reproducible. Instead of calling a real model during tests, the test suite starts a fake model server on localhost. The server accepts any chat completion request and returns a canned response. This catches JSON parsing errors, wrong payload shapes, and timeout handling in seconds, without spending a single token.

# test_digest.py
import json
import threading
import unittest
from http.server import BaseHTTPRequestHandler, HTTPServer

from digest import filter_issues, summarize


class FakeModelHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        self.rfile.read(length)
        body = json.dumps(
            {"choices": [{"message": {"content": "summary from fake model"}}]}
        ).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


class DigestTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.server = HTTPServer(("127.0.0.1", 0), FakeModelHandler)
        cls.port = cls.server.server_address[1]
        threading.Thread(target=cls.server.serve_forever, daemon=True).start()

    @classmethod
    def tearDownClass(cls):
        cls.server.shutdown()
        cls.server.server_close()

    def test_filter_issues_removes_pull_requests(self):
        issues = [
            {"number": 1, "title": "Bug report"},
            {"number": 2, "title": "Add feature", "pull_request": {}},
        ]
        self.assertEqual(len(filter_issues(issues)), 1)

    def test_summarize_parses_model_response(self):
        issues = [{"number": 7, "title": "Login fails on Safari"}]
        base = f"http://127.0.0.1:{self.port}"
        result = summarize(issues, base, "fake-model")
        self.assertEqual(result, "summary from fake model")


if __name__ == "__main__":
    unittest.main()

Running the suite is one command:

python -m unittest test_digest.py -v

The fake server starts on a random port, serves the request, and shuts down. The test for the pull request filter uses a list with one real issue and one pull request, and asserts that only the issue survives. The test for the summarizer verifies that the response is parsed correctly. Both tests run offline.

A real run looks like this:

GITHUB_REPO=yourname/yourrepo python digest.py

The bot logs the remaining GitHub rate limit from the X-RateLimit-Remaining response header. Unauthenticated requests are rate limited, and the exact number changes over time, so reading the header is more reliable than hard-coding a value. For a personal bot that runs a few times a day, the limit is a non-issue. For anything with more than a handful of users, it would be the first constraint to hit.

The results of this case study are best described as quiet. The test suite runs in under a second and spends zero tokens. A real digest costs one chat completion per run, typically a few hundred tokens, and at that rate a 10 million token allowance is effectively infinite for this workload. The constraint that actually shapes the project is not the token budget; it is the discipline of keeping the prompt small and the tests offline.

Three lessons came out of this project. First, mock the model before touching the real endpoint; the fake server catches integration mistakes in milliseconds, and it makes the test suite safe to run in CI without an API key. Second, send titles, not bodies; the model does not need the full text of an issue to produce a useful summary, and every unnecessary token is a small tax on the allowance. Third, a free server option changes the nature of a project, because a script that runs on a laptop is a demo while the same script running on a timer is a tool. MonkeyCode's free server option is what moves this bot from the first category to the second.

This approach is not for everyone. A team that needs guaranteed latency, high-throughput batch processing, or strict data residency should pay for an SLA and run on infrastructure they control. A free tier is a constraint, and constraints only help when they match the workload. For a personal bot that summarizes five issues a few times a day, the constraint is invisible; for a production pipeline, it would be the first thing to break.

The real output of this project is not the digest. It is the discovery that a small, well-tested bot can run on free infrastructure for a long time. The token allowance is generous, but the discipline required to stay inside it is the actual skill. Anyone with a small repository and a zero-dollar budget can copy this pattern and run it today; the current free-tier details in the MonkeyCode README are the only thing that needs checking before starting.

The Case of the 3 a.m. Outage Nobody Saw

Charlie Zhu — Sat, 22 Aug 2026 14:28:07 +0000

A small server ran three small services. A status page, an RSS digest fed by cron, and an API that one mobile app depended on. Nothing critical, until a database filled its disk at 3 a.m. and all three went down together. Nobody noticed for nine hours. The postmortem was short and uncomfortable: no alerting, no history, no way to know when the silence started.

This is the case study of the fix. The constraint was real money — the services were cheap, so the monitoring had to be cheaper. The developer set a budget of zero dollars and one evening. The goal was a minimal uptime monitor: check three endpoints every sixty seconds, record every result in SQLite, and fire a webhook after two consecutive failures. No dashboard. No pager duty. No metrics pipeline.

The build used MonkeyCode, an open-source AI coding assistant. Disclosure: This article was prepared as part of MonkeyCode's product outreach. At the time of writing, the project's free tier includes a 10-million-token model allowance and a free server option, and both played a role in the workflow. The model access wrote the scaffold; the server option gave the monitor a home. Token numbers and server terms can change, so the repository README is the only source of truth.

The workflow had three passes. First, a plain-language spec was written before any code: inputs, outputs, failure rule, storage schema. Second, that spec went into a single prompt, and the assistant returned a working Python scaffold. Third, the scaffold was reviewed line by line and tightened. The spec did the heavy lifting; the model did the typing.

The core script stayed under a hundred lines.

# monitor.py — minimal uptime checker with SQLite history
import json
import sqlite3
import sys
import time
import urllib.request
from datetime import datetime, timezone

FAIL_THRESHOLD = 2
TIMEOUT = 5


def probe(url: str) -> tuple[bool, float]:
    started = time.monotonic()
    try:
        with urllib.request.urlopen(url, timeout=TIMEOUT) as resp:
            ok = 200 <= resp.status < 400
    except Exception:
        ok = False
    return ok, time.monotonic() - started


def main() -> int:
    with open("endpoints.json") as fh:
        endpoints = json.load(fh)["endpoints"]

    con = sqlite3.connect("status.db")
    con.execute(
        "CREATE TABLE IF NOT EXISTS checks ("
        "url TEXT, ts TEXT, ok INTEGER, latency REAL)"
    )
    con.execute(
        "CREATE TABLE IF NOT EXISTS alerts (url TEXT, ts TEXT)"
    )

    for ep in endpoints:
        ok, latency = probe(ep["url"])
        ts = datetime.now(timezone.utc).isoformat()
        con.execute(
            "INSERT INTO checks VALUES (?, ?, ?, ?)",
            (ep["url"], ts, int(ok), round(latency, 3)),
        )

        recent = con.execute(
            "SELECT ok FROM checks WHERE url=? ORDER BY ts DESC LIMIT ?",
            (ep["url"], FAIL_THRESHOLD),
        ).fetchall()
        if len(recent) == FAIL_THRESHOLD and all(row[0] == 0 for row in recent):
            con.execute("INSERT INTO alerts VALUES (?, ?)", (ep["url"], ts))
            webhook = ep.get("webhook")
            if webhook:
                payload = json.dumps(
                    {"text": f"DOWN: {ep['url']} at {ts}"}
                ).encode()
                req = urllib.request.Request(
                    webhook,
                    data=payload,
                    headers={"Content-Type": "application/json"},
                )
                urllib.request.urlopen(req, timeout=TIMEOUT)

    con.commit()
    con.close()
    return 0


if __name__ == "__main__":
    sys.exit(main())

The config file is the contract between the script and reality.

{
  "endpoints": [
    {
      "url": "https://status.example.com/health",
      "webhook": "https://hooks.example.com/notify"
    },
    {
      "url": "https://api.example.org/ping"
    },
    {
      "url": "https://blog.example.net/"
    }
  ]
}

Two tables hold the whole history. checks stores one row per probe with a timestamp, a boolean, and latency. alerts records every time the threshold trips. That is enough to answer the two questions that matter: is it down now, and when did it start. A log file would have been simpler, but SQLite made the failure window queryable, and that difference mattered later.

Deployment on the free server took one cron line.

* * * * * cd /srv/uptime && /usr/bin/python3 monitor.py >> monitor.log 2>&1

The developer did not trust the first green run. Verification came first. A local mock server stood in for the real endpoints, and the monitor pointed at it.

python3 -m http.server 8080 --bind 127.0.0.1 &
python3 monitor.py
sqlite3 status.db "SELECT url, ts, ok FROM checks ORDER BY ts DESC LIMIT 3;"

The first probe recorded ok=1. Then the mock was killed and the monitor ran twice more.

kill %1
python3 monitor.py
python3 monitor.py
sqlite3 status.db "SELECT * FROM alerts;"

The alert row appeared exactly as specified. Two consecutive failures, one webhook, zero surprises. The threshold logic was the part most worth testing, because a single failed probe is normal and two in a row is news.

The monitor ran for ten days. On day six, the RSS digest service failed during a routine deploy. The webhook fired at 02:14, the failure window showed up in the log, and the issue was fixed before breakfast. Nine hours of silence became nine minutes of noise. The latency column turned out to be a quiet early-warning signal: response time had crept from 120 ms to 900 ms over three days before the crash. The alert caught the hard failure; the history showed the slow climb that led to it.

The first lesson from the case: the model was not the bottleneck, the spec was. Every vague prompt produced vague code. Every prompt that named a schema, a threshold, and a webhook produced code that matched the prompt. The free model access handled a well-scoped project without drama, and the 10-million-token allowance covered the whole build with room to spare.

The second lesson: the free server is a fine home for a cron job and a bad place for a promise. The monitor is a single process with a single point of failure. If the box dies, the monitor dies with it. For a side project that is acceptable. For a customer-facing SLA it is not even close. The script also does not deduplicate alerts; a service that stays down fires every minute. That is acceptable noise for a hobby tool, and a good reason to add a cooldown before this pattern touches anything serious.

The approach is not for every team. Teams that need multi-region checks, on-call rotation, or real paging should look elsewhere. Anyone monitoring production revenue should look elsewhere too. The approach also assumes the monitored services live somewhere other than the monitoring box; a monitor that shares a disk with its targets misses the exact failure that started this story.

The whole project — script, config, cron line — fits in one directory and costs nothing to run. The quiet lesson is that constraint is not the enemy of quality. A tiny tool, a clear spec, and a free tier caught an outage that nine hours of silence had hidden. The same recipe is easy to repeat, and the free tier makes the first run cost nothing.

The Edge Where Free AI Code Breaks

Charlie Zhu — Fri, 21 Aug 2026 13:04:47 +0000

Somewhere in a codebase, a function died on an empty string. The tests were green. The review was clean. The function had been generated by a model, checked against three examples, and merged. Then a user submitted a form with a blank field, and the function raised an exception that no test had ever seen.

This is the boundary-condition gap. Example-based tests prove a function works for the inputs a human thought of. They do not prove it works for the inputs a model never considered. And models, left to their own devices, tend to write code that handles the happy path beautifully and the edge path not at all.

The gap is not specific to free models. Paid models have the same blind spot. But free models make the gap easier to reach. The cost of iterating is zero, so more generated code enters the codebase, and more boundary conditions arrive uninvited.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below uses MonkeyCode's free model access — which includes a 10-million-token allowance as of this writing — and its free server option. Token allowances and server terms change; the current details live in the project's documentation.

The fix is not more examples. The fix is a different kind of test. Property-based testing flips the question. Instead of asking "does this function return the right value for this input?", it asks "does this function preserve the right invariant for every input in this domain?"

Consider a concrete case. A model was asked to write a function that parses a date string and returns a weekday name. The model produced:

from datetime import datetime

def weekday(date_str: str) -> str:
    return datetime.strptime(date_str, "%Y-%m-%d").strftime("%A")

Three example tests passed. "2026-08-21" returned "Friday". "2026-01-01" returned "Thursday". "2024-02-29" returned "Thursday". The function looked correct.

A property test asks a different question. For any valid date string in the supported range, the result should be one of seven weekday names. For any invalid input, the function should raise a clear error rather than crash with a stack trace. The Hypothesis library makes this trivial:

from hypothesis import given, strategies as st
from datetime import date

@given(st.dates())
def test_weekday_is_always_a_weekday(d: date):
    result = weekday(d.isoformat())
    assert result in {"Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"}

@given(st.text())
def test_invalid_inputs_raise_cleanly(s: str):
    try:
        weekday(s)
    except ValueError:
        pass  # expected
    except Exception as exc:
        raise AssertionError(f"unexpected exception type: {type(exc).__name__}") from exc

The first property passed. The second property failed within seconds. The model's function raised a TypeError for None, an AttributeError for empty strings, and a ValueError with a raw strptime message for malformed input. None of those failures appeared in the example tests, because nobody had written an example for "" or None.

The failure mode is instructive. The model had learned the shape of a date parser — import, call, format — without learning the contract. A human who writes a date parser from scratch usually thinks about the contract because the runtime forces the conversation. A model that generates code from patterns skips that conversation. The result is code that looks right and is right, except at the edges.

Property-based testing does not fix the model. It fixes the pipeline. The property tests become a permanent gate. Every time the model regenerates the function, the properties run. If the new version violates an invariant, the gate fails, and the human sees exactly which property broke. This turns "trust the model" into "verify the model against the contract", which is a much better position to be in.

The same idea scales beyond single functions. A small project that generates several utility functions can define a property file for each module. The properties are the real specification; the model's output is a candidate implementation. This separation is the durable pattern.

The workflow fits neatly on free compute. Generation happens through MonkeyCode's free model access. Property tests run in a plain Python environment. The whole thing can live on the free server option as a scheduled job. The resource footprint is tiny — Hypothesis generates thousands of cases, but each case is a single function call, so a few seconds of CPU covers a full property suite.

Not every project needs property tests. A script that runs once and is thrown away does not need a contract. A library that other code depends on, or a function that handles user input, absolutely does. The cost of writing properties is small. The cost of a boundary-condition crash in production is large. The asymmetry is the argument.

The empty string is patient. It waits in form fields, in config files, in API responses. Example tests will not find it, because example tests only look where the human pointed. Property tests sweep the whole floor. For generated code, that sweep is not a luxury. It is the difference between code that looks right and code that is right.

If you want to try this pattern, start with one function, write three properties, and run Hypothesis against the model's output. The properties take ten minutes to write, and they will outlive the model, the token allowance, and the server. That is the point.

Comparing AI Coding Models Without Burning Budget: A Reproducible Harness on Free Compute

Charlie Zhu — Thu, 13 Aug 2026 03:26:43 +0000

Every few weeks a new coding model drops, the timeline fills with cherry-picked screenshots, and I'm left with the same question: is it actually better for my repo, or just better at demo prompts?

In my last article I described a sandbox loop for safely executing AI-generated code: generate, isolate, assert. This post is the natural next step — using that same loop as a comparison harness so I can evaluate candidate models on my own tasks instead of trusting leaderboard vibes. The trick that makes this sustainable is doing it on free model access and free server time, so the cost of curiosity is zero and I can rerun the whole thing whenever a new model appears.

Why leaderboards don't answer my question

Public benchmarks measure performance on tasks I don't have. My actual workload is boring and specific:

Small refactors inside a legacy Flask app with inconsistent naming
Writing pytest fixtures for functions with messy side effects
Explaining stack traces from a CI job I can only partially reproduce

A model that wins a competitive-programming benchmark can still be mediocre at "add a regression test for this 200-line view function without breaking the other 40 tests." So the unit of evaluation has to be my tasks, my assertions, my pass criteria.

The setup: one harness, swappable models

The harness has three parts:

A task file — real prompts drawn from my actual work, each with an assertion script.
A runner — sends each prompt to whatever model endpoint is configured, captures the output.
The sandbox loop from my previous post — executes each generated solution in isolation and runs the assertions.

For compute, I've been running this through MonkeyCode, which offers free model access and a free server option, so the runner and the sandbox can live on their side without me paying for idle time or burning a paid API quota on experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness itself is plain Python and works against any OpenAI-compatible endpoint — nothing below is locked to one provider.

tasks.yaml — real tasks, mechanical assertions

- id: regression_test_view
  prompt: |
    Here is a Flask view function (pasted below). Write a pytest
    regression test that covers the ValueError branch. Do not modify
    the view. Use only pytest and the standard library.
    ...
  assert_script: checks/test_regression_view.py
  pass_criteria: exit_code_zero

- id: explain_ci_failure
  prompt: |
    This CI log tail shows a failure (pasted below). Identify the
    most likely root cause in two sentences and name the file to fix.
  assert_script: checks/keyword_and_file.py
  pass_criteria: mentions_expected_file

Note that not every assertion is "run the code." For explanation tasks I check cheaper mechanical signals (does it name the right file? does it avoid inventing a config key that doesn't exist in the repo?) and reserve manual review for borderline cases.

runner.py — the swappable part

import subprocess, tempfile, pathlib, yaml

def run_task(client, model, task):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": task["prompt"]}],
        temperature=0,
    )
    output = resp.choices[0].message.content
    code = extract_code_block(output)  # returns None if no code block

    with tempfile.TemporaryDirectory() as tmp:
        workdir = pathlib.Path(tmp)
        if code:
            (workdir / "solution.py").write_text(code)
        result = subprocess.run(
            ["python", task["assert_script"]],
            cwd=workdir, capture_output=True, timeout=60,
        )
        return {
            "task": task["id"],
            "model": model,
            "passed": result.returncode == 0,
            "stderr_tail": result.stderr.decode()[-500:],
        }

def compare(clients_and_models, tasks_file="tasks.yaml"):
    tasks = yaml.safe_load(open(tasks_file))
    rows = []
    for client, model in clients_and_models:
        for task in tasks:
            rows.append(run_task(client, model, task))
    return rows

Everything is deterministic-ish by design: temperature=0, a fixed prompt file, fixed assertions. When a new model shows up, I add one line to the client list and rerun — that rerun is free, which is the whole point.

A decision table, not a verdict

After a couple of runs, I stopped asking "which model is best" and started recording where each one fails. My current table looks like this (model names anonymized because your table should be yours):

Task type	Model A	Model B	Model C
Regression test generation	pass 4/5	pass 5/5	pass 2/5
Refactor with constraints	pass 3/5	pass 2/5	pass 4/5
CI log diagnosis	wrong file 2/5	correct 4/5	correct 3/5
Hallucinated imports/config keys	rare	occasional	frequent

The actionable output isn't a winner — it's a routing rule: "use B for test generation, C for constrained refactors, never trust C's imports without a compile check." That rule survives the next model release; a hype-driven ranking doesn't.

Limitations, honestly

Small-N problem. Five tasks per category is a smoke test, not statistics. I treat failures as signals to investigate, not proof.
Temperature 0 isn't fully deterministic. Same prompt can still vary run to run. I rerun failures once before recording them.
Mechanical assertions miss quality. A test can pass my checker and still be unreadable. I spot-review anything I'm going to actually adopt.
Free tiers have limits. Rate limits and queue times mean I batch runs instead of iterating interactively, and I don't assume today's free access is permanent — the harness is provider-agnostic precisely so I can move it.
Who shouldn't bother: if you only use AI autocomplete for single lines, a task-level harness is overkill. This pays off when you're choosing a model for sustained, multi-step work.

Where I'd start

If you want to try this, don't build the whole thing at once. Pick five prompts from work you did last week, write one assertion script, and run two models against them. If you're looking for somewhere to run it without a budget, MonkeyCode's free model access and free server option is one place to host the loop — but the harness above will run against whatever endpoint you already have.

The durable skill here isn't picking the right model this month. It's owning the evaluation, so every "new model just dropped" announcement becomes a ten-minute rerun instead of a leap of faith.

A Reproducible Sandbox Loop for AI-Generated Code: Generate, Isolate, Assert

Charlie Zhu — Mon, 10 Aug 2026 08:03:08 +0000

AI coding assistants are getting more tool access — shell commands, file writes, network calls. The recent discussions on DEV about agent boundaries failing resonate with a practical problem I keep running into: how do you evaluate code an AI just wrote without trusting it on your own machine?

This article describes a repeatable, boring-on-purpose workflow: generate code with an AI assistant, run it inside a locked-down container, and assert on the output. No trust required. The artifact below is a working harness you can adapt; treat it as a starting point, not production policy.

The problem

When an assistant produces a script, the usual options are:

Read it carefully, then run it locally — slow, and you will eventually miss something.
Run it and hope — fast, and eventually catastrophic.
Never run generated code — which throws away most of the value.

Option 4 is a fixed evaluation loop with a hard isolation boundary. The point is not that containers are a perfect security boundary (they are not — see limitations). The point is that the process becomes reproducible: same prompt, same harness, same assertions, comparable results across models and sessions.

Where the free tier fits

The loop has two costs: model calls and a machine to run untrusted code on. Both can gatekeep experimentation. MonkeyCode currently offers free model access and a free server option, which removes those two gates for this kind of workflow — you can iterate on prompts and run the generated code somewhere that is not your laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A few honest caveats: I am not going to quote specific model names, quotas, or hardware specs here, because availability on free tiers changes and you should check the current offering yourself. The workflow below is tool-agnostic — it works with any assistant that can emit code, and any host where you can run containers. If the free options disappear tomorrow, the harness still stands.

The artifact: a minimal isolation harness

The core idea: every generated artifact runs in a container with no network, a memory cap, a CPU cap, a read-only root filesystem, and a wall-clock timeout. Here is the runner (bash, requires Docker):

#!/usr/bin/env bash
# run_sandboxed.sh — execute an untrusted file in a locked-down container.
# Usage: ./run_sandboxed.sh <file> <image> <timeout_seconds>
set -euo pipefail

FILE="$1"
IMAGE="${2:-python:3.12-slim}"
TIMEOUT="${3:-10}"

if [ ! -f "$FILE" ]; then
  echo "No such file: $FILE" >&2
  exit 64
fi

# --network none   : no exfiltration, no package installs mid-run
# --read-only      : root fs immutable; /tmp is a small tmpfs
# --memory / --cpus: resource caps so runaway loops don't eat the host
# --pids-limit     : blunts fork bombs
# timeout          : hard wall-clock kill
exec timeout "$TIMEOUT" docker run --rm \
  --network none \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 256m \
  --cpus 0.5 \
  --pids-limit 64 \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -v "$(realpath "$FILE")":/work/task:ro \
  "$IMAGE" /work/task

And a tiny evaluation driver. The point of this file is not the code — it is that the assertions are written before you look at the generated solution, so you are testing the model's output rather than rationalizing it:

# eval_loop.py — proposal-grade harness; extend for your own tasks.
import json, subprocess, sys
from pathlib import Path

TASKS = json.loads(Path("tasks.json").read_text())
# tasks.json: [{"id": "fizzbuzz", "prompt": "...", "stdin": "", "expect": "1\n2\nFizz\n..."}]

results = []
for task in TASKS:
    candidate = Path(f"gen/{task['id']}.py")
    if not candidate.exists():
        results.append({"id": task["id"], "status": "missing"})
        continue
    proc = subprocess.run(
        ["./run_sandboxed.sh", str(candidate), "python:3.12-slim", "10"],
        input=task["stdin"], capture_output=True, text=True,
    )
    ok = proc.returncode == 0 and proc.stdout == task["expect"]
    results.append({
        "id": task["id"],
        "status": "pass" if ok else "fail",
        "stderr_tail": proc.stderr[-300:] if not ok else "",
    })

print(json.dumps(results, indent=2))

The loop in practice:

Write the task spec and expected output yourself, first.
Have the assistant (in my case, running through MonkeyCode's free model access) produce gen/<task>.py.
Run the driver. Every execution is isolated, time-boxed, and resource-capped.
Record pass/fail. Now you can meaningfully compare prompt variants or models, because the measurement is stable.

Which sandbox level do you actually need?

Threat you care about	Minimum sensible setup	The harness above enough?
Accidental damage (rm in wrong dir, huge allocation)	Restricted container, resource caps	Yes
Nosy generated code (reading env vars, home dir)	No network + dropped capabilities + clean image	Mostly
Deliberately adversarial code	Separate VM or host you can rebuild	No — escalate isolation
Compliance / customer data in scope	Dedicated infra, audit trail, legal review	Absolutely not

Limitations

Containers share the host kernel. A kernel exploit escapes this sandbox. For genuinely adversarial code, use a separate VM or a disposable machine — which is one reason running this on a free remote server instead of your daily driver is attractive.
No network means no pip installs. If generated code needs dependencies, prebuild an image containing them rather than opening the network.
Timeouts cut both ways. A 10-second limit will kill legitimate slow tasks; tune per task, and treat timeouts as a signal, not noise.
The harness does not judge code quality, only behavior against your assertions. Tests you wrote badly will pass bad code.

Who should skip this

If your assistant only ever produces single functions you fully review before use, this is overhead. If you need to run generated code that must access production credentials or live APIs, do not adapt this harness — that is a secrets-management and permissions problem, not a sandboxing problem. And if your organization has an approved execution environment for AI output, use that instead.

Closing thought

The useful shift is treating AI output like any other untrusted artifact: fixed inputs, isolated execution, assertions written in advance. Once the loop is reproducible, model and prompt comparisons stop being vibes. If you want to try this without paying for model calls or a throwaway host while you experiment, MonkeyCode's free model access and free server option are a reasonable place to start — but the harness matters more than the vendor.