Free Model + Free Server: A Reproducible 12-Task Eval for AI Coding Agents

A free model tier can handle scoped coding tasks. It breaks on open-ended refactors and long context. A 12-task harness makes that boundary measurable.

Why evaluate before you adopt

Free model access changes how teams prototype. Free servers remove the cost barrier for agent experiments. Neither removes the need for evidence.

A vibe check is not a test suite. The harness below turns "it feels smart" into a pass rate. That pass rate decides where the free tier belongs.

What gets evaluated

MonkeyCode is an open-source project with free model access and a free server option. The server exposes an OpenAI-compatible endpoint.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That endpoint matters. It means the eval uses a standard client. No vendor SDK is required.

The eval targets three workload classes:

1. Scoped tasks: one file, one function, a clear test.
2. Medium tasks: two files, a small API change.
3. Open tasks: multi-file refactors with no acceptance test.

Twelve tasks total. Four per class. Each task has a written pass criterion.

Why twelve tasks

Four tasks per class is enough to see a pattern. One task is noise. Three runs per task smooth the variance.

The suite runs in under an hour. The token cost is zero by design. The server cost is zero by design.

The task suite

ID	Class	Task	Pass criterion
t01	scoped	Add a zero-division test	Test asserts ZeroDivisionError
t02	scoped	Rename a variable in one file	No old name remains
t03	scoped	Fix a failing regex	Existing test passes
t04	scoped	Add input validation	Invalid input returns 400
t05	medium	Add an endpoint to a Flask app	curl returns 201
t06	medium	Split a module into two files	Imports resolve
t07	medium	Add pagination to a query	Page size respected
t08	medium	Refactor a function into a class	Behavior unchanged
t09	open	Extract a service layer	Tests pass, logic unchanged
t10	open	Migrate callbacks to async	All tests pass
t11	open	Split a monolith module	No circular imports
t12	open	Add error handling codebase-wide	No uncaught exceptions

Every pass criterion is binary. No partial credit. That keeps scoring honest.

The harness

The harness is one Python file. It uses the OpenAI SDK against the free server's base URL.

# eval_free_tier.py
import json
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MONKEYCODE_BASE_URL"],
    api_key=os.environ["MONKEYCODE_API_KEY"],
)

TASKS = [
    {
        "id": "t01",
        "prompt": (
            "Add a pytest test to src/math_utils.py. "
            "Assert that divide(1, 0) raises ZeroDivisionError."
        ),
        "pass": "ZeroDivisionError in test file",
    },
    # Add t02..t12 from the table above.
]

def run_task(task):
    start = time.time()
    response = client.chat.completions.create(
        model=os.environ.get("MONKEYCODE_MODEL", "default"),
        messages=[{"role": "user", "content": task["prompt"]}],
        temperature=0,
    )
    return {
        "id": task["id"],
        "output": response.choices[0].message.content,
        "pass_marker": task["pass"],
        "seconds": round(time.time() - start, 1),
        "tokens": response.usage.total_tokens,
    }

def main():
    results = [run_task(t) for t in TASKS]
    print(json.dumps(results, indent=2))

if __name__ == "__main__":
    main()

Run it with two environment variables:

export MONKEYCODE_BASE_URL="https://your-server.example"
export MONKEYCODE_API_KEY="your-key"
python eval_free_tier.py > results.json

The base URL comes from the server settings. The model id defaults to "default". Change it if the server lists a specific model. Temperature is zero. The eval measures capability, not creativity.

Scoring with a checker

Manual scoring is fine for twelve tasks. A checker is better for repeated runs.

# score.py
import json
import sys

def main(path):
    with open(path) as fh:
        results = json.load(fh)
    for r in results:
        passed = r["pass_marker"] in r["output"]
        print(f'{r["id"]}: {"PASS" if passed else "FAIL"}')

if __name__ == "__main__":
    main(sys.argv[1])

Each task carries a pass_marker. The checker greps the model output. The result is a clean pass/fail column.

Reading the results

The table below shows the output shape. Values are illustrative until you run the suite.

Class	Pass rate	Median tokens	Median seconds
Scoped	4/4	1,200	18
Medium	3/4	2,800	41
Open	1/4	6,500	96

The pattern matters more than the numbers. Scoped tasks pass. Open tasks drift. The free tier earns its place on the left side of the table.

Where free tiers break

Three failure modes show up consistently:

1. Context drift. Long prompts push the model past its useful window. Output becomes generic.
2. Tool loops. The agent repeats the same edit. No progress between calls.
3. Confident edits. The code compiles. The tests still fail.

Each failure maps to a workload class. Scoped tasks avoid all three. Open tasks invite all three.

A decision table for free tiers

Situation	Verdict
One-file task with a clear test	Use the free tier
Prototype or spike	Use the free tier
Multi-file refactor without tests	Avoid
Production migration	Avoid
Long-context analysis	Chunk it first

The boundary is not about intelligence. It is about verification. Free tiers work when a test can judge the output.

Extending the suite

Add tasks as your workflow changes.

1. Write a failing test first.
2. Convert the test into a prompt.
3. Add the pass marker to the task.
4. Run the suite.
5. Record the result.

The suite becomes a regression check for your tooling. When the free tier changes, the pass rate tells you.

Limitations of this eval

This is not a benchmark. It measures one server, one day, one task suite. Free tiers change without notice.

The eval does not measure latency under load. It does not measure security or compliance. It does not measure code review quality.

Teams with production SLAs should not rely on a free tier. Teams with compliance requirements should not either. Use this harness for prototypes and internal tools.

The takeaway

Free model access lowers the cost of experimentation. A free server lowers the cost of automation. Neither lowers the cost of verification.

Run the 12-task suite before you build a workflow on any free tier. MonkeyCode's free model access and free server are a reasonable place to start. The harness is the point. The pass rate is the decision.