Benchmarks are useful only when they predict behavior on a team's own code, and most public tables do not reach that bar.
A recent MiniMax H3 release has been moving through developer feeds, but a model release is not yet a reason to change a workflow.
The practical next step is to compare the new model against a free baseline on a fixed set of failing tests.
This article describes a small reproducible harness that does exactly that without vendor lock-in.
The account's earlier evaluations used a two-model regression harness, and this version adds one constraint that matters for small teams.
MonkeyCode's free model access and free server option remove the budget excuse for skipping a controlled comparison.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The free endpoint is used here only as the baseline, while the MiniMax H3 response becomes the candidate.
No benchmark score is copied from a public leaderboard, because leaderboards do not see a team's hidden edge cases.
Think of a model evaluation as a code review, not as a race.
A reviewer does not ask which candidate once won a contest; the reviewer asks which candidate passes the current patch under the project's lint rules and test suite.
The harness treats both models as command-line responders, so the evaluation can be rerun whenever a prompt changes.
Keeping the comparison in git means every conclusion has an audit trail.
The five-minute baseline
Create a project directory, install pytest, and leave two environment variables with placeholder adapter commands.
The following commands build a clean workspace and keep the baseline and candidate outputs separate.
Each model receives the same prompt through standard input, and each response is stored as a timestamped JSON line.
The shell commands do not depend on any provider SDK, which keeps the evaluation portable.
mkdir model-eval && cd model-eval
python -m venv .venv && source .venv/bin/activate
pip install pytest
mkdir -p tasks runs baseline candidate tests
Create a concrete task that can fail in ordinary code generation.
The task asks for a small parser and a test for invalid input, which makes a generated answer easy to judge.
A broad prompt would hide differences, while a narrow one exercises the same parsing edge cases that break in real projects.
cat > tasks/code_task_1.txt <<'EOF'
Write a Python function `parse_time` that accepts a string such as "1h20m"
and returns an integer number of minutes. Include a short test suite for invalid input.
EOF
The harness below reads a prompt file and invokes both adapters from environment variables.
It writes the full stdout, stderr, and return code to runs/, then stores separate text copies in baseline/ and candidate/ for easy diffing.
The script avoids hard-coding any vendor API because a model comparison should not require rewriting the runner when a provider changes.
import json
import os
import shlex
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
def run_adapter(env_name: str, prompt: str) -> dict:
raw = os.getenv(env_name)
if not raw:
raise SystemExit(f"Set {env_name} before running the harness.")
parts = shlex.split(raw)
proc = subprocess.run(
parts,
input=prompt,
text=True,
capture_output=True,
check=False,
)
return {
"stdout": proc.stdout,
"stderr": proc.stderr,
"returncode": proc.returncode,
}
def main() -> None:
if len(sys.argv) != 2:
raise SystemExit("Usage: python run_pair.py tasks/code_task_1.txt")
prompt = Path(sys.argv[1]).read_text()
baseline = run_adapter("BASELINE_CMD", prompt)
candidate = run_adapter("CANDIDATE_CMD", prompt)
record = {
"task": sys.argv[1],
"run_at": datetime.now(timezone.utc).isoformat(),
"baseline": baseline,
"candidate": candidate,
}
runs_dir = Path("runs")
runs_dir.mkdir(exist_ok=True)
json_path = runs_dir / (Path(sys.argv[1]).stem + ".json")
json_path.write_text(json.dumps(record, indent=2), encoding="utf-8")
base_dir = Path("baseline")
cand_dir = Path("candidate")
base_dir.mkdir(exist_ok=True)
cand_dir.mkdir(exist_ok=True)
base_path = base_dir / Path(sys.argv[1]).name
cand_path = cand_dir / Path(sys.argv[1]).name
base_path.write_text(baseline["stdout"], encoding="utf-8")
cand_path.write_text(candidate["stdout"], encoding="utf-8")
print(f"wrote {json_path}, {base_path}, {cand_path}")
if __name__ == "__main__":
main()
An adapter keeps the harness independent of a particular SDK.
When an endpoint exposes an OpenAI-compatible chat route, the same small adapter can drive both the free baseline and the candidate.
The adapter reads a prompt from standard input, sends it as a user message, and prints the assistant reply.
import argparse
import json
import sys
import urllib.request
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--base-url", required=True)
parser.add_argument("--model", required=True)
args = parser.parse_args()
prompt = sys.stdin.read()
payload = {
"model": args.model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
}
request = urllib.request.Request(
args.base_url.rstrip("/") + "/chat/completions",
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request, timeout=120) as response:
body = json.load(response)
print(body["choices"][0]["message"]["content"])
if __name__ == "__main__":
main()
Run both responses through the same adapter and compare the raw text before evaluating the code.
Set the endpoint and model values from the provider's current documentation, because the harness deliberately avoids embedding secrets or stale URLs.
A visible diff exposes formatting differences, code-block padding, and refusal patterns that numeric scores often bury.
export BASELINE_CMD="python adapters/http_chat.py --base-url $BASELINE_URL --model $BASELINE_MODEL"
export CANDIDATE_CMD="python adapters/http_chat.py --base-url $CANDIDATE_URL --model $CANDIDATE_MODEL"
python run_pair.py tasks/code_task_1.txt
diff -u baseline/code_task_1.txt candidate/code_task_1.txt || true
For a code task the next step is to extract the Python block from the candidate output and run a test against it.
The sample test below checks the format promised by the task, including an invalid input that should raise instead of returning a wrong total.
This is the point where a leaderboard score becomes an engineering result.
cat > tests/test_parse_time.py <<'EOF'
import pytest
from candidate_solution import parse_time
def test_basic_minutes():
assert parse_time("1h20m") == 80
def test_minutes_only():
assert parse_time("45m") == 45
def test_invalid_input_raises():
with pytest.raises(ValueError):
parse_time("tomorrow")
EOF
python - <<'PY'
from pathlib import Path
text = Path("candidate/code_task_1.txt").read_text()
code = text.split("```
python")[-1].split("
```")[0]
Path("candidate_solution.py").write_text(code)
PY
pytest -q tests/test_parse_time.py
The open-source value here is not a marketing label; it is the ability to read the diff and rerun the same input.
When a model's score comes from a private harness, readers inherit someone else's conclusion.
When the harness is a few Python files, readers can change the task, add a failing test, and repeat the comparison without paying for access.
A free baseline is therefore not merely a cost decision; it makes independent verification cheaper than trust.
This harness does not measure latency under load, long-context retrieval, safety, or license compatibility.
The results only cover the chosen prompt set, which means a different team may see different behavior.
The free baseline availability and quota are operator-supplied and should be confirmed from current MonkeyCode documentation before relying on them.
Teams that require a private on-premises model, guaranteed support, or strict output licensing should not treat a free server evaluation as a production readiness test.
Developers who want the exact endpoint values should check MonkeyCode's current documentation; the harness itself assumes nothing beyond a chat route.
Top comments (0)