DEV Community: Quinn Li

Reproducible Evaluation Harness for New Model Releases: MiniMax H3

Quinn Li — Fri, 14 Aug 2026 13:57:36 +0000

You don't need a leaderboard screenshot to judge a new model release. I build a deterministic evaluation harness—a small hand-labelled dataset, a fixed prompt, and a scoring function—and use MiniMax H3 as a case study for measuring whether a model works for a specific team task. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. Start with a narrow task, not a benchmark

A vague evaluation produces vague conclusions. I shrink the evaluation to one decision the application actually needs. In this case it is GitHub issue triage: given an issue title and body, return a JSON object with one field, label, whose value is one of bug, feature, or question.

The scoring contract has three parts:

The dataset must be hand-labelled.
The output must be valid JSON.
The model must be retried once on malformed JSON with the original prompt and an instruction to fix the format.

This is not a general intelligence benchmark; it is a narrow task test. A small hand-labelled set is enough to expose formatting failures and basic label confusion.

2. Build a small labelled dataset and a strict prompt

I keep the layout simple so raw predictions stay separate from scoring code and the audit trail is clear.

eval-minimax-h3/
  data/
    cases.jsonl
  prompts/
    system.txt
  scripts/
    run_eval.py
    report.py
  results/
    raw_predictions.jsonl

Each line in data/cases.jsonl uses JSON Lines format:

{"id":"issue-001","input":"Title: Login button does nothing on Safari\nBody: Clicking login returns no error and the console is clean.","expected":"bug"}
{"id":"issue-002","input":"Title: Add CSV export to invoice list\nBody: Users need to export visible columns.","expected":"feature"}
{"id":"issue-003","input":"Title: Where is the rate limit documented?\nBody: The API docs do not show the limit.","expected":"question"}

The system prompt is deliberately strict:

You are an issue triage assistant.
Return only a JSON object in this form:
{"label":"bug|feature|question"}
Do not include markdown fences or extra text.

A strict prompt is a form of experimental control: it makes the scoring rule easier to reproduce than a natural-language prompt that rewards style as much as correctness.

3. Write a deterministic runner

The runner uses the OpenAI Python client so the same script can target a local server, a hosted gateway, or a provider endpoint. The endpoint, model name, and key come from environment variables.

import json
import os
import time
from openai import OpenAI

with open('data/cases.jsonl', encoding='utf-8') as f:
    cases = [json.loads(line) for line in f if line.strip()]

with open('prompts/system.txt', encoding='utf-8') as f:
    SYSTEM_PROMPT = f.read().strip()

client = OpenAI(
    base_url=os.environ['MODEL_BASE_URL'],
    api_key=os.environ['MODEL_API_KEY'],
)

def complete(user_input: str) -> str:
    resp = client.chat.completions.create(
        model=os.environ['MODEL_NAME'],
        temperature=0,
        messages=[
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': user_input},
        ],
    )
    return resp.choices[0].message.content

def parse_label(text: str) -> str:
    text = text.strip()
    if text.startswith('`' * 3):
        text = text.strip('`')
        if text.startswith('json'):
            text = text[4:]
    start, end = text.find('{'), text.rfind('}')
    if start != -1 and end != -1:
        text = text[start:end + 1]
    try:
        data = json.loads(text)
        return str(data.get('label', '')).strip().lower()
    except json.JSONDecodeError:
        return ''

Then result loop:

results = []
for case in cases:
    started = time.perf_counter()
    raw = complete(case['input'])
    latency_ms = (time.perf_counter() - started) * 1000
    label = parse_label(raw)
    if label == '':
        raw = complete(case['input'] + '\nYour previous response was not valid JSON. Return only the JSON object.')
        label = parse_label(raw)
    results.append({
        'id': case['id'],
        'expected': case['expected'],
        'raw': raw,
        'label': label,
        'correct': label == case['expected'].lower(),
        'latency_ms': round(latency_ms, 2),
    })

with open('results/raw_predictions.jsonl', 'w', encoding='utf-8') as f:
    for row in results:
        f.write(json.dumps(row, ensure_ascii=False) + '\n')

Using temperature=0 keeps the output deterministic for a fixed prompt. The Chat Completions reference documents the parameter.

4. Report honest numbers and run it cheaply

A single accuracy percentage hides formatting problems. I print accuracy, remaining invalid JSON, and latency percentiles:

import json
import statistics

rows = [json.loads(line) for line in open('results/raw_predictions.jsonl', encoding='utf-8')]
total = len(rows)
correct = sum(r['correct'] for r in rows)
invalid = sum(not r['label'] for r in rows)
latencies = [r['latency_ms'] for r in rows]

print(f'accuracy: {correct}/{total} = {correct/total:.3f}')
print(f'invalid_json_remaining: {invalid}')
print(f'p50_latency_ms: {statistics.median(latencies):.1f}')
print(f'p95_latency_ms: {sorted(latencies)[int(total * 0.95) - 1]:.1f}')

These numbers are only valid for the labelled set and the fixed prompt. They do not measure creativity, factual accuracy, or agentic behavior.

A run looks like this:

export MODEL_BASE_URL='https://example.com/v1'
export MODEL_API_KEY='...'
export MODEL_NAME='minimax-h3'
python scripts/run_eval.py
python scripts/report.py

Replace the endpoint and model name with values from the provider's current documentation. Because the client is OpenAI-compatible, the same artifact runs against several backends:

Local open-weight endpoint through Ollama or vLLM.
Any hosted gateway by setting the environment variables.
A scheduled runner on a free server option, when available, for repeated evaluation.

MonkeyCode's operator-supplied free model access can be used when the model under test is exposed through its endpoint; MonkeyCode's free server option can host the runner for scheduled jobs. The article does not state specific quotas, hardware, latency, or permanence, so those details should be confirmed with the operator.

5. Publish the artifact, not the number

Open-source practice is not only about model weights; it is about making measurement reproducible. Publishing the prompts, raw predictions, scoring code, and the exact model revision is more useful than posting a single leaderboard number. A reader can then re-run the same cases.jsonl against a different endpoint and compare results without guessing what the original author did.

Limitations:

The hand-labelled set is small, so the result is not a general ranking.
Temperature 0 is deterministic but hides sampling variance; repeat at temperature 1 for consistency checks.
Exact-match scoring is too strict for free-text tasks; use normalized semantic agreement when the output is not a fixed label.
The harness does not evaluate safety, factual accuracy, long-horizon agent behavior, or prompt-injection resistance.
API schema changes can break the runner, so the adapter should be versioned.
Teams with no labelled examples should invest in labelling before running this harness.
Teams that need a universal model comparison should use multiple benchmarks and human review rather than one narrow task.

Copy this harness, replace the prompt with your own taxonomy, and run it against the current MiniMax H3 endpoint. If you hit a formatting failure or want to compare backends, share the raw predictions file and scoring code—that's the artifact worth reviewing. Check MonkeyCode's free model access and free server option if you need a low-cost runner.

A Reproducible Tool-Call Gatekeeper for AI Agents

Quinn Li — Fri, 14 Aug 2026 10:51:47 +0000

Autonomous agents are often connected to real tools: shell commands, email clients, database handles, and file operations. A single unsafe call can delete data or send an external message, so teams add a policy layer between the agent and the executor.

This article builds a small, testable tool-call gatekeeper. It uses deterministic allow and deny rules first, then falls back to an LLM classifier for unknown tool calls. The prototype is written to run against MonkeyCode's free model access and free server option during local experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the gatekeeper does

The gatekeeper receives a proposed tool call, checks it against a local policy file, and returns one of three decisions:

allow: the call is safe to forward to the executor.
deny: the call is blocked before execution.
escalate: the call needs human review.

Deterministic rules handle known tools, while the LLM fallback handles only the long tail. That separation makes the behavior easier to debug than a single end-to-end classifier.

Step 1: Create the project and install dependencies

mkdir agent-gatekeeper
cd agent-gatekeeper
python -m venv .venv
source .venv/bin/activate
pip install requests

Step 2: Define a readable policy file

Create policy.json:

{
  "version": 1,
  "default_decision": "escalate",
  "allowlist": ["read_file", "list_directory", "search_docs"],
  "denylist": ["delete_file", "drop_table", "send_email"],
  "llm_fallback": {
    "enabled": true,
    "timeout_seconds": 10,
    "max_tokens": 120
  }
}

The default decision matters when the LLM fallback is disabled or returns malformed output. Escalating unknown calls is safer than allowing them by default.

Step 3: Implement the gatekeeper

Create gatekeeper.py:

import argparse
import json
import os

import requests


def load_policy(path):
    with open(path, 'r', encoding='utf-8') as handle:
        return json.load(handle)


def deterministic_decision(policy, tool_name):
    if tool_name in policy.get('denylist', []):
        return {'decision': 'deny', 'reason': 'denylist match'}
    if tool_name in policy.get('allowlist', []):
        return {'decision': 'allow', 'reason': 'allowlist match'}
    return None


def classify_with_llm(tool_call, policy, env):
    url = env['GATE_LLM_URL']
    headers = {'Content-Type': 'application/json'}
    if env.get('GATE_LLM_API_KEY'):
        headers['Authorization'] = 'Bearer ' + env['GATE_LLM_API_KEY']

    system_message = (
        'You are a tool-call policy classifier. Reply with JSON only. '
        'Choose allow, deny, or escalate. Deny destructive or high-risk calls. '
        'Escalate when intent is unclear.'
    )
    payload = {
        'model': env.get('GATE_LLM_MODEL', 'default'),
        'temperature': 0,
        'messages': [
            {'role': 'system', 'content': system_message},
            {'role': 'user', 'content': json.dumps(tool_call)}
        ],
        'max_tokens': policy['llm_fallback']['max_tokens']
    }

    response = requests.post(
        url,
        headers=headers,
        json=payload,
        timeout=policy['llm_fallback']['timeout_seconds']
    )
    response.raise_for_status()
    content = response.json()['choices'][0]['message']['content']
    parsed = json.loads(content)
    return {
        'decision': parsed['decision'],
        'reason': parsed.get('reason', 'llm fallback')
    }


def evaluate(tool_call, policy, env):
    tool_name = tool_call.get('name', '')
    result = deterministic_decision(policy, tool_name)
    if result is not None:
        return result

    fallback = policy.get('llm_fallback', {})
    if not fallback.get('enabled', True):
        return {
            'decision': policy.get('default_decision', 'escalate'),
            'reason': 'default policy, fallback disabled'
        }

    try:
        return classify_with_llm(tool_call, policy, env)
    except Exception as exc:
        return {
            'decision': policy.get('default_decision', 'escalate'),
            'reason': 'llm error: ' + str(exc)
        }


def main():
    parser = argparse.ArgumentParser(description='Evaluate a proposed tool call')
    parser.add_argument('--policy', default='policy.json')
    parser.add_argument('--tool-call', required=True)
    parser.add_argument('--dry-run', action='store_true')
    args = parser.parse_args()

    with open(args.tool_call, 'r', encoding='utf-8') as handle:
        tool_call = json.load(handle)

    policy = load_policy(args.policy)
    env = {
        'GATE_LLM_URL': os.environ.get('GATE_LLM_URL', ''),
        'GATE_LLM_MODEL': os.environ.get('GATE_LLM_MODEL', ''),
        'GATE_LLM_API_KEY': os.environ.get('GATE_LLM_API_KEY', '')
    }

    result = evaluate(tool_call, policy, env)
    if args.dry_run:
        print('DRY RUN')
    print(json.dumps(result, indent=2))


if __name__ == '__main__':
    main()

The try/except around the LLM call escalates on timeout or malformed JSON instead of failing open.

Step 4: Configure the endpoint

export GATE_LLM_URL='https://your-endpoint-here.example/chat/completions'
export GATE_LLM_MODEL='the-model-id-from-the-provider-dashboard'
export GATE_LLM_API_KEY='your-api-key-here'
python gatekeeper.py --policy policy.json --tool-call samples/delete_file.json --dry-run

The URL path, model identifier, and authentication format must come from the provider's current documentation. The gatekeeper does not hard-code those values. For a non-OpenAI-compatible API, the request payload and response parsing inside classify_with_llm should be replaced.

Step 5: Run a small test plan

Create three sample files.

samples/read_file.json:

{
  "name": "read_file",
  "arguments": {"path": "/tmp/notes.txt"},
  "agent_id": "agent-1",
  "intent": "load notes for a summary"
}

samples/delete_file.json:

{
  "name": "delete_file",
  "arguments": {"path": "/tmp/cache.txt"},
  "agent_id": "agent-1",
  "intent": "remove cache file"
}

samples/unknown_query.json:

{
  "name": "run_sql",
  "arguments": {"query": "SELECT COUNT(*) FROM events"},
  "agent_id": "agent-1",
  "intent": "check event volume"
}

Run the deterministic cases first:

python gatekeeper.py --policy policy.json --tool-call samples/read_file.json --dry-run
python gatekeeper.py --policy policy.json --tool-call samples/delete_file.json --dry-run

Expected output: allow for the read, deny for the delete, without an LLM request.

Then run the unknown case against the configured LLM endpoint. The classifier should return allow, deny, or escalate with a plain-text reason. If the endpoint is unavailable, the gatekeeper should return escalate rather than crash.

Why the fallback should be conservative

An LLM classifier is useful for unusual tool names, but it is not a security boundary. The model can be confused by indirect instructions, prompt injection, or an unfamiliar tool. For that reason, the policy escalates on malformed output and keeps a short denylist for the riskiest actions.

Limitations

The code assumes an OpenAI-compatible chat completions response shape. Other APIs need a different parser.
Free model access and free server options may have rate limits, latency, and usage caps. This article does not test those limits and does not claim specific quotas or guaranteed availability.
The classifier is probabilistic. Identical requests can produce different results if the provider changes the model or sampling settings.
The prototype is synchronous and not suitable for high-throughput tool traffic.

Who should not use this approach

This pattern is not appropriate when the gatekeeper protects high-privilege production actions such as database deletion, cloud resource teardown, or customer-facing messaging. It is not a replacement for access control, audit logs, or signed tool allowlists. The gatekeeper is a prototyping aid, not a hardened security control.

Closing

The full gatekeeper is small enough to port into an existing agent harness. Replace the endpoint with a documented free server during early experiments, keep the dry-run mode enabled, and review escalated calls by hand before opening any real tool.