Reproducible Evaluation Harness for New Model Releases: MiniMax H3

You don't need a leaderboard screenshot to judge a new model release. I build a deterministic evaluation harness—a small hand-labelled dataset, a fixed prompt, and a scoring function—and use MiniMax H3 as a case study for measuring whether a model works for a specific team task. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. Start with a narrow task, not a benchmark

A vague evaluation produces vague conclusions. I shrink the evaluation to one decision the application actually needs. In this case it is GitHub issue triage: given an issue title and body, return a JSON object with one field, label, whose value is one of bug, feature, or question.

The scoring contract has three parts:

• The dataset must be hand-labelled.
• The output must be valid JSON.
• The model must be retried once on malformed JSON with the original prompt and an instruction to fix the format.

This is not a general intelligence benchmark; it is a narrow task test. A small hand-labelled set is enough to expose formatting failures and basic label confusion.

2. Build a small labelled dataset and a strict prompt

I keep the layout simple so raw predictions stay separate from scoring code and the audit trail is clear.

eval-minimax-h3/
  data/
    cases.jsonl
  prompts/
    system.txt
  scripts/
    run_eval.py
    report.py
  results/
    raw_predictions.jsonl

Each line in data/cases.jsonl uses JSON Lines format:

{"id":"issue-001","input":"Title: Login button does nothing on Safari\nBody: Clicking login returns no error and the console is clean.","expected":"bug"}
{"id":"issue-002","input":"Title: Add CSV export to invoice list\nBody: Users need to export visible columns.","expected":"feature"}
{"id":"issue-003","input":"Title: Where is the rate limit documented?\nBody: The API docs do not show the limit.","expected":"question"}

The system prompt is deliberately strict:

You are an issue triage assistant.
Return only a JSON object in this form:
{"label":"bug|feature|question"}
Do not include markdown fences or extra text.

A strict prompt is a form of experimental control: it makes the scoring rule easier to reproduce than a natural-language prompt that rewards style as much as correctness.

3. Write a deterministic runner

The runner uses the OpenAI Python client so the same script can target a local server, a hosted gateway, or a provider endpoint. The endpoint, model name, and key come from environment variables.

import json
import os
import time
from openai import OpenAI

with open('data/cases.jsonl', encoding='utf-8') as f:
    cases = [json.loads(line) for line in f if line.strip()]

with open('prompts/system.txt', encoding='utf-8') as f:
    SYSTEM_PROMPT = f.read().strip()

client = OpenAI(
    base_url=os.environ['MODEL_BASE_URL'],
    api_key=os.environ['MODEL_API_KEY'],
)

def complete(user_input: str) -> str:
    resp = client.chat.completions.create(
        model=os.environ['MODEL_NAME'],
        temperature=0,
        messages=[
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': user_input},
        ],
    )
    return resp.choices[0].message.content

def parse_label(text: str) -> str:
    text = text.strip()
    if text.startswith('`' * 3):
        text = text.strip('`')
        if text.startswith('json'):
            text = text[4:]
    start, end = text.find('{'), text.rfind('}')
    if start != -1 and end != -1:
        text = text[start:end + 1]
    try:
        data = json.loads(text)
        return str(data.get('label', '')).strip().lower()
    except json.JSONDecodeError:
        return ''

Then result loop:

results = []
for case in cases:
    started = time.perf_counter()
    raw = complete(case['input'])
    latency_ms = (time.perf_counter() - started) * 1000
    label = parse_label(raw)
    if label == '':
        raw = complete(case['input'] + '\nYour previous response was not valid JSON. Return only the JSON object.')
        label = parse_label(raw)
    results.append({
        'id': case['id'],
        'expected': case['expected'],
        'raw': raw,
        'label': label,
        'correct': label == case['expected'].lower(),
        'latency_ms': round(latency_ms, 2),
    })

with open('results/raw_predictions.jsonl', 'w', encoding='utf-8') as f:
    for row in results:
        f.write(json.dumps(row, ensure_ascii=False) + '\n')

Using temperature=0 keeps the output deterministic for a fixed prompt. The Chat Completions reference documents the parameter.

4. Report honest numbers and run it cheaply

A single accuracy percentage hides formatting problems. I print accuracy, remaining invalid JSON, and latency percentiles:

import json
import statistics

rows = [json.loads(line) for line in open('results/raw_predictions.jsonl', encoding='utf-8')]
total = len(rows)
correct = sum(r['correct'] for r in rows)
invalid = sum(not r['label'] for r in rows)
latencies = [r['latency_ms'] for r in rows]

print(f'accuracy: {correct}/{total} = {correct/total:.3f}')
print(f'invalid_json_remaining: {invalid}')
print(f'p50_latency_ms: {statistics.median(latencies):.1f}')
print(f'p95_latency_ms: {sorted(latencies)[int(total * 0.95) - 1]:.1f}')

These numbers are only valid for the labelled set and the fixed prompt. They do not measure creativity, factual accuracy, or agentic behavior.

A run looks like this:

export MODEL_BASE_URL='https://example.com/v1'
export MODEL_API_KEY='...'
export MODEL_NAME='minimax-h3'
python scripts/run_eval.py
python scripts/report.py

Replace the endpoint and model name with values from the provider's current documentation. Because the client is OpenAI-compatible, the same artifact runs against several backends:

1. Local open-weight endpoint through Ollama or vLLM.
2. Any hosted gateway by setting the environment variables.
3. A scheduled runner on a free server option, when available, for repeated evaluation.

MonkeyCode's operator-supplied free model access can be used when the model under test is exposed through its endpoint; MonkeyCode's free server option can host the runner for scheduled jobs. The article does not state specific quotas, hardware, latency, or permanence, so those details should be confirmed with the operator.

5. Publish the artifact, not the number

Open-source practice is not only about model weights; it is about making measurement reproducible. Publishing the prompts, raw predictions, scoring code, and the exact model revision is more useful than posting a single leaderboard number. A reader can then re-run the same cases.jsonl against a different endpoint and compare results without guessing what the original author did.

Limitations:

• The hand-labelled set is small, so the result is not a general ranking.
• Temperature 0 is deterministic but hides sampling variance; repeat at temperature 1 for consistency checks.
• Exact-match scoring is too strict for free-text tasks; use normalized semantic agreement when the output is not a fixed label.
• The harness does not evaluate safety, factual accuracy, long-horizon agent behavior, or prompt-injection resistance.
• API schema changes can break the runner, so the adapter should be versioned.
• Teams with no labelled examples should invest in labelling before running this harness.
• Teams that need a universal model comparison should use multiple benchmarks and human review rather than one narrow task.

Copy this harness, replace the prompt with your own taxonomy, and run it against the current MiniMax H3 endpoint. If you hit a formatting failure or want to compare backends, share the raw predictions file and scoring code—that's the artifact worth reviewing. Check MonkeyCode's free model access and free server option if you need a low-cost runner.