Pick a Free AI Model by Score, Not by Reputation: A 20-Prompt Harness

Model selection is the new dependency pinning. You would not add a library to your lockfile because a blog post praised it; you would run its tests against your own code first. Most teams do the opposite with AI models: they pick one from a trending article, configure it once, and never re-score it. A 20-prompt harness turns that decision back into evidence.

Here is the concrete situation I am working from. MonkeyCode is an open-source project whose free tier, at the time of writing, includes access to free models, a token allocation of 10 million, and a free server instance you can use for evaluation runs. Model lists and quotas move, so verify the current numbers in the docs before you depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness design matters more than the product behind it. You define 20 prompts from real tasks in your repository, send them to every model you are considering, and score the outputs with a rubric you can defend in a code review. The output is a decision table: model, prompt, pass, fail, and score. No opinions, no release-note reading.

Step 1: Write 20 prompts that look like your workload

A generic prompt set measures generic skill. Your prompts should come from commits, issues, and failures that actually happened in your repo. Keep the list small but representative:

# eval/prompts.yaml
prompts:
  - id: commit-007
    task: commit_message
    input: fix webhook parser; handle empty body regression from PR 412
  - id: test-013
    task: generate_test
    input: parseRetryAfter(value) returns null on absent header
  - id: review-021
    task: review_diff
    input: review the unified diff in patches/pr-118.diff
  - id: explain-034
    task: explain_error
    input: TypeError Cannot read properties of undefined reading map

Twenty prompts is a sample, not a census. It is enough to expose a stable ranking for one narrow task family, and small enough that a human can read every output in under an hour.

Step 2: Send each prompt to every candidate model

Keep the runner provider-agnostic. The script below expects an endpoint and an API key from environment variables, so you can point it at any compatible model service, including the free models available through MonkeyCode's tier:

#!/usr/bin/env bash
# eval/run.sh - evaluate every model against every prompt and store raw output
set -euo pipefail
MODELS="${MODELS:-model-a,model-b}"
PROMPTS="eval/prompts.yaml"
OUTDIR="eval/results"
mkdir -p "$OUTDIR"
mapfile -t ids < <(yq -r '.prompts[].id' "$PROMPTS")

for model in ${MODELS//,/ }; do
  for id in "${ids[@]}"; do
    prompt=$(yq -r ".prompts[] | select(.id == \"$id\") | .input" "$PROMPTS")
    ts=$(date +%s)
    curl -sS "$MODEL_ENDPOINT/v1/chat/completions" \
      -H "Authorization: Bearer $MODEL_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"model":"'"$model"'","messages":[{"role":"user","content":"'"$prompt"'"}]}' \
      > "$OUTDIR/${model}-${id}-${ts}.json"
    echo "finished $model / $id"
  done
done

This is a template, not a proven production tool: adjust the YAML parsing to your environment and test it against one model before running the full batch.

Step 3: Score with a rubric that rewards what you need

Human reading is the only honest scorer, so make the rubric mechanical enough to stay consistent. Score each output from 0 to 5 on four criteria:

• Compiles: the code runs or the text matches the expected format.
• Fits bounds: the output stays under a length cap you set per task.
• Uses context: the output references the actual input symbol or diff.
• No invented API: it does not import a function you have never heard of.

Sum the four categories for a maximum of 20 points per prompt. A decision table, printed after the run, is the whole deliverable:

Model	Compiles	Fits bounds	Uses context	No invented API	Total / 100
free-model-a	4	3	4	2	65
free-model-b	5	4	5	5	95
free-model-c	3	5	2	4	70

The ranking belongs to your prompts, not to the model. Re-run the harness when your workload changes, because a commit-message winner is not necessarily a test-generation winner.

Step 4: Deploy the winner on a free server, with a guard

The second free resource is useful here. After the harness picks a winner, expose it through a small scheduled job on the free server instance: every night, generate outputs for ten new prompts and post the score to a channel. That server is an evaluation box, not a production host. Rate limits and cold starts make it a poor place for user-facing traffic, so treat it as a measurement device.

A cron entry is enough:

0 3 * * * cd /srv/model-eval && ./run.sh && ./score.py > report.md

If the score drops a full standard deviation below the baseline, the workflow should open an issue automatically. That alert is the whole point: models change silently, and your decision table goes stale without anyone noticing.

Who should not use this harness

• Teams that cannot review 400 outputs by hand. Automated scoring looks scientific and is usually wrong.
• Teams that ship one model everywhere. A single score across commit messages, test generation, and code review hides severe per-task failures.
• Teams looking for "the best model" in general. This method answers only one question: which model wins on these 20 prompts, on this codebase, this month.

Limitations

Implementing this harness will touch real API costs. With a free tier the token budget is a constraint you should respect: 10 million tokens covers hundreds of evaluation runs, but a careless loop can burn it in an afternoon, so the runner records usage per call and fails early at 80 percent. The free server adds a real but modest compute ceiling; long batch runs may hit timeout walls, which is why the cron job is split per prompt rather than run as one giant request.

None of this validates correctness in a deep sense. A 95-point model can still generate a plausible but wrong test. The harness measures consistency, format fit, and contextual recall, not whether the logic matches the business requirements.

The takeaway

Next time someone proposes a model by name in a planning meeting, do not argue about reputation. Ask which 20 prompts it passed. The free tier from MonkeyCode is a reasonable place to run the gauntlet, and the free server gives you a cheap way to keep the score fresh. Neither detail changes the method: score, compare, decide, then re-score when the prompt set changes.