From Model Selection to First Run

TL;DR

• Model selection is not about finding the best model. It is about finding the first model worth testing.
• Start with the task, user, input, output, quality bar, and deployment constraints.
• Run a baseline before optimizing prompts, adding retrieval, or fine-tuning.
• The first run is not proof. It is orientation.

Abstract

Model selection can become a research hole.

There are leaderboards, benchmark claims, model cards, context windows, parameter counts, quantization formats, licensing constraints, serving options, and community opinions. All of that matters. None of it replaces running the model on your task.

The goal of early model selection is not to find the final model.

The goal is to find a reasonable first candidate, run it against representative examples, inspect the outputs, and learn what the problem really requires.

This post gives a practical workflow for moving from model selection to first run.

Table of Contents

• Define the Task
• Pick Candidate Models
• Create a Small Test Set
• Run the Baseline
• Inspect Results
• Tune the Obvious Things
• Decide the Next Step
• Summary
• Next Steps

Define The Task

Do not start with the model list.

Start with the task.

Write down:

• input
• output
• user
• quality bar
• risk level
• deployment constraint
• expected volume
• latency need
• privacy requirement

For example:

Input: support ticket conversation. Output: three-sentence summary, category, and suggested next action. User: support agent. Quality bar: agent accepts or lightly edits 80% of outputs. Constraint: no customer data sent to unapproved services.

That description narrows the model search.

Without task definition, "best model" is meaningless.

Pick Candidate Models

Pick a small set of candidate models.

Evaluate:

• capability
• size
• cost
• licensing
• context length
• modality
• serving requirements
• ecosystem support
• data privacy constraints

For text workflows, you may compare a hosted model, an open-weight instruct model, and a smaller local model.

For multimodal workflows, you may compare a vision-language model against OCR plus text-only processing.

For domain behavior, you may test prompting first before deciding whether fine-tuning is needed. A workflow like instruction tuning Qwen3 should usually come after a baseline, not before one.

Do not compare ten models at the start. Compare two or three.

Create A Small Test Set

Build a test set before running the model.

Include:

• normal cases
• edge cases
• bad inputs
• short inputs
• long inputs
• real examples
• examples users care about

The test set does not need to be large. It needs to be representative enough to expose obvious failure modes.

If you are testing visual tasks, include images with variation: lighting, clutter, scale, occlusion, labels, and ambiguous objects. Posts like detect what you can name and modern VQA fine-tuning show why task-specific examples matter.

Run The Baseline

The first run should be boring.

Use:

• simple prompt
• default-ish parameters
• clear output format
• no elaborate orchestration
• no premature fine-tuning

Save:

• input
• prompt
• parameters
• model version
• output
• runtime
• cost where possible
• reviewer notes

The point is to create a baseline you can improve from.

If you change too many things before the first run, you will not know what mattered.

Inspect Results

Inspect outputs carefully.

Ask:

• What worked?
• What failed?
• What was inconsistent?
• What was too verbose?
• What was unsupported?
• What format broke?
• What required missing context?
• What surprised us?

Do not only count wins. The failures are the map.

If the model gets the task mostly right but lacks current knowledge, try retrieval.

If it follows the task but fails the format, improve instructions or output parsing.

If it cannot perform the task even with good context, try a stronger model or different model family.

If it performs well but is too expensive, test smaller models, quantization, caching, or routing.

Tune The Obvious Things

After the baseline, tune the obvious variables:

• prompt instructions
• examples
• output schema
• temperature
• max tokens
• retrieved context
• system message
• post-processing

Keep changes small. Run the same test set again.

The goal is not to perfect the model. It is to learn which lever matters.

If prompt changes solve most failures, you may not need fine-tuning. If retrieval solves knowledge gaps, you may not need a larger model. If output parsing keeps breaking, the product may need a stricter schema or different interaction pattern.

Decide The Next Step

End the first run with a decision.

Try Another Model

Do this if the model lacks core capability.

Add Retrieval

Do this if the model needs private, current, or source-specific information.

Fine-Tune

Do this if the model needs repeated task behavior, format adherence, or domain style that prompting does not reliably produce.

Add Human Review

Do this if the task is useful but risk requires oversight.

Stop

Do this if the task is not valuable, not measurable, or not safe enough.

Want this workflow applied to your model shortlist? We can help you move from model options to evidence.

Summary

Model selection is not a leaderboard exercise.

It is a workflow: define the task, pick a small candidate set, create representative examples, run a baseline, inspect outputs, tune the obvious things, and decide the next step.

The first run is not proof. It is orientation.

Next Steps

Before testing a model, write:

• the task
• the user
• the input
• the output
• the quality bar
• the constraints
• the first 20 examples

Then run the simplest baseline you can defend.

Need help turning an AI idea into a working system? We build, break, and explain AI. Then we hand it back working.

Frequently asked questions

How should I choose the first AI model to test for a new task?

Model selection is about finding the first model worth testing, not the best one. Define the task, user, input, output, quality bar, and deployment constraints first, then pick a small candidate set. Compare two or three models at the start rather than ten.

What should a baseline run of an AI model look like?

The first run should be boring: a simple prompt, default-ish parameters, a clear output format, no elaborate orchestration, and no premature fine-tuning. Save the input, prompt, parameters, model version, output, runtime, cost where possible, and reviewer notes so later changes can be compared against it.

How do I decide between retrieval, fine-tuning, or a different model after the first run?

Add retrieval if the model needs private, current, or source-specific information. Fine-tune if repeated task behavior, format adherence, or domain style is not reliably produced by prompting. Try another model if core capability is lacking, add human review if risk demands oversight, and stop if the task is not valuable.

How large does a test set need to be for the first AI model run?

It does not need to be large, only representative enough to expose obvious failure modes. Include normal cases, edge cases, bad inputs, short and long inputs, and real examples users care about. Having the first 20 examples ready before testing a model is the suggested starting point.

---

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.