How I Actually Use OpenRouter Compare to Pick a Model (Not Just Trust the Benchmarks)

I needed to pick a model for a support-ticket triage tool — classify incoming tickets by urgency and route them, nothing fancy, but wrong classifications are annoying enough in production that "good enough on a benchmark" wasn't a satisfying way to choose. My first instinct was to open OpenRouter's compare page, sort by whatever leaderboard looked most impressive, and grab the top result. That took about four minutes and produced a pick I ended up not using, because the leaderboard I was looking at was answering a different question than the one I actually had.

Here's what I mean by that, and the process I landed on instead.

What the compare and rankings pages actually tell you

OpenRouter's /compare page lets you put models side by side on benchmarks, price, context length, latency, uptime, and throughput — genuinely useful for narrowing a large catalog down to a shortlist fast. Separately, /rankings shows usage data: which models are getting the most token volume, which are trending week over week, and how spend is distributed across models and providers.

The part that tripped me up initially is that these two pages answer different questions, and it's easy to conflate them. Rankings tell you what's popular or growing — that's a signal about adoption and trust, not about whether a model is good at your specific task. A model can rank highly because it's cheap and widely used for simple chat, which tells you very little about whether it's the right pick for structured ticket classification. The rankings page itself is explicit that it doesn't rank by accuracy or reasoning ability — that's what the separate benchmarks page covers, with independently run evaluations across things like tool-calling under policy constraints and multi-step research tasks.

So the realistic use of these pages is: benchmarks and rankings together get you from "hundreds of models" to a shortlist of maybe four or five plausible candidates. Neither one picks the winner for your actual job.

Where the real comparison happens: your own prompts

Once I had a shortlist — a couple of general-purpose models and a couple of cheaper, faster ones that looked like they might be "good enough" for classification specifically — the only comparison that mattered was running my actual prompts against each one. Not a clean eval set, which makes every model look competent, but the messy real tickets that usually cause the failures I actually cared about: ambiguous wording, tickets that mix two issues, tickets in a second language mixed into an otherwise English conversation.

OpenRouter's Chat Playground supports this directly — you can add multiple models and send the same prompt to all of them side by side, which is the fastest way to eyeball differences before writing any code. For anything beyond a quick eyeball check, though, I wanted numbers I could actually compare across a batch of real tickets, so I scripted it:

// compare-models.js — sends the same real prompts to a shortlist of models
// and reports latency and output for manual scoring

const SHORTLIST = [
  "anthropic/claude-haiku-4.5",
  "openai/gpt-4o-mini",
  "qwen/qwen3.6-flash",
  "deepseek/deepseek-v4-flash",
];

async function callModel(model, prompt) {
  const start = Date.now();
  const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model,
      messages: [{ role: "user", content: prompt }],
    }),
  });
  const elapsedMs = Date.now() - start;
  const data = await response.json();
  return {
    model,
    elapsedMs,
    output: data.choices?.[0]?.message?.content ?? null,
    usage: data.usage ?? null,
  };
}

async function compareOnPrompt(prompt) {
  const results = await Promise.all(
    SHORTLIST.map((model) => callModel(model, prompt))
  );
  console.table(
    results.map((r) => ({
      model: r.model,
      ms: r.elapsedMs,
      tokens: r.usage?.total_tokens ?? "n/a",
      output: (r.output ?? "").slice(0, 80),
    }))
  );
  return results;
}

// Run this against a batch of real, messy examples — not a clean eval set
const realTickets = [
  "my invoice is wrong AND the app crashed twice today, need help asap",
  "just curious when the next feature update is coming out",
  // ...pull a real sample from your own ticket queue here
];

(async () => {
  for (const ticket of realTickets) {
    console.log(`\n--- ${ticket} ---`);
    await compareOnPrompt(ticket);
  }
})();

This isn't sophisticated — no scoring rubric baked in, no statistical significance testing. What it gave me was the actual thing I needed: side-by-side outputs and latency on the exact kind of input my tool would see, instead of a benchmark score computed on a task that wasn't mine.

The step most people skip: checking providers within a model

Something the compare page doesn't fully surface, but that turned out to matter for the model I eventually picked: the same model is often served by multiple upstream providers through OpenRouter, and they don't perform identically. Calling the list-model-endpoints part of the API for a specific model returns every provider currently serving it, along with price, context length, throughput, and latency over a recent window, plus uptime and quantization details. Two providers hosting the "same" model can differ meaningfully in speed and reliability — worth checking before assuming the model-level benchmark applies uniformly regardless of which provider actually serves your request.

For my use case this mattered more than I expected: one provider serving my chosen model had noticeably higher latency during the hours my ticket volume actually spikes, which isn't something any static benchmark would have caught.

What I actually picked, and why

I ended up going with a smaller, cheaper model than the one topping the general leaderboard, because on my actual ticket samples it classified urgency correctly just as often as the larger model, at a fraction of the per-token cost and with lower latency — the exact kind of result that "compare by benchmark" alone wouldn't have surfaced, since benchmark leaderboards aren't graded on my specific classification task.

That's really the whole lesson: OpenRouter's compare and rankings pages are genuinely good at narrowing hundreds of models down to a short, sane list fast. They are not a substitute for running your own messy, real inputs through that shortlist and looking at what actually comes back — and once you're down to a few candidates, that step is cheap enough that skipping it is the actual mistake, not a shortcut.

Where a routing decision comes in after the model decision

Once you've picked a model this way, a separate question shows up: which gateway you call it through. That's a different comparison than the one this post is about — model selection versus provider/gateway selection are two different decisions that get conflated a lot. If OpenRouter's routing and pricing already work for you, there's no reason to complicate that. If cost or provider mix become the actual friction point later, that's worth its own evaluation — gateways like RouteAI, for instance, expose some of the same underlying model families through a different fee structure — but that's a downstream decision, not something to solve at the same time as picking the model itself.

TL;DR: OpenRouter's compare and rankings pages are good for narrowing hundreds of models to a shortlist, but they answer "what's popular or benchmarks well" — not "what performs best on my specific task." Running real, messy prompts from your own use case through that shortlist, and checking per-provider stats via list-model-endpoints, is what actually decided my pick.

Website: https://www.fastrouteai.com