Gemini 4 Argon hit #1 on the Vals Index at 68.9% this week — the same week Nate B. Jones published a video titled "AI benchmarks are dead." Both are true. Benchmarks measure a model's ability on abstract, constructed tasks. Your business runs on specific, concrete ones: the quote you send a contractor, the follow-up email to a customer who ghosted, the cash-flow read on a slow month. A model that tops a leaderboard can still lose on all three.
The leaderboard exists to rank models against each other. It was never designed to answer the only question that matters to you: which of these tools should I actually pay for? That's a question about your work, and the only valid instrument is your work.
So stop watching the leaderboards. Run the test instead. Thirty minutes, three tasks, one honest scorecard.
Step 1: Pick three real tasks from your actual week
Not invented ones. Not idealized ones. Open your calendar and your sent folder from last week and pull three:
- One email thread. A customer reply that took you real effort — an objection, a complaint, a price pushback. Copy the incoming message, paste in the context you'd need (customer history, your pricing stance), and ask the tool to draft the reply.
- One quote or document. Something with structure and numbers — a quote, an estimate, an invoice summary, a statement of work. Feed it the inputs it needs and ask for the deliverable.
- One analysis. A judgment call: "which of these two suppliers is better for us and why," "review this month's numbers and tell me what changed," "summarize this lease and flag anything unusual."
Three different task types, on purpose: drafting, structured output, and analysis. Tools have different strengths across them, and you want to know where the tool fails, not just where it wins.
Step 2: Score against what you would have produced
This is the part most people skip, and it's the whole test. Run each task through the tool, then answer two questions per task:
- Usable as-is? Would you send/act on this output without meaningful edits? Yes or no.
- Better or worse than your version? Take five minutes, write your own version of the email (you know what you'd say), and compare honestly.
Then a 1–5 score per task, with a simple rubric: 1 = unusable, 3 = usable with edits, 5 = better than what I'd have written.
Fifteen points total across three tasks. That's your number. A tool that scores 13+ on your work is a tool worth licensing. A tool that scores 8 with all its points in one category is a specialist — useful, but not your daily driver. A tool that scores below 7 across the board isn't ready for your business, no matter what the index says.
Step 3: Buy the winner of YOUR leaderboard
Here's the counterintuitive part: the tool that wins your test might not be the one at the top of anyone's public index. It might be the free tier model that happened to nail your customer-email tone. It might be the mid-tier one that got the numbers right and the analysis sharper than yours. That's not a fluke — that's the measurement working. Benchmarks average across millions of tasks; you run thousands. Your sample is small but it's the only sample that pays you.
One honest caveat: your test measures today's model on this week's tasks. If the tool's model updates quarterly, your scores go stale too — rerun the three-task test each quarter before you renew. That's not a burden; it's a built-in escape hatch from lock-in you didn't notice you had.
The protocol at a glance
- Pull three real tasks from last week (one email, one structured document, one analysis).
- Run each through the tool with the real inputs and real context.
- Score each 1–5: unusable → usable-with-edits → better-than-mine.
- Total: 13+ = license it. 8–12 = specialist, keep it scoped. <7 = pass.
- Rerun quarterly, before every renewal.
Thirty minutes. Cheaper than a wrong annual license, and more honest than any benchmark. The leaderboard can tell you which model the industry loves this week. Only your own work can tell you which one your business should buy.
Want the full tool-selection protocol as a checklist? The AI Automation Starter Kit includes the three-task test template, scoring rubric, and quarterly rerun checklist that turns AI vendor selection into a 30-minute decision instead of a week of tab-hopping.
Top comments (0)