Last quarter I watched a team of eight spend three weeks jumping between AI coding assistant trials, argue about model benchmarks the whole time, and then quietly go back to writing code by hand. Nothing they tested was bad. They'd just started the search in the wrong place — with the model, instead of with the work.
That's the mistake this checklist fixes. I've run this order on my own tool picks and with teams I've advised, and while it won't tell you which assistant is "best" in the abstract, it will stop you from paying for the wrong one. Work through it before you open a single pricing page.
Start with the tasks, not the job title
"Software development" isn't a use case. Spend a week jotting down the moments where you'd actually want help:
- writing tests for that ancient module nobody touches
- figuring out why CI broke overnight
- explaining a 40k-line repo to a new hire
- scaffolding the boring parts of a new service
- cleaning up a refactor you've been avoiding
Rank the list honestly. Assistants have genuinely different centres of gravity — some shine at generating new features, others at navigating and explaining what's already there, and the newer agent-style ones are best at multi-step chores you'd otherwise delegate to a junior. If most of your list is "help me understand and change this existing codebase," you're shopping for a different tool than someone whose list is mostly greenfield scripts.
Then check the data policy (yes, before the features)
This is the step people skip and regret later. For each candidate, find written answers — on a docs page, not in a sales call — to these:
- Is our code used for training by default? Can we turn that off, and on which plan?
- Where does code get processed, and how long is it kept?
- Is zero retention available, and does it cost extra?
- Who are the subprocessors?
A coding assistant sees your repository, your half-hidden secrets, and your internal architecture. "Trust us" isn't a data policy. If the answers only exist behind a demo request, that's an answer too.
Look at what's actually behind the model dropdown
Plenty of assistants now let you switch models — proprietary, open-weight, occasionally self-hosted. Two questions worth asking:
- Is the default model named, or served behind an alias the vendor never identifies?
- If the vendor upgrades it under you, how much notice do you get?
An unidentified model can't be compared on published specs, which means any benchmark claim about "our assistant" might not describe what you'll be using next month. (We run into this exact problem a lot at Oryxen when we re-check records — half the industry's model pages describe the marketing version, not the endpoint.) If you want to see how we handle it, here's our guide to choosing an AI coding assistant with the full evaluation frame.
Trial it in your repo, not theirs
Every trial should look the same, or you're comparing vibes:
- Pick one real task from your week-one list.
- Run it on a sandbox branch of your actual codebase — not the vendor's sample project.
- Note four things: time saved, edits accepted, edits rewritten, and how often it confidently invented an API that doesn't exist.
- Repeat with a different kind of task.
Two tasks are enough to expose a mismatch. If the tool flails on your framework, your conventions, your monorepo layout, a bigger model won't rescue it.
Count the review cost
Generated code is cheap. Reading it is not — so during the trial, notice whether the diffs arrive small and skimmable or as wall-of-text patches, whether the assistant explains its changes, and whether your junior engineers actually understand what they're approving.
I'd rather have 60 lines I can verify over a coffee than 400 I have to re-derive from scratch. On a complex repo, the second tool is slower, no matter what the token counts say.
Decide the kill criteria before you pay
Write this down while you're still objective:
- "If fewer than X trial tasks pass, we don't buy."
- "If the data terms change, we're out within days."
- "We check per-seat spend against real usage every month."
Assistant pricing in 2026 mixes seats, tokens, and rate-limited free tiers, so a plan that looks cheap in week one can surprise you at team scale. (Worth a read before you commit: how to verify a "free forever" tier before it becomes load-bearing.) Exit criteria turn a subscription back into a decision.
The order is the whole point
Workflow, then data policy, then model, then trial, then review cost, then exit criteria. Teams that start at step three end up benchmarking tools that were never going to fit. Teams that follow the order often find the less hyped assistant wins — because the choice was never really about whose demo was flashiest.
If you want a shortcut: Oryxen keeps a source-attributed index of 36,000+ AI and developer tools, each record with its source, fit, and last re-check date — so you can compare without wading through another listicle. Check it before you subscribe, not after.
Top comments (0)