Sooner or later, every engineer gets pulled into a software selection: a new ERP, inventory system, backup tool, CRM or contract management platform. And most teams start the same way — googling "best X software" and reading a ranked list that was written to earn affiliate commissions.
At Fratech, where I publish independent B2B software reviews, comparisons and how-to guides, the core idea is the opposite: the best software depends on your fit, not on a ranking. Here's the evaluation method, written as something you can run yourself.
1. Write requirements before you look at vendors
Once you've seen demos, every requirement quietly bends toward the product you liked. Start with a list and split it into two kinds:
- Must-haves — hard gates. If a product fails one, it's out, no matter how good it is elsewhere.
- Weighted criteria — things that matter to different degrees.
interface Criterion { id: string; label: string; weight: number } // weights sum to 1
interface Gate { id: string; label: string }
const gates: Gate[] = [
{ id: "sso", label: "SAML SSO" },
{ id: "export", label: "Full data export in an open format" },
];
const criteria: Criterion[] = [
{ id: "fit", label: "Industry/module fit", weight: 0.30 },
{ id: "integration", label: "APIs and integrations", weight: 0.20 },
{ id: "tco", label: "3-year total cost", weight: 0.20 },
{ id: "admin", label: "Admin effort", weight: 0.15 },
{ id: "vendor", label: "Vendor viability/support", weight: 0.15 },
];
"Who owns the data, and can we get it out?" belongs in the gates for almost any system of record.
2. Score on anchored scales
A bare 1–5 score means something different to every reviewer. Anchor each level with a description, so "3 on integrations" means the same thing to everyone:
| Score | Integrations |
|---|---|
| 1 | CSV import/export only |
| 3 | REST API covering core objects |
| 5 | Full API + webhooks + maintained connectors for our stack |
3. Compute — and keep the gates separate
type Scores = Record<string, number>; // criterion id -> 1..5
export function evaluate(
vendor: string,
passes: Record<string, boolean>,
scores: Scores
) {
const failed = gates.filter((g) => !passes[g.id]).map((g) => g.label);
if (failed.length) return { vendor, eligible: false, failed };
const total = criteria.reduce((sum, c) => sum + c.weight * (scores[c.id] ?? 0), 0);
return { vendor, eligible: true, score: Number(total.toFixed(2)) };
}
Never let a high weighted score compensate for a failed gate. That's the most common way teams end up with software that can't do the one thing they needed.
4. Stress-test the weights
If a small change in weights flips the winner, your decision is fragile — and the conversation should be about priorities, not scores:
function sensitivity(base: Criterion[], delta = 0.05) {
return base.map((c) => {
const bumped = base.map((x) =>
x.id === c.id ? { ...x, weight: x.weight + delta }
: { ...x, weight: x.weight - delta / (base.length - 1) });
return { bumped: c.id, ranking: rankVendors(bumped) };
});
}
5. Count the cost that isn't on the pricing page
Three-year total cost of ownership includes implementation, data migration, integrations, training and admin time — often more than the license itself. Put real numbers against each, even rough ones.
6. Prove it with your own data
The final step is a scripted proof of concept with your real workflows and a sample of your real data. Demos show the happy path; your data shows the edge cases.
Takeaways
- Separate must-have gates from weighted criteria.
- Anchor scores so reviewers mean the same thing.
- Check sensitivity before declaring a winner.
- Evaluate total cost and data ownership, not just features.
More buyer guides — ERP, inventory management, backup, CRM, contract lifecycle management and data governance — are at fratech.net. How does your team run software evaluations?
Top comments (0)