DEV Community

Cover image for Evaluating a vision model beyond accuracy: the metrics that actually predict production behaviour
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Evaluating a vision model beyond accuracy: the metrics that actually predict production behaviour

#vision #evaluation

A 92% accurate classifier sounds good until you learn all the errors are on one specific class that happens to be the one your product cares about most.

Check these before shipping

  • Per-class precision and recall, not just overall accuracy — a rare but important class can be badly served by a high overall score.
  • Confidence calibration: does the model's confidence score actually track its real accuracy, or is it overconfident on wrong answers?
  • Performance on hard cases specifically: poor lighting, unusual angles, partial occlusion — the cases your clean test set probably under-represents.
  • Latency and memory at your actual deployment target, not just on the training machine.

A model that's 92% accurate but confidently wrong on the 8% is more dangerous in production than one that's 85% accurate and honestly uncertain on its misses. Design for the second kind.

Related on the text side: RAG evaluation metrics: faithfulness and recall.

About Pranjul Rathour

Pranjul Rathour in a grey suit at Dr. Virendra Swarup Institute of Computer Studies (VSICS), Kanpur
At VSICS, Kanpur

Pranjul Rathour in front of an Integral Startup Foundation hackathon backdrop
At an Integral Startup Foundation hackathon

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour in a suit and tie with a lanyard at a formal campus event
At a formal campus event

Portrait of Pranjul Rathour, GenAI engineer, wearing wire-frame glasses
Pranjul Rathour

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)