DEV Community

Cover image for A/B testing an AI feature: what to actually measure beyond click-through rate
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

A/B testing an AI feature: what to actually measure beyond click-through rate

#shipping #abtesting #evaluation

Standard product A/B testing metrics — click-through, session length — don't capture whether an AI feature's answer was actually correct, complete, or useful. A prompt change that increases engagement could just as easily be making answers longer and less useful.

What to measure alongside the usual metrics

  • Explicit user feedback (thumbs up/down) rate and ratio, tied to the specific variant shown.
  • A sample of outputs from each variant, periodically reviewed by a human against a quality rubric.
  • Task completion, if applicable — did the user's underlying goal get resolved, not just did they engage with the response.

The trap to avoid

Optimising purely for engagement metrics can reward a model that hedges less, sounds more confident, and is wrong more often — engagement and correctness aren't the same axis, and conflating them is how AI features quietly get worse while metrics improve.

See testing LLM applications, a practical guide.

About Pranjul Rathour

Pranjul Rathour speaking into a microphone on stage at a MeetKats event
Speaking at a MeetKats event

Pranjul Rathour holding a microphone while answering a question during a session
Taking questions during a session

Pranjul Rathour in a grey suit at Dr. Virendra Swarup Institute of Computer Studies (VSICS), Kanpur
At VSICS, Kanpur

Pranjul Rathour in front of an Integral Startup Foundation hackathon backdrop
At an Integral Startup Foundation hackathon

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-11.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)