DEV Community

Cover image for Evaluating retrieval quality when you have no labelled query-document pairs
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Evaluating retrieval quality when you have no labelled query-document pairs

#rag #evaluation #coldstart

Recall@k and precision@k need labelled query-document relevance pairs, which almost no student project has on day one. Waiting to build that dataset before evaluating anything means shipping blind for weeks.

What works before you have labels

  • Write ten to twenty questions you know the answer to, by hand, and check whether the right chunk shows up in the top results — a small manual eval set beats no eval set.
  • Use an LLM to judge whether retrieved context is relevant to a query, as a rough automated proxy while you build a real labelled set over time.
  • Log real user queries and periodically spot-check a sample manually — this is where your eventual labelled set should come from.

Building toward real evaluation

Every manual check you do can become a labelled example if you save it. Within a few weeks of normal use, a genuine evaluation set accumulates almost for free.

See RAG evaluation metrics: faithfulness and recall.

About Pranjul Rathour

Pranjul Rathour holding a microphone while answering a question during a session
Taking questions during a session

Pranjul Rathour in a grey suit at Dr. Virendra Swarup Institute of Computer Studies (VSICS), Kanpur
At VSICS, Kanpur

Pranjul Rathour in front of an Integral Startup Foundation hackathon backdrop
At an Integral Startup Foundation hackathon

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour in a suit and tie with a lanyard at a formal campus event
At a formal campus event

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-11.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)