DEV Community

Cover image for Testing LLM applications: unit tests, golden sets and regression prompts
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Testing LLM applications: unit tests, golden sets and regression prompts

#testing #llmapps #ci #codequality

"You can't test AI" is what students say before they ship something that breaks on a Tuesday. You can test most of an LLM application exactly as you test any software, and the non-deterministic part you test statistically. FineTune Studio ships with 107 passing tests; here is how the layers divide.

Test the deterministic parts like normal software

  • Chunking: given this document, these chunks with these boundaries and this heading context.
  • Retrieval plumbing: the query goes to both indexes; results merge with the right fusion.
  • Schema validation: malformed model output is caught and reported, not passed downstream.
  • Fallback logic: a provider error triggers the next provider; a mid-stream error emits a typed event.

Mock the model here. You are testing your code, not the provider.

Golden prompts with expected properties

For the model-facing parts, keep a set of prompts with properties rather than exact strings: the answer cites at least one chunk; the answer contains the number 5; the answer refuses when the context is empty. Run them against the real model in a scheduled job, not on every commit, and alert on failures.

The evaluation set as a test

Your RAG evaluation set is a test with thresholds: recall@5 must not drop below 0.85, false-refusal rate must stay under 5%. Run it before a deploy that touches retrieval or prompts. When it fails, you have caught a regression before a user did.

In CI

  1. Unit and schema tests on every push — fast, mocked.
  2. Golden prompts and the evaluation set on a schedule and before release — slower, real model, budgeted.
  3. A dataset validation step for fine-tuning projects, so a bad file never reaches a GPU.

What students get wrong

Asserting exact model output, which breaks on every model update; and skipping tests for the deterministic 80% because "it's an AI project". The plumbing is where most bugs live, and it is entirely testable. I teach this as the last hour of the build workshop, because a project with tests is the one a recruiter trusts.

About Pranjul Rathour

Pranjul Rathour holding a microphone while answering a question during a session
Taking questions during a session

Pranjul Rathour in a grey suit at Dr. Virendra Swarup Institute of Computer Studies (VSICS), Kanpur
At VSICS, Kanpur

Pranjul Rathour in front of an Integral Startup Foundation hackathon backdrop
At an Integral Startup Foundation hackathon

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour in a suit and tie with a lanyard at a formal campus event
At a formal campus event

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)