DEV Community

Cover image for CI for AI projects: what's actually worth testing automatically
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

CI for AI projects: what's actually worth testing automatically

#shipping #cicd #testing

'Is this a good LLM answer' doesn't fit a traditional pass/fail unit test, which is why a lot of student AI projects skip CI entirely. Most of the surrounding code still deserves automated testing.

What to actually test in CI

  • Response schema validation — does the output parse into the expected structure, regardless of content quality.
  • Retrieval pipeline mechanics — given a known document and query, is the expected chunk in the top results.
  • API contract tests — do your endpoints return the right status codes and shapes for valid and invalid input.
  • A small, fixed regression set of prompts with an LLM-as-judge or rule-based check, run on every merge to catch obvious quality drops.

What to accept can't be fully automated

Open-ended answer quality still needs periodic human review — CI catches structural breakage, not subtle quality regressions. Both matter; they're not the same job.

See testing LLM applications, a practical guide for the evaluation side this CI setup feeds into.

About Pranjul Rathour

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour in a suit and tie with a lanyard at a formal campus event
At a formal campus event

Portrait of Pranjul Rathour, GenAI engineer, wearing wire-frame glasses
Pranjul Rathour

Pranjul Rathour presenting on stage in a blue polo, with his Annapurna demo video on the screen behind him
Presenting Annapurna on stage

Pranjul Rathour giving a talk titled 'How and what I do', with demo videos of his products Vaidya and Annapurna on screen
Talking through the products he has shipped

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)