DEV Community

Cover image for Error handling for flaky LLM APIs: what to do when the provider times out mid-demo
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Error handling for flaky LLM APIs: what to do when the provider times out mid-demo

#shipping #reliability #errors

Every LLM provider occasionally times out, rate-limits, or returns a response that doesn't parse as expected. An app with no plan for this fails visibly at the worst possible moment — usually during a demo.

A practical pattern

  1. Set a sane request timeout, shorter than your users' patience, not the provider's maximum.
  2. Retry transient failures (timeouts, 5xx errors) with exponential backoff, capped at 2-3 attempts.
  3. On a parse failure (malformed JSON, missing field), retry once with a stricter prompt before giving up.
  4. Always have a final fallback message — 'something went wrong, try again' — rather than a raw stack trace reaching the user.

For a demo specifically

Test what your app does when the API is deliberately unreachable, the night before you present it. A graceful failure message reads as professional; a crash reads as unfinished.

See multi-provider LLM fallback for handling this at the provider level, not just the request level.

About Pranjul Rathour

Pranjul Rathour in a black t-shirt holding a microphone in front of a chequered wall
On the mic

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour speaking from the podium at VSICS, Kanpur
At the VSICS podium, Kanpur

Pranjul Rathour holding a trophy and a certificate of merit after a win
Trophy and certificate after a win

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)