What happens to your app when your LLM provider goes down

Most apps built on AI today call one provider. One API key, one endpoint, one company you are trusting to stay up. It works fine right until it doesn't.

I spent the last few months on one question. What happens when that provider fails? Not if. When. Every API goes down eventually, or gets slow, or rate limits you at the worst possible moment. And when it does, your users don't see "the provider had an outage." They see that your app broke.

So I started looking at how to serve LLM requests across more than one provider. The idea is old. We have done it for databases and servers for years. If one fails, route to another. Treat the model like the unreliable dependency it actually is.

The hard part is the middle. Providers speak different APIs, return different shapes, and fail in different ways. You need a layer that hides all of that from your app and makes the right call the moment something breaks. That is where most of my time went, and where most of the interesting problems live.

I am writing this because I could not find much on it said plainly. It is either a vendor promising you reliability, or a paper that takes an hour to read. I wanted the version in between.

This is my first post here. I plan to write about the unglamorous side of AI infrastructure, the parts that decide
whether a system holds up when real traffic hits it. If that is your kind of thing, stick around.

Prashant. I build reliable infrastructure for AI.