My prompts work fine. Until they don't.

There's this great post making rounds on HN today called "Please stop the AI confidence theater." The line that hit me hardest: "The first prompt is fun. The next thousand prompts is where real value lives."

I felt that. Because the thing nobody talks about is that prompts have a shelf life.

Last month I had a system prompt that worked flawlessly for weeks. Model responses were consistent, structured output parsed clean, everything in production. Then one Tuesday morning, without touching a single line of code, the outputs started looking different. Format changed. Content drifted. What used to work now needed extra hand-holding.

Turns out the upstream model had been quietly updated. No announcement, no changelog, no deprecation notice. Just... different behavior.

This happens more often than people realize. API providers push model updates silently. Rate limit thresholds change. Endpoints get deprecated with little warning. If you're running a production workload on a third-party model API, you're building on shifting sand.

Here's what I've learned from getting burned a few times:

Pin your model versions when possible. Some providers let you specify a fixed version tag instead of just the model name. That way a "latest" deployment doesn't silently change your outputs. If pinning isn't available, run regression tests after any provider maintenance window.

Monitor output quality, not just uptime. Your API might return HTTP 200 every time, but the content quality could be degrading. I now run a simple weekly script that feeds a fixed set of prompts to production models and diffs the responses. If they change more than a threshold, I investigate before customers notice.

Build fallback logic. If model A changes behavior, have model B ready to swap in. This means keeping multiple API keys warm and your code abstracted enough that switching models doesn't require a code deploy.

Log everything. When something breaks at 2 AM and you need to figure out what changed, you'll wish you had the last 1000 responses logged with timestamps. I use structured logging that captures model name, prompt hash, response hash, latency, and HTTP status for every call. When something drifts, I can pinpoint exactly when it started.

The dirty secret of AI development is that "it works" is a temporary state. Model behavior drifts, APIs change, pricing shifts, rate limits get stricter. The real skill isn't writing the first prompt — it's building systems that survive the next thousand.