Every "build an AI app" tutorial stops at the demo. Prompt goes in, response comes out, ship it. Nobody covers the part where that demo has real users and you're staring at a Gemini or OpenAI invoice trying to figure out which feature did that.
I've spent the last several months building the AI layer for a consumer app that fires vision and language calls on almost every user action. Not a chatbot getting occasional traffic. A product where the model call basically is the product. Here's what I actually had to build, in the order I had to build it.
Four problems, not one
Cost first, obviously. Tokens are metered, and past a certain volume, calling the model on every request means paying for answers you already gave someone five minutes ago.
Latency next. A cache hit lands in milliseconds. A cold model call takes seconds. Users feel that, especially anything camera-driven where they're staring at a loading spinner over their own kitchen counter.
Reliability too. Your provider will have an outage or a degraded day at some point. Not if, when.
And blast radius. One bug, one bot, one traffic spike, and a $50/day bill becomes a $5,000/day one while everyone's asleep.
You don't see any of this in a demo. It shows up with real traffic, and by then it's a lot more expensive to fix than it would've been to build right the first time.
Cache on what the query means, not its exact string
Key your cache off the literal request text and you've built something close to useless. "What can I make with chicken and rice" and "chicken and rice, what should I cook" mean the same thing and share almost no characters.
So embed the query, run a vector similarity search against everything already answered, and if something clears a high threshold, serve that instead of paying for another call. I use 0.95 cosine similarity as the bar.
async function checkSemanticCache(embedding: number[], taskType: string, threshold = 0.95) {
const { data } = await db.rpc("find_similar_response", {
query_embedding: embedding,
task_type: taskType,
similarity_threshold: threshold,
});
return data?.[0] ?? null;
}
Cut spend by 40-50%. Zero drop in answer quality anyone noticed. Turns out most real queries cluster around way fewer distinct intents than the raw text makes it look.
Here's the bug that ate an afternoon. Gemini's gemini-embedding-001 supports Matryoshka-style truncation, so you can ask for 768 dimensions instead of the native 3072 (which is what you want if your vector column is sized for 768). Only the full 3072-dim output comes pre-normalized to unit length though. Truncate it and normalization doesn't carry over automatically. Skip renormalizing and cosine similarity ordering degrades silently, no error, no crash, just slowly-worse nearest neighbors as the cache fills up. Fix is two lines:
const norm = Math.sqrt(values.reduce((sum, v) => sum + v * v, 0));
const normalized = norm > 0 ? values.map(v => v / norm) : values;
Would've saved myself that afternoon if I'd read the model card more carefully instead of assuming truncation preserves whatever guarantee the full vector has.
For images, hash before you even embed
If your product looks at photos, caching the prompt text misses the real opportunity. The same image, or something close enough, gets sent more than you'd guess. Retries, re-crops, someone photographing the same shelf twice because the app felt slow the first time and they assumed it didn't register.
A perceptual hash catches that before you've paid for an embedding, let alone a vision call. You want near-duplicates to collide, which is the opposite of what a cryptographic hash gives you, so check it with a Hamming distance tolerance instead of an exact match.
async function checkVisionCache(imageHash: string, taskType: string) {
const { data } = await db.rpc("find_by_hash", {
image_hash: imageHash,
task_type: taskType,
max_hamming_distance: 10,
});
return data?.[0] ?? null;
}
Vision calls are usually the priciest line item since images chew through prompt tokens fast. This layer pays for itself faster than anything else here.
Circuit breakers should know who's paying
Standard circuit breaker: open after N failures, half-open probe after a cooldown, closed again on success. Fine, that part's solved. What's less talked about: once the model degrades, deciding who gets the remaining capacity is a product decision dressed up as an infra one.
Mine holds back free-tier traffic first when degraded, so paying users keep priority on cached responses and whatever slots the half-open probe allows. "Our provider had a bad day" becomes "free users saw a fallback screen" instead of "everyone got a 503." Better line to write in a postmortem, and a better thing for a paying user to experience.
function shouldRejectRequest(breaker: CircuitBreaker, isPremiumUser: boolean): boolean {
return breaker.state !== "closed" && !isPremiumUser;
}
Put a real dollar number on the ceiling
Rate limits protect latency. They do nothing for your bank balance. You need a separate, literal daily USD cap that cuts all AI traffic the second it's hit, whether or not any single request looked abusive on its own.
if (await getGlobalDailySpend() >= DAILY_BUDGET_USD_CAP) {
return respond(503, { retryAfter: secondsUntilMidnightUtc() });
}
I alert at 80% as a warning and again at 100% when it actually kills traffic, deduplicated so one bad day doesn't page the team on every request that gets rejected. Cheapest insurance against a viral spike or a bug you haven't found yet.
One more bug worth flagging, because I've seen versions of it in more than one codebase. If you run multiple model tiers, cheap-and-fast next to pricier-and-better, and your cost tracker has one hardcoded price constant, every response gets billed at that single rate. The day your app actually calls the pricier model, spend tracking quietly under-counts, and the "hard" ceiling above stops being hard without anyone noticing until the real bill lands. Price by the model actually used, at request time. Not the one you had in mind when you wrote the constant six months ago.
What it added up to
Caching cut AI spend by 40-50%, no tradeoff anyone could feel. A degraded provider turned into a tiered slowdown instead of a full outage. And there's now an actual ceiling on how much damage one bad bug can do, separate from every other safeguard in the stack.
Caching, circuit breakers, budget caps. None of it is new. It's the layer that decides whether an AI feature is a product or a liability once real people depend on it. If the model call sits in the critical path of most requests your app handles, build this before you need it. The invoice is a worse teacher than the code review.
I'm building Shelfie, an AI-native Kitchen OS. This is the set of problems I ended up solving to run it at consumer scale.
Top comments (0)