Deploying AI Apps to Production (Costs, Scaling, and Not Getting Blindsided)

Written by Syed Muhammad Ali Raza

Seven articles into this series and I've been quietly avoiding the least glamorous but most expensive lesson of them all. Everything we've built so far, RAG, agents, multi-agent pipelines, all of it, worked great running on my laptop with me as the only user, patiently waiting a couple seconds per response, not caring about cost because I ran maybe fifty requests a day.

Production is a different animal entirely. I found this out the fun way, checking a bill I did not expect and staring at it for a solid minute before I understood what happened. This article is everything I wish I'd known before that moment, real numbers, real code, real mistakes, so you can skip the expensive lesson and go straight to the useful one.

A real life example before the scary bill

Think about the difference between cooking dinner for your own family and catering a three hundred person wedding.

Cooking for four people, you can wing a lot of things. Run out of an ingredient, walk to the store, no big deal. Meal takes ten extra minutes because you got distracted, nobody's really bothered. You taste everything yourself before serving it, so quality control is just, well, you.

Catering a wedding for three hundred people is an entirely different operation. You can't just "walk to the store" if you run out of something mid service, you need to have calculated exactly how much of everything you need ahead of time, with buffer. If plating one dish takes ten extra seconds and you're doing it three hundred times, that's fifty minutes you didn't plan for, and guests are now standing around hungry. You genuinely cannot taste every single plate yourself before it goes out, you need a system, a head chef doing spot checks, a process that catches problems without you personally inspecting every single serving.

Every problem in this article is some version of that same jump, from cooking for four to catering for three hundred. Cost, that used to not matter because you were only making a few requests, suddenly matters enormously at scale. Latency that felt fine once becomes a real bottleneck across thousands of requests. And quality control that used to just be you eyeballing outputs now genuinely needs an actual system, which is exactly why the last article in this series was about evals, that's not a coincidence, it's a direct prerequisite for this one.

Problem one, the bill that makes you sit down

Here's the trap I fell into personally. Testing locally, I'd send maybe fifty requests a day, and even a slightly wasteful prompt costs pennies at that volume, so I genuinely never noticed. Then a small feature I built got actual daily users, a few hundred of them, each triggering several model calls per session, and the exact same "slightly wasteful" prompt I'd been running all along suddenly added up to real money, fast.

The fix starts with actually measuring what you're spending before you're surprised by it, not after.

Solution, actually track your token usage and cost per request

import anthropic

client = anthropic.Anthropic(api_key="your-api-key-here")

# rough per million token pricing, check current pricing for your
# exact model since this changes and varies by model
PRICING_PER_MILLION_TOKENS = {
    "input": 3.00,
    "output": 15.00
}

def call_with_cost_tracking(prompt, system=None):
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        system=system or "",
        messages=[{"role": "user", "content": prompt}]
    )

    input_tokens = response.usage.input_tokens
    output_tokens = response.usage.output_tokens

    input_cost = (input_tokens / 1_000_000) * PRICING_PER_MILLION_TOKENS["input"]
    output_cost = (output_tokens / 1_000_000) * PRICING_PER_MILLION_TOKENS["output"]
    total_cost = input_cost + output_cost

    print(f"Tokens: {input_tokens} in, {output_tokens} out, cost: ${total_cost:.5f}")

    return response.content[0].text, total_cost

Once I actually had this number sitting in front of me on every single call, a few very fixable problems jumped out immediately, ones that were invisible when I was just eyeballing responses.

Solution, stop resending things you don't need to

The single biggest fix for me personally was realizing how much repeated, unnecessary content I was stuffing into every single prompt. If your system prompt is long and mostly identical across requests, or you're re-sending the same reference documents on every single call in a conversation, you're paying full price for the same tokens over and over.

# BEFORE, wasteful, resending the entire reference document
# on every single question in a RAG style setup
def answer_wastefully(question, full_document):
    prompt = f"Document:\n{full_document}\n\nQuestion: {question}"
    # this document might be thousands of tokens, paid for
    # again on every single question asked about it


# BETTER, only retrieve and send the specific relevant chunks,
# exactly like we covered back in the RAG article in this series
def answer_efficiently(question, relevant_chunks_only):
    prompt = f"Context:\n{relevant_chunks_only}\n\nQuestion: {question}"
    # dramatically fewer tokens per call, same or better quality,
    # because the model isn't wading through irrelevant text either

If you skipped the RAG article earlier in this series, this is genuinely another reason that pattern matters, it's not just about accuracy, it directly controls your cost per request too.

Solution, cache responses you're going to get asked again

A huge chunk of real traffic repeats itself, the same or very similar questions asked by different users. Paying full price to regenerate an identical answer every single time is money left on the table.

import hashlib
import json

# a simple in memory cache, a real production system would use
# something like Redis so the cache survives restarts and is
# shared across multiple servers
response_cache = {}

def get_cache_key(prompt, system):
    combined = f"{system}||{prompt}"
    return hashlib.sha256(combined.encode()).hexdigest()

def call_with_caching(prompt, system=None):
    cache_key = get_cache_key(prompt, system or "")

    if cache_key in response_cache:
        print("Cache hit, no API call made, cost: $0.00000")
        return response_cache[cache_key]

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        system=system or "",
        messages=[{"role": "user", "content": prompt}]
    )
    result = response.content[0].text
    response_cache[cache_key] = result
    return result

For a genuinely dynamic conversational assistant this has limited value since exact repeats are rare, but for things like FAQ style questions, common lookups, or repeated tool inputs, caching alone can cut a meaningful chunk of your bill without touching quality at all.

Solution, don't use your most expensive model for every single task

This one felt obvious in hindsight but I genuinely didn't think about it until my bill forced me to. Not every task needs your most capable, most expensive model. Simple classification, basic extraction, short factual lookups, these often work perfectly well on a smaller, cheaper model, while you save the expensive one specifically for the genuinely hard reasoning tasks that actually need it.

def route_to_appropriate_model(task_type, prompt):
    # simple tasks go to a smaller, cheaper, faster model
    simple_tasks = ["classification", "extraction", "simple_lookup"]

    if task_type in simple_tasks:
        model = "claude-haiku-4-5"  # smaller and meaningfully cheaper
    else:
        model = "claude-sonnet-4-6"  # save the bigger model for genuinely hard reasoning

    response = client.messages.create(
        model=model,
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

This single change, routing easy tasks to a cheaper model and reserving the expensive one for genuinely hard work, ended up being one of the largest cost reductions I made, and it's honestly one of the easiest to actually implement.

Problem two, it worked fine with me testing it, then real traffic hit

A single developer testing locally sends one request, waits for the response, sends the next one. Real production traffic doesn't politely wait in line like that, dozens or hundreds of requests can hit your system at nearly the same moment, and AI APIs have rate limits, a maximum number of requests or tokens you're allowed to send per minute.

Solution, handle rate limits gracefully instead of just crashing

import time
import random

def call_with_retry(prompt, max_retries=5):
    for attempt in range(max_retries):
        try:
            response = client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1000,
                messages=[{"role": "user", "content": prompt}]
            )
            return response.content[0].text

        except anthropic.RateLimitError:
            # exponential backoff, wait longer after each failed
            # attempt, with a little randomness so many waiting
            # requests don't all retry at the exact same moment
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            print(f"Rate limited, waiting {wait_time:.1f}s before retry {attempt + 1}")
            time.sleep(wait_time)

        except anthropic.APIError as e:
            print(f"API error, {e}")
            if attempt == max_retries - 1:
                raise

    raise Exception("Max retries exceeded")

That randomness in the wait time matters more than it looks like, it's called jitter, and it exists specifically to stop a thundering herd problem, where a bunch of requests all failed at once and then all retry at the exact same instant, immediately triggering the rate limit again, in a loop.

Solution, don't let one slow request hold up everything else

If your app handles multiple users, one slow AI response absolutely should not freeze the entire application for everyone else waiting. This is exactly the kind of thing async handling and background job queues solve, letting requests genuinely happen in parallel instead of one at a time in a single blocking line.

import asyncio
from anthropic import AsyncAnthropic

async_client = AsyncAnthropic(api_key="your-api-key-here")

async def call_model_async(prompt):
    response = await async_client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

async def handle_multiple_requests(prompts):
    # these all run concurrently instead of one after another,
    # a real difference between a few seconds and a few minutes
    # once you're dealing with dozens of simultaneous requests
    tasks = [call_model_async(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    return results

# example usage
prompts = ["Summarize topic A", "Summarize topic B", "Summarize topic C"]
results = asyncio.run(handle_multiple_requests(prompts))

Problem three, you genuinely have no idea what's happening in production

This is the one that scared me the most once I actually thought about it seriously. Locally, if something goes wrong, I see it immediately, right there in my terminal. In production, with real users, on the other side of the world, at three in the morning, something can quietly break and you might not find out until a user complains days later, if they even bother complaining at all instead of just leaving.

Solution, log every single call with enough detail to actually debug it later

import logging
import json
import time
import uuid

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ai_calls")

def call_with_full_logging(prompt, user_id, system=None):
    request_id = str(uuid.uuid4())
    start_time = time.time()

    log_entry = {
        "request_id": request_id,
        "user_id": user_id,
        "prompt_preview": prompt[:150],
        "timestamp": time.time()
    }

    try:
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1000,
            system=system or "",
            messages=[{"role": "user", "content": prompt}]
        )

        duration = time.time() - start_time

        log_entry.update({
            "status": "success",
            "duration_seconds": round(duration, 2),
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens
        })
        logger.info(json.dumps(log_entry))

        return response.content[0].text

    except Exception as e:
        duration = time.time() - start_time
        log_entry.update({
            "status": "error",
            "duration_seconds": round(duration, 2),
            "error": str(e)
        })
        logger.error(json.dumps(log_entry))
        raise

That request_id matters a lot more than it looks like sitting there. When a user reports "something weird happened," being able to trace the exact request, exact prompt, exact response, exact timing, turns a vague complaint into something you can actually investigate, rather than trying to guess what might have gone wrong from a fuzzy description.

Solution, set up actual alerts, don't rely on noticing problems yourself

Logs sitting in a file nobody reads are basically decoration. You want automatic alerts when something crosses a threshold you've decided actually matters.

def check_and_alert(recent_calls):
    total = len(recent_calls)
    if total == 0:
        return

    errors = sum(1 for c in recent_calls if c["status"] == "error")
    error_rate = errors / total

    avg_duration = sum(c.get("duration_seconds", 0) for c in recent_calls) / total

    if error_rate > 0.05:
        send_alert(f"Error rate spiked to {error_rate * 100:.1f}% over last {total} calls")

    if avg_duration > 8:
        send_alert(f"Average response time hit {avg_duration:.1f}s, users are likely feeling this")


def send_alert(message):
    # in a real system this would post to Slack, PagerDuty, email,
    # whatever your team actually watches, not just a print statement
    print(f"ALERT: {message}")

Picking those specific threshold numbers, five percent error rate, eight seconds average response time, isn't something you get right on day one, you tune them based on what actually matters for your specific users. But having any concrete threshold that triggers a real notification is dramatically better than "I'll notice if something feels off," which is exactly the instinct that already failed me once.

Problem four, quality quietly degrading and nobody catching it

This connects directly to the evals article right before this one in the series. In production, models get updated by the provider, your own prompts drift as you make small tweaks over time, and traffic patterns shift as users ask things you didn't originally design for. Nothing crashes, nothing throws an error, the whole system just slowly gets worse in ways a simple uptime check will never catch.

Solution, run your eval suite continuously against live traffic, not just before deploying

import random

def sample_and_evaluate_production_traffic(recent_requests, sample_rate=0.05):
    # grab a random small slice of real production requests to
    # actually check quality on, running all of them would be
    # expensive and usually isn't necessary to catch real problems
    sample_size = max(1, int(len(recent_requests) * sample_rate))
    sample = random.sample(recent_requests, min(sample_size, len(recent_requests)))

    quality_scores = []
    for request in sample:
        # reuse the exact llm_judge function built in the evals article
        score_result = llm_judge(
            request["response"],
            "The response should directly and accurately address what the user asked"
        )
        quality_scores.append(score_result.get("score", 0))

    if quality_scores:
        average_quality = sum(quality_scores) / len(quality_scores)
        print(f"Sampled {len(sample)} live requests, average quality score: {average_quality:.2f}/5")

        if average_quality < 3.5:
            send_alert(f"Live traffic quality dropped to {average_quality:.2f}/5, investigate recent changes")

This is genuinely the piece most teams skip, and it's the one that catches the quietest, most dangerous kind of failure, nothing crashing, nothing erroring, just slowly getting worse while every dashboard still shows green.

Bringing it all together, a genuinely production ready call

Here's roughly how all of these pieces stack into one function that you could actually trust with real traffic.

def production_ready_call(prompt, user_id, system=None):
    # 1, check cache first, potentially save the call entirely
    cache_key = get_cache_key(prompt, system or "")
    if cache_key in response_cache:
        return response_cache[cache_key]

    # 2, route to the appropriately sized model for the task
    # 3, call with retry and backoff for rate limit resilience
    # 4, log everything with a traceable request id
    # 5, a small percentage of these results later get sampled
    #    into the quality eval check shown above

    result, cost = call_with_cost_tracking(prompt, system)
    response_cache[cache_key] = result
    return result

No single piece here is complicated on its own. Cost tracking, caching, retries, logging, alerting, continuous quality sampling, each one is a genuinely small, understandable addition. What actually makes something production ready isn't one clever trick, it's having all of these small, boring, unglamorous pieces in place together, so that when something inevitably goes wrong at three in the morning, you find out from an alert instead of from an angry user, and you have the actual data sitting there to fix it fast instead of guessing.

Bringing this back to the whole series

We started this series with a plain model just answering from memory, and we've ended up here, with a full production system, grounded in your own data, capable of taking real actions, defended against manipulation, coordinated across multiple specialized agents, continuously evaluated for quality, and now, actually operable at real scale without quietly bankrupting you or failing silently. That entire arc, from a fun local demo to something you'd genuinely trust in front of real users and a real budget, is honestly the exact journey most serious AI products go through, and now you've seen the whole thing, piece by piece.

---

If you've gotten a surprise AI bill of your own, I'd genuinely like to hear what caused it, misconfigured caching, a runaway agent loop, or something else entirely, there's always a good lesson buried in there.