DEV Community: 李丽娜

How I Built a Cost-Aware Layer for LLM APIs: Caching, Routing, and Fallback

李丽娜 — Tue, 08 Sep 2026 08:48:28 +0000

Last month my API spend roughly tripled while traffic grew about 40%. Nothing was broken — no runaway loop, no leaked key. The bill was just... honest. I was paying the strongest model to answer "what's the status of order 12345," re-paying for the same prompt hundreds of times a day, and quietly paying again every time a request timed out and got retried.
This post is about the three layers I added to fix that. None of them are exotic, and all of them are things you can drop into an existing codebase this afternoon.
Why the bill grows faster than the traffic
Four things compound, and they're all invisible until you look:

Everything goes to the strongest model. A default model="gpt-4o" set once and never revisited turns your entire product into a premium-tier product.
Identical prompts get billed repeatedly. Support bots, batch jobs, CI runs — a large share of production traffic is the same request wearing a different timestamp.
Failures still cost money. A timeout at 30 seconds may have already consumed the tokens. Retry it and you pay twice for one answer.
Long static prefixes are charged every single call. A 3,000-token system prompt is billed on every request, forever. The fix isn't "use a cheaper model." That just moves the cost into quality complaints. The fix is three layers, in this order: cache → route → fall back. Layer 1: Caching There are two kinds, and most people only know one. Provider-side caching is automatic on most major APIs now. If the prefix of your request matches a recent one, the cached portion is billed at a steep discount and returns faster. You don't enable it — you qualify for it by structuring your prompts correctly. And here's the part almost everyone gets wrong: the static part must come first. # Wrong — the user's question sits in front of the docs, # so every request has a different prefix and never hits cache messages = [ {"role": "user", "content": user_question}, {"role": "system", "content": long_policy_docs}, ]

Right — stable prefix first, dynamic input last

messages = [
{"role": "system", "content": long_policy_docs},
{"role": "user", "content": user_question},
]
That one reordering is usually worth more than any other single change in this post. Put system prompts, tool definitions, and reference documents at the top. Put anything that changes per request at the bottom.
Your own cache handles the cases provider caching can't — exact repeats of a full request. Here's a small one with no dependencies:
import hashlib
import json
import time

class ResponseCache:
def init(self, ttl: int = 3600, max_items: int = 10_000):
self.ttl = ttl
self.max_items = max_items
self._store: dict[str, tuple[float, str]] = {}

@staticmethod
def key(model: str, messages: list, **kwargs) -> str:
    payload = json.dumps(
        {"model": model, "messages": messages, **kwargs},
        sort_keys=True,
        ensure_ascii=False,
        default=str,
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def get(self, key: str) -> str | None:
    hit = self._store.get(key)
    if hit and time.time() - hit[0] < self.ttl:
        return hit[1]
    self._store.pop(key, None)
    return None

def set(self, key: str, value: str) -> None:
    if len(self._store) >= self.max_items:
        self._store.pop(next(iter(self._store)), None)  # drop oldest
    self._store[key] = (time.time(), value)

Two rules for using it safely:

Only cache deterministic calls. If temperature is above 0, or the prompt contains a timestamp, a request ID, or anything time-sensitive, don't cache it. You'll serve stale or wrong answers and won't notice for weeks.
Start with the boring workloads. CI suites, eval runs, and batch jobs are where the hit rate is highest and the risk is lowest. Caching those costs you nothing and can remove a surprising line item. Layer 2: Routing by complexity Not every request deserves your most expensive model. The trick is picking a tier without spending a call to decide. import re

TIERS = {
"cheap": "gpt-4o-mini",
"balanced": "claude-3-5-sonnet",
"strong": "gpt-4o",
}

def route(prompt: str, has_tools: bool = False, turns: int = 1) -> str:
"""Pick a model tier from cheap signals — no extra LLM call required."""
if len(prompt) < 120 and not _STRONG_SIGNALS.search(prompt) and not has_tools:
return TIERS["cheap"]
if _STRONG_SIGNALS.search(prompt) or turns > 6:
return TIERS["strong"]
return TIERS["balanced"]
Short, no tools, no reasoning keywords → cheap tier. Explicit reasoning language or a long multi-turn conversation → strong tier. Everything else lands in the middle.
You can use a small model as a classifier instead of regex, and it works better on messy input. But be honest about the math: the classifier call costs money too. It only pays off when the price gap between your tiers is wide and your traffic is high enough to amortize it. For most teams, start with heuristics and only upgrade when you can see the heuristics failing.
One thing worth doing from day one: log which tier handled each request. That distribution is the single most useful artifact you'll have when someone asks "why is the bill going up?"
Layer 3: Fallback
Fallback gets filed under reliability, but it belongs in a cost post too — because the cheap fallback is almost always better than a failed request.
def call_with_fallback(client, messages, chain, **kwargs):
"""Try each model in order. First success wins."""
last_error = None
for attempt, model in enumerate(chain, start=1):
try:
resp = client.chat.completions.create(
model=model, messages=messages, **kwargs
)
resp._attempt = attempt # handy for the metrics layer below
return resp
except Exception as e:
last_error = e
continue
raise RuntimeError(f"all models failed: {chain}") from last_error

FALLBACK_CHAIN = ["gpt-4o", "claude-3-5-sonnet", "deepseek-chat"]
Keep retry and fallback as separate concepts, because they solve different problems:
Retry
Fallback
Same model?
Yes
No
Solves
Transient errors (429, 502, blip)
Sustained unavailability
Budget
2 attempts, with backoff
Walk the chain once
Retrying the same model three times against a provider that's genuinely down just burns money and latency. Detect the difference by error class: rate limits and 5xx are worth retrying; a persistent 400 or a hard outage is worth falling back on.
Putting it together
Here's the wrapper I actually run. It does all three layers and records enough to answer "where did the money go":
import os
import time
from dataclasses import dataclass, field
from openai import OpenAI

@dataclass
class CallRecord:
model: str
prompt_tokens: int
completion_tokens: int
cached: bool
attempt: int
latency_ms: int

class CostAwareClient:
def init(self, chain: list[str], cache_ttl: int = 3600):
self.client = OpenAI(
api_key=os.getenv("LLM_API_KEY"),
base_url=os.getenv("LLM_BASE_URL", "https://easy88ai.com/v1"),
)
self.chain = chain
self.cache = ResponseCache(ttl=cache_ttl)
self.records: list[CallRecord] = []

def complete(self, messages: list, **kwargs):
    model = route(messages[-1].get("content", ""))
    started = time.perf_counter()

    # 1. cache
    ck = self.cache.key(model, messages, **kwargs)
    if (hit := self.cache.get(ck)) is not None:
        self.records.append(CallRecord(model, 0, 0, True, 0, 0))
        return hit

    # 2. route + 3. fall back
    chain = [model] + [m for m in self.chain if m != model]
    try:
        resp = call_with_fallback(self.client, messages, chain, **kwargs)
    except Exception:
        raise

    text = resp.choices[0].message.content
    self.cache.set(ck, text)

    usage = getattr(resp, "usage", None)
    self.records.append(
        CallRecord(
            model=resp.model,
            prompt_tokens=getattr(usage, "prompt_tokens", 0) if usage else 0,
            completion_tokens=getattr(usage, "completion_tokens", 0) if usage else 0,
            cached=False,
            attempt=getattr(resp, "_attempt", 1),
            latency_ms=int((time.perf_counter() - started) * 1000),
        )
    )
    return text

def report(self) -> dict:
    total = len(self.records)
    if not total:
        return {}
    cached = sum(1 for r in self.records if r.cached)
    by_model: dict[str, int] = {}
    for r in self.records:
        by_model[r.model] = by_model.get(r.model, 0) + 1
    return {
        "calls": total,
        "cache_hit_rate": round(cached / total, 3),
        "tier_distribution": by_model,
        "retry_rate": round(
            sum(1 for r in self.records if r.attempt > 1) / total, 3
        ),
        "prompt_tokens": sum(r.prompt_tokens for r in self.records),
        "completion_tokens": sum(r.completion_tokens for r in self.records),
    }

How to measure whether it worked
I'm not going to paste a pricing table here — per-token prices change often, vary by tier and region, and any numbers I write will be stale within a few weeks. Pull current rates from your provider's pricing page, or better, read actual charges off your console.
What matters is the metric, not the unit price:
Cost per resolved task = total spend ÷ number of tasks actually completed successfully.
That's the number that moves when you do this right. Everything else is diagnostic:
Metric
Healthy
What a bad value tells you
Cache hit rate
20–40% on support/batch workloads
Your static prefix isn't actually first, or TTL is too short
Cheap-tier share
50%+ of calls
Your router is too conservative
Retry rate
< 5%
Timeouts are set too tight, or you're retrying non-retryable errors
Fallback rate
< 2%
A provider is degrading and you haven't noticed
Track these before you change anything. Without a baseline you can't tell a 40% saving from a slow week.
What I'd skip

Semantic caching, at first. Embedding-similarity caching sounds great and is genuinely powerful, but tuning the similarity threshold takes longer than the money it saves at low volume. Exact-match caching gets you most of the value with none of the false-positive debugging. Add semantic later, once you know your hit rate ceiling.
An LLM-based router on day one. Start with heuristics. You can always upgrade, and by then you'll have the logs to prove it's worth it.
Trimming prompts until quality breaks. Cutting a system prompt from 3,000 tokens to 400 saves real money until the model starts ignoring your output format. Measure quality alongside cost, not instead of it. Three things I'd tell myself at the start
Prompt order is free money. Moving static content to the front of your messages costs ten minutes and can be the single largest win available to you.
Route before you optimize. Knowing which tier handled each request turns cost conversations from guesswork into a pie chart.
Retry and fallback are different tools. Retry absorbs blips; fallback absorbs outages. Conflating them is how you end up paying for three failed attempts. None of this requires a framework or a vendor. It's a cache, a regex, and a loop — and it's the difference between a bill that scales with your product and one that scales with your inattention.

I'm building easy88ai, a unified API gateway that routes GPT, Claude, Gemini and 200+ models through one OpenAI-compatible endpoint — which is what I use as the base_url in the examples above. Happy to swap notes on LLM tooling in the comments.

How I Built a Tool-Calling AI Agent from Scratch (Python) — No Framework Required

李丽娜 — Mon, 07 Sep 2026 07:26:27 +0000

The hottest topic in 2026 might be: "AI can already write code, but how do we make it actually do things?" For the past two months I've offloaded a few repetitive manual tasks on my team — checking the weather, querying inventory, sending notifications — to an agent. The biggest lesson: whether an agent can actually work isn't about how smart the model is, it's about how cleanly you hand it the "tools."

This post skips the theory and walks through a runnable example of how Function Calling (a.k.a. tool calling) actually works, plus the real pitfalls I hit. The code is copy-paste ready — once you run it, you'll understand the layer underneath every major agent framework (LangChain, AutoGen, OpenAI Agents SDK).

1. First, what is Function Calling?

One sentence: the model doesn't execute your function — it outputs a structured "call request," and your code does the actual execution.

A lot of first-timers assume "the model runs the function for me." It doesn't. The real flow is:

You tell the model "here are the tools you can use" — what each one looks like (name, parameters, description).
The user asks something. The model decides "this needs a tool" and returns a JSON block: {"name": "get_weather", "arguments": {"city": "Xi'an"}}
Your program receives that JSON and actually calls get_weather("Xi'an"), then gets the result.
You feed the result back into the conversation, and the model composes a natural-language answer based on it.

The model only decides which tool and what arguments. Execution always stays in your hands. Once you internalize this, every agent framework is just a wrapper around this loop.

2. Environment setup

You only need an OpenAI-compatible Python SDK:

pip install openai

The key parameter when initializing the client is base_url. Any endpoint that follows the OpenAI API spec can plug in here — a local gateway, a cloud inference service, or your team's existing unified access layer. Swap in your address:

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-key",
    base_url="https://easy88ai.com/v1"  # replace with your OpenAI-compatible endpoint
)

Tip: Hardcoding your key like this is only for demo. In real projects use an env var os.getenv("OPENAI_API_KEY") and never commit keys to a repo.

3. Define a tool: give the model a "capability list"

Tools are described with JSON Schema. The model uses description to decide when to call a tool, so writing a clear description matters more than anything. Here's a weather-lookup tool:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Query the current weather for a specified city. Use when the user asks about the weather, temperature, or rainfall of a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "City name, e.g.: Xi'an, Shanghai, Beijing"
                    }
                },
                "required": ["city"]
            }
        }
    }
]

The description says when to use it, not what the function does. Those are two different things — the model uses the former to make decisions and the latter to understand boundaries.

4. Full runnable code

This is a minimal closed loop: send a message → the model may return a tool call → you execute it → feed the result back → the model summarizes. Copy and run:

import json
from openai import OpenAI

client = OpenAI(
    api_key="sk-your-key",
    base_url="https://easy88ai.com/v1"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Query the current weather for a specified city. Use when the user asks about the weather, temperature, or rainfall of a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name, e.g.: Xi'an"}
                },
                "required": ["city"]
            }
        }
    }
]

# A mock weather function (swap in your real API call in production)
def get_weather(city: str) -> str:
    fake_db = {"Xi'an": "Sunny, 26°C, southeast wind 2级", "Shanghai": "Cloudy, 30°C, high humidity"}
    return fake_db.get(city, f"{city}: weather data unavailable")

messages = [{"role": "user", "content": "How's the weather in Xi'an today? Good for going out?"}]

# Round 1: model decides whether to call a tool
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    tools=tools
)

choice = response.choices[0].message

# If the model wants to call a tool
if choice.tool_calls:
    # 1. Append the model's reply (with tool_calls) back — many people miss this step
    messages.append(choice)

    for call in choice.tool_calls:
        args = json.loads(call.function.arguments)
        result = get_weather(args["city"])

        # 2. Feed the tool result back as a tool message
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": result
        })

    # 3. Round 2: model generates the final answer based on the tool result
    final = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages
    )
    print(final.choices[0].message.content)
else:
    print(choice.content)

Output looks roughly like:

Xi'an is sunny today, 26°C with a light southeast wind — feels quite pleasant, good for going out.

See? The model never actually "checked" the weather. It just correctly decided to call get_weather('Xi'an'). The real lookup was done by your get_weather function.

5. The pitfalls I actually hit (the valuable part)

Running the example above is easy. Wiring it into real business logic is hard. Here are the ones that bit me:

Pitfall 1: Vague tool description → random calls. Early on I wrote the description as "get weather info," and the model tried to call the weather tool even when the user asked "where should I travel tomorrow?" Later I changed it to "Use when the user asks about the weather, temperature, or rainfall of a location," and false calls dropped sharply. The description is the model's decision boundary — spend 10 minutes polishing it.

Pitfall 2: Forgetting to echo back the assistant's tool_calls message. This is the #1 beginner error — tool_call_id mismatch. The message returned in round 1 carries tool_calls; you must append it to messages verbatim, then append the role: "tool" result. Skip any step and round 2 throws a 400.

Pitfall 3: No validation before executing. The model returns a JSON string — after parsing, always validate field types and required fields. I had one production incident: the model passed city as an array ["Xi'an", "Shanghai"], but my function only accepted a string and crashed. Now every tool entry gets a pydantic / hand-written validation layer first.

Pitfall 4: Tools need timeout and retry. Real tools sit behind external APIs — they hang, they're slow. The time I didn't set a timeout, a weather API blocked for 40 seconds and the whole agent froze. Now every tool call is wrapped with timeout + at most 2 retries; on failure it returns "tool temporarily unavailable" so the model degrades gracefully instead of the whole chain dying.

Pitfall 5: Multi-turn tool calls need a loop, not an if. The example above calls one tool. Real scenarios may chain several (check weather → check transit → check ticket price). The correct pattern wraps "model decides → execute → feed back" in a while loop that runs until the model stops returning tool_calls.

6. Advanced: organizing multiple tools

In production you'll have a dozen tools. Two lessons:

Don't overload tools. Stuffing 20 tools at once drops decision quality. Load relevant tools dynamically per scenario (e.g., the "ordering" scenario only gets menu/payment tools).
Use a strategy pattern to pick different models for different tasks. Cheap model for simple queries, big model for complex reasoning. Same idea as "multi-model routing" — but that's a topic for another post.

7. Three conclusions from actually building this

Function Calling is the foundation of an agent, not decoration. To make AI actually do work, cleanly describe your tools in JSON Schema first — bigger ROI than buying a more expensive model.
Execution power always stays on your side. The model only makes decisions; the real side effects (sending messages, mutating databases, calling APIs) must be guarded by your code with validation and timeouts. That's the production-ready baseline.
Hand-write the minimal loop before adopting a framework. LangChain / Agents SDK are just wrappers around that while loop above. Run it yourself once, then read the framework docs — comprehension speed is completely different.

The code in this post is the most bare-bones version. Once you internalize it, you'll read any agent framework's source and think "oh, so that's all it is."

I'm building easy88ai, a unified API gateway that routes GPT, Claude, Gemini and 200+ models through one OpenAI-compatible endpoint — which is what I use as the base_url in the examples above. Happy to swap notes on LLM tooling in the comments.