I built a small Python library to add retries, caching, fallbacks, budgets, and guardrails around native LLM SDK calls

Tanbir Ramim — Wed, 16 Sep 2026 12:54:33 +0000

I got tired of writing the same LLM boilerplate in every project, so I made a library

Three lines to call an LLM in a prototype. Then you go to production and suddenly you're writing the same 200 lines you wrote last time:

Retry logic because OpenAI throws 429s at you. Cost tracking because someone left a loop running and burned $40 on a Saturday. Caching because your support bot answers "how do I reset my password?" eight hundred times a day and you're paying for each one. PII scrubbing because customer emails keep showing up in prompts. Output parsing because the model returns markdown when you asked for JSON, and now your frontend is on fire.

I kept copy-pasting this stuff between projects until I finally pulled it into a library: callm.

It's just a decorator

I didn't want to learn a new client or replace the SDK. The whole idea is that you keep writing normal OpenAI/Anthropic code and slap a decorator on top:



python
import openai
from pydantic import BaseModel
from callm import callm

client = openai.OpenAI(max_retries=0)

class Summary(BaseModel):
    title: str
    bullets: list[str]

@callm(
    cache=True,
    retry=3,
    fallback=["anthropic/claude-sonnet-5"],
    max_cost=0.25,
    block_pii=True,
    detect_injection=True,
    output_schema=Summary,
)
def summarize(text: str):
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": text}],
    )

You call summarize(), you get back a validated Summary. That's it. Outside the decorator, your SDK works exactly like before — nothing monkey-patched, nothing weird.

What's actually going on under the hood

When your function runs, the call passes through a stack of middleware:

Security first — prompt injection scoring, then PII gets masked (jane@acme.com → [EMAIL_1]). This happens before anything hits the cache or the network.
Cache check — exact-match lookup based on the masked request, params, and schema.
Validation — response gets parsed into your Pydantic model. If it fails, the model gets called again with the validation errors attached (usually fixes it on the second try).
Fallback — if OpenAI is just not having it today, the request gets translated and sent to Claude instead. Your code still gets back an OpenAI ChatCompletion object, so nothing breaks downstream.
Cost guard — estimated cost is checked against max_cost before the request actually goes out. No more surprise bills.
Retry — exponential backoff with jitter, and it actually reads the provider's rate-limit headers instead of guessing.

Everything gets logged locally (tokens, cost, cache hits, retries — never your actual prompts), and you can run callm stats to see what you're spending per provider, model, or function.

A couple of design choices I want to explain

The cache is exact-match on purpose. I know semantic caching sounds cool, but think about it: "Summarize https://example.com/post-1" and "Summarize https://example.com/post-2" look almost identical to an embedding model. A semantic cache would cheerfully hand you the wrong summary and you wouldn't notice for a while. callm does support semantic matching if you want it, but you have to opt in, and it'll never match across different system prompts or conversation histories.

Fallback won't silently break your request. If you're using tools, images, or a structured response format, callm will only fall back to another model from the same provider. Translating tool schemas across providers is a can of worms, and I'd rather fail loudly than give you a subtly wrong result.

"Cool, but does it add latency?"

The repo has an offline benchmark that runs the real OpenAI SDK against a fake server:

What I measured

Overhead per call (default settings) +0.06 ms
Cost savings with cache, FAQ-style traffic 91% lower
Cost savings with cache, mostly unique prompts 2% lower
Success rate with 20% random 503s: plain SDK → retry=2 → + fallback 79.9% → 99.4% → 100%

The cache numbers are the obvious takeaway: caching is huge if your requests repeat, and basically irrelevant if they don't. Run callm stats on your own traffic before you count on those savings.

Give it a spin
bash
pip install "callm-toolkit[openai,anthropic,validation]"

There's an offline demo in the repo (examples/offline_demo.py) that walks through a retry, a cache hit, a fallback, a blocked expensive call, and a flagged injection — no API key needed.

GitHub: https://github.com/TanbirRamim/callm
Docs: https://tanbirramim.github.io/callm/

Author: Tanbir Hossain Ramim

It's MIT-licensed and pretty new. If you try it on a real workload and something breaks or feels off, open an issue — that kind of feedback is exactly what I need right now.

DEV Community: Tanbir Ramim

I built a small Python library to add retries, caching, fallbacks, budgets, and guardrails around native LLM SDK calls