DEV Community: Antoinette C. Lennox

ToolOps - Most Developers Building AI Agents Are Solving the Wrong Problem. I Was One of Them

Antoinette C. Lennox — Mon, 01 Jun 2026 08:21:33 +0000

A genuine note to the community — not a product review.

There's a particular kind of developer frustration that doesn't have a name yet.

It's not a bug. It's not a deployment failure. It's not a model hallucination or a broken API contract. It's the feeling you get when you've built something technically correct — something that works, something users actually want — and you're still losing. Slowly, quietly, in ways that don't show up in your error logs.

You're losing to your own architecture.

I want to talk about that. And somewhere in the middle of talking about it, I'm going to mention a tool. When I do, I want you to notice something: you'll probably feel a reflex to discount what I'm saying, the way you discount anything that sounds like a recommendation. That reflex is correct. It has kept you from wasting time on overhyped libraries a hundred times.

But I'm going to ask you to hold it for a few minutes. Not because I want to sell you anything — I'm not affiliated with this project, I receive nothing for writing this — but because I spent months with that reflex firmly intact, solving the wrong problem in my own agent infrastructure, and I want to spare you the same detour.

The Problem That Doesn't Look Like a Problem

Here's what production AI agent development actually looks like, once you're past the demo phase.

You're making external calls — LLMs, APIs, databases, third-party tools. Those calls are slow, expensive, and unreliable. You know this. Every developer building in this space knows this. The standard response is to optimize the obvious things: compress your prompts, choose the right model tier, cache where you can.

The trap is that these optimizations feel sufficient. Your error rate is low. Your latency is acceptable. Your system, by most observable measures, is performing.

What you're not seeing — because it doesn't surface as a failure — is the structural waste underneath. In a multi-agent system, multiple agents fire identical or semantically equivalent queries to the same endpoints, independently, simultaneously, with no shared memory between them. Each one pays the full price for a result that already exists. The system isn't broken. It's just forgetting, constantly, at scale, and you're paying for every instance of that forgetting.

The reason this doesn't get talked about enough is simple: it doesn't produce errors. It produces invoices.

And because invoices are a business problem rather than an engineering problem, engineers often don't feel responsible for solving them — until the number gets large enough that someone asks a question in a meeting that's hard to answer.

I've been in that meeting. I've watched other developers sit through it. And I've noticed that every time, the real answer — the architectural answer — wasn't part of the conversation.

What the Architectural Answer Looks Like

The correct fix operates at the layer between your business logic and your external calls.

Not at the prompt level. Not at the model selection level. At the infrastructure layer — the one that manages what happens when a call is made, how results are stored, whether a redundant call is even necessary, and what happens when an endpoint fails.

Most teams build this layer themselves, from scratch, for every project. Custom cache managers. Hand-rolled retry logic. Circuit breakers copy-pasted from a Stack Overflow answer three projects ago. Pages of scaffolding that wraps three lines of actual work, grows beyond anyone's full understanding, and has to be rebuilt the next time.

A few months ago, I stopped rebuilding it.

The tool is called ToolOps. It's an open-source Python middleware SDK — a single decorator that wraps any async function and provides the full resilience layer automatically. Caching, retry logic, circuit breaking, request coalescing, semantic cache for natural language inputs, observability. Framework-agnostic. One install command.

pip install toolops

I'm not going to spend the rest of this article listing features. You can read the documentation. What I want to do instead is tell you what I think is actually interesting about this project — and why I've been thinking about it long after I integrated it.

The Part That's Worth Thinking About

Here's what stayed with me.

When I added ToolOps to a client's multi-agent system — a chatbot handling over ten thousand conversations per day, running paid tool integrations across a network of sub-agents — the cost reduction was significant. Real numbers, real impact. But that's not what I keep thinking about.

What I keep thinking about is that the fix took a weekend.

Everything my client needed — caching, resilience, request coalescing across concurrent agents — was already built, already tested, already production-ready. The integration was decorator placement and backend configuration. The agents didn't change. The business logic didn't change. The entire infrastructure problem, which had been bleeding money for months, was resolved in two days.

And then I thought: how many production AI systems are running right now with this exact problem unfixed, not because the solution is hard, but because nobody told the team it existed?

That question is why I'm writing this.

What I'm Actually Asking You to Do

Nothing dramatic.

If you're building AI agents in Python — with LangChain, CrewAI, LlamaIndex, raw OpenAI calls, anything — I'm asking you to spend twenty minutes with ToolOps. Install it. Wrap one tool function. Watch what happens to your observability output.

Not because you'll be amazed. Maybe you won't be. Maybe your infrastructure is already handled. Maybe this solves a problem you don't have yet.

But if you're at the scale where external call costs are becoming real — or you're heading there — I think you'll recognize the problem it solves immediately. And that recognition is usually followed by a specific thought:

I wish someone had told me about this six months ago.

That thought is the one I'm trying to put in front of you now, rather than later.

And If You Recognize the Problem

Tell someone.

Not as a product recommendation. As a practitioner. In the same way you'd mention a debugging technique that saved you hours, or an architectural pattern that finally made a messy codebase legible. The developer community runs on that kind of knowledge transfer — the informal, peer-to-peer kind that doesn't show up in documentation.

The teams who'd benefit most from knowing this exists are the ones currently hand-rolling their own infrastructure, burning through API credits, sitting through that billing meeting, wondering if the problem is their model choice when the problem is actually the layer underneath.

They're not going to find it through a product page. They're going to find it because someone on their team read something, tried something, and said — in a Slack message, a forum comment, a short post, a conversation at a conference — have you looked at this?

That's all I'm doing here.

GitHub: github.com/hedimanai-pro/toolops

Documentation: hedimanai.vercel.app/projects/toolops.html

If you try it and form a view — positive, skeptical, or somewhere in between — I'd genuinely like to read it in the comments. The most useful thing you can do for a project this early isn't to endorse it. It's to engage with it honestly, publicly, where other developers can find the conversation.

I Just Upgraded My AI Agent Stack — ToolOps Is Now the First Thing I Install on Every New Project

Antoinette C. Lennox — Mon, 01 Jun 2026 08:12:17 +0000

I don't usually write posts like this. I'm not a library evangelist. But every once in a while something in the tooling layer changes meaningfully enough that I feel like I'd be doing the community a disservice by staying quiet about it.

Toolops just hit stable release, and it's worth knowing about.

Quick background, if you're not familiar

ToolOps is a Python middleware SDK for AI agent infrastructure. The concept is simple: you wrap any async tool function in a single decorator, and it instantly gets production-grade caching, retry logic, circuit breaking, request coalescing, and observability — without touching your business logic.

I've been using it in client work for a few months. I've written about specific use cases before — a startup handling 10,000+ conversations a day that was quietly bleeding money on redundant API calls, multi-agent systems where sub-agents were independently firing the same paid tool queries with no shared memory between them. In both cases, ToolOps was the fix.

The stable release changes the installation story significantly, and that's what I want to talk about.

What just changed — and why it matters

Up until now, using ToolOps with specific database backends meant managing optional extras. PostgreSQL required toolops[postgres]. Semantic caching required toolops[semantic]. If you wanted everything, you installed toolops[all] and hoped your dependency resolver didn't complain.

That's gone now.

pip install toolops

That's the whole command. One install gives you:

PostgreSQL via asyncpg
SQLite via aiosqlite
Valkey / Redis via redis
MySQL and MariaDB via aiomysql
Semantic caching via sentence-transformers and numpy
OpenAI embeddings support
OpenTelemetry and Prometheus telemetry

All of it. Out of the box. No extras flags, no dependency juggling, no "why is semantic caching not working" debugging sessions at midnight because you forgot to install the right variant.

For anyone who has spent time managing Python dependency extras in CI pipelines or Docker images, you know how much hidden friction this removes.

The new backends are the real headline

Four new first-class cache backends shipped alongside this release, and the expansion matters more than it sounds.

SQLite is the one I'm most immediately glad to have. For local development, single-process tools, or serverless deployments where standing up a Redis instance is overkill — SQLite now works out of the box with full tag-based invalidation. It uses a two-table relational schema with indexed lookups, so it's not a toy implementation. It's genuinely fast for the use cases it fits.

Valkey is the open-source Redis fork that's been gaining serious traction since Redis changed its licensing. If your infrastructure team has already migrated — or is planning to — ToolOps now supports it natively with an async connection pool and O(1) tag-based invalidation using Sets.

RedisCache is provided as a clean alias that inherits from the Valkey backend. If your existing deployment scripts reference Redis by name, nothing breaks. The nomenclature is preserved; the backend is unified.

MySQL and MariaDB round out the database support. Compatible with MySQL 8.0+ and MariaDB 10.5+, normalized dual-table schema, transactional commits, upsert semantics via ON DUPLICATE KEY UPDATE. For teams already running MySQL in production — which is most teams, if you're honest about the industry — this removes the last remaining reason to reach for a separate caching solution.

What this means in practice

Before this release, choosing a cache backend was a deployment decision that also became an install decision. PostgreSQL for one project, Redis for another — each one required a different install command, different CI configuration, different Dockerfile lines.

Now you pick the backend at configuration time, not install time:

ToolOpsManager.register_backend(
    "main_cache",
    MySQLCache(
        host="localhost",
        database="toolops_cache",
        user="root",
        password="password"
    )
)

The decorator stays identical regardless of backend:

@readonly(cache_backend="main_cache", cache_ttl=3600, retry_count=3)
async def run_tool(query: str) -> str:
    return await paid_tool.call(query)

Backend is a configuration choice. Decorator is business logic. They don't bleed into each other.

Who should care about this right now

Teams already using ToolOps: Upgrade is a single command — pip install --upgrade toolops — and requires zero code changes. Your existing decorators, backends, and CLI commands work exactly as before. You can simplify your requirements.txt or pyproject.toml immediately.

Teams running MySQL or MariaDB in production: You now have a first-class, native cache backend that integrates directly into the infrastructure you already operate. No Redis sidecar, no extra managed service, no additional monthly cost.

Teams doing local AI development: SQLite backend means you can run a fully-featured, properly cached agent pipeline with zero external infrastructure. It's the fastest possible path from "I want to test this tool" to a working, resilient, observable local environment.

Teams building multi-agent systems at scale: None of the core features changed. Request coalescing, semantic caching, circuit breaking — all still there, all still the core reason to use this. The stable release just means you can trust the foundation underneath them.

A note on the codebase itself

For anyone who evaluates libraries by looking at the internals before adopting them — the caching subsystem was refactored into a clean modular package structure as part of this release. Each backend lives in its own module. The interface contracts are properly typed and pass mypy strict mode across all nine cache modules with zero errors. The test matrix covers each new backend with unit and integration tests.

It's the kind of release that signals a project transitioning from "promising experiment" to "something you can actually build on." The internals match the ambitions.

Final thought

The tooling layer for AI agents has been a mess for a long time — not because people weren't trying, but because the problem surface is broad and the solutions kept arriving piecemeal. Caching here, retry logic there, a circuit breaker you wrote yourself two projects ago and copy-paste ever since.

ToolOps has been quietly assembling those pieces into a coherent whole. The stable release, and the database expansion that came with it, is the point where I'd say it's no longer a library you evaluate — it's a library you add to the stack and stop thinking about.

That's the highest compliment I can give infrastructure.

GitHub: github.com/hedimanai-pro/toolops

Happy to answer questions about specific integration patterns or use cases in the comments — particularly around multi-agent setups and high-volume pipelines.

ToolOps, The Night the Dashboard Turned Red

Antoinette C. Lennox — Mon, 25 May 2026 08:23:36 +0000

The alert came at 2:17 a.m.

Marcus didn't hear it at first. He was already awake, sitting at the kitchen table in the blue light of his laptop, watching numbers refresh that he no longer wanted to see. Three months since launch. Forty thousand users. A waitlist that kept growing. By every measure that was supposed to matter, Aria — his AI-powered research assistant — was a success.

The billing dashboard said otherwise.

He'd built Aria to do what human researchers did, only faster. You asked her a question, she deployed a team of sub-agents — each one specialized, each one pulling from a different paid data source — and within seconds you had an answer that would have taken a junior analyst two hours to compile. The product was elegant. The demo converted. Investors had used words like inevitable.

What investors don't model for, Marcus had learned, is what happens when something inevitable reaches ten thousand conversations in a single day.

His phone buzzed. Then buzzed again. Then held a continuous, low vibration that meant the alert had escalated from warning to critical. He looked at the screen.

API spend: $4,340. Today.

He set the phone face-down on the table.

The problem, he knew, wasn't that the system was broken. It was that the system was working exactly as designed. Every sub-agent was doing its job. Every tool call was legitimate. Somewhere inside those ten thousand daily conversations, the same searches were being fired independently, simultaneously, by agents that had no way of knowing another agent had asked the same question four seconds earlier. Three agents, three API calls, three invoices — for one piece of information that had already been retrieved.

At scale, that math became its own kind of catastrophe.

He'd tried to fix it himself, six weeks earlier. Stayed up three nights writing a custom cache layer, proud of the architecture, satisfied with the elegance of the solution. It held for eleven days. Then a memory leak he hadn't anticipated took down the entire pipeline at peak traffic, and he spent the following morning explaining to users why Aria had gone silent for four hours.

The custom fix was now disabled. The billing clock was running again.

At 3:05 a.m., he sent me a message. We'd worked together briefly the year before, on an earlier project that never launched. The message was two sentences.

I think I've built something people actually want. I'm not sure I can afford to keep running it.

I called him the next morning.

He walked me through the architecture slowly, with the particular exhaustion of someone who has explained a problem so many times that the explanation itself has started to feel like the problem. The sub-agent network. The paid tool integrations. The volume. The redundancy he couldn't eliminate without building infrastructure he didn't have time to build.

I let him finish. Then I asked him one question.

"What's sitting between your agents and your APIs?"

Silence.

"Nothing," he said. "Just the calls."

That was the problem. Not the product, not the model choices, not the architecture of the agents themselves. The layer between the business logic and the external world was empty — no caching, no coalescing, no circuit breaking, no shared memory. Every call landed cold. Every duplicate query cost real money as if it were the first time it had ever been asked.

I told him about ToolOps. I'd been using it in my own work for a few months — a Python middleware SDK that wraps tool functions in a single decorator and handles the entire resilience layer automatically. Caching, retry logic, circuit breaking, and observability. For a multi-agent system like his, the critical feature was request coalescing: when multiple agents fire the same endpoint simultaneously, ToolOps executes the call once and distributes the result. Semantic caching meant that queries with identical intent but different phrasing — the kind a chatbot generates by the thousands — hit the same cache entry rather than triggering separate calls.

He was quiet for a moment.

"How long to integrate?"

"A decorator per tool function," I said. "The agents don't change. The business logic doesn't change. You're wrapping the calls, not rewriting the system."

We shipped the integration over a weekend.

I remember watching Marcus go quiet on the call as the first full day of data came in. Not the anxious quiet of someone bracing for bad news. Something slower and more private — the particular stillness of a person watching a problem they'd carried for months simply stop.

The duplicate calls that had been firing in parallel were coalescing into single upstream requests. The semantic cache was catching intent matches his exact-match logic had never seen. The circuit breaker — which he'd never had before — flagged one unreliable third-party endpoint that had been silently degrading his response quality for weeks, long before it showed up as an error.

His spend that day was a fraction of what it had been.

He didn't say much. He didn't need to. He took a screenshot of the dashboard — the same dashboard he'd been watching turn red every morning for three months — and sent it to me without a caption.

The numbers were green.

A few weeks later, over coffee, he told me what he'd done with the savings.

Not pocketed them. Not used them to extend runway. He'd hired a front-end developer he'd been putting off, shipped a feature that had been sitting in the backlog since January, and started building an integration his enterprise users had been asking about since launch.

The infrastructure efficiency hadn't just saved the product. It had funded the next version of it.

He said something I've thought about since.

"I kept thinking the problem was that we were growing too fast. But it wasn't that. The problem was that nothing was remembering anything."

That's it, really. That's the whole lesson, told better than I could have told it.

At scale, memory is money. And the systems that forget — the ones that fire every call cold, that treat every question as if it's never been asked before — pay for that forgetting, every single day, in ways that only show up when you're staring at a dashboard at 2 a.m. trying to understand how something this successful can feel this fragile.

Aria is still running. Still growing. Still handling her ten thousand conversations a day.

She just remembers now.

GitHub: https://github.com/hedimanai-pro/toolops

ToolOps Saved My Client’s Startup. Here’s the Architecture Problem Nobody Talks About.

Antoinette C. Lennox — Fri, 22 May 2026 07:17:14 +0000

A field report from the production layer.

The call came at a bad time — or maybe exactly the right time.

My client had built something that was actually working. An AI-powered chatbot handling web searches, pulling from multiple paid tool integrations, serving real users at real volume. The product was live. Users were engaged. By every surface metric, the startup was on track.

Except the infrastructure was silently bleeding money.

I've spent years helping teams build production-ready AI applications. I've seen the full range: systems that collapse under their first real traffic spike, systems that work beautifully at demo scale and become unmanageable at ten times that, and systems like my client's — architecturally sound, functionally impressive, and quietly unsustainable because of a single layer nobody had addressed.

When we got on the call and he walked me through the numbers, it clicked immediately.

The Architecture Behind the Problem

The system wasn't simple. It was never going to be simple — the product didn't allow for it.

The chatbot operated through a network of sub-agents. Each conversation didn't trigger one process; it triggered a cascade. Each sub-agent had its own set of tools — search APIs, data services, third-party integrations — and every single one of those tools billed per call. The architecture was correct for the product requirements. But there was no shared intelligence between the agents. No layer that could recognize when the same query had already been answered sixty seconds ago. No mechanism to prevent three sub-agents in three parallel conversations from independently firing the same API call, paying three times for one piece of information.

At 10,000 conversations a day, that redundancy compounds fast.

Here's what makes this problem invisible until it isn't: every individual call looks justified. The sub-agent needed that data. The tool returned the right result. Nothing failed. The system log shows clean executions from top to bottom. The billing dashboard tells a different story — one that only becomes legible when you step back and look at the aggregate, at the patterns, at the sheer volume of duplicate intent spread across thousands of simultaneous conversations.

This is the infrastructure problem nobody talks about, because it doesn't produce errors. It produces invoices.

The Standard Fix — And Why It Doesn't Scale

Before I found a better solution, I would have approached this the way I'd always approached it: write a custom cache layer per tool.

I've done it enough times to know the real cost of that approach. A proper cache implementation for a single tool — one that handles cache logic correctly, manages TTL, deals with edge cases, and doesn't introduce new failure modes — requires at minimum 20 lines of code. For a system with multiple paid tools spread across multiple sub-agents, you're writing that infrastructure over and over again, for every tool, maintained separately, tested separately, debugged separately.

That's weeks of engineering time that produces no product value. It makes the system more complex. It gives you more surface area for failure. And it still doesn't solve the multi-agent problem cleanly, because hand-rolled cache layers don't naturally share state across independently running sub-agents.

The deeper issue is philosophical: caching, retry logic, circuit breaking, and observability aren't features you bolt onto a production AI system after the fact. They're the foundation. But the tooling to implement that foundation properly hadn't existed in a form that was fast to integrate — until recently.

Why ToolOps Was the Right Call

I'd been using ToolOps in my own work before this client came to me. It's a Python middleware SDK built specifically for AI agent infrastructure — it wraps any async function in a single decorator and handles caching, retry logic, circuit breaking, and observability automatically, without touching your business logic.

For a multi-agent system running paid tools at high volume, the critical feature is request coalescing: when multiple agents call the same endpoint simultaneously, ToolOps executes the actual API call once and distributes the result across all callers. In a system handling thousands of daily conversations with overlapping query patterns — which is exactly what my client had — this collapses cascading duplicate calls into a fraction of the original volume.

The semantic caching layer compounds the effect. Unlike exact-match caching, it recognizes intent rather than literal string matches. A chatbot fielding 10,000 conversations a day generates enormous natural language variety around a relatively finite set of underlying queries. Most caching systems miss that entirely. Semantic caching catches it.

The integration required no architectural overhaul. One decorator per tool function:

@readonly(cache_backend="semantic", cache_ttl=3600, retry_count=3)
async def run_tool(query: str) -> str:
    return await paid_tool.call(query)

Every tool in the system, wrapped. The sub-agents kept running exactly as before. The layer between them and the APIs changed everything.

What Actually Changed

The cost reduction was significant — significant enough that my client didn't just stabilize the unit economics of his existing system. He had runway he hadn't had before.

What he did with it matters more than the savings themselves: he reinvested directly into the product. Better capabilities. Improvements that had been on the roadmap for months, waiting for budget that kept getting consumed by infrastructure overhead. The efficiency gain at the tooling layer funded the next stage of the build.

That's the outcome that's hard to explain to someone who hasn't seen it happen. Optimizing your token count gets you incremental savings on one line of the bill. Fixing the infrastructure layer changes what the business can do.

There's something else that changed, quieter but just as real: the operational experience of running the system. Fewer unexpected spikes. A circuit breaker that detects failing endpoints and stops hammering them before the errors cascade. A single CLI command — toolops doctor — that validates backend health and reports state without digging through logs. For a startup at this scale, that kind of operational clarity isn't a convenience. It's the difference between a system you can manage and one that manages you.

The Pattern I Keep Seeing

This client's situation wasn't unusual. It's representative of a failure mode I encounter consistently in production AI systems: the product architecture is solid, the model selection is thoughtful, and the infrastructure layer — the one that sits between the business logic and the external world — is either missing entirely or stitched together from custom code that's grown beyond anyone's full understanding.

The mistake isn't negligence. It's sequencing. You build the product first, which is correct. You defer the infrastructure, which is understandable. And then the system scales, and the infrastructure debt becomes the most expensive line on the bill.

Multi-agent architectures make this worse by nature. Every agent you add multiplies the external call volume. Every paid tool you integrate adds another billing surface. The redundancy that's invisible at demo scale becomes structurally significant at production scale — not because anything broke, but because nothing in the system was built to recognize and eliminate it.

The teams that will run efficiently at scale — as models get cheaper, as agent architectures grow more complex, as API-dependent products become the norm — are the ones who treat the infrastructure layer as a first-class concern from the beginning. Not an afterthought, not a future sprint, not something to fix when the bill becomes impossible to ignore.

The caching layer is not a performance optimization. It's an architectural decision about how much of your operating cost you're willing to pay twice.

I work with teams building production AI systems and help them move from prototype to production-ready architecture. If this pattern sounds familiar in your own stack, I'd be glad to hear about it in the comments.

Stack: ToolOps: github.com/hedimanai-pro/toolops

ToolOps: The Python Middleware That's Quietly Cutting AI Infrastructure Costs for Teams Running at Scale

Antoinette C. Lennox — Wed, 20 May 2026 09:20:13 +0000

There's a number most AI teams discover too late.

It's not in the documentation. It's not in the LLM provider's pricing FAQ. It shows up on the bill — usually during a routine review, usually after a production deployment that "went well." According to CloudZero's research, average monthly AI spend jumped from $63,000 in 2024 to $85,500 in 2025 — a 36% increase. And for the teams that figure out what's actually driving that number, the culprit is almost never the model they chose. It's the calls they didn't need to make.

This article is about a Python SDK called ToolOps that I started using a few months ago. I'm not affiliated with the project. I'm a developer who was burning through LLM credits faster than I should have been, tried a few solutions, and eventually found one that actually worked.

The Real Cost of Production AI Agents

Token prices are falling. LLM API prices dropped approximately 80% between early 2025 and early 2026 — GPT-4o input pricing fell from $5.00 to $2.50 per million tokens, and newer models offer input at just $0.55/MTok. On paper, that sounds like great news for anyone building AI systems.

In practice, it barely moves the needle if your architecture is inefficient.

Here's why: each tool call in an agent adds the full message history back into the prompt. A 5-step agent with a 30,000-token system prompt can pay for that prompt five or more times per request. Now multiply that by concurrent agents, parallel pipelines, and repetitive queries that ask effectively the same thing in slightly different words. The token price per million is irrelevant. You're paying for the same computation over and over.

The cheapest API call is the one you don't make. Efficient prompts, smart caching, and appropriate model selection matter more than provider choice. That principle sounds obvious until you're the one writing the infrastructure to enforce it — at which point you realize it's neither simple nor fast.

What Most Teams Do (And Why It Doesn't Scale)

The standard approach to managing these costs involves writing custom infrastructure: a cache layer, retry logic, a circuit breaker for when APIs go down, observability hooks so you can debug what's happening, and concurrency controls to prevent 40 agents from hammering the same endpoint in parallel.

Every piece of that is necessary. And every piece of it is code you write yourself, from scratch, for each project.

When you build AI agents, external calls — LLMs, APIs, databases — are expensive, unreliable, and slow. ToolOps eliminates the boilerplate: it's a framework-agnostic middleware SDK that wraps any Python function in a single decorator, instantly upgrading it with caching, resilience, observability, and concurrency control.

That's the pitch. Here's what it actually looks like in code.

One Decorator. Everything Else Is Handled.

The before/after is stark.

Before ToolOps, a properly resilient LLM tool call involves cache management, retry logic, circuit breaker state, timeout handling, and tracing — spread across dozens of lines of infrastructure code that wraps three lines of actual work.

After:

@readonly(cache_backend="semantic", cache_ttl=3600, retry_count=3)
async def ask_llm(query: str) -> str:
    return await llm.complete(query)

Automatically cached, retried, and traced. Every agent developer hits a wall when moving from demo to production — and that one decorator is what stands between a clean codebase and an unmaintainable nest of infrastructure scaffolding.

The @readonly decorator signals that this function is idempotent — safe to cache and retry. The @readonly / @sideeffect decorator split is opinionated in a good way: it forces you to be explicit about whether a tool call is idempotent or not, which matters a lot when deciding what's safe to cache and retry.

The Feature That Makes the Biggest Difference at Scale

For teams running multi-agent systems — which is increasingly the default architecture for any serious AI workflow — there's one ToolOps feature that changes the economics of high-volume operations more than anything else.

Request coalescing.

If 50 agents call the same endpoint simultaneously, ToolOps executes the real API call once and multicasts the result.

At first pass, this sounds like a minor optimization. It's not. In a production pipeline where multiple agents are processing similar inputs concurrently, this collapses what would be dozens of identical upstream requests into a single one. In a 50-concurrent-call benchmark, 50 calls collapsed to 1 upstream request — the thundering herd problem on cache miss is real, and this handles it cleanly.

One request. One credit charge. One point of failure.

For large-scale document processing, RAG pipelines, customer-facing AI products, or any architecture that handles bursty, repetitive loads — this is a structural cost reduction that no amount of model-switching will replicate.

Semantic Caching: Catching Costs That Exact-Match Misses

Standard caching is binary: the input either matches a cached key or it doesn't. That works well for structured data. For natural language queries — which is most of what LLM-powered agents process — it misses an enormous opportunity.

The semantic caching in ToolOps uses an intent-matching approach that's genuinely useful for NLP tool inputs. Queries like "Check status of invoice #442" and "Is invoice 442 paid?" hit the same cache entry, reducing LLM token usage noticeably.

This matters more than it might seem. In customer support agents, document analysis pipelines, and data extraction workflows, users phrase the same underlying question dozens of different ways. Every variation that misses an exact-match cache is a redundant API call. Semantic caching eliminates that category of waste entirely.

Production-Grade Resilience Without the Ceremony

Beyond cost reduction, there's the reliability side of production AI infrastructure.

LLM APIs go down. External services rate-limit. Downstream databases return transient errors. The naive response is to let your agent fail. The correct response is a circuit breaker that detects consistent failures, temporarily halts calls to the affected service, and allows recovery — without you having to build that logic yourself.

ToolOps includes this out of the box. A single CLI command — toolops doctor — validates all your backends and reports circuit breaker state. It's exactly what you want to wire into a health check endpoint.

That kind of operational visibility — knowing the status of every backend, every circuit breaker, without digging through logs — is the difference between an agent that fails silently and one you can actually run in production with confidence.

Framework Compatibility: It Works With What You Already Use

The natural concern when evaluating any new piece of infrastructure is migration cost. How much do I have to change?

ToolOps decorates plain Python async functions, making it 100% compatible with your favorite agent frameworks. It works across LangGraph, CrewAI, LlamaIndex, and MCP natively.

You don't rewrite your agents. You don't change your business logic. You add a decorator to the functions that make external calls and configure backends once at startup.

You register backends once at application startup, then reference them by name. ToolOps supports multiple backends simultaneously. Redis for persistent caching, in-memory for low-latency hot paths, semantic backends for NLP tools — you configure the combination that fits your architecture. Then you stop thinking about it.

The core package has zero external dependencies. You only install what you need. No forced opinions on your stack, no transitive dependency conflicts on day one, no bloat.

Who Benefits Most From This

ToolOps is most valuable in three specific situations.

High-volume production pipelines. If your system makes thousands or tens of thousands of API calls per day, even modest cache hit rates translate to significant cost reductions. At scale, organizations can achieve cost reductions of 50% to 90% while maintaining or even improving the quality of their AI applications.

Multi-agent architectures. The request coalescing feature was built for this. The more agents you run in parallel on overlapping workloads, the more redundant upstream calls you're generating without it.

Teams who've been hand-rolling infrastructure. If your codebase currently has a custom retry wrapper, a homemade cache manager, and a circuit breaker you wrote yourself — that's infrastructure debt ToolOps replaces directly. The integration is one decorator per function, with zero changes to business logic.

Getting Started

pip install "toolops[all]"

From there, it's backend configuration at startup and decorator placement on your tool functions. The GitHub repository covers the full setup, and the official documentation walks through backend configuration and the decorator API in detail.

The project is early — a web dashboard and budget control features are still on the roadmap — but the core resilience layer is solid. It's Apache 2.0 licensed. Open source, production-ready for its current feature set, actively developed.

The Architecture Principle It Enforces

There's something more fundamental happening here than a useful library.

ToolOps is built on the idea that every external call an AI agent makes should be treated as a first-class operation — not an afterthought. Caching, retry logic, circuit breaking, observability, and concurrency control aren't optional production concerns you bolt on later. They're the minimum viable infrastructure for anything that talks to an LLM or an external API.

Most teams know this. Most teams also don't have time to build it properly for every project. ToolOps packages that infrastructure into a decorator and gets out of the way.

Don't over-optimize for today's prices. What matters is building the architecture that can take advantage of future pricing improvements. The teams that will operate efficiently as models get cheaper, as APIs multiply, as agent systems scale — are the ones who built the right plumbing early. ToolOps is that plumbing.

If you're building production AI agents and you've hit the credit-burn problem, I'd genuinely like to hear how you've handled it. Drop a comment below.

GitHub: github.com/hedimanai-pro/toolops