Small models guard every data hop between my agents — here's why

Most agent frameworks I've looked at put guardrails at the obvious places: input from the user, output back to the user. What gets skipped almost everywhere is the stuff that happens between agents — memory writes, memory recall, and handoffs when one agent delegates work to another. Those are just as real a leak surface, and I think they get ignored because they're internal, so nobody's watching.

In AladdinAI, every one of those transfer points has its own small model sitting in front of it, doing one narrow job:

• Handoff filter — runs before one agent delegates to another (or calls ask_agent). Strips irrelevant context and checks for anything that shouldn't cross the boundary, using nvidia/llama-3.1-nemoguard-8b-topic-control.
• Memory write classifier — decides whether a fact is even worth persisting via remember, and screens it for PII before it ever touches storage. Same NemoGuard model, different job.
• Recall reranker — when memory is queried back, this reorders results by actual semantic relevance instead of raw vector-search order, using meta/llama-3.2-3b-instruct. It's not just ranking, it's a second filter pass on what comes back out.

Each of these is toggleable per-agent, each logs its decisions (there's a "recent decisions" view per gate), and memory itself is split private/shared so agents don't leak facts across boundaries that were never meant to cross.

Why I bring this up now: there's a lot of noise this week about a new class of small, fast "decision models" for exactly this kind of narrow structured judgment (routing, classification, confidence scoring) as a novel idea — a model that doesn't generate text, just returns typed decisions cheaply and fast. It's being framed as a new paradigm. I've had the same pattern — small specialized models making narrow structured calls at specific points in a pipeline — running self-hosted via NIM in AladdinAI for months, just applied specifically to the multi-agent handoff/memory problem instead of a general "decision layer" product.

I'm not claiming I invented the idea of small classifiers doing narrow jobs — that's old ML practice. What I think is less common is applying it specifically as a privacy boundary between agents, not just as a speed/cost optimization over a big LLM. The threat model is different: it's not "how do I avoid an expensive LLM call," it's "how do I stop agent A from leaking something into agent B's context, or into long-term memory, that shouldn't be there."

Curious if anyone else here has built something similar, or if there's an obvious failure mode in this approach I haven't hit yet at larger scale.

Repo: AladdinAI