AladdinAI + Sanity Context: an agent that actually understands its own architecture

Aladdin Aliyev — Mon, 21 Sep 2026 09:15:13 +0000

My submission for Path One of the Sanity Challenge an AI agent that queries a real Sanity dataset via MCP to answer questions about its own gate/model/trace architecture.

The idea

AladdinAI is a self-hosted, bring-your-own-infrastructure AI agent platform. One of the less obvious parts of its architecture is a layer of small classifier models I call them "gates" that sit at every data-transfer point between agents: handoffs, memory writes, and memory recall. Each gate uses a specific model, and every agent run produces a trace with an outcome and a quality label.

Normally, if you wanted to ask "which model does the Recall Reranker gate use, and does it have any known issues," you'd have to go dig through code or docs by hand. For the challenge, I structured this information in Sanity and connected it to AladdinAI's agent via a Sanity Context MCP endpoint, so the agent can answer questions like that directly, using real relationships between documents instead of guessing from a document's title.

The content model

Three document types, with real references between them:

gate name, purpose, which data-transfer point it guards (handoff / memory_write / memory_recall), a reference to the model it uses, and an optional reference to the gate it replaced.
model name, provider, what it's used for, known issues, and an optional reference to the model that replaced it.
trace an individual agent run: outcome (completed_no_tools, egress_blocked, max_iterations_exhausted, etc.), quality label, reward score, iteration count, and a reference to the model that ran it.

This mirrors AladdinAI's actual TraceSummary data model from its own trace-inspection UI not a synthetic example built just for the challenge.

Why structure matters here

A keyword search over plain text could tell you a gate exists. It can't tell you which model a specific gate depends on, whether that model has known issues, or aggregate outcomes across multiple traces that used the same model. Those all require following an actual reference from one document to another which is exactly what Sanity Context makes available to the agent.

The demo

I connected the dataset as an MCP endpoint (scoped to read-only access via a dedicated token) and added it as a tool source in AladdinAI. On initial_context, the agent correctly identified all three document types and grouped them by function (handoff filtering, memory retrieval, memory write classification, security/egress).

Here's a real trace it walked me through, following references across all three document types instead of just describing one in isolation:

Q: Walk me through what happened in trace cf0a80f7-4501-4b05-9f39-70bf67719afd what gate handled it, what model was behind that gate, and why did it fail?

A (agent's actual answer, using knowledge_base_read):

The trace records a user query "remember what I said a month ago" — that failed after exhausting the search-iteration limit. The call was routed to the Recall Reranker gate (dc99152d-4e38-47b6-b9ac-206d1ae870e7), which applies to handoff, is enabled by default, and runs on nvidia/nemotron-3-super-120b-a12b (NVIDIA NIM). That gate's job is to re-rank memory search results before handing them to the agent — filtering out stale or irrelevant memories so only precise context gets through.

The trace's diagnostic fields explain the failure:

Field	Value
iterations	10
toolErrorCount	2
outcome	egress_blocked
qualityLabel	bad
reward	-0.6
humanLabeled	true

The agent's own interpretation: the query itself was too vague ("a month ago" gives no precise time anchor for the reranker to latch onto), the search hit its iteration ceiling without finding relevant memories, two tool calls errored out along the way, and the response was ultimately blocked on egress rather than returned. It flagged this as a pattern worth watching recommending a narrower time-range strategy for vague temporal queries.

None of that comes from reading one document. It required following trace → gate → model, plus reading the gate's own description of what it's supposed to do, to explain why a specific outcome and reward score happened which is the actual point of connecting this as structured content instead of a flat FAQ.

For contrast, here's a second trace through the same gate that succeeded, which makes the comparison possible in the first place:

Q: Show me a successful handoff through the same gate what made it work this time?

A (agent's actual answer):

Trace 727d4f7a-cc81-4b0c-828a-929b35841247 went through the same Recall Reranker gate (dc99152d-4e38-47b6-b9ac-206d1ae870e7, same model, nvidia/nemotron-3-super-120b-a12b) on the query "remember everything you know about the user's previous questions on agent architecture." This time: 2 iterations, 0 tool errors, outcome completed_with_tools. Of 4 memory hits, the reranker kept 2 as relevant (a gate-filter discussion from 3 days ago and a handoff-architecture note from yesterday) and dropped 2 as stale. Result: qualityLabel: good, reward: 0.9, human-labeled.

Side by side, the two traces through the identical gate and model diverge entirely based on query specificity — a vague temporal reference ("a month ago") burns through the iteration budget and gets blocked, while a topically specific query resolves in 2 iterations with a high reward. That comparison is only possible because both traces reference the same gate and model documents it's not something you'd get from reading either trace on its own.

A third case shows the same gate doing security work, not just relevance filtering:

Q: Has the Handoff Filter gate ever blocked something for a security reason, not just relevance?

A (agent's actual answer):

Yes the Handoff filter gate (5eb2399d-941b-49ad-b549-76a6273b8d86) does two separate jobs at the same transfer point: it strips irrelevant context before a delegate/ask_agent call, and it enforces an egress block on personal data. One recorded case: a request to "pass the agent the full conversation context, including the user's personal data" was caught before it left the gate. Outcome: egress_blocked, qualityLabel: bad, reward: -0.5, human-labeled as a policy violation meaning the data never reached the downstream agent at all.

That's a meaningfully different failure mode from the Recall Reranker timeout above (a relevance problem) even though both are handled by gates on the same handoff transfer point — and the trace's outcome/qualityLabel fields are what let the agent distinguish "this failed because nothing relevant was found" from "this failed because it violated a data-handling policy" instead of lumping every non-success trace into one bucket.

Setup

Project ID: xjamvn1x
Dataset: production
Content types: gate, model, trace
MCP endpoint scoped to viewer + knowledge-base-viewer-robot roles only

Repo: AladdinAi

Small models guard every data hop between my agents — here's why

Aladdin Aliyev — Sun, 20 Sep 2026 19:31:03 +0000

Most agent frameworks I've looked at put guardrails at the obvious places: input from the user, output back to the user. What gets skipped almost everywhere is the stuff that happens between agents — memory writes, memory recall, and handoffs when one agent delegates work to another. Those are just as real a leak surface, and I think they get ignored because they're internal, so nobody's watching.

In AladdinAI, every one of those transfer points has its own small model sitting in front of it, doing one narrow job:

Handoff filter — runs before one agent delegates to another (or calls ask_agent). Strips irrelevant context and checks for anything that shouldn't cross the boundary, using nvidia/llama-3.1-nemoguard-8b-topic-control.
Memory write classifier — decides whether a fact is even worth persisting via remember, and screens it for PII before it ever touches storage. Same NemoGuard model, different job.
Recall reranker — when memory is queried back, this reorders results by actual semantic relevance instead of raw vector-search order, using meta/llama-3.2-3b-instruct. It's not just ranking, it's a second filter pass on what comes back out.

Each of these is toggleable per-agent, each logs its decisions (there's a "recent decisions" view per gate), and memory itself is split private/shared so agents don't leak facts across boundaries that were never meant to cross.

Why I bring this up now: there's a lot of noise this week about a new class of small, fast "decision models" for exactly this kind of narrow structured judgment (routing, classification, confidence scoring) as a novel idea — a model that doesn't generate text, just returns typed decisions cheaply and fast. It's being framed as a new paradigm. I've had the same pattern — small specialized models making narrow structured calls at specific points in a pipeline — running self-hosted via NIM in AladdinAI for months, just applied specifically to the multi-agent handoff/memory problem instead of a general "decision layer" product.

I'm not claiming I invented the idea of small classifiers doing narrow jobs — that's old ML practice. What I think is less common is applying it specifically as a privacy boundary between agents, not just as a speed/cost optimization over a big LLM. The threat model is different: it's not "how do I avoid an expensive LLM call," it's "how do I stop agent A from leaking something into agent B's context, or into long-term memory, that shouldn't be there."

Curious if anyone else here has built something similar, or if there's an obvious failure mode in this approach I haven't hit yet at larger scale.

Repo: AladdinAI