Routing one request across many small models: what the demo doesn't show you

Everyone shows the same demo. A request comes in, it hits a router, the router picks a small specialist model instead of a giant generalist one, and the slide moves on to the next bullet point about cost savings. I spent most of this year living in the part the demo skips: the arbitration layer that has to decide, for every request, which specialist (if any) is even allowed to touch it, and how the pieces of a multi-step request get stitched back together afterwards.

A single big generalist model doesn't need any of this. It has one context window, one set of weights, and whatever access you've already granted it. The moment you split the work across several smaller, narrower models, you've created a coordination problem that didn't exist before. That problem, not the models themselves, is where most of the engineering effort actually goes.

Arbitration is a permissions question before it's a quality question

The instinct when you first build a router is to treat it like a classifier: look at the request, work out its intent, pick the model that's best at that intent. That's backwards. The first question has to be which models are even permitted to see this particular input and take this particular action, and only then do you ask which of the permitted ones is best.

I keep these as two separate checks for a reason. A contract clause extractor might be the best tool in the building for parsing a lease, but if the request includes a field tagged as personal data and that specialist's permission set doesn't cover personal data, it never gets to compete on quality. It's excluded before ranking starts, not after. With three specialists you can eyeball the permission matrix by hand. With thirty, the exclusion check has to be a first-class step in the pipeline, not something a prompt gently asks the model to respect. A generalist model that already has broad access skips this problem entirely, which is exactly why it's harder to audit what it actually used that access for on any given request.

A single request is rarely a single task

Once you accept that requests get split across specialists, you notice most real requests were never one task to begin with. 'Summarize this contract, flag anything that needs legal review, and draft a holding reply' is three jobs with a dependency order: the flagging step needs the summary, the drafting step needs to know what got flagged. That's a small directed acyclic graph, not a single function call.

I don't author these graphs by hand per request type. The arbitration layer builds the DAG dynamically: the incoming request gets decomposed into sub-tasks, each sub-task declares what it needs as input and what permissions it requires, and the graph only commits to an execution order once every node has a specialist it's actually allowed to run. One of the specialists I run as part of SIOS, the platform I've been building, does nothing but contract clause extraction, nothing else, and that narrowness is deliberate. If clause language in a particular jurisdiction changes, I retrain or replace that one node and nothing else in the graph has to move.

What the interface actually looks like

Stripped down to the part that matters, the arbitration interface is smaller than people expect. Roughly:

interface Specialist {
  id: string
  domain: string          // e.g. "clause-extraction", "pii-redaction", "drafting"
  permissions: string[]   // data classes / actions this model may touch
  costTier: "low" | "medium" | "high"
}

interface SubTask {
  id: string
  intent: string
  dependsOn: string[]        // ids of upstream nodes this needs output from
  requiredPermissions: string[]
}

interface ArbitrationResult {
  subTaskId: string
  eligible: Specialist[]     // passed the permission check
  chosen: Specialist | null
  reason: "chosen" | "no_eligible_specialist" | "generalist_fallback"
}

function arbitrate(task: SubTask, registry: Specialist[]): ArbitrationResult {
  const eligible = registry.filter(s =>
    task.requiredPermissions.every(p => s.permissions.includes(p))
  )

  if (eligible.length === 0) {
    return { subTaskId: task.id, eligible, chosen: null, reason: "no_eligible_specialist" }
  }

  const ranked = eligible
    .filter(s => s.domain === task.intent)
    .sort((a, b) => costWeight(a.costTier) - costWeight(b.costTier))

  return {
    subTaskId: task.id,
    eligible,
    chosen: ranked[0] ?? null,
    reason: ranked[0] ? "chosen" : "generalist_fallback",
  }
}

The detail worth noticing is the order of operations inside arbitrate(). Permission filtering happens first and narrows the field before any quality ranking runs. Flip that order, rank everything by quality first and check permissions second, and you'll eventually ship a request where the 'best' model got chosen before anyone checked whether it should have seen the input at all. I've seen that bug in an early version of this, and it's the kind of thing that only shows up once you've got enough specialists that the top-ranked one isn't always the permitted one.

The tradeoff nobody puts in the demo

The case for many small specialists is real. Each one is narrow enough to audit properly, you can point at a specific model and say exactly what data classes it's allowed near. Each one is independently updatable, a regulatory change in one domain means retraining one node, not re-evaluating a monolith's behavior across every domain it touches. And you can run the cheap model for the common case and only wake the expensive one when the router decides the task actually needs it.

None of that is free. Every hop between specialists is a network call, a permission check, and a schema translation between what one model outputs and what the next one expects as input. Say each hop costs somewhere on the order of 100 to 200 milliseconds once you include the arbitration check, a four-node DAG can easily spend close to a second just coordinating, before any model has done its actual work. A single generalist model never pays that coordination tax because it never had to hand off to anyone. You're trading a hard-to-audit black box for several auditable ones, plus a new black box sitting in front of them (the router and arbitration layer itself), which now needs its own logging, its own test suite, and its own failure modes worked out.

Still unresolved

The part I haven't settled is where the arbitration log should live so it's actually useful during an audit without turning into its own compliance burden. Logging every permission check at every node gives you full traceability and a volume of logs nobody will ever read end to end. Logging only at the branch points is cheaper but means you're trusting the bits in between. If you've built anything with a routing layer in front of multiple models, I'd like to know which way you landed and why.