Engineering a Multi-Agent AI Platform — Part 1: Adaptive Depth

This is Part 1 of a series on engineering AI systems that learn in production. This part covers the classification and strategy dispatch layer.

The problem nobody talks about

Here's a question that keeps coming up in production AI systems: should your model think before it answers?

Not philosophically. Mechanically. When a user asks "What's my account balance?", firing up a chain-of-thought reasoning engine with 16,000 thinking tokens is like hiring a PhD mathematician to count change. You burn compute, you add latency, and the answer isn't any better for it. But when someone asks "Compare the ROI of three deployment architectures under peak load, factoring in failure modes and regional latency differences," a zero-reasoning direct call produces garbage.

Most platforms pick one approach and apply it everywhere. The expensive ones over-reason on simple queries, bleeding money on thinking tokens that produce nothing useful. The cheap ones under-reason on hard problems, producing responses that sound confident but miss the point entirely.

Consider a platform where dozens of autonomous agents serve thousands of tenants across different industries. A scheduling agent handles "move my 3pm to Thursday" alongside an analytics agent wrestling with "why did churn spike in Q3 among enterprise accounts." These requests hit the same infrastructure, but they need radically different cognitive investment.

The system we're building doesn't pick a single depth. It scores each request and routes it to the appropriate reasoning tier. And then, as we'll see in Parts 2 and 3, it learns from outcomes and human feedback to get better at that scoring over time.

Three tiers of thinking

Think of reasoning depth as a spectrum with three practical zones.

Tier 1: No intermediate reasoning (NONE)

The model receives the prompt and generates a response directly. No thinking tokens, no structured intermediate steps. Input goes in, output comes out. This is your workhorse for factual lookups, simple status checks, greetings, and FAQ-style questions. Latency sits around 200ms. Cost is minimal.

When does this fail? When the question requires the model to hold multiple constraints in working memory simultaneously, or when the answer depends on comparing several options against each other. The model will produce something plausible-sounding, but it hasn't actually reasoned through the tradeoffs.

Tier 2: Lightweight structured reasoning (DRAFT)

Here, the system injects a structured instruction into the prompt: "Before answering, write your reasoning inside <draft> tags, limited to 512 tokens. Then provide your final answer inside <answer> tags."

The model gets a small scratchpad. It can organize its thoughts, check a few constraints, maybe outline a comparison. But the budget is tight. 512 tokens of reasoning is enough to decompose a moderately complex question into sub-parts and address each one, but not enough to explore rabbit holes or consider five alternative framings.

The system parses the response, extracts what's between the <answer> tags, discards the draft. The user never sees the intermediate reasoning unless the operator wants to inspect it for debugging.

Tier 3: Full chain-of-thought (FULL_COT)

This tier hands the problem to a model with native thinking-token support. Models like o3 or DeepSeek-R1 have a separate "thinking" channel where they can reason at length before producing a visible response. The system sets a reasoning_effort parameter and allocates a thinking budget (more on budgets in a moment).

The model can now spend thousands of tokens working through a problem. It can consider alternatives, backtrack when it hits a dead end, verify its own logic. For genuinely hard analytical questions, this produces qualitatively different answers than the other two tiers.

The tradeoff is obvious: latency climbs to 2-5 seconds and token costs multiply. You don't want this running on "What time do you close?"

Scoring complexity

So how does the system decide which tier to use? It runs the incoming request through a multi-signal classifier that produces a single score between 0 and 1.

Five signals feed the score:

Task type (weight: 0.30). Is this a factual lookup? A comparison? An analysis? A debugging request? A creative task? The classifier maintains a taxonomy of task types with associated difficulty priors. A status check scores low. A multi-criteria comparison scores high.

Ambiguity (weight: 0.20). How many ways could this request be interpreted? "Tell me about performance" is ambiguous (performance of what? the system? the team? a specific metric?). "What was the p95 latency for the /checkout endpoint last Tuesday" is not. Ambiguity markers include vague pronouns, missing context, and underspecified scope.

Domain vocabulary density (weight: 0.20). Requests loaded with specialized terminology signal that the answer needs domain expertise and careful reasoning. A message full of financial modeling terms or medical terminology likely needs deeper thinking than a conversational greeting.

Lexical complexity (weight: 0.15). Sentence length, subordinate clause depth, vocabulary sophistication. A proxy for how much cognitive load the request itself carries. Not a perfect signal (a short request can demand complex reasoning), but a useful one in combination with the others.

Context dependency (weight: 0.15). How much does the correct answer depend on prior conversation history, tenant-specific data, or cross-referencing multiple information sources? A standalone question scores lower than one that requires synthesizing three previous messages and two documents.

The composite score:

C(x) = 0.30 · S_task(x) + 0.20 · S_amb(x) + 0.20 · S_dom(x) + 0.15 · S_lex(x) + 0.15 · S_ctx(x)

Each signal function S(x) outputs a value in [0, 1]. The weighted sum produces the final complexity score C(x), also in [0, 1].

The routing thresholds:

• C(x) < 0.30 → NONE (direct call, no reasoning)
• 0.30 ≤ C(x) < 0.65 → DRAFT (lightweight scratchpad, 512-token cap)
• C(x) ≥ 0.65 → FULL_COT (native thinking tokens with budget)

A concrete example makes this tangible. "What is my account balance?" scores roughly: task_type = 0.1 (factual lookup), ambiguity = 0.1 (clear intent), domain = 0.05 (no specialized vocabulary), lexical = 0.1 (short, simple), context = 0.2 (needs account data but straightforward). Composite: 0.30(0.1) + 0.20(0.1) + 0.20(0.05) + 0.15(0.1) + 0.15(0.2) = 0.03 + 0.02 + 0.01 + 0.015 + 0.03 = 0.105. NONE tier. Just answer it.

Now try "Compare the ROI of three deployment architectures considering latency, cost, and failure modes under peak load." Task_type = 0.9 (multi-criteria analysis), ambiguity = 0.4 (what counts as "ROI" needs interpretation), domain = 0.8 (infrastructure terminology), lexical = 0.7 (complex sentence, multiple constraints), context = 0.6 (needs architecture details from prior context). Composite: 0.30(0.9) + 0.20(0.4) + 0.20(0.8) + 0.15(0.7) + 0.15(0.6) = 0.27 + 0.08 + 0.16 + 0.105 + 0.09 = 0.705. FULL_COT. This one needs to think.

Why this matters in multi-agent systems

A single-agent chatbot can get away with a fixed reasoning depth. But multi-agent platforms have a topology that makes static approaches break down.

Picture a supervisor agent that receives every inbound message. It classifies intent and routes to specialist agents: a scheduling agent, a billing agent, a technical support agent, an analytics agent. Each specialist handles a different slice of the problem space.

Here's what's interesting: the same specialist agent might need different reasoning depths depending on the specific request. The billing agent handles "what's my next payment date?" (NONE) alongside "explain why my invoice increased 40% this month considering the three pricing tiers, the mid-cycle plan change, and the prorated adjustments" (FULL_COT).

The reasoning strategy is selected per-request, not per-agent. The billing agent doesn't have a fixed reasoning depth. It gets classified fresh on every interaction based on what the user actually asked.

It gets more layered when you have agent teams. A lead agent might decompose a complex request into subtasks, delegating each to a different specialist. One subtask might be simple data retrieval (NONE), another might require cross-referencing multiple sources (DRAFT), and a third might involve genuine analytical reasoning (FULL_COT). The system scores each subtask independently.

This per-request granularity is what makes the whole thing tractable economically. If you set every agent to FULL_COT because some of their tasks are hard, you're paying ten times more for the easy ones. If you set them all to NONE because most tasks are easy, the hard ones produce bad answers and you lose user trust. Scoring at the request level means you pay for reasoning exactly where it creates value.

The thinking budget

Even within FULL_COT, not every hard problem deserves the maximum thinking allocation. A moderately complex question might need 4,096 thinking tokens to work through. A genuinely difficult analytical task might need 16,384. Allocating 32,000 thinking tokens to a question that only needs 4,000 wastes compute and sometimes degrades quality (the model can over-think, going in circles or second-guessing correct intermediate steps).

The system maps complexity sub-ranges to budget tiers:

• Simple (below threshold): 0 tokens (NONE/DRAFT handle it)
• Moderate (0.65-0.80): 4,096 tokens
• Complex (0.80-1.0): 16,384 tokens

Budget enforcement works as a state machine. When the model starts thinking, the state is ACTIVE. If the model reaches the budget limit, the state transitions to EXHAUSTED, and the system signals the model to wrap up its reasoning. A grace window of 512 tokens allows the model to reach a natural stopping point rather than being cut off mid-thought. After the grace window, the state transitions to TERMINATED and the model must produce its final answer from whatever reasoning it completed.

This prevents runaway thinking. Without budget enforcement, a model given native thinking tokens can sometimes spend 20,000+ tokens reasoning in circles on a problem that doesn't benefit from it. The budget creates a ceiling without preventing deep reasoning when it's needed.

The diagram

Here's the full request lifecycle through strategy selection:

A request enters the complexity classifier, which evaluates five signal dimensions and produces a weighted score. The score crosses one of two thresholds (0.30 and 0.65), routing the request into one of three strategy lanes. Each lane uses a different mechanical approach to produce the final response: a simple pass-through call, a structured draft envelope with tag parsing, or a full thinking-token API with budget enforcement. All three converge on a response delivered to the user.

What's missing

The classifier works. It routes most requests to reasonable tiers. But it's a heuristic, and heuristics don't learn.

If the system consistently under-reasons on a particular type of request for a specific agent, the classifier keeps making the same mistake. If tenant patterns shift (maybe a business changes industries and their support queries become more technical), the static weights don't adapt.

What we actually want is a system that observes outcomes. Did the DRAFT strategy produce a good result for this context, or did it fall short? Was the FULL_COT response worth the extra latency and cost, or would DRAFT have been equally good?

That's the reinforcement learning router, and it's the subject of Part 2. The core idea: maintain a probabilistic belief about how well each (model, strategy) pair performs in each context, sample from that belief to select an action, observe the outcome, and update the belief. A contextual bandit that learns the terrain.

The classifier gives us a reasonable starting position. The bandit learns to do better.

Next in the series: Part 2 explores how Thompson Sampling learns optimal model+strategy routing from production outcomes.

You can follow me on LinkedIn and Twitter for more updates.