It's 2 a.m. The alert channel blows up.
Someone needs error rates. Someone else wants traces. Another person suspects a full disk on a container. You open one AI chat — it either hand-waves, or mixes metrics, logs, and SSH into one muddy answer. The more you ask, the messier it gets.
Open more windows? Worse. Each bot talks past the others. You still don't have one incident-ready summary you can paste into the war room.
DataBuff treats multi-expert coordination like a hospital: triage desk + specialist consults. You speak to one entry point. Data Query, Inspection, Ops, and Product Q&A work in parallel behind the scenes, then roll up into an evidence-backed report. This post covers two things: why on-call actually needs multiple experts, and how to dispatch, parallelize, and merge without chaos — framework only, no source-code dump.
Why one bigger prompt is not enough
The instinct is: one stronger model, one longer prompt, done. On a real incident, that breaks — for the same reason a hospital does not send one doctor to run the ECG, draw blood, read the CT, and operate at once.
Chest pain in the ER: you do not expect a single generalist to do every specialty with every machine. Wrong tools, wrong permissions, shallow on everything. Lab values and surgical plans get mixed up. Bad outcomes follow.
On-call is the same. Ask "anything wrong with the cluster?" and you may need:
- Data Query — latency and error rates in Doris, slow traces (labs)
- Inspection — JVM/GC/dependency sweeps across dozens of services (screening)
- Ops Expert — SSH, logs, disk, processes (surgery)
- Product Q&A — where OTLP ports and alert menus live (admin desk)
Four different toolchains, permissions, and report formats. Stuffing them into one mega-prompt balloons context: metric semantics bleed into shell commands. Models cross-contaminate, overreach, hallucinate.
More chat tabs do not fix it. That is the family running between departments with no triage: no routing, no shared chart, you stitch the story at 2 a.m.
Multi-expert is not hype. Ops problems are cross-domain, parallel, and evidence-heavy — same shape as triage → specialty work → attending synthesis.
One front door, specialists in the back
Think of DataBuff's AI platform as on-call consult:
- You talk only to the triage desk (AI Brain)
- The desk does not run labs or operate — it pages the right specialty
- Each expert queries real telemetry and returns evidence
- The desk merges into something you can act on
This is not "sidebar ChatGPT." Classic APM + chat often cannot see your live metrics and traces. DataBuff is AI-native APM on OpenTelemetry — experts must call tools; guessing is not the design.
Three layers, not one giant prompt
New capability is added in three layers:
- Tools — instruments: service lists, traces, inspections, controlled commands
- Experts — roles: Data Query knows metrics, Inspection scans the fleet, Ops goes on-host, Q&A reads product docs
- Playbooks — how each role investigates and writes reports; plus routing rules for the Brain
A new "department" ≈ tools + playbook + expert registration. The hospital building (your observability pipeline) stays put. You still talk to one desk.
How the Brain dispatches
The Brain loads routing rules, reads every expert's charter, and semantically matches your request. It emits a dispatch: which expert + task brief. Two hard rules: faithful to your intent, and the Brain does not do the work — no metrics, no inspection runs, no shell on its own. Triage and synthesis only.
Pain point 1: what does dispatch actually look like?
A ticket-shaped loop:
- You message AI Brain; it reads routing rules and picks experts.
- Brain issues one dispatch: target expert + task brief (your words preserved).
- System immediately replies "accepted, please wait" — ticket created, work not finished.
- The expert runs in the background: tools, traces, inspection, controlled commands.
- Expert returns a delivery artifact to the same session and wakes Brain for another turn.
Pain point 2: why not wait synchronously?
Expert jobs are slow. Long-lived SSE connections time out. True parallelism dies if you block serially. Experts are multi-step tool loops. So dispatch is async by design: Brain ends its turn, experts run, callbacks wake Brain when done.
Pain point 3: four guardrails so async does not sprawl
- Same expert, serial only — duplicate dispatch while busy is rejected.
- Different experts may run in parallel — Inspection + Data Query together.
- Session-level pending counter — dispatch +1, delivery −1.
- Final answer only when pending = 0 — Brain must rewrite a complete reply, not "as above."
Live demo: one sentence, two experts
On demo.databuff.ai, open AI Brain. Do not pick an expert manually. Paste:
Any cluster anomalies in the last hour? Run a joint diagnosis with Data Query and Intelligent Inspection: Data Query checks latency, error rate, and slow traces; Inspection runs tiered health checks; summarize into an incident report I can forward.
Brain dispatches twice in parallel — Data Query and Inspection.
Inspection returns an S/A/B tiered view: most of 34 services in S/A; service-b at B with InsufficientStockException on SKU DEMO-10001. Brain merges into HTML with P0/P1 action items — ready for the incident channel:
Most products still make you pick an expert first
Many "multi-agent" UIs show four tiles — Data Query, Inspection, Ops, Q&A. You still guess which door to knock on. One conversation belongs to one expert. Switch experts, new window, context lost. Ask for error rates and a JVM sweep in one breath? You play dispatcher across tabs.
No triage desk — self-serve specialty signup. Wrong line wastes time; right line still means you collect reports from every window.
The gap is not four avatars vs one. It is who routes work: you picking single-expert chats, vs Brain accepting, dispatching, parallelizing, and merging with an evidence chain. That is what on-call needs: one entry, Brain dispatch, legal parallelism, traceable rollup.
Good systems do not make you guess the specialty. Hand off like shift change — one sentence in, triage and consults happen backstage. Try it on demo.databuff.ai with the prompt above.
Try DataBuff
Open source · multi-expert orchestration · one entry, parallel dispatch
Online Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
If this helped, star the repo — and run the demo prompt yourself once.
Top comments (0)