DEV Community: databufflabs

LangGraph: Orchestrate a Swarm of AI Agents with One Graph

databufflabs — Mon, 24 Aug 2026 02:18:12 +0000

A single AI assistant is already standard in many teams: ask a question, get an answer, like chatting with a seasoned hand. But the moment you want several AIs to work together — one checks metrics, one digs through logs, one draws the conclusion — it falls apart: who hands off to whom? Can they run in parallel? Should it stop and ask you midway? If it crashes halfway, is everything before it wasted?

LangGraph exists to answer those questions. It's LangChain's open-source multi-agent orchestration framework (MIT). But the essence first: it is, before anything else, a general-purpose graph execution engine — nodes, edges, conditional edges, parallelism, checkpointing, waiting on a human are all generic graph capabilities, with no inherent tie to AI; it just so happens that a node can hold an AI, and once it does, you're "orchestrating multiple AI agents." The core is simply: you draw the graph, it runs the graph.

This article uses a single case — a 2 a.m. alert: "order-service error rate spiking" — to explain it end to end: the graph first, then the code, then the execution model and context passing, and finally a comparison with Dify and Claude Code's dynamic workflow.

A case: 2 a.m., order-service error rate spikes

Your on-call phone buzzes. The alert says order-service error rate just hit 12%, and your AI assistant has to walk the whole investigation flow itself. This graph is the entire logic it will execute:

Fig 1 · A late-night investigation: one graph exercises nodes, edges, conditional edges, Send, interrupt, and checkpoint.

A single run goes roughly like this: the AI diagnoses first, finds it's serious, and fans out 12 parallel subtasks to check instances one by one; after checking, it summarizes "these 3 are bad"; then it stops and asks whether to restart — you approve, and only then does it execute the fix, finally re-checking and reporting.

This graph uses almost every core capability LangGraph has: nodes (each box), edges (arrows), conditional edges (serious?), Send (batch checks), interrupt (await your approval), checkpoint (persist each step). Let's draw it in code.

Drawing this graph in code

Build a graph, register each box as a node, connect arrows as edges:

from langgraph.graph import StateGraph, START, END

def diagnose(state):   # 1 diagnose: check error rate, pull Trace, return severity
    error_rate = query_error_rate(state["service"])
    return {"abnormal": error_rate > 5}

def fan_out(state):    # 3 batch check: Send dispatches 12 subtasks at once
    return [Send("check_host", {"host": h}) for h in state["hosts"]]

def check_host(state): # 4 check one instance (each Send is its own small task)
    return {"results": [f"{state['host']}: ok"]}

def ask_human(state):  # 5 await your approval
    decision = interrupt("3 bad instances found — auto-restart?")
    return {"decision": decision}

def fix(state):        # 6 execute the fix
    restart(state["hosts"])
    return {}

def report(state):     # 7 produce the report
    return {"report": "handled"}

g = StateGraph(dict)
g.add_node("diagnose", diagnose)          # register nodes
g.add_node("fan_out", fan_out)
g.add_node("check_host", check_host)
g.add_node("ask_human", ask_human)
g.add_node("fix", fix)
g.add_node("report", report)

g.add_edge(START, "diagnose")             # start -> diagnose
g.add_conditional_edges("diagnose", route)# 2 conditional edge: serious? -> fan_out or report
g.add_conditional_edges("fan_out", fan_out)  # 3 Send fan-out
g.add_edge("check_host", "ask_human")     # checked -> await approval
g.add_edge("ask_human", "fix")            # approved -> fix
g.add_edge("fix", "report")               # fixed -> report
g.add_edge("report", END)

graph = g.compile(checkpointer=InMemorySaver())   # compile + enable persistence

A few key points, one by one:

1 · A node is just a plain function. diagnose, check_host, fix are ordinary Python functions. Whether to call an LLM is entirely up to the function body — diagnose can call a model, check_host can be pure computation.

2 · A conditional edge picks the path. route returns "fan_out" or "report", and the graph walks to the matching node:

def route(state):
    return "fan_out" if state.get("abnormal") else "report"

3 · Send = dispatch in bulk. fan_out returns 12 Send("check_host", {...}), executed in parallel within the same superstep, results auto-merged:

def fan_out(state):
    return [Send("check_host", {"host": h}) for h in state["hosts"]]

Fig 2 · Send zoomed: dispatch N independent subtasks in one superstep, then merge.

4 · interrupt = stop and wait for a human. Call interrupt() inside a node and the graph halts, surfacing the question to you; after you approve and resume with an answer, the graph resumes from its checkpoint:

decision = interrupt("3 bad instances found — auto-restart?")
# after you approve:
graph.invoke(Command(resume="yes"),
             {"configurable": {"thread_id": "t1"}})

5 · checkpoint = persist every step. The graph stores progress after each step (the thread_id acts as a ticket id). If it crashes midway, re-invoke with the same thread_id and it resumes from the breakpoint — officially called durable execution.

How does LangGraph actually execute this graph?

Running the graph isn't a headlong dash from start to end; it moves forward round by round. The official term is superstep, and each round does four things:

Fig 3 · One superstep: figure out who runs -> run in parallel -> refresh the board -> persist.

Nodes don't pass messages to each other. Everyone faces the same whiteboard: this round only reads what's already on the board, and writes are set aside — only after the whole round finishes does the board refresh. The source comment is one line: what step N writes, step N+1 sees.

Fig 4 · How context passes: no talking between nodes, just read/write the same whiteboard.

Walk the Fig 1 on-call case through it, and the rhythm is:

Fig 5 · The on-call case unrolled by superstep: writes land on the board next round.

Chat history works the same way: put messages on the board, and the next step sees them.

Compared to other options — where's the difference?

Dify and Claude Code are both often called "multi-agent," but set beside LangGraph, one table each is enough.

Dify: visual platform vs code library. Dify is a web canvas where you drag nodes; LangGraph is a graph you draw in Python code.

Dimension	Dify	LangGraph
Form	Web canvas, drag nodes	Draw the graph in Python
Who it's for	Non-developers / quick prototypes	Developers / fine control
License	Apache 2.0 modified (commercial conditions apply)	MIT (commercial use OK)

Claude Code dynamic workflow: the AI writes a LangGraph on the spot. You just say what to do; in the background it auto-decomposes the task and spins up tens to hundreds of agents in parallel.

Dimension	Claude Code dynamic workflow	LangGraph
Who orchestrates	The AI itself: reads the request, decomposes dynamically	You: draw a fixed graph in code
Orchestration artifact	An agent tree in memory, gone after the run	A compilable, replayable graph definition
Predictability	Same input may produce different graphs	Graph fixed, behavior predictable

One line to remember: Dify hands you a ready-made car, Claude Code has the AI write the orchestration on the spot, LangGraph hands you the engine and the blueprint — the most freedom, and the most work.

Our open-source DataBuff — how do its agents collaborate?

Our open-source DataBuff (AI-native APM, GitHub: https://github.com/databufflabs/databuff) also does multi-agent collaboration. You talk to a single entry point; an AI brain dispatches the work to experts — query, inspection, ops, Q&A — in parallel, then assembles their findings into a conclusion with an evidence chain:

Fig 6 · You face one entry point; the complex collaboration happens behind it.

The core purpose

LangGraph's core purpose isn't "to hand you a ready-made multi-agent solution," but to provide a controllable runtime for agent flows that need to run long, hold state, persist, and loop in a human — it turns the low-level dirty work — parallelism, checkpointing, waiting, recovery — into primitives, so you only worry about the business flow itself.

Takeaway: copy those 6 functions and the graph structure from the case and run them locally, and you'll understand what multi-agent orchestration is about.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs, and AI troubleshooting in one.

GitHub: https://github.com/databufflabs/databuff

Live demo: https://demo.databuff.ai

Grafana's LGTM Stack — Tempo and Loki Are Starting to Feel Dated

databufflabs — Wed, 19 Aug 2026 02:45:11 +0000

Same slow checkout: four hops from metrics to Tempo to Loki. DataBuff drills down from the service list, then you can ask the AI and check the platform itself.

DataBuff is an AI-native APM. It ingests OpenTelemetry and SkyWalking, and ships seven AI capabilities. GitHub: https://github.com/databufflabs/databuff

Same slow checkout: Grafana needs three systems. DataBuff finishes the path in one.

What you check	Grafana: 3 systems / 3 separate pages	DataBuff: 1 system / 3 linked pages
Metric: who is slow	Grafana: Service Map	Service list
Trace: which hop	Tempo: trace list, waterfall	Tracing: drill down from the service page
Log: what happened then	Loki: query by traceId	Logs: open Trace and jump back

On-call, whether those three sit on one path matters more than whether you have all three pillars.

This LGTM: Metric / Trace / Log on three lines

The scene is service-a GET /demo/checkout getting slow. Find who is slow in Metric, then the request and span in Trace, then context in Log. In this stack the three entries are different datasources, with different filters.

Metric · ① Who is slow: metrics-generator → metrics store → Service Map

Tempo stores spans. To draw a service map, metrics-generator aggregates spans into time series, writes them to a separate metrics store, then Grafana can paint the Service Map.

Metric · Service Map (from metrics-generator, not Tempo itself)

Trace · ② Find the request: Explore → Tempo, search the list

Leave Service Map, switch Explore to the Tempo datasource, filter by service / operation for the slow checkout, and copy the traceId.

Trace · Explore → Tempo: different query box, different syntax

Trace · ③ Waterfall: where the time went

Open the trace. See which hop of service-a → service-b → service-c is slow. Still in Tempo. Logs not touched yet.

Trace · Tempo waterfall: find the slow span

Log · ④ Logs: Explore → Loki, paste the traceId

Log is another line: Explore → Loki, put the traceId into LogQL. If tracesToLogs is not wired, this hop is manual.

Log · Explore → Loki: LogQL, paste the traceId yourself

In this stack, Metric, Trace, and Log are separate datasources

This run: four hops, and logs still needed a hand-copied traceId.

DataBuff: same incident, drill down from the service list

Same slow checkout. After OTLP lands in DataBuff, Metric / Trace / Log are already joined. Stay in one product. You do not paste a traceId into a new query language to open logs.

Metric · ① App performance → service list

The service list shows service-a latency, errors, and traffic. Click the name. Metrics and traces share one store — no extra extract from spans.

Metric · service health: latency / errors / traffic

Trace · ② Tracing: open the slow waterfall

From the service page, open the 240ms checkout. Middleware spans sit on the same picture. No system switch.

Trace · waterfall + call chain

Log · ③ Logs: Trace on the row, one click back

Each log row has a Trace button. That is the on-call step you skip: copy the id, change the query.

Log · logs → trace, same product

Topology, service health, and waterfall share one dataset — no extra metrics-generator layer

By here the gap is no longer “can you see the data.” DataBuff still has ask-the-AI and platform health. This LGTM stack did not expose those two entries.

After that path: AI and self-monitoring

After services, traces, and logs, two more things sit on the same menu: ask the data in natural language, and see the platform's own health. Those were the biggest feel differences on this run.

AI · Q&A / inspect / docs — same data you just drilled

Open the AI platform. You do not pick Tempo or Loki first, and you do not write TraceQL / LogQL. The page already has ask-data, inspect, product Q&A, and an ops expert. Sample prompts are concrete: last-hour service list, service-b upstream/downstream, request and error trends.

You still bring your own model. This lab used DeepSeek. The difference is the experts and tools are already in the product, so you do not assemble three datasource contexts first.

AI platform · ask, inspect, Q&A, ops expert — same menu as Metric / Trace / Log

This Grafana 11.5 does not ship that as a chat entry. To ask the same way you add an LLM plugin, then wire Tempo, Loki, and the metrics store into the model context.

Self-monitoring · deploy status: treat the platform as a business system

Install → Deploy status. One page: ingest TPS, write failures, Doris disk, query failures; legend split by trace / metric / log. Inbound requests, bytes, latency, and outbound drops share one timeline.

Deploy status · ingest overview: 32.9/s in, Doris disk 49%; trace / metric / log on one page

At 2 a.m., “is the app slow, or is the platform stuck?”, this page knocks out a batch: are the three lines still rising, did writes fail, is Doris disk full. In this LGTM stack those checks still mean Tempo, Loki, the metrics store, and Grafana, each on its own page.

LGTM: stop when you can see the data. DataBuff: after that you can still ask, and still see the platform.

What you want	Grafana LGTM	DataBuff
Ask “who is slow, what does the topology look like”	Write the query yourself; or add an LLM plugin plus data context	AI chat: sample prompts are service list / topology / traffic
An inspect report you can forward	Stitch dashboards and alert rules	Inspect: the expert reads the same metrics and writes the report
Is the platform itself stuck	Open each component health page	One deploy-status page: ingest + Doris
Component bill	Tempo + metrics-generator + metrics store + Loki + Grafana	ingest + one engine + Web (AI and deploy status included)

How to try: OTLP dual-write, no rip-and-replace

The collector already speaks OTLP. Add a DataBuff exporter in Alloy / Collector, write to Tempo in parallel for a few days, confirm the same spans line up, then drop the Tempo path if you want. The whole thing rolls back.

Walk it once: service health → trace → logs → ask the AI for a service list or topology → deploy status, three ingest lines. LGTM can stay. Just see if this on-call path feels shorter.

DataBuff

AI-native APM. Ingests OpenTelemetry and SkyWalking. Seven AI capabilities on the same stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

DeepSeek Harness Hit 126k Stars — What Self-Evolution Means for Software (and Observability)

databufflabs — Mon, 17 Aug 2026 01:50:53 +0000

A repo on GitHub has been exploding lately. DeepSeek open-sourced an agent framework called deepseek-harness (often shortened to dsh). It went public on August 13; when I took this screenshot, it was already at 126k stars.

GitHub repo page, captured 2026-08-16. Top-right shows 126k stars; About lists 125.9k.

Repo: https://github.com/deepseek-ai/deepseek-harness

Its one-line pitch: Everything is a Plugin. You're not stuck with whatever shipped in the box. The AI can see what it already has, and when something is missing it can add it to itself. They call that self-evolution.

With stars climbing that fast, I wanted to see what it actually does. So I opened the web UI and typed one request.

I asked it to add a sidebar item

Roughly: build a plugin, inject it into yourself. Change the left menu — add a Workflow item above Workspace. Click it and show a complex flow chart; click a node and open a drawer on the right.

It actually did it. Bottom-left showed Cordis Plugin: 1 running. The three screenshots below are from the live page.

Red box is the original prompt. It inspected its own sidebar first — didn't jump straight to editing files.

The left **Workflow* item is live. This ticket canvas wasn't built-in — the plugin it just wrote drew it.*

Clicking the red **External API Query* node: timeout and retry 3/5 in the drawer.*

That extra Workflow entry wasn't in a release. It wrote a plugin on the spot and mounted it.

That made me think about on-call pages

The troubleshooting pages you open on shift are usually fixed at ship time. How topology switches, which metrics sit where, what the drawer shows first — often one team's playbook baked into the UI. Banks want ledger latency; e-commerce wants checkout funnels; one layout rarely fits both.

Changing that in traditional software is hard: tweak a page, wait for the next release. Low-code is better — drag and drop — but you're still picking from the vendor's blocks; step outside and you're stuck.

dsh is different. You say what you need; it adds a piece to itself right then. The block is fabricated on site, not chosen from a catalog.

Fig 1 · Traditional software, low-code, and dsh in one glance

Three capabilities — we've shipped two

We've been thinking about this for a while. Software that takes care of itself, in my view, comes down to three things: self-ops, self-Q&A, and self-evolution. DataBuff already has the first two in product.

Project on GitHub: https://github.com/databufflabs/databuff

Self-ops isn't just flashing a red chart. In our demo, logs were dropping — thousands to fifteen thousand per minute. A human said two things: let it investigate, then allow it on the box. It checked its own self-monitoring, found the write queue capped at 16 batches, bumped it to 32, restarted ingest. After that, write drops went to zero.

Self-ops · SSH to change params and restart ingest; write drops back to zero. It fixed itself.

Self-Q&A: installed but don't know where to click? You used to hunt external docs. In DataBuff, switch to product support and ask: how do I wire OpenTelemetry, where do I set alert thresholds? It answers with menu paths and fields from in-product docs.

Self-Q&A · Ask how to onboard OTel and configure alerts — get paths, not a link dump.

Both are visible the moment you open the product. Self-evolution we haven't built yet. dsh just demonstrated the third.

Fig 2 · Three capabilities — DataBuff has the first two

Self-evolution in observability

Observability isn't a chat window. Collection, storage, query — touch the wrong layer and your data is dirty. dsh can rewrite sidebars and canvases because almost everything lives in plugins. Observability can't copy that wholesale. The ingest/store/query pipeline must not be edited live.

What can grow on demand is the human-facing layer. "Show me UnionPay transactions first" — add a troubleshooting panel or drawer on the spot. The pipeline underneath stays the same.

Fig 3 · Observability: lock the base, extend the pages

Fig 4 · dsh can change the whole building; observability only the room users see

Same telemetry as before. Pages can be tailored per customer on the spot.

Beyond the hype — still those three

Most of dsh's star rush is probably about "AI adding features to itself."

Self-ops — when it breaks, it fixes itself. Not just a red dashboard.
Self-Q&A — don't know how to use it? Ask; get menu paths.
Self-evolution — not enough? Add a piece for your request on the spot.

We've shipped the first two in DataBuff. The third — for observability — we're not there yet.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too

databufflabs — Wed, 12 Aug 2026 01:42:13 +0000

The page is slow, APIs time out randomly, and you've already checked Java logs and slow SQL — still nothing. The gap is often the Nginx in front: it proxy-passes the request, but classic access logs won't tell you whether the hop succeeded, how long it took, or where it stalled.

OpenTelemetry turns that into "load a module, write a few lines of config." This post walks a Demo we actually ran: instrument Nginx and Java with OpenTelemetry, then open one Trace in DataBuff that spans Nginx → Java → Redis.

What OpenTelemetry is doing

OpenTelemetry stitches each service hop into one call chain. A full request is a Trace; each processing segment is a Span. Nginx uses the official module, Java uses the official agent — each records its hop, exports with the same TraceID, and the platform merges the fragments.

curl / browser
   │   GET /hello
   ▼
[ Nginx  nginx:1.27-alpine-otel ]   ngx_otel_module   → OTLP gRPC 4317
   │   reverse proxy proxy_pass
   ▼
[ Java   JDK HttpServer + Jedis ]   opentelemetry-javaagent → OTLP HTTP 4318
   │   set + get
   ▼
[ Redis  192.168.50.120:16379 ]     Jedis auto-instrumented
   ▼
[ DataBuff ]  ← gRPC 4317 + HTTP 4318, merged under one TraceID

Hop 1 · Nginx — two install paths

The official OTel module needs a recent Nginx (1.21+). Our host still runs 1.20.1, so path A uses the official image that already ships the module; path B installs the module package on an existing Nginx and adds one load_module line. OTel directives are the same either way.

Path A — official image:

docker run -d --name nginx-otel \
  -p 8090:80 \
  --add-host host.docker.internal:host-gateway \
  -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro \
  nginx:1.27-alpine-otel

Path B — Alpine example: apk add --repository https://nginx.org/packages/mainline/alpine/v3.21/main nginx-module-otel, then load_module /usr/lib/nginx/modules/ngx_otel_module.so; and nginx -s reload.

Config points that matter:

Export over gRPC host.docker.internal:4317 (HTTP 4318 fails with OTel export failure ... Socket closed)
otel_service_name nginx-otel-demo
otel_trace on

The line people miss: otel_trace_context propagate;. By default the module receives upstream context but does not inject it downstream. Without this line Nginx reports its own Span, but Java never sees the TraceID — you get two unrelated Traces. After adding propagate, they joined immediately.

otel_exporter {
    endpoint host.docker.internal:4317;   # gRPC
}
otel_service_name nginx-otel-demo;
otel_trace_context propagate;             # required: inject traceparent

Hop 2 · Java agent, zero code changes

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=java-redis-demo \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318 \
  -cp "libs/*:classes" OtelDemoServer

Java uses HTTP OTLP (4318), Nginx uses gRPC (4317) — two protocols, one DataBuff, one Trace later.

Hop 3 · Two exporters, one Trace

Both nginx-otel-demo and java-redis-demo show up with volume, error rate, and latency.

Topology draws nginx-otel-demo → java-redis-demo → Redis.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/epov100r6v9hxqllxlv9.png

Waterfall: /hello (nginx) → GET /hello (java) → AUTH/SET/GET (redis), total 975ms, TraceID 7dff87e865b7fe975877a506db26a2ab. The Demo sleeps about one second in Java on purpose — nginx Span (975ms) and Java Span (974.75ms) rise together, so Nginx duration really covers the whole request. Redis even records AUTH as its own Span when the connection uses a password.

Don't trust the UI alone — check Doris

SELECT service, name, span_id, parent_id, is_parent, duration
FROM trace_dc_span
WHERE trace_id='7dff87e865b7fe975877a506db26a2ab'
ORDER BY startTime;

nginx-otel-demo  /hello      (empty, root)  975000000
java-redis-demo  GET /hello  <nginx span>   974745285
[redis]…:16379   AUTH        <java span>     638593
[redis]…:16379   SET         <java span>     361212
[redis]…:16379   GET         <java span>     164941

Filter the trace list by nginx / java when you're hunting edge issues.

Try it tonight — three steps

docker run nginx:1.27-alpine-otel with otel_trace_context propagate;, expose http://<host>:8090/hello.
Start Java with -javaagent:opentelemetry-javaagent.jar pointing at http://127.0.0.1:4318.
curl http://192.168.50.140:8090/hello a few times, wait ~30s for batch export, open DataBuff Traces, filter nginx-otel-demo.

Self-check: nginx root Span present; Java under nginx (not a sibling Trace); Redis SET/GET under Java. If you see two Traces, you almost certainly forgot propagate.

Fifteen minutes on the Nginx module and the Java agent turns the first hop from a blind spot into a span you can scroll.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

What Does a Self-Caring Software Look Like? (DataBuff Self-Monitoring & Self-Troubleshooting)

databufflabs — Mon, 10 Aug 2026 01:22:18 +0000

When you're on call, the worst thing isn't that your business went down — it's that your monitoring platform itself broke and you had no idea. Log volume suddenly drops, a chunk of data silently disappears — and after hours of searching you realize it was never the business at fault: the collection pipeline choked on itself. To make it worse, traditional monitoring software is always watching "everything else," while being a black box itself: CPU spikes, memory full, queues blocked — it never says a single word.

First, what is DataBuff: an open-source AI-native APM where the AI lives directly on top of your OpenTelemetry data — asking questions and troubleshooting are done by AI experts calling tools against real data. Project on GitHub (github.com/databufflabs/databuff). Its approach is different: it treats itself as a business system being monitored. Once installed, you can see whether it's healthy, why it's abnormal, which parameter to tune — you can even say one sentence and it will run an inspection on itself. This article walks through a real case.

1. It watches itself: the platform's self-monitoring page

DataBuff's "Deployment Status" page exposes metrics for all three core components: ingest (collects data), web (queries data), and Doris (stores data). The overview opens with four cards that state the conclusion up front: inbound events per second, any write failures, Doris disk remaining, any query errors.

Scrolling down gets more granular: every signal path in ingest — trace / metric / log — is tracked individually with "how many, how big, how slow, any dropped"; Doris disk and CPU are broken out too.

2. It understands itself: every metric ships with its own manual

More metrics isn't the point — every metric comes with an explanation and its tuning parameters. Click the title of any chart and a drawer pops up: how this metric is computed, whether you should worry, which env var to tune when it goes wrong, and the default value. All explained at once.

Take the "Write Drop" metric: the drawer tells you it only occurs in two situations — the queue is full and a whole batch is dropped, or writes fail consecutively and are discarded; it reminds you that "any sustained drop means data loss — check queue depth, write failures, and Doris liveness first"; and finally it lists the tunable parameters directly.

3. It does health checkups: one sentence, AI inspects the whole platform

Don't want to flip through page after page? Just tell the AI platform "inspect the DataBuff platform and produce an HTML inspection report". The product support expert reads the metric catalog, queries the platform's own self-monitoring metrics, picks out the anomalies, and produces an HTML report you can forward directly.

We ran it live: within minutes it automatically checked ingest, writes, pipeline, query domain, Doris, and process resources. The conclusion matched manual troubleshooting — one query-domain failure was still ongoing, and it flagged it as-is.

4. It diagnoses: logs are dropping, and it finds the cause itself

This data loss isn't made up — it actually happened and was actually fixed in our demo environment. The "Write Drop" metric on the Deployment Status page went red — only the log signal was dropping continuously, thousands to fifteen thousand per minute:

At this point, no one needs to flip through pages. Just say the word in the AI platform and let the expert investigate. It first rules out two "non-issues": the business side is normal — not the app's fault; Doris storage is alive and writes are fast — not the storage's fault. The real cause lands in ingest. Logs are batched and pushed into a write queue — this environment's queue holds only 16 batches, normally 32. Sudden bursts don't fit, and the overflow gets dropped as whole batches. The log shows Doris ready queue full (16/16) every minute — confirmed.

It packages "why it dropped, where it dropped, which param to tune, and to what value" into a conclusion with evidence, and hands over the fix suggestion:

5. It fixes itself: changing params and restarting, all on its own

The suggested fix is exactly this group of INGEST_DORIS_* parameters — the core problem is a too-small queue, just make it bigger:

Parameter	Before	After	Effect
`INGEST_DORIS_MAX_READY_BATCHES`	16	32	Write queue too small for bursts; double it
`INGEST_DORIS_FLUSH_TIMEOUT_MS`	30s	60s	Restore default write timeout
`INGEST_DORIS_FLUSH_BATCH_BYTES`	50 MiB	50 MiB (unchanged)	Not touched this time

And you don't change the params or restart by hand. The product support expert SSHs in, edits the config, and restarts ingest per the suggestion, then re-checks to confirm:

After the restart takes effect, drops return to zero — before the fix, thousands dropped per minute; after the restart, several consecutive minutes at 0, the curve back to normal:

All a human did: ask it to diagnose, then ask it to fix.

Getting started — you can use it the same way

After install, check Deployment Config → Deployment Status; scan the four overview cards for health at a glance
To confirm whether data is being lost: ingest page → "Write Drop", click the title for the explanation
Add "how to fix dropping data" to your on-call runbook: trigger troubleshooting in the AI platform for a fix suggestion, let the product support expert SSH in to change INGEST_DORIS_* and restart ingest
Periodically have the support expert run a platform inspection and forward the report to the team

Closing — the self-discipline software should have

Software used to be done the moment it shipped; problems were handled by people watching, investigating, and restarting. What DataBuff does isn't complicated: it builds the operational work into the software itself, using the same methods you already use to watch your business.

This data loss — from discovery to on-box fix — nobody opened a runbook or guessed a direction even once.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

The OpenTelemetry Ecosystem Is Strong — DataBuff Can Plug Into Its eBPF Trace Path

databufflabs — Thu, 06 Aug 2026 01:24:44 +0000

The OpenTelemetry community packaged eBPF trace collection as OBI (OpenTelemetry eBPF Instrumentation). On Kubernetes you run a DaemonSet per node; eBPF watches HTTP calls between local services, assembles traces, and exports them — app Pods stay unchanged. Image: otel/ebpf-instrument. Project: opentelemetry-ebpf-instrumentation.

Where do traces land? DataBuff is an open-source APM platform for services, topology, and call chains. Point OBI at DataBuff and you're done.

One DaemonSet, one collector Pod per node — ns obi, app Pods untouched, eBPF on local HTTP → export to DataBuff.

Step 1 · Install DataBuff

curl -fsSL https://databuff.ai/databuff/ai-apm-k8s-install.sh | bash
kubectl -n databuff get pods

All Pods Running and the UI opens — you're good.

Step 2 · Check eBPF readiness on app nodes

uname -r
ls /sys/kernel/btf/vmlinux

Kernel 5.8+ recommended; the second command must list a file. Without BTF, collector Pods won't capture.

Step 3 · Image

Use otel/ebpf-instrument:latest (worked in our test; pin a version in production). Skip if the cluster can pull; offline clusters need docker load on nodes.

Step 4 · Apply the DaemonSet

Replace YOUR_APP_NAMESPACE and YOUR_DATABUFF_HOST (ai-apm-ingest.databuff.svc). hostPID + privileged are required.

Key config:

discovery.instrument.k8s_namespace
ebpf.context_propagation: headers
otel_traces_export.endpoint: http://YOUR_DATABUFF_HOST:4318

kubectl apply -f obi.yaml
kubectl -n obi get ds,pods -o wide

DESIRED / READY should match node count.

Step 5 · Logs

kubectl -n obi logs -l app=obi --tail=80 | grep -iE "instrumenting|process|error" | head -30

Look for instrumenting process.

Step 6 · Traffic

for i in $(seq 1 80); do
  curl -sS -m 2 "http://your-app-url/" >/dev/null || true
  sleep 0.2
done

Step 7 · Verify in DataBuff

Open APM → Services, then topology and trace detail.

eBPF vs language Agent

Approach	Better when
eBPF + DaemonSet	No injection/restart; HTTP-first; polyglot quick layer
Language Agent	Dubbo, slow SQL, method stacks; kernel < 5.8 / no BTF

Limitations: no Dubbo RPC yet; no method stacks or custom business spans; privileged Pod + BTF required.

How multi-hop traces connect

With context_propagation: headers and no app code changes:

Ingress — read Traceparent: on incoming HTTP; attach or create trace context.
Local correlate — match outbound HTTP to the inbound request.
Egress — sockmap sk_msg inserts this hop's Traceparent after the request line.

HTTPS uses a separate TCP Option path upstream; this walkthrough is plain HTTP + headers.

Full bilingual post (with more YAML detail): databuff.ai/blog/en/databuff-obi-ebpf-traces

Palantir Made Ontology Hot — Can APM Troubleshooting Stop at Ontology Alone?

databufflabs — Fri, 31 Jul 2026 08:25:28 +0000

"Ontology" is back in fashion. Palantir Foundry is not pitched as yet another data warehouse — it builds an ontology for the enterprise: tables, streams, and sensors become "Customer," "Order," and "Device," with relationships and actions defined so data becomes a world model programs can reason over. LLMs made the pitch louder — models still need to know what exists in the world.

For our field: what is the APM ontology? And if you build it, does incident triage just work? Two separate questions.

Q1 · How to build an APM ontology

An APM ontology is two steps: extract entities and relationships from traces, then materialize them into metric tables.

Setup: service A has instances A-1 and A-2; two endpoints; both call database DB and run the same SQL1. Four traces:

#	service	instance	endpoint	call	sql
T1	A	A-1	ep1	DB	SQL1
T2	A	A-1	ep2	DB	SQL1
T3	A	A-2	ep1	DB	SQL1
T4	A	A-2	ep2	DB	SQL1

Step 1 — extract. From T1: entities are service A, instance A-1, endpoint ep1, database DB, SQL1. Relationships: A has instance A-1; A exposes ep1; A-1 serves requests on ep1; ep1 calls DB; ep1 executes SQL1; DB has statement SQL1. P99, latency, and call count are not entities — they are observations attached to entities.

All four traces together:

Service hasInstance Instance: A → A-1, A-2
Service exposes Endpoint: A → ep1, ep2
Instance serves request on Endpoint: A-1 / A-2 both serve ep1 and ep2
Endpoint calls DB: ep1 → DB; ep2 → DB
Endpoint executes SQL: ep1 → SQL1; ep2 → SQL1
DB hasStatement SQL: DB → SQL1

Step 2 — materialize. Extracted entities become queryable metric tables. In DataBuff's Doris layer (schema is open source), dimension columns are entities; call_count / resp_time are observations:

Table	Dimensions	Observations	Entities
metric_service	service	call_count / resp_time / error_count	Service
metric_service_instance	service + instance	same	Service, Instance
metric_service_http	service + instance + endpoint	same	Service, Instance, Endpoint
metric_service_db	service + instance + db + sql	db calls / db latency	Service, Instance, DB, SQL

Aggregating the four traces:

metric_service: service A → 1 row, 4 entry calls
metric_service_instance: by instance → 2 rows
metric_service_http: by instance + endpoint → 4 rows
metric_service_db: by instance + DB + SQL → 2 rows

Ontology done: traces tell you what exists and what connects; metric tables materialize those entities.

Q2 · Is ontology enough?

Entities and relationships complete — is troubleshooting solved? Consider how a connection pool actually behaves:

Even if you add ConnectionPool as Service → ConnectionPool → Database, you only label the big boxes. The branches and states in between cannot be covered by adding one more entity — that layer is logic:

a) Acquire: idle → return; can grow → create; pool full → lock wait; timeout → error
b) Use / return: run SQL → return → wake waiters
c) Background maintenance: heartbeat, evict stale connections

Entity-only reasoning fails when an endpoint slows down:

	Ontology only	With logic
Reasoning	endpoint → DB, slow ⇒ DB slow	stuck in lock wait, SQL never ran
Symptom	endpoint latency up	DB execution may be fine
Conclusion	root cause is DB	root cause is waiting for a connection

First principles: program = data + logic.

	Ontology	Logic
Manages	what exists, who connects	how requests flow, where they stall
Locks onto	target entities	last-mile root cause
Without it	no drill-down target	stuck at "something is wrong"

Connection pool is one slice — it shows entity-only is not enough. See how DataBuff breaks down latency for one endpoint:

Entry average ~240ms; breakdown shows HTTP service-b ~100ms, RPC service-b ~80ms, remainder in MySQL, ES, Redis, Kafka. That is logic modeling for one entry — track entry latency and state, then each operation type after the entry.

Q3 · How to model logic

Scale to a real app: track latency and state for every entry, and for each operation type after every entry. Entries include RPC, HTTP, MQ consumers, scheduled jobs, and more.

After a request enters, it typically lands in 5 operation types + 1 behavior:

Type	Key dimensions
DB remote operation	dal group / table / operation / sql
Redis remote operation	command
MQ remote operation	exchange / routingKey / vhost
RPC remote operation	downstream service / remote method
Local operation	no extra attributes yet
Thrown exception	exception name

For remote operations, split latency into three parts:

a) Client: connect, send, receive
b) Network: transport
c) Server: peer execution

Pool-full lock wait accrues on the client — waiting for a connection ≠ slow SQL.

Incidents then follow one chain: which entries → local ops → remote ops (down to SQL/method) → network vs server → exceptions.

Ontology locks entities; logic locks root cause. Logic models entry → operation → remote three-part latency → exception, with latency and state at each layer. For AI to separate "waiting for a connection" from "slow SQL," it must land on entities via ontology, then walk this logic chain to the actual reason.

Install Failed but the Page Still Opens — Ops Expert Restores Doris

databufflabs — Tue, 28 Jul 2026 01:27:27 +0000

Almost every product hits two kinds of user pain:

Can't use it: install fails, won't start, or dies in production — errors only, then humans grep logs and guess commands.
Don't know how: docs elsewhere, menus buried — ask a helper and get "see the official documentation."

Self-healing ops fixes the first: when storage is down the product stays usable, SSH for evidence, change config, restart, verify.

In-product Q&A fixes the second: ask the product in natural language; answers come from in-product docs and your deployment.

DataBuff is an open-source, AI-native OpenTelemetry APM — metrics, traces, logs first; AI on the same telemetry. Three pieces: Ingest, Doris, AI platform / Web.

curl -fsSL https://databuff.ai/install.sh | bash

Self-healing in practice: Doris breaks during install. Many products go dark; you SSH and guess. We inject mem_limit: 256m on Doris BE so start.sh exits non-zero. Troubleshooting mode keeps Web up while Doris is unhealthy.

Configure an LLM, open AI chat, pick Ops Expert, grant SSH, ask for locate-and-fix:

I installed DataBuff on 192.168.50.140 (dir /opt/databuff-ai-apm-failover).
install/start failed but Web opens.
SSH to root@192.168.50.140 (password Databuff@123)
Find why Doris FE/BE is not ready; fix root cause and report results.
Install dir: /opt/databuff-ai-apm-failover

On the host: BE Restarting → mem_limit: 256m OOM → memory raised to 4g, config persisted, ingest restarted, full-stack Healthy.

Terminal matches: four containers healthy, Doris SELECT 1 passes. Troubleshooting mode exits automatically.

In-product Q&A on the same AI entry — OTel ingest and alert thresholds:

Then LLM setup — menu paths, required fields, common pitfalls:

In one line: self-healing fixes "can't use it"; in-product Q&A fixes "don't know how."

One Entry, Many Experts: How DataBuff Orchestrates Parallel Ops AI

databufflabs — Sun, 26 Jul 2026 01:26:58 +0000

It's 2 a.m. The alert channel blows up.

Someone needs error rates. Someone else wants traces. Another person suspects a full disk on a container. You open one AI chat — it either hand-waves, or mixes metrics, logs, and SSH into one muddy answer. The more you ask, the messier it gets.

Open more windows? Worse. Each bot talks past the others. You still don't have one incident-ready summary you can paste into the war room.

DataBuff treats multi-expert coordination like a hospital: triage desk + specialist consults. You speak to one entry point. Data Query, Inspection, Ops, and Product Q&A work in parallel behind the scenes, then roll up into an evidence-backed report. This post covers two things: why on-call actually needs multiple experts, and how to dispatch, parallelize, and merge without chaos — framework only, no source-code dump.

Why one bigger prompt is not enough

The instinct is: one stronger model, one longer prompt, done. On a real incident, that breaks — for the same reason a hospital does not send one doctor to run the ECG, draw blood, read the CT, and operate at once.

Chest pain in the ER: you do not expect a single generalist to do every specialty with every machine. Wrong tools, wrong permissions, shallow on everything. Lab values and surgical plans get mixed up. Bad outcomes follow.

On-call is the same. Ask "anything wrong with the cluster?" and you may need:

Data Query — latency and error rates in Doris, slow traces (labs)
Inspection — JVM/GC/dependency sweeps across dozens of services (screening)
Ops Expert — SSH, logs, disk, processes (surgery)
Product Q&A — where OTLP ports and alert menus live (admin desk)

Four different toolchains, permissions, and report formats. Stuffing them into one mega-prompt balloons context: metric semantics bleed into shell commands. Models cross-contaminate, overreach, hallucinate.

More chat tabs do not fix it. That is the family running between departments with no triage: no routing, no shared chart, you stitch the story at 2 a.m.

Multi-expert is not hype. Ops problems are cross-domain, parallel, and evidence-heavy — same shape as triage → specialty work → attending synthesis.

One front door, specialists in the back

Think of DataBuff's AI platform as on-call consult:

You talk only to the triage desk (AI Brain)
The desk does not run labs or operate — it pages the right specialty
Each expert queries real telemetry and returns evidence
The desk merges into something you can act on

This is not "sidebar ChatGPT." Classic APM + chat often cannot see your live metrics and traces. DataBuff is AI-native APM on OpenTelemetry — experts must call tools; guessing is not the design.

Three layers, not one giant prompt

New capability is added in three layers:

Tools — instruments: service lists, traces, inspections, controlled commands
Experts — roles: Data Query knows metrics, Inspection scans the fleet, Ops goes on-host, Q&A reads product docs
Playbooks — how each role investigates and writes reports; plus routing rules for the Brain

A new "department" ≈ tools + playbook + expert registration. The hospital building (your observability pipeline) stays put. You still talk to one desk.

How the Brain dispatches

The Brain loads routing rules, reads every expert's charter, and semantically matches your request. It emits a dispatch: which expert + task brief. Two hard rules: faithful to your intent, and the Brain does not do the work — no metrics, no inspection runs, no shell on its own. Triage and synthesis only.

Pain point 1: what does dispatch actually look like?

A ticket-shaped loop:

You message AI Brain; it reads routing rules and picks experts.
Brain issues one dispatch: target expert + task brief (your words preserved).
System immediately replies "accepted, please wait" — ticket created, work not finished.
The expert runs in the background: tools, traces, inspection, controlled commands.
Expert returns a delivery artifact to the same session and wakes Brain for another turn.

Pain point 2: why not wait synchronously?

Expert jobs are slow. Long-lived SSE connections time out. True parallelism dies if you block serially. Experts are multi-step tool loops. So dispatch is async by design: Brain ends its turn, experts run, callbacks wake Brain when done.

Pain point 3: four guardrails so async does not sprawl

Same expert, serial only — duplicate dispatch while busy is rejected.
Different experts may run in parallel — Inspection + Data Query together.
Session-level pending counter — dispatch +1, delivery −1.
Final answer only when pending = 0 — Brain must rewrite a complete reply, not "as above."

Live demo: one sentence, two experts

On demo.databuff.ai, open AI Brain. Do not pick an expert manually. Paste:

Any cluster anomalies in the last hour? Run a joint diagnosis with Data Query and Intelligent Inspection: Data Query checks latency, error rate, and slow traces; Inspection runs tiered health checks; summarize into an incident report I can forward.

Brain dispatches twice in parallel — Data Query and Inspection.

Inspection returns an S/A/B tiered view: most of 34 services in S/A; service-b at B with InsufficientStockException on SKU DEMO-10001. Brain merges into HTML with P0/P1 action items — ready for the incident channel:

Most products still make you pick an expert first

Many "multi-agent" UIs show four tiles — Data Query, Inspection, Ops, Q&A. You still guess which door to knock on. One conversation belongs to one expert. Switch experts, new window, context lost. Ask for error rates and a JVM sweep in one breath? You play dispatcher across tabs.

No triage desk — self-serve specialty signup. Wrong line wastes time; right line still means you collect reports from every window.

The gap is not four avatars vs one. It is who routes work: you picking single-expert chats, vs Brain accepting, dispatching, parallelizing, and merging with an evidence chain. That is what on-call needs: one entry, Brain dispatch, legal parallelism, traceable rollup.

Good systems do not make you guess the specialty. Hand off like shift change — one sentence in, triage and consults happen backstage. Try it on demo.databuff.ai with the prompt above.

Try DataBuff

Open source · multi-expert orchestration · one entry, parallel dispatch

Online Demo: https://demo.databuff.ai

GitHub: https://github.com/databufflabs/databuff

If this helped, star the repo — and run the demo prompt yourself once.

Is Your Data Quality Good Enough for the AI Era?

databufflabs — Thu, 23 Jul 2026 01:37:09 +0000

When AI troubleshooting fails, people blame the Prompt, then the model. More often the problem is upstream: the telemetry you feed the AI is not good enough — APIs, SQL, entrypoints, and hops don’t line up, so even a strong model can only tell you to “go dig the traces yourself.” Same rule as RAG and AI support: bad data, fancy layers don’t help.

How do you know you “pass”? This article does three things:

Set a four-question bar
Score DataBuff, SkyWalking, Jaeger, Pinpoint, SigNoz, and OpenObserve with the same ruler
Show how DataBuff uses 12 fixed metric_service_* tables to meet that bar

1. A scenario you’ve probably hit

Checkout is slow in production. You plug AI into your APM and ask:

“Why is checkout slow? Which SQL is the bottleneck?”

Fail

“Please open the trace detail, find the root span manually, then correlate the DB span…”

AI becomes a fancy search box — barely better than no AI.

Pass

“Slow on SELECT … FROM orders, triggered by /checkout; the payment hop has the highest latency.”

No raw-trace digging — a direct answer.

In the AI era, data quality wins. Everyone can copy Prompts; a curated metric schema can’t be copied overnight. Bad datasets just make stronger models tell you to dig yourself.

Analogy: raw telemetry is unlabeled video footage; a high-quality dataset is a fixed-column spreadsheet — which API, which SQL, who triggered it — AI can answer from that.

2. The bar: four questions AI must answer

Forget field names. Ask your APM these four (with or without AI) — can it answer directly? That’s the bar.

Which API is broken?

HTTP, DB, and MQ must not share one Span-name pile. Redis GET and checkout POST need separate stats.
Which SQL / which database is slow?

“MySQL avg 50ms” is not enough — you need the exact statement.
Which page triggered the slow SQL?

The critical question. Many stacks can’t answer — you’re sent back to hunt the entry by hand.
Which hop on the call path is dragging?

order → payment → MySQL: traffic and errors per hop — not just a topology thumbnail.

Pass = all four answered directly. If one answer is “go dig the traces yourself,” the dataset fails.

3. App-performance matrix: where the six diverge

Same-environment application performance matrix. Basics (topology / service list / Trace) are common; the highlighted capability rows open the gap: call analysis, service flow, middleware pages.

Legend: ✅ verified · △ entry exists / limited depth · ❌ no equivalent

Versions tested: DataBuff v0.1.4 · SkyWalking 10.4.0 · Jaeger 1.76 · Pinpoint 3.1.0 · SigNoz 0.133 · OpenObserve 0.91-rc1.

Capability	DataBuff	SkyWalking	Jaeger	Pinpoint	SigNoz	OpenObserve
1. Global topology	✅	✅	△	✅	✅	❌
2. Service list / golden metrics	✅	✅	❌	✅	✅	✅
3. Service-level topology	✅	✅	△	✅	△	❌
4. Service call analysis	✅	❌	❌	❌	❌	❌
5. Instance golden metrics	✅	✅	❌	△	❌	❌
6. Instance topology	✅	❌	❌	❌	❌	❌
7. Instance call analysis	✅	❌	❌	❌	❌	❌
8. API-level topology	✅	❌	❌	❌	❌	❌
9. API call analysis	✅	❌	❌	△	❌	❌
10. Service flow	✅	❌	❌	❌	❌	❌
11. Middleware pages (DB / cache / MQ)	✅	❌	❌	△	❌	❌
12. Error analysis	✅	❌	❌	△	❌	△
13. Trace list / search	✅	✅	✅	✅	✅	✅
14. Trace detail	✅	✅	✅	✅	✅	✅
15. Span ↔ logs	✅	✅	△	❌	✅	✅
16. Log list / search	✅	✅	❌	❌	✅	✅
17. Log detail	✅	✅	❌	❌	✅	✅
18. Logs ↔ Trace	✅	✅	❌	❌	✅	✅

How to read it

All six can search traces; the gap is call analysis, service flow, middleware pages.
For humans staring at UI, SkyWalking / Pinpoint often suffice; for AI to answer the four questions, you need that depth.
Those green cells aren’t extra menus — they’re the backbone of the causality chain: which API, which SQL, who triggered it, which hop dragged.

4. How do you prepare a passing dataset?

Those capabilities aren’t pages bolted on later — incoming spans are written into fixed tables by type. DataBuff ships this as 12 metric_service_* tables.

Spans arrive
↓ Split by type: HTTP / DB / Redis / MQ / RPC… one table each
↓ Freeze key columns: entry API, SQL digest, path hops with the metrics
↓ Query joins the chain; UI grows middleware pages / service flow / call analysis

#	Table	Stores	Answers
1	`metric_service`	Service-entry RED	Is this service healthy?
2	`metric_service_trace`	Trace root	End-to-end success and duration
3	`metric_service_http`	HTTP API typing	Which URL / method / status is bad?
4	`metric_service_db`	DB calls	Which SQL is slow, who triggered (entry on same row)
5	`metric_service_flow`	Entry path tree	From entry, which hop drags?
6	`metric_service_rpc`	RPC calls	gRPC / Dubbo method and status
7	`metric_service_redis`	Cache calls	Who issues GET/SET, is it slow?
8	`metric_service_mq`	Messaging	Topic produce/consume, lag
9	`metric_service_remote`	External deps	External API QPS / latency
10	`metric_service_exception`	Entry exceptions	Exception name / code
11	`metric_service_config`	Config reads	Are Nacos / ZK reads slow?
12	`metric_service_instance`	Instance metadata	Pod / host / Java version (JOIN)

For the four questions, lead with tables 1 / 3 / 4 / 5.

Three words before you query:

tag — filter columns (url, sqlContent, rootResource)
field — numeric columns (cnt, sumDuration)
virtual service — e.g. [mysql]demo_apm as a topology node

HTTP / DB / MQ / cache typed into topology — from component tables, not one Span-name dump

5. Four questions → which table

Question	Primary table	Key tags
Is the service healthy? (warmup)	`metric_service`	`service`, `errorType`
1. Which API is broken?	`metric_service_http`	`url`, `httpMethod`, `httpCode`
2. Which SQL is slow?	`metric_service_db`	`sqlContent`, `isSlow`
3. Who triggered the slow SQL?	`metric_service_db`	`rootResource` (same row as sqlContent)
4. Which hop drags?	`metric_service_flow`	`entryInterfacePathId`, `pathId`, `parentService`

① metric_service — warmup

Answers: “service-a QPS, error rate, avg latency today?”

Only entry requests write a row — filter out DB / Redis / MQ spans.

Service list — product face of metric_service

② metric_service_http — Q1

Answers: “Slowest URL? GET or POST? 4xx or 5xx?”

URLs like /demo/checkout as their own rows

③ metric_service_db — Q2 and Q3

Answers: “Which SQL is slow? Which entry triggered it?” — same table, same row.

sqlContent + rootResource on one row is the most important design for the bar: no Trace hunt for the trigger.

[mysql]demo_apm / ES as virtual services

④ metric_service_flow — Q4

Answers: “From service-a entry, how much response each hop contributes?”

The path tree is computed once a Trace is complete — not one hop per arriving Span.

Entry service-a expands downstream with response share

6. End-to-end: slow checkout

Entry health — metric_service: did service-a error rate / avg latency spike?
Pin the slow API — metric_service_http: confirm url is /demo/checkout
Slow SQL + trigger — metric_service_db: rootResource='/demo/checkout' AND isSlow=1
Which hop drags — metric_service_flow: expand from service-a, compare response share
(Optional) which machine — JOIN metric_service_instance

That’s data quality: filter and aggregate on fixed columns — not guessing Span names, not sending humans back into raw traces.

7. Three-step self-check

Run the four questions — if one answer is “dig the traces yourself,” you fail. Chat ≠ troubleshooting.
Check the highlighted matrix rows — call analysis / service flow / middleware pages — especially “slow SQL → entry.”
If you fail — add fixed columns (entry API, SQL digest) at write time, or switch to a more complete dataset.

Close: In the AI era, ask is your data quality good enough — before Prompts and models. Four questions set the bar; the matrix shows the product surface; the 12 tables are why.

Try DataBuff

Open source · OpenTelemetry · datasets designed for AI querying

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full post: https://databuff.ai/blog/en/trace-metrics-data-quality/

If this helped, leave a ❤️ or a Star on GitHub — and tell us which of the four questions your stack still can’t answer.

DataBuff vs Jaeger: Same-Host Lab Comparison

databufflabs — Thu, 23 Jul 2026 01:33:56 +0000

Same-host lab: DataBuff (OTLP :4318) and Jaeger all-in-one (OTLP / UI :16686) side by side on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · Jaeger v1.76.0. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

1. Capability matrices

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

Capability	Jaeger v1.76.0	DataBuff v0.1.4
① See · natural-language questions	❌	✅ Ask about services / topology / trends; AI reads telemetry
② Squad · multi-agent collaboration	❌	✅ Parallel evidence gathering; reusable task orchestration
③ Inspect · service inspection + report	❌	✅ One-shot inspection with evidence and actions
④ Diagnose · bottleneck / RCA evidence	❌	✅ Trace / metrics / topology evidence (not a black-box “root cause”)
⑤ Repair · Ops Expert actions	❌	✅ Repair under policy + human approval; dangerous-command denylist
⑥ Predict · capacity / trends	❌	✅ Capacity and trend analysis — from after-the-fact to ahead-of-time
⑦ Answer · product Q&A	❌	✅ Answers deploy / ingest / config from docs and code
Extend · MCP / Skill / custom experts	❌	✅ External MCP / Skill and custom digital experts

Largest gap: Jaeger is a distributed tracing backend with no equivalent AI platform; DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

Capability	Jaeger v1.76.0	DataBuff v0.1.4
1. Global topology	△ Dependencies (service DAG; this lab shows service-a → service-b)	✅ Topology + health colors + drill-down (incl. middleware)
2. Service list & golden metrics	❌ Search dropdown only; no dedicated service list / golden-metric charts	✅ Service list + charts; same demo shows service-a / b
3. Service-level topology	△ Via Dependencies only	✅ Dedicated service topology
4. Service call analysis (up/downstream + Trace)	❌	✅ Upstream/downstream structure, latency/contribution; drill to Trace
5. Instance golden metrics	❌	✅ Instance golden-metric charts / list
6. Instance topology	❌	✅ Dedicated instance topology
7. Instance call analysis (up/downstream + Trace)	❌	✅ Per-instance up/downstream + Trace
8. Endpoint topology	❌	✅ Dedicated endpoint topology
9. Endpoint call analysis (up/downstream + Trace)	❌ Mostly Trace search filters	✅ Per-endpoint caller/callee + Trace
10. Service flow (service / endpoint Trace contribution)	❌ Dependencies answers “who connects” only	✅ Response contribution from entry; service / endpoint Trace view
11. Middleware / external pages (DB / cache / MQ / external)	❌	✅ Dedicated pages: DB / cache / MQ / external
12. Error analysis (stats + endpoint)	❌ Mostly Trace status filters	✅ Error stats + endpoint drill-down
13. Trace list / search	✅ Service / operation / Tags / time — mature search UX	✅ Charts + list, multi-dimension filters
14. Trace detail	✅ Classic Waterfall + Tags + Span Logs	✅ Call-order waterfall + Span attributes
15. Trace Span → logs	△ Span Logs (instrumentation events) only; no OTLP app-log link	✅ Top “Log analysis” + Span Logs / Logs tab
16. Log list / search	❌	✅ Log analysis list / search
17. Log detail	❌	✅
18. Log → Trace	❌	✅ Log → Trace, down to Span

Jaeger is strong on pure Trace search and waterfall. Most other APM surfaces (golden metrics, multi-level topology / call analysis, service flow, middleware pages, logs) are absent. DataBuff leads there and on Span↔log linkage.

Alerting

Capability	Jaeger v1.76.0	DataBuff v0.1.4
How rules are configured	❌ No built-in alerting product	✅ Alert center in product
Threshold alerts	❌ Needs Prometheus / Alertmanager, etc.	✅ Managed in platform
Smart alerts	❌	✅ Linked with APM metrics
Alert event list	❌	✅ Non-empty in this lab
Alerts linked to service / middleware	❌	✅ List links back into APM

Jaeger itself does not alert; threshold / notify stacks are external. DataBuff keeps rule config, event list, and service context in one alert center.

When to pick which

Scenario	Better fit	Note
Already on OTLP, want AI / APM depth first	DataBuff (side-by-side)	Point ingest at DataBuff
Need the seven AI capabilities	DataBuff	No Jaeger AI platform
MCP / Skill / custom experts	DataBuff	Jaeger has no such layer
See who slows the entry response	DataBuff	Service flow + contribution
Call analysis → Trace (service / instance / endpoint)	DataBuff	No Jaeger path
Slow SQL / cache / MQ pages	DataBuff	Jaeger has no middleware pages
Log + Trace correlation	DataBuff	Jaeger has no log product surface
Built-in / smart alerts	DataBuff	Jaeger needs external stack
Lightweight Trace storage + waterfall only	Jaeger / either	No need to migrate for brand
Already on ES / Cassandra and Trace-only	Jaeger	Reuse storage; DataBuff can still OTLP side-by-side

Boundary: Deep Jaeger search workflow lock-in, or Trace-only needs → stay on Jaeger. DataBuff fits same OTLP data + AI + APM depth + alerts, side-by-side or gradual switch.

2. Screenshot evidence (explains the tables)

Screenshots from the same lab (Jaeger UI :16686; DataBuff v0.1.4). Captions map to capability rows. Focus on DataBuff’s AI / call analysis / dedicated pages / alerts. Jaeger’s strength is pure Trace search and waterfall.

Seven AI capabilities (no Jaeger equivalent UI)

DataBuff AI chat home and seven capability entries (no Jaeger equivalent)

DataBuff ① See: ask about service-a calling service-b; AI reads telemetry

DataBuff ② Squad: digital expert / multi-agent entries

Services & topology

Jaeger Dependencies: service-a → service-b (“who connects”)

DataBuff Global topology + health colors (incl. mysql / redis)

DataBuff Service list + golden-metric charts (Jaeger has Search dropdown only)

Call analysis + service flow (matrix rows 4 / 9 / 10)

Jaeger Dependencies only answers “who connects”. DataBuff goes from “who connects” to “who slows the response, then drill into Trace”.

DataBuff Service call analysis: service-a → service-b (drill to Trace)

DataBuff Endpoint call analysis for /demo/checkout

DataBuff Service flow: entry service-a → downstream response contribution

Trace (Jaeger mature surface)

Jaeger Search: service / operation / Tags filters

Jaeger Trace list: service-a results + scatter

DataBuff Trace list: charts + table

Jaeger Waterfall + Tags + Span Logs

DataBuff Call-order waterfall; can link to application logs

Logs (matrix rows 16–18; no Jaeger equivalent)

DataBuff Log analysis: Log → Trace down to Span

DataBuff dedicated pages (matrix rows 11 / 12)

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis

DataBuff Error analysis

These pages are the depth after Dependencies shows “who connects” — the APM gap most worth verifying side-by-side with Jaeger.

Alerting (no Jaeger built-in alerts)

DataBuff Alert center; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Try DataBuff

Open source · OpenTelemetry APM with AI-native troubleshooting

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full comparison: https://databuff.ai/blog/en/databuff-vs-jaeger/

If this helped, a ❤️ or a GitHub Star is appreciated.

DataBuff vs OpenObserve: Same-Host Lab Comparison

databufflabs — Thu, 23 Jul 2026 01:33:54 +0000

Same-host lab: DataBuff (OTLP :4318) and OpenObserve (OTLP HTTP :5080/api/default) on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · OpenObserve v0.91.0-rc1. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

Positioning: DataBuff = AI-native APM depth; OpenObserve = unified observability platform (Logs / Metrics / Traces / RUM), strong on log search and object-storage cost.

1. Capability matrix

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
① See · natural-language questions	❌	✅ Ask about services / topology / trends; AI reads telemetry
② Squad · multi-agent collaboration	❌	✅ Parallel evidence gathering; serial context preservation; reusable task orchestration
③ Inspect · service inspection + report	❌	✅ One-shot inspection with evidence and recommended actions
④ Diagnose · bottleneck / RCA evidence	❌	✅ Trace / metrics / topology evidence (not a black-box “root cause”)
⑤ Repair · Ops Expert actions	❌	✅ Repair under policy + human approval; dangerous-command denylist
⑥ Predict · capacity / trends	❌	✅ Capacity and trend analysis — from after-the-fact to ahead-of-time
⑦ Answer · product Q&A	❌	✅ Answers deploy / ingest / config from docs and code
Extend · MCP / Skill / custom experts	❌	✅ External MCP / Skill and custom digital experts

Largest gap: OpenObserve has no equivalent AI platform (Traces has an LLM Insights entry, not validated here as APM triage); DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
1. Global topology	❌ No service dependency topology	✅ Global topology + health colors + node drill-down
2. Service list & golden metrics	✅ Service Catalog (Requests / Error Rate / P99, etc.)	✅ Service list + charts; same demo shows service-a / b
3. Service-level topology	❌	✅ Dedicated service topology
4. Service call analysis (up/downstream + Trace)	❌	✅ Upstream/downstream structure, latency/contribution; drill to Trace
5. Instance golden metrics	❌	✅ Instance golden-metric charts / list
6. Instance topology	❌	✅ Dedicated instance topology
7. Instance call analysis (up/downstream + Trace)	❌	✅ Per-instance up/downstream + Trace
8. Endpoint topology	❌	✅ Dedicated endpoint topology
9. Endpoint call analysis (up/downstream + Trace)	❌	✅ Per-endpoint caller/callee + Trace
10. Service flow (service / endpoint Trace contribution)	❌	✅ Response contribution from entry; service / endpoint Trace view
11. Middleware / external pages (DB / cache / MQ / external)	❌ db/http visible on Span fields; no dedicated pages	✅ Dedicated pages: DB / cache / MQ / external
12. Error analysis (stats + endpoint)	△ Can filter ERROR spans / logs	✅ Error stats + endpoint drill-down
13. Trace list / search	✅ Spans/Traces + flexible query; this lab shows service-a · GET /demo/checkout	✅ Charts + list, multi-dimension filters
14. Trace detail	✅ Waterfall / Flame Graph / Trace Graph	✅ Call-order waterfall + Span attributes
15. Trace Span → logs	✅ Trace / Span can link to logs	✅ Top “Log analysis” + Span Logs / Logs tab
16. Log list / search	✅ Strength: SQL / full-text + histogram; hundreds of events in this lab	✅
17. Log detail	✅	✅
18. Log → Trace	✅ Log → Trace (down to Span)	✅ Log → Trace, down to Span
19. Flexible Metrics query (SQL / PromQL)	✅ Metrics page: SQL / PromQL / Builder	△ Internal SQL; no public PromQL entry
20. Custom dashboards	✅ Dashboards can be created (list may be empty in this lab; capability present)	❌ Not yet
21. Unified storage cost (object store + compression)	✅ Home shows Ingested / Compressed (~96MB → 10.5MB in this lab)	△ Doris columnar; not an object-storage cost story
22. RUM	✅ Built-in RUM (Real User Monitoring)	❌ Not yet
23. Pipelines	✅ Realtime / Scheduled: transform / enrich / filter / route after ingest (VRL); logs→metrics, etc.	❌ Not yet
24. Reports	✅ Scheduled / Cached reports; timed generate & distribute	❌ Not yet

Shared base: service list & golden metrics, Trace list/waterfall, logs, and Span↔log links. DataBuff leads on topology / service·instance·endpoint call analysis / service flow / middleware pages. OpenObserve leads on log search & cost, SQL/PromQL, custom dashboards, Pipelines, Reports, unified L/M/T + RUM.

Alerting

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
How rules are configured	✅ Alerts UI (needs Destination / Template first)	✅ Alert center in product
Threshold alerts	✅ Scheduled / Realtime	✅ Managed in platform
Smart alerts	❌ No equivalent smart-alert product	✅ Smart alerts linked with APM metrics
Alert event list	✅ Alerts UI for triggered alerts / rules	✅ Alert list (severity / service / time)
Alerts linked to service / middleware	△ Stream-oriented alerts; APM context must be stitched manually	✅ List links service / middleware back into APM

Both can configure alerts in UI; difference is smart alerts and alert → APM service context. OpenObserve alerts lean Logs/Metrics streams; DataBuff leans APM triage loop.

When to pick which

Scenario	Better fit	Note
Already on OTLP; want AI / APM depth first	DataBuff	Point exporters at DataBuff; no need to migrate off OpenObserve first
Need the seven AI capabilities	DataBuff	OpenObserve has no equivalent AI platform
Need MCP / Skill or custom digital experts	DataBuff	AI platform is extensible; OO has no such layer
Need global topology + health colors at a glance	DataBuff	OO has no service dependency topology
Need “who slowed the response” from entry service	DataBuff	Service flow + contribution; OO has no equivalent page
Need service / instance / endpoint call analysis → Trace	DataBuff	Three-level call analysis all link to Trace; OO has no path
Need instance golden metrics / instance topology	DataBuff	OO has no equivalent instance pages
Need slow SQL / cache / MQ / external service pages	DataBuff	OO mostly Span fields; no dedicated pages
Need dedicated error analysis	DataBuff	OO requires manual ERROR filtering
Need smart alerts tied to service / middleware	DataBuff	OO alerts are stream-oriented; no smart-alert APM loop
Huge log volume; need object-storage cost control	OpenObserve	Compression / storage story is a strength
Need SQL / PromQL Metrics + custom dashboards	OpenObserve	DataBuff has no custom dashboards yet
Need post-ingest transform / enrich / filter / route	OpenObserve	Pipelines (Realtime / Scheduled + VRL)
Need scheduled / cached reports	OpenObserve	Reports (Scheduled / Cached)
Need unified Logs + Metrics + Traces + RUM	OpenObserve	DataBuff focuses on APM depth
Only need the same Demo Trace waterfall	Either	No need to migrate for branding

Boundary: Staying on OpenObserve is reasonable when deeply tied to its log pipelines / dashboards / Pipelines / Reports / cost story. DataBuff fits OTLP + seven AI + topology / call analysis / service flow / dedicated pages / smart alerts. Dashboards, Pipelines, Reports, and large-scale log cost are not peer capabilities yet.

2. Screenshot evidence (explains the tables)

Screenshots from the same lab (192.168.50.140). Captions map to capability rows. Focus on DataBuff AI / topology / dedicated pages / alerts, plus OpenObserve logs / Trace / Metrics / dashboard entry points.

Seven AI capabilities (no OpenObserve equivalent UI; DataBuff evidence)

DataBuff AI chat home and seven capability entries: See / Squad / Inspect / Diagnose / Repair / Predict / Answer

DataBuff ① See: real question on service-a checkout / service-b; AI reads telemetry

DataBuff ② Squad: digital expert / multi-agent entries (no OpenObserve equivalent)

Overview & data plane

OpenObserve Home: Streams≈38 · Events≈205K · Ingested 96MB → Compressed 10.5MB (matrix row 21)

OpenObserve Streams: unified Logs / Metrics / Traces data plane

Services & topology

DataBuff Service list + golden-metric charts (peers with OO Service Catalog)

DataBuff Global topology + health colors (mysql can show red); OpenObserve has no topology view

Service / endpoint call analysis + service flow (matrix rows 4 / 9 / 10)

OpenObserve can list Trace / Span, but has no service / instance / endpoint call analysis, and no service-flow contribution view. DataBuff goes from “who connects” to “who slows, then into Trace”.

DataBuff Service call analysis: service-a → service-b (drill to Trace)

DataBuff Endpoint call analysis for /demo/checkout: volume / latency / error rate

DataBuff Service flow: entry service-a → downstream response contribution; no OpenObserve equivalent

Trace

OpenObserve Traces: service-a · GET /demo/checkout · 240ms · 13 Spans

DataBuff Trace list: charts + table

OpenObserve Trace Waterfall: service-a → service-b · includes SQL / Dubbo Spans

DataBuff Call-order waterfall + Span attributes; can link to logs in-product

Log / Metric

OpenObserve Logs: histogram + event table; Trace / Span correlation (matrix rows 15–18)

DataBuff Log analysis: Log → Trace down to Span

OpenObserve Metrics: SQL / PromQL / Builder (matrix row 19; DataBuff has no public PromQL)

Dashboards / Pipelines / Reports (matrix rows 20 / 23 / 24; OpenObserve strengths)

OpenObserve Dashboards: create custom boards

OpenObserve Pipelines: Realtime / Scheduled data pipelines

OpenObserve Reports: Scheduled / Cached

DataBuff dedicated pages (matrix rows 11 / 12; no OO equivalents)

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis list

DataBuff Error analysis (stats + endpoint)

These pages are depth beyond “middleware Spans in Trace” — the APM differences most worth validating against OpenObserve.

Alerting

OpenObserve Alerts UI: configure and view rules / events

DataBuff Alert center: rules in-product; list links service context; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Online Demo: https://demo.databuff.ai

Try DataBuff

Open source · OpenTelemetry APM with AI-native troubleshooting

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full comparison: https://databuff.ai/blog/en/databuff-vs-openobserve/

If this helped, a ❤️ or a GitHub Star is appreciated.