DEV Community: databufflabs

The OpenTelemetry Ecosystem Is Strong — DataBuff Can Plug Into Its eBPF Trace Path

databufflabs — Thu, 06 Aug 2026 01:24:44 +0000

The OpenTelemetry community packaged eBPF trace collection as OBI (OpenTelemetry eBPF Instrumentation). On Kubernetes you run a DaemonSet per node; eBPF watches HTTP calls between local services, assembles traces, and exports them — app Pods stay unchanged. Image: otel/ebpf-instrument. Project: opentelemetry-ebpf-instrumentation.

Where do traces land? DataBuff is an open-source APM platform for services, topology, and call chains. Point OBI at DataBuff and you're done.

One DaemonSet, one collector Pod per node — ns obi, app Pods untouched, eBPF on local HTTP → export to DataBuff.

Step 1 · Install DataBuff

curl -fsSL https://databuff.ai/databuff/ai-apm-k8s-install.sh | bash
kubectl -n databuff get pods

All Pods Running and the UI opens — you're good.

Step 2 · Check eBPF readiness on app nodes

uname -r
ls /sys/kernel/btf/vmlinux

Kernel 5.8+ recommended; the second command must list a file. Without BTF, collector Pods won't capture.

Step 3 · Image

Use otel/ebpf-instrument:latest (worked in our test; pin a version in production). Skip if the cluster can pull; offline clusters need docker load on nodes.

Step 4 · Apply the DaemonSet

Replace YOUR_APP_NAMESPACE and YOUR_DATABUFF_HOST (ai-apm-ingest.databuff.svc). hostPID + privileged are required.

Key config:

discovery.instrument.k8s_namespace
ebpf.context_propagation: headers
otel_traces_export.endpoint: http://YOUR_DATABUFF_HOST:4318

kubectl apply -f obi.yaml
kubectl -n obi get ds,pods -o wide

DESIRED / READY should match node count.

Step 5 · Logs

kubectl -n obi logs -l app=obi --tail=80 | grep -iE "instrumenting|process|error" | head -30

Look for instrumenting process.

Step 6 · Traffic

for i in $(seq 1 80); do
  curl -sS -m 2 "http://your-app-url/" >/dev/null || true
  sleep 0.2
done

Step 7 · Verify in DataBuff

Open APM → Services, then topology and trace detail.

eBPF vs language Agent

Approach	Better when
eBPF + DaemonSet	No injection/restart; HTTP-first; polyglot quick layer
Language Agent	Dubbo, slow SQL, method stacks; kernel < 5.8 / no BTF

Limitations: no Dubbo RPC yet; no method stacks or custom business spans; privileged Pod + BTF required.

How multi-hop traces connect

With context_propagation: headers and no app code changes:

Ingress — read Traceparent: on incoming HTTP; attach or create trace context.
Local correlate — match outbound HTTP to the inbound request.
Egress — sockmap sk_msg inserts this hop's Traceparent after the request line.

HTTPS uses a separate TCP Option path upstream; this walkthrough is plain HTTP + headers.

Full bilingual post (with more YAML detail): databuff.ai/blog/en/databuff-obi-ebpf-traces

Palantir Made Ontology Hot — Can APM Troubleshooting Stop at Ontology Alone?

databufflabs — Fri, 31 Jul 2026 08:25:28 +0000

"Ontology" is back in fashion. Palantir Foundry is not pitched as yet another data warehouse — it builds an ontology for the enterprise: tables, streams, and sensors become "Customer," "Order," and "Device," with relationships and actions defined so data becomes a world model programs can reason over. LLMs made the pitch louder — models still need to know what exists in the world.

For our field: what is the APM ontology? And if you build it, does incident triage just work? Two separate questions.

Q1 · How to build an APM ontology

An APM ontology is two steps: extract entities and relationships from traces, then materialize them into metric tables.

Setup: service A has instances A-1 and A-2; two endpoints; both call database DB and run the same SQL1. Four traces:

#	service	instance	endpoint	call	sql
T1	A	A-1	ep1	DB	SQL1
T2	A	A-1	ep2	DB	SQL1
T3	A	A-2	ep1	DB	SQL1
T4	A	A-2	ep2	DB	SQL1

Step 1 — extract. From T1: entities are service A, instance A-1, endpoint ep1, database DB, SQL1. Relationships: A has instance A-1; A exposes ep1; A-1 serves requests on ep1; ep1 calls DB; ep1 executes SQL1; DB has statement SQL1. P99, latency, and call count are not entities — they are observations attached to entities.

All four traces together:

Service hasInstance Instance: A → A-1, A-2
Service exposes Endpoint: A → ep1, ep2
Instance serves request on Endpoint: A-1 / A-2 both serve ep1 and ep2
Endpoint calls DB: ep1 → DB; ep2 → DB
Endpoint executes SQL: ep1 → SQL1; ep2 → SQL1
DB hasStatement SQL: DB → SQL1

Step 2 — materialize. Extracted entities become queryable metric tables. In DataBuff's Doris layer (schema is open source), dimension columns are entities; call_count / resp_time are observations:

Table	Dimensions	Observations	Entities
metric_service	service	call_count / resp_time / error_count	Service
metric_service_instance	service + instance	same	Service, Instance
metric_service_http	service + instance + endpoint	same	Service, Instance, Endpoint
metric_service_db	service + instance + db + sql	db calls / db latency	Service, Instance, DB, SQL

Aggregating the four traces:

metric_service: service A → 1 row, 4 entry calls
metric_service_instance: by instance → 2 rows
metric_service_http: by instance + endpoint → 4 rows
metric_service_db: by instance + DB + SQL → 2 rows

Ontology done: traces tell you what exists and what connects; metric tables materialize those entities.

Q2 · Is ontology enough?

Entities and relationships complete — is troubleshooting solved? Consider how a connection pool actually behaves:

Even if you add ConnectionPool as Service → ConnectionPool → Database, you only label the big boxes. The branches and states in between cannot be covered by adding one more entity — that layer is logic:

a) Acquire: idle → return; can grow → create; pool full → lock wait; timeout → error
b) Use / return: run SQL → return → wake waiters
c) Background maintenance: heartbeat, evict stale connections

Entity-only reasoning fails when an endpoint slows down:

	Ontology only	With logic
Reasoning	endpoint → DB, slow ⇒ DB slow	stuck in lock wait, SQL never ran
Symptom	endpoint latency up	DB execution may be fine
Conclusion	root cause is DB	root cause is waiting for a connection

First principles: program = data + logic.

	Ontology	Logic
Manages	what exists, who connects	how requests flow, where they stall
Locks onto	target entities	last-mile root cause
Without it	no drill-down target	stuck at "something is wrong"

Connection pool is one slice — it shows entity-only is not enough. See how DataBuff breaks down latency for one endpoint:

Entry average ~240ms; breakdown shows HTTP service-b ~100ms, RPC service-b ~80ms, remainder in MySQL, ES, Redis, Kafka. That is logic modeling for one entry — track entry latency and state, then each operation type after the entry.

Q3 · How to model logic

Scale to a real app: track latency and state for every entry, and for each operation type after every entry. Entries include RPC, HTTP, MQ consumers, scheduled jobs, and more.

After a request enters, it typically lands in 5 operation types + 1 behavior:

Type	Key dimensions
DB remote operation	dal group / table / operation / sql
Redis remote operation	command
MQ remote operation	exchange / routingKey / vhost
RPC remote operation	downstream service / remote method
Local operation	no extra attributes yet
Thrown exception	exception name

For remote operations, split latency into three parts:

a) Client: connect, send, receive
b) Network: transport
c) Server: peer execution

Pool-full lock wait accrues on the client — waiting for a connection ≠ slow SQL.

Incidents then follow one chain: which entries → local ops → remote ops (down to SQL/method) → network vs server → exceptions.

Ontology locks entities; logic locks root cause. Logic models entry → operation → remote three-part latency → exception, with latency and state at each layer. For AI to separate "waiting for a connection" from "slow SQL," it must land on entities via ontology, then walk this logic chain to the actual reason.

Install Failed but the Page Still Opens — Ops Expert Restores Doris

databufflabs — Tue, 28 Jul 2026 01:27:27 +0000

Almost every product hits two kinds of user pain:

Can't use it: install fails, won't start, or dies in production — errors only, then humans grep logs and guess commands.
Don't know how: docs elsewhere, menus buried — ask a helper and get "see the official documentation."

Self-healing ops fixes the first: when storage is down the product stays usable, SSH for evidence, change config, restart, verify.

In-product Q&A fixes the second: ask the product in natural language; answers come from in-product docs and your deployment.

DataBuff is an open-source, AI-native OpenTelemetry APM — metrics, traces, logs first; AI on the same telemetry. Three pieces: Ingest, Doris, AI platform / Web.

curl -fsSL https://databuff.ai/install.sh | bash

Self-healing in practice: Doris breaks during install. Many products go dark; you SSH and guess. We inject mem_limit: 256m on Doris BE so start.sh exits non-zero. Troubleshooting mode keeps Web up while Doris is unhealthy.

Configure an LLM, open AI chat, pick Ops Expert, grant SSH, ask for locate-and-fix:

I installed DataBuff on 192.168.50.140 (dir /opt/databuff-ai-apm-failover).
install/start failed but Web opens.
SSH to root@192.168.50.140 (password Databuff@123)
Find why Doris FE/BE is not ready; fix root cause and report results.
Install dir: /opt/databuff-ai-apm-failover

On the host: BE Restarting → mem_limit: 256m OOM → memory raised to 4g, config persisted, ingest restarted, full-stack Healthy.

Terminal matches: four containers healthy, Doris SELECT 1 passes. Troubleshooting mode exits automatically.

In-product Q&A on the same AI entry — OTel ingest and alert thresholds:

Then LLM setup — menu paths, required fields, common pitfalls:

In one line: self-healing fixes "can't use it"; in-product Q&A fixes "don't know how."

One Entry, Many Experts: How DataBuff Orchestrates Parallel Ops AI

databufflabs — Sun, 26 Jul 2026 01:26:58 +0000

It's 2 a.m. The alert channel blows up.

Someone needs error rates. Someone else wants traces. Another person suspects a full disk on a container. You open one AI chat — it either hand-waves, or mixes metrics, logs, and SSH into one muddy answer. The more you ask, the messier it gets.

Open more windows? Worse. Each bot talks past the others. You still don't have one incident-ready summary you can paste into the war room.

DataBuff treats multi-expert coordination like a hospital: triage desk + specialist consults. You speak to one entry point. Data Query, Inspection, Ops, and Product Q&A work in parallel behind the scenes, then roll up into an evidence-backed report. This post covers two things: why on-call actually needs multiple experts, and how to dispatch, parallelize, and merge without chaos — framework only, no source-code dump.

Why one bigger prompt is not enough

The instinct is: one stronger model, one longer prompt, done. On a real incident, that breaks — for the same reason a hospital does not send one doctor to run the ECG, draw blood, read the CT, and operate at once.

Chest pain in the ER: you do not expect a single generalist to do every specialty with every machine. Wrong tools, wrong permissions, shallow on everything. Lab values and surgical plans get mixed up. Bad outcomes follow.

On-call is the same. Ask "anything wrong with the cluster?" and you may need:

Data Query — latency and error rates in Doris, slow traces (labs)
Inspection — JVM/GC/dependency sweeps across dozens of services (screening)
Ops Expert — SSH, logs, disk, processes (surgery)
Product Q&A — where OTLP ports and alert menus live (admin desk)

Four different toolchains, permissions, and report formats. Stuffing them into one mega-prompt balloons context: metric semantics bleed into shell commands. Models cross-contaminate, overreach, hallucinate.

More chat tabs do not fix it. That is the family running between departments with no triage: no routing, no shared chart, you stitch the story at 2 a.m.

Multi-expert is not hype. Ops problems are cross-domain, parallel, and evidence-heavy — same shape as triage → specialty work → attending synthesis.

One front door, specialists in the back

Think of DataBuff's AI platform as on-call consult:

You talk only to the triage desk (AI Brain)
The desk does not run labs or operate — it pages the right specialty
Each expert queries real telemetry and returns evidence
The desk merges into something you can act on

This is not "sidebar ChatGPT." Classic APM + chat often cannot see your live metrics and traces. DataBuff is AI-native APM on OpenTelemetry — experts must call tools; guessing is not the design.

Three layers, not one giant prompt

New capability is added in three layers:

Tools — instruments: service lists, traces, inspections, controlled commands
Experts — roles: Data Query knows metrics, Inspection scans the fleet, Ops goes on-host, Q&A reads product docs
Playbooks — how each role investigates and writes reports; plus routing rules for the Brain

A new "department" ≈ tools + playbook + expert registration. The hospital building (your observability pipeline) stays put. You still talk to one desk.

How the Brain dispatches

The Brain loads routing rules, reads every expert's charter, and semantically matches your request. It emits a dispatch: which expert + task brief. Two hard rules: faithful to your intent, and the Brain does not do the work — no metrics, no inspection runs, no shell on its own. Triage and synthesis only.

Pain point 1: what does dispatch actually look like?

A ticket-shaped loop:

You message AI Brain; it reads routing rules and picks experts.
Brain issues one dispatch: target expert + task brief (your words preserved).
System immediately replies "accepted, please wait" — ticket created, work not finished.
The expert runs in the background: tools, traces, inspection, controlled commands.
Expert returns a delivery artifact to the same session and wakes Brain for another turn.

Pain point 2: why not wait synchronously?

Expert jobs are slow. Long-lived SSE connections time out. True parallelism dies if you block serially. Experts are multi-step tool loops. So dispatch is async by design: Brain ends its turn, experts run, callbacks wake Brain when done.

Pain point 3: four guardrails so async does not sprawl

Same expert, serial only — duplicate dispatch while busy is rejected.
Different experts may run in parallel — Inspection + Data Query together.
Session-level pending counter — dispatch +1, delivery −1.
Final answer only when pending = 0 — Brain must rewrite a complete reply, not "as above."

Live demo: one sentence, two experts

On demo.databuff.ai, open AI Brain. Do not pick an expert manually. Paste:

Any cluster anomalies in the last hour? Run a joint diagnosis with Data Query and Intelligent Inspection: Data Query checks latency, error rate, and slow traces; Inspection runs tiered health checks; summarize into an incident report I can forward.

Brain dispatches twice in parallel — Data Query and Inspection.

Inspection returns an S/A/B tiered view: most of 34 services in S/A; service-b at B with InsufficientStockException on SKU DEMO-10001. Brain merges into HTML with P0/P1 action items — ready for the incident channel:

Most products still make you pick an expert first

Many "multi-agent" UIs show four tiles — Data Query, Inspection, Ops, Q&A. You still guess which door to knock on. One conversation belongs to one expert. Switch experts, new window, context lost. Ask for error rates and a JVM sweep in one breath? You play dispatcher across tabs.

No triage desk — self-serve specialty signup. Wrong line wastes time; right line still means you collect reports from every window.

The gap is not four avatars vs one. It is who routes work: you picking single-expert chats, vs Brain accepting, dispatching, parallelizing, and merging with an evidence chain. That is what on-call needs: one entry, Brain dispatch, legal parallelism, traceable rollup.

Good systems do not make you guess the specialty. Hand off like shift change — one sentence in, triage and consults happen backstage. Try it on demo.databuff.ai with the prompt above.

Try DataBuff

Open source · multi-expert orchestration · one entry, parallel dispatch

Online Demo: https://demo.databuff.ai

GitHub: https://github.com/databufflabs/databuff

If this helped, star the repo — and run the demo prompt yourself once.

Is Your Data Quality Good Enough for the AI Era?

databufflabs — Thu, 23 Jul 2026 01:37:09 +0000

When AI troubleshooting fails, people blame the Prompt, then the model. More often the problem is upstream: the telemetry you feed the AI is not good enough — APIs, SQL, entrypoints, and hops don’t line up, so even a strong model can only tell you to “go dig the traces yourself.” Same rule as RAG and AI support: bad data, fancy layers don’t help.

How do you know you “pass”? This article does three things:

Set a four-question bar
Score DataBuff, SkyWalking, Jaeger, Pinpoint, SigNoz, and OpenObserve with the same ruler
Show how DataBuff uses 12 fixed metric_service_* tables to meet that bar

1. A scenario you’ve probably hit

Checkout is slow in production. You plug AI into your APM and ask:

“Why is checkout slow? Which SQL is the bottleneck?”

Fail

“Please open the trace detail, find the root span manually, then correlate the DB span…”

AI becomes a fancy search box — barely better than no AI.

Pass

“Slow on SELECT … FROM orders, triggered by /checkout; the payment hop has the highest latency.”

No raw-trace digging — a direct answer.

In the AI era, data quality wins. Everyone can copy Prompts; a curated metric schema can’t be copied overnight. Bad datasets just make stronger models tell you to dig yourself.

Analogy: raw telemetry is unlabeled video footage; a high-quality dataset is a fixed-column spreadsheet — which API, which SQL, who triggered it — AI can answer from that.

2. The bar: four questions AI must answer

Forget field names. Ask your APM these four (with or without AI) — can it answer directly? That’s the bar.

Which API is broken?

HTTP, DB, and MQ must not share one Span-name pile. Redis GET and checkout POST need separate stats.
Which SQL / which database is slow?

“MySQL avg 50ms” is not enough — you need the exact statement.
Which page triggered the slow SQL?

The critical question. Many stacks can’t answer — you’re sent back to hunt the entry by hand.
Which hop on the call path is dragging?

order → payment → MySQL: traffic and errors per hop — not just a topology thumbnail.

Pass = all four answered directly. If one answer is “go dig the traces yourself,” the dataset fails.

3. App-performance matrix: where the six diverge

Same-environment application performance matrix. Basics (topology / service list / Trace) are common; the highlighted capability rows open the gap: call analysis, service flow, middleware pages.

Legend: ✅ verified · △ entry exists / limited depth · ❌ no equivalent

Versions tested: DataBuff v0.1.4 · SkyWalking 10.4.0 · Jaeger 1.76 · Pinpoint 3.1.0 · SigNoz 0.133 · OpenObserve 0.91-rc1.

Capability	DataBuff	SkyWalking	Jaeger	Pinpoint	SigNoz	OpenObserve
1. Global topology	✅	✅	△	✅	✅	❌
2. Service list / golden metrics	✅	✅	❌	✅	✅	✅
3. Service-level topology	✅	✅	△	✅	△	❌
4. Service call analysis	✅	❌	❌	❌	❌	❌
5. Instance golden metrics	✅	✅	❌	△	❌	❌
6. Instance topology	✅	❌	❌	❌	❌	❌
7. Instance call analysis	✅	❌	❌	❌	❌	❌
8. API-level topology	✅	❌	❌	❌	❌	❌
9. API call analysis	✅	❌	❌	△	❌	❌
10. Service flow	✅	❌	❌	❌	❌	❌
11. Middleware pages (DB / cache / MQ)	✅	❌	❌	△	❌	❌
12. Error analysis	✅	❌	❌	△	❌	△
13. Trace list / search	✅	✅	✅	✅	✅	✅
14. Trace detail	✅	✅	✅	✅	✅	✅
15. Span ↔ logs	✅	✅	△	❌	✅	✅
16. Log list / search	✅	✅	❌	❌	✅	✅
17. Log detail	✅	✅	❌	❌	✅	✅
18. Logs ↔ Trace	✅	✅	❌	❌	✅	✅

How to read it

All six can search traces; the gap is call analysis, service flow, middleware pages.
For humans staring at UI, SkyWalking / Pinpoint often suffice; for AI to answer the four questions, you need that depth.
Those green cells aren’t extra menus — they’re the backbone of the causality chain: which API, which SQL, who triggered it, which hop dragged.

4. How do you prepare a passing dataset?

Those capabilities aren’t pages bolted on later — incoming spans are written into fixed tables by type. DataBuff ships this as 12 metric_service_* tables.

Spans arrive
↓ Split by type: HTTP / DB / Redis / MQ / RPC… one table each
↓ Freeze key columns: entry API, SQL digest, path hops with the metrics
↓ Query joins the chain; UI grows middleware pages / service flow / call analysis

#	Table	Stores	Answers
1	`metric_service`	Service-entry RED	Is this service healthy?
2	`metric_service_trace`	Trace root	End-to-end success and duration
3	`metric_service_http`	HTTP API typing	Which URL / method / status is bad?
4	`metric_service_db`	DB calls	Which SQL is slow, who triggered (entry on same row)
5	`metric_service_flow`	Entry path tree	From entry, which hop drags?
6	`metric_service_rpc`	RPC calls	gRPC / Dubbo method and status
7	`metric_service_redis`	Cache calls	Who issues GET/SET, is it slow?
8	`metric_service_mq`	Messaging	Topic produce/consume, lag
9	`metric_service_remote`	External deps	External API QPS / latency
10	`metric_service_exception`	Entry exceptions	Exception name / code
11	`metric_service_config`	Config reads	Are Nacos / ZK reads slow?
12	`metric_service_instance`	Instance metadata	Pod / host / Java version (JOIN)

For the four questions, lead with tables 1 / 3 / 4 / 5.

Three words before you query:

tag — filter columns (url, sqlContent, rootResource)
field — numeric columns (cnt, sumDuration)
virtual service — e.g. [mysql]demo_apm as a topology node

HTTP / DB / MQ / cache typed into topology — from component tables, not one Span-name dump

5. Four questions → which table

Question	Primary table	Key tags
Is the service healthy? (warmup)	`metric_service`	`service`, `errorType`
1. Which API is broken?	`metric_service_http`	`url`, `httpMethod`, `httpCode`
2. Which SQL is slow?	`metric_service_db`	`sqlContent`, `isSlow`
3. Who triggered the slow SQL?	`metric_service_db`	`rootResource` (same row as sqlContent)
4. Which hop drags?	`metric_service_flow`	`entryInterfacePathId`, `pathId`, `parentService`

① metric_service — warmup

Answers: “service-a QPS, error rate, avg latency today?”

Only entry requests write a row — filter out DB / Redis / MQ spans.

Service list — product face of metric_service

② metric_service_http — Q1

Answers: “Slowest URL? GET or POST? 4xx or 5xx?”

URLs like /demo/checkout as their own rows

③ metric_service_db — Q2 and Q3

Answers: “Which SQL is slow? Which entry triggered it?” — same table, same row.

sqlContent + rootResource on one row is the most important design for the bar: no Trace hunt for the trigger.

[mysql]demo_apm / ES as virtual services

④ metric_service_flow — Q4

Answers: “From service-a entry, how much response each hop contributes?”

The path tree is computed once a Trace is complete — not one hop per arriving Span.

Entry service-a expands downstream with response share

6. End-to-end: slow checkout

Entry health — metric_service: did service-a error rate / avg latency spike?
Pin the slow API — metric_service_http: confirm url is /demo/checkout
Slow SQL + trigger — metric_service_db: rootResource='/demo/checkout' AND isSlow=1
Which hop drags — metric_service_flow: expand from service-a, compare response share
(Optional) which machine — JOIN metric_service_instance

That’s data quality: filter and aggregate on fixed columns — not guessing Span names, not sending humans back into raw traces.

7. Three-step self-check

Run the four questions — if one answer is “dig the traces yourself,” you fail. Chat ≠ troubleshooting.
Check the highlighted matrix rows — call analysis / service flow / middleware pages — especially “slow SQL → entry.”
If you fail — add fixed columns (entry API, SQL digest) at write time, or switch to a more complete dataset.

Close: In the AI era, ask is your data quality good enough — before Prompts and models. Four questions set the bar; the matrix shows the product surface; the 12 tables are why.

Try DataBuff

Open source · OpenTelemetry · datasets designed for AI querying

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full post: https://databuff.ai/blog/en/trace-metrics-data-quality/

If this helped, leave a ❤️ or a Star on GitHub — and tell us which of the four questions your stack still can’t answer.

DataBuff vs Jaeger: Same-Host Lab Comparison

databufflabs — Thu, 23 Jul 2026 01:33:56 +0000

Same-host lab: DataBuff (OTLP :4318) and Jaeger all-in-one (OTLP / UI :16686) side by side on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · Jaeger v1.76.0. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

1. Capability matrices

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

Capability	Jaeger v1.76.0	DataBuff v0.1.4
① See · natural-language questions	❌	✅ Ask about services / topology / trends; AI reads telemetry
② Squad · multi-agent collaboration	❌	✅ Parallel evidence gathering; reusable task orchestration
③ Inspect · service inspection + report	❌	✅ One-shot inspection with evidence and actions
④ Diagnose · bottleneck / RCA evidence	❌	✅ Trace / metrics / topology evidence (not a black-box “root cause”)
⑤ Repair · Ops Expert actions	❌	✅ Repair under policy + human approval; dangerous-command denylist
⑥ Predict · capacity / trends	❌	✅ Capacity and trend analysis — from after-the-fact to ahead-of-time
⑦ Answer · product Q&A	❌	✅ Answers deploy / ingest / config from docs and code
Extend · MCP / Skill / custom experts	❌	✅ External MCP / Skill and custom digital experts

Largest gap: Jaeger is a distributed tracing backend with no equivalent AI platform; DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

Capability	Jaeger v1.76.0	DataBuff v0.1.4
1. Global topology	△ Dependencies (service DAG; this lab shows service-a → service-b)	✅ Topology + health colors + drill-down (incl. middleware)
2. Service list & golden metrics	❌ Search dropdown only; no dedicated service list / golden-metric charts	✅ Service list + charts; same demo shows service-a / b
3. Service-level topology	△ Via Dependencies only	✅ Dedicated service topology
4. Service call analysis (up/downstream + Trace)	❌	✅ Upstream/downstream structure, latency/contribution; drill to Trace
5. Instance golden metrics	❌	✅ Instance golden-metric charts / list
6. Instance topology	❌	✅ Dedicated instance topology
7. Instance call analysis (up/downstream + Trace)	❌	✅ Per-instance up/downstream + Trace
8. Endpoint topology	❌	✅ Dedicated endpoint topology
9. Endpoint call analysis (up/downstream + Trace)	❌ Mostly Trace search filters	✅ Per-endpoint caller/callee + Trace
10. Service flow (service / endpoint Trace contribution)	❌ Dependencies answers “who connects” only	✅ Response contribution from entry; service / endpoint Trace view
11. Middleware / external pages (DB / cache / MQ / external)	❌	✅ Dedicated pages: DB / cache / MQ / external
12. Error analysis (stats + endpoint)	❌ Mostly Trace status filters	✅ Error stats + endpoint drill-down
13. Trace list / search	✅ Service / operation / Tags / time — mature search UX	✅ Charts + list, multi-dimension filters
14. Trace detail	✅ Classic Waterfall + Tags + Span Logs	✅ Call-order waterfall + Span attributes
15. Trace Span → logs	△ Span Logs (instrumentation events) only; no OTLP app-log link	✅ Top “Log analysis” + Span Logs / Logs tab
16. Log list / search	❌	✅ Log analysis list / search
17. Log detail	❌	✅
18. Log → Trace	❌	✅ Log → Trace, down to Span

Jaeger is strong on pure Trace search and waterfall. Most other APM surfaces (golden metrics, multi-level topology / call analysis, service flow, middleware pages, logs) are absent. DataBuff leads there and on Span↔log linkage.

Alerting

Capability	Jaeger v1.76.0	DataBuff v0.1.4
How rules are configured	❌ No built-in alerting product	✅ Alert center in product
Threshold alerts	❌ Needs Prometheus / Alertmanager, etc.	✅ Managed in platform
Smart alerts	❌	✅ Linked with APM metrics
Alert event list	❌	✅ Non-empty in this lab
Alerts linked to service / middleware	❌	✅ List links back into APM

Jaeger itself does not alert; threshold / notify stacks are external. DataBuff keeps rule config, event list, and service context in one alert center.

When to pick which

Scenario	Better fit	Note
Already on OTLP, want AI / APM depth first	DataBuff (side-by-side)	Point ingest at DataBuff
Need the seven AI capabilities	DataBuff	No Jaeger AI platform
MCP / Skill / custom experts	DataBuff	Jaeger has no such layer
See who slows the entry response	DataBuff	Service flow + contribution
Call analysis → Trace (service / instance / endpoint)	DataBuff	No Jaeger path
Slow SQL / cache / MQ pages	DataBuff	Jaeger has no middleware pages
Log + Trace correlation	DataBuff	Jaeger has no log product surface
Built-in / smart alerts	DataBuff	Jaeger needs external stack
Lightweight Trace storage + waterfall only	Jaeger / either	No need to migrate for brand
Already on ES / Cassandra and Trace-only	Jaeger	Reuse storage; DataBuff can still OTLP side-by-side

Boundary: Deep Jaeger search workflow lock-in, or Trace-only needs → stay on Jaeger. DataBuff fits same OTLP data + AI + APM depth + alerts, side-by-side or gradual switch.

2. Screenshot evidence (explains the tables)

Screenshots from the same lab (Jaeger UI :16686; DataBuff v0.1.4). Captions map to capability rows. Focus on DataBuff’s AI / call analysis / dedicated pages / alerts. Jaeger’s strength is pure Trace search and waterfall.

Seven AI capabilities (no Jaeger equivalent UI)

DataBuff AI chat home and seven capability entries (no Jaeger equivalent)

DataBuff ① See: ask about service-a calling service-b; AI reads telemetry

DataBuff ② Squad: digital expert / multi-agent entries

Services & topology

Jaeger Dependencies: service-a → service-b (“who connects”)

DataBuff Global topology + health colors (incl. mysql / redis)

DataBuff Service list + golden-metric charts (Jaeger has Search dropdown only)

Call analysis + service flow (matrix rows 4 / 9 / 10)

Jaeger Dependencies only answers “who connects”. DataBuff goes from “who connects” to “who slows the response, then drill into Trace”.

DataBuff Service call analysis: service-a → service-b (drill to Trace)

DataBuff Endpoint call analysis for /demo/checkout

DataBuff Service flow: entry service-a → downstream response contribution

Trace (Jaeger mature surface)

Jaeger Search: service / operation / Tags filters

Jaeger Trace list: service-a results + scatter

DataBuff Trace list: charts + table

Jaeger Waterfall + Tags + Span Logs

DataBuff Call-order waterfall; can link to application logs

Logs (matrix rows 16–18; no Jaeger equivalent)

DataBuff Log analysis: Log → Trace down to Span

DataBuff dedicated pages (matrix rows 11 / 12)

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis

DataBuff Error analysis

These pages are the depth after Dependencies shows “who connects” — the APM gap most worth verifying side-by-side with Jaeger.

Alerting (no Jaeger built-in alerts)

DataBuff Alert center; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Try DataBuff

Open source · OpenTelemetry APM with AI-native troubleshooting

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full comparison: https://databuff.ai/blog/en/databuff-vs-jaeger/

If this helped, a ❤️ or a GitHub Star is appreciated.

DataBuff vs OpenObserve: Same-Host Lab Comparison

databufflabs — Thu, 23 Jul 2026 01:33:54 +0000

Same-host lab: DataBuff (OTLP :4318) and OpenObserve (OTLP HTTP :5080/api/default) on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · OpenObserve v0.91.0-rc1. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

Positioning: DataBuff = AI-native APM depth; OpenObserve = unified observability platform (Logs / Metrics / Traces / RUM), strong on log search and object-storage cost.

1. Capability matrix

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
① See · natural-language questions	❌	✅ Ask about services / topology / trends; AI reads telemetry
② Squad · multi-agent collaboration	❌	✅ Parallel evidence gathering; serial context preservation; reusable task orchestration
③ Inspect · service inspection + report	❌	✅ One-shot inspection with evidence and recommended actions
④ Diagnose · bottleneck / RCA evidence	❌	✅ Trace / metrics / topology evidence (not a black-box “root cause”)
⑤ Repair · Ops Expert actions	❌	✅ Repair under policy + human approval; dangerous-command denylist
⑥ Predict · capacity / trends	❌	✅ Capacity and trend analysis — from after-the-fact to ahead-of-time
⑦ Answer · product Q&A	❌	✅ Answers deploy / ingest / config from docs and code
Extend · MCP / Skill / custom experts	❌	✅ External MCP / Skill and custom digital experts

Largest gap: OpenObserve has no equivalent AI platform (Traces has an LLM Insights entry, not validated here as APM triage); DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
1. Global topology	❌ No service dependency topology	✅ Global topology + health colors + node drill-down
2. Service list & golden metrics	✅ Service Catalog (Requests / Error Rate / P99, etc.)	✅ Service list + charts; same demo shows service-a / b
3. Service-level topology	❌	✅ Dedicated service topology
4. Service call analysis (up/downstream + Trace)	❌	✅ Upstream/downstream structure, latency/contribution; drill to Trace
5. Instance golden metrics	❌	✅ Instance golden-metric charts / list
6. Instance topology	❌	✅ Dedicated instance topology
7. Instance call analysis (up/downstream + Trace)	❌	✅ Per-instance up/downstream + Trace
8. Endpoint topology	❌	✅ Dedicated endpoint topology
9. Endpoint call analysis (up/downstream + Trace)	❌	✅ Per-endpoint caller/callee + Trace
10. Service flow (service / endpoint Trace contribution)	❌	✅ Response contribution from entry; service / endpoint Trace view
11. Middleware / external pages (DB / cache / MQ / external)	❌ db/http visible on Span fields; no dedicated pages	✅ Dedicated pages: DB / cache / MQ / external
12. Error analysis (stats + endpoint)	△ Can filter ERROR spans / logs	✅ Error stats + endpoint drill-down
13. Trace list / search	✅ Spans/Traces + flexible query; this lab shows service-a · GET /demo/checkout	✅ Charts + list, multi-dimension filters
14. Trace detail	✅ Waterfall / Flame Graph / Trace Graph	✅ Call-order waterfall + Span attributes
15. Trace Span → logs	✅ Trace / Span can link to logs	✅ Top “Log analysis” + Span Logs / Logs tab
16. Log list / search	✅ Strength: SQL / full-text + histogram; hundreds of events in this lab	✅
17. Log detail	✅	✅
18. Log → Trace	✅ Log → Trace (down to Span)	✅ Log → Trace, down to Span
19. Flexible Metrics query (SQL / PromQL)	✅ Metrics page: SQL / PromQL / Builder	△ Internal SQL; no public PromQL entry
20. Custom dashboards	✅ Dashboards can be created (list may be empty in this lab; capability present)	❌ Not yet
21. Unified storage cost (object store + compression)	✅ Home shows Ingested / Compressed (~96MB → 10.5MB in this lab)	△ Doris columnar; not an object-storage cost story
22. RUM	✅ Built-in RUM (Real User Monitoring)	❌ Not yet
23. Pipelines	✅ Realtime / Scheduled: transform / enrich / filter / route after ingest (VRL); logs→metrics, etc.	❌ Not yet
24. Reports	✅ Scheduled / Cached reports; timed generate & distribute	❌ Not yet

Shared base: service list & golden metrics, Trace list/waterfall, logs, and Span↔log links. DataBuff leads on topology / service·instance·endpoint call analysis / service flow / middleware pages. OpenObserve leads on log search & cost, SQL/PromQL, custom dashboards, Pipelines, Reports, unified L/M/T + RUM.

Alerting

Capability	OpenObserve v0.91.0-rc1	DataBuff v0.1.4
How rules are configured	✅ Alerts UI (needs Destination / Template first)	✅ Alert center in product
Threshold alerts	✅ Scheduled / Realtime	✅ Managed in platform
Smart alerts	❌ No equivalent smart-alert product	✅ Smart alerts linked with APM metrics
Alert event list	✅ Alerts UI for triggered alerts / rules	✅ Alert list (severity / service / time)
Alerts linked to service / middleware	△ Stream-oriented alerts; APM context must be stitched manually	✅ List links service / middleware back into APM

Both can configure alerts in UI; difference is smart alerts and alert → APM service context. OpenObserve alerts lean Logs/Metrics streams; DataBuff leans APM triage loop.

When to pick which

Scenario	Better fit	Note
Already on OTLP; want AI / APM depth first	DataBuff	Point exporters at DataBuff; no need to migrate off OpenObserve first
Need the seven AI capabilities	DataBuff	OpenObserve has no equivalent AI platform
Need MCP / Skill or custom digital experts	DataBuff	AI platform is extensible; OO has no such layer
Need global topology + health colors at a glance	DataBuff	OO has no service dependency topology
Need “who slowed the response” from entry service	DataBuff	Service flow + contribution; OO has no equivalent page
Need service / instance / endpoint call analysis → Trace	DataBuff	Three-level call analysis all link to Trace; OO has no path
Need instance golden metrics / instance topology	DataBuff	OO has no equivalent instance pages
Need slow SQL / cache / MQ / external service pages	DataBuff	OO mostly Span fields; no dedicated pages
Need dedicated error analysis	DataBuff	OO requires manual ERROR filtering
Need smart alerts tied to service / middleware	DataBuff	OO alerts are stream-oriented; no smart-alert APM loop
Huge log volume; need object-storage cost control	OpenObserve	Compression / storage story is a strength
Need SQL / PromQL Metrics + custom dashboards	OpenObserve	DataBuff has no custom dashboards yet
Need post-ingest transform / enrich / filter / route	OpenObserve	Pipelines (Realtime / Scheduled + VRL)
Need scheduled / cached reports	OpenObserve	Reports (Scheduled / Cached)
Need unified Logs + Metrics + Traces + RUM	OpenObserve	DataBuff focuses on APM depth
Only need the same Demo Trace waterfall	Either	No need to migrate for branding

Boundary: Staying on OpenObserve is reasonable when deeply tied to its log pipelines / dashboards / Pipelines / Reports / cost story. DataBuff fits OTLP + seven AI + topology / call analysis / service flow / dedicated pages / smart alerts. Dashboards, Pipelines, Reports, and large-scale log cost are not peer capabilities yet.

2. Screenshot evidence (explains the tables)

Screenshots from the same lab (192.168.50.140). Captions map to capability rows. Focus on DataBuff AI / topology / dedicated pages / alerts, plus OpenObserve logs / Trace / Metrics / dashboard entry points.

Seven AI capabilities (no OpenObserve equivalent UI; DataBuff evidence)

DataBuff AI chat home and seven capability entries: See / Squad / Inspect / Diagnose / Repair / Predict / Answer

DataBuff ① See: real question on service-a checkout / service-b; AI reads telemetry

DataBuff ② Squad: digital expert / multi-agent entries (no OpenObserve equivalent)

Overview & data plane

OpenObserve Home: Streams≈38 · Events≈205K · Ingested 96MB → Compressed 10.5MB (matrix row 21)

OpenObserve Streams: unified Logs / Metrics / Traces data plane

Services & topology

DataBuff Service list + golden-metric charts (peers with OO Service Catalog)

DataBuff Global topology + health colors (mysql can show red); OpenObserve has no topology view

Service / endpoint call analysis + service flow (matrix rows 4 / 9 / 10)

OpenObserve can list Trace / Span, but has no service / instance / endpoint call analysis, and no service-flow contribution view. DataBuff goes from “who connects” to “who slows, then into Trace”.

DataBuff Service call analysis: service-a → service-b (drill to Trace)

DataBuff Endpoint call analysis for /demo/checkout: volume / latency / error rate

DataBuff Service flow: entry service-a → downstream response contribution; no OpenObserve equivalent

Trace

OpenObserve Traces: service-a · GET /demo/checkout · 240ms · 13 Spans

DataBuff Trace list: charts + table

OpenObserve Trace Waterfall: service-a → service-b · includes SQL / Dubbo Spans

DataBuff Call-order waterfall + Span attributes; can link to logs in-product

Log / Metric

OpenObserve Logs: histogram + event table; Trace / Span correlation (matrix rows 15–18)

DataBuff Log analysis: Log → Trace down to Span

OpenObserve Metrics: SQL / PromQL / Builder (matrix row 19; DataBuff has no public PromQL)

Dashboards / Pipelines / Reports (matrix rows 20 / 23 / 24; OpenObserve strengths)

OpenObserve Dashboards: create custom boards

OpenObserve Pipelines: Realtime / Scheduled data pipelines

OpenObserve Reports: Scheduled / Cached

DataBuff dedicated pages (matrix rows 11 / 12; no OO equivalents)

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis list

DataBuff Error analysis (stats + endpoint)

These pages are depth beyond “middleware Spans in Trace” — the APM differences most worth validating against OpenObserve.

Alerting

OpenObserve Alerts UI: configure and view rules / events

DataBuff Alert center: rules in-product; list links service context; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Online Demo: https://demo.databuff.ai

Try DataBuff

Open source · OpenTelemetry APM with AI-native troubleshooting

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full comparison: https://databuff.ai/blog/en/databuff-vs-openobserve/

If this helped, a ❤️ or a GitHub Star is appreciated.

DataBuff vs SigNoz: Same-Host Lab Comparison

databufflabs — Thu, 23 Jul 2026 01:33:19 +0000

Same-host lab: DataBuff (OTLP :4318) and SigNoz (OTLP :24318) side by side on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · SigNoz v0.133.0. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

1. Capability matrices

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

Capability	SigNoz v0.133.0	DataBuff v0.1.4
① See · natural-language questions	❌	✅ Ask about services / topology / trends; AI reads telemetry
② Squad · multi-agent collaboration	❌	✅ Parallel evidence gathering; reusable task orchestration
③ Inspect · service inspection + report	❌	✅ One-shot inspection with evidence and actions
④ Diagnose · bottleneck / RCA evidence	❌	✅ Trace / metrics / topology evidence (not a black-box “root cause”)
⑤ Repair · Ops Expert actions	❌	✅ Repair under policy + human approval; dangerous-command denylist
⑥ Predict · capacity / trends	❌	✅ Capacity and trend analysis — from after-the-fact to ahead-of-time
⑦ Answer · product Q&A	❌	✅ Answers deploy / ingest / config from docs and code
Extend · MCP / Skill / custom experts	❌	✅ External MCP / Skill and custom digital experts

Largest gap: SigNoz has no equivalent AI platform; DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

Capability	SigNoz v0.133.0	DataBuff v0.1.4
1. Global topology	✅ Service Map (incl. middleware nodes)	✅ Topology + health colors + drill-down
2. Service list & golden metrics	✅ Services (P99 / Error / OPS)	✅ Service list + charts; same demo shows service-a / b
3. Service-level topology	△ Via Service Map only; no dedicated page	✅ Dedicated service topology
4. Service call analysis (up/downstream + Trace)	❌	✅ Upstream/downstream structure, latency/contribution; drill to Trace
5. Instance golden metrics	❌	✅ Instance golden-metric charts / list
6. Instance topology	❌	✅ Dedicated instance topology
7. Instance call analysis (up/downstream + Trace)	❌	✅ Per-instance up/downstream + Trace
8. Endpoint topology	❌	✅ Dedicated endpoint topology
9. Endpoint call analysis (up/downstream + Trace)	❌ Mostly Traces filters	✅ Per-endpoint caller/callee + Trace
10. Service flow (service / endpoint Trace contribution)	❌ Service Map answers “who connects”	✅ Response contribution from entry
11. Middleware / external pages (DB / cache / MQ / external)	❌ Nodes only, no dedicated depth	✅ Dedicated pages: DB / cache / MQ / external
12. Error analysis (stats + endpoint)	❌ Mostly Traces filters	✅ Error stats + endpoint drill-down
13. Trace list / search	✅ Traces Explorer	✅ Charts + list, multi-dimension filters
14. Trace detail	✅ Span timeline / attributes	✅ Call-order waterfall + Span attributes
15. Trace Span → logs	✅ From Trace detail	✅ Top “Log analysis” + Span Logs / Logs tab
16. Log list / search	✅ Logs Explorer	✅ Log analysis
17. Log detail	✅	✅
18. Log → Trace	✅ Log → Trace	✅ Log → Trace, down to Span
19. Custom dashboards	✅ Dashboards V2 (Perses / PromQL)	❌ Not yet

Basics (incl. Span↔logs) exist on both sides; DataBuff leads on call analysis, service flow, dedicated pages, error depth, and Log→Trace down to Span. SigNoz’s clear strength is custom dashboards.

Alerting

Capability	SigNoz v0.133.0	DataBuff v0.1.4
How rules are configured	✅ Alert Rules UI	✅ Alert center
Threshold alerts	✅	✅ Managed in platform
Period-over-period (WoW/MoM) alerts	❌	✅ Period-over-period entry linked with APM metrics
Alert event list	✅ Triggered Alerts; firing in this lab	✅ Non-empty in this lab
Alerts linked to service / middleware	△ Notifications; stitch APM yourself	✅ List links back into APM

Both sides have alerting; this lab has a SigNoz threshold rule firing. Gaps remain on period-over-period alerts and alerts that carry service / middleware APM context.

When to pick which

Scenario	Better fit	Note
Same OTel data, want AI / APM pages first	DataBuff (side-by-side)	Point OTLP at DataBuff
Need the seven AI capabilities	DataBuff	No SigNoz AI platform
MCP / Skill / custom experts	DataBuff	No such layer in SigNoz
See who slows the entry response	DataBuff	Service flow + contribution
Call analysis → Trace (service / instance / endpoint)	DataBuff	No SigNoz path
Slow SQL / cache / MQ pages	DataBuff	SigNoz mostly map nodes
Custom dashboards / PromQL boards	SigNoz	Dashboards V2; DataBuff not yet
Mature Trace / Logs Explorer only	Either / lean SigNoz	No need to migrate for brand

Boundary: Deep SigNoz Dashboard / PromQL workflows → stay on SigNoz. DataBuff fits same OTel data + AI + APM depth, side-by-side or gradual switch.

2. Screenshot evidence (explains the tables)

Screenshots from 192.168.50.140. Captions map to capability rows.

Seven AI capabilities

DataBuff AI chat home and seven capability entries (no SigNoz equivalent)

DataBuff ① See: natural language over telemetry

DataBuff ② Squad: digital expert entries

Services & topology

SigNoz Services: application P99 / Error / OPS

DataBuff Service list + golden-metric charts

SigNoz Service Map: middleware nodes appear here too

DataBuff Global topology + health colors; drill into dedicated pages

Fact check: SigNoz Service Map also draws middleware nodes. The gap is dedicated-page depth plus call analysis / service flow.

Call analysis & service flow

DataBuff Service call analysis → Trace

DataBuff Endpoint call analysis (no SigNoz page)

DataBuff Service flow: response contribution from entry

Trace

SigNoz Traces Explorer

DataBuff Tracing: charts + list

SigNoz Trace detail: Span timeline

DataBuff Trace→Log productized path

Log

SigNoz Logs Explorer

DataBuff Log→Trace, down to Span

Dashboards (SigNoz strength)

SigNoz Dashboards V2; DataBuff has no peer yet

DataBuff dedicated pages

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis

DataBuff Error analysis

Alerting

SigNoz Triggered Alerts: demo-calls-threshold firing

DataBuff Alert center; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Try DataBuff

Open source · OpenTelemetry APM with AI-native troubleshooting

Live Demo: https://demo.databuff.ai
GitHub: https://github.com/databufflabs/databuff
Full comparison: https://databuff.ai/blog/en/databuff-vs-signoz/

If this helped, a ❤️ or a GitHub Star is appreciated.

Open-Source AIOps, Finally Someone Built It

databufflabs — Mon, 20 Jul 2026 02:40:45 +0000

Gartner coined AIOps back in 2016. For years the gut feeling was “vaporware”: commercial tools are expensive, closed, and black-box; in open source you get alert-noise scripts or log analyzers — nothing that truly wires AI into live monitoring data and can fix problems. The moment I got DataBuff running, my first thought was — someone finally built it.

This post skips architecture slides and protocol names. I walk through seven scenarios I ran hands-on in a test environment. Each starts with one sentence; the AI pulls data, draws conclusions, and in some cases SSHs in to repair. After these, AIOps stops being a deck concept and becomes something you can use.

What it is: open source, AI-native, OpenTelemetry APM

One-line positioning: DataBuff is an open-source, AI-native APM built on OpenTelemetry. Plain English — it is a monitoring platform for metrics, traces, and logs; unlike classic APM, AI is not a bolt-on chat box on the side. It lives on the data: you ask a question, it reads live metrics, follows traces, queries logs, and returns evidence-backed answers instead of dumping charts on you.

Minimal stack: ingest + storage + AI platform — one command to start

The gap is “AI-native.” Many products are “APM + chat widget” where the model just tells you to look at a dashboard. DataBuff’s AI reads data, calls tools, and can reach hosts — the seven scenarios below are proof. The arc to keep in mind:

Glance the roadmap first, then walk each case

Scenario 1 · Ask your system in plain language

Start with the obvious. To find slow services you used to open Grafana, memorize PromQL, write queries, read charts, and sort yourself. In DataBuff I asked in plain English: “Which service was slowest in the last hour? List the top 3.”

One question — AI queried 20 services and returned a ranked table with latency and error rate

It did not flash a chart and leave me guessing. It actually scanned 20 services, computed average latency, and returned a table: slowest service-a at 240ms avg , then service-b 70ms, skyWalking-service-a 35.7ms, with volume and error rate. Twenty-three seconds, zero query language.

For newcomers, that is what AIOps should feel like — no query DSL, no metric math — just ask.

Takeaway: natural-language Q&A is not “search.” The model reads intent, hits live data, and answers with evidence. The more casual the question, the more concrete the reply.

Scenario 2 · Multi-agent: not one bot, a squad

If scenario one felt “nice,” this one reframes the product — DataBuff is not a single AI, it is a squad. The experts on the roster: AI Brain, Data Query, Intelligent Inspection, Ops Expert, Product Q&A. The right pattern is not inventing fake “metrics expert / trace expert” labels — leave the expert picker alone, throw the hard task at AI Brain , and let it dispatch real experts. I said one sentence:

“Any cluster anomalies in the last hour? Run a joint diagnosis with Data Query and Intelligent Inspection: Data Query checks latency, error rate, and slow traces; Inspection runs tiered health checks; summarize into an incident report I can forward.”

AI Brain calls dispatchExpertTask twice — Data Query (data) and Inspection in parallel

What followed is not typical APM. AI Brain said it would dispatch two tasks, then successfully called dispatchExpertTask twice — latency/errors/slow traces to Data Query , tiered health checks to Intelligent Inspection. Both worked in parallel. Inspection reported JVM/GC/thread metrics healthy across 34 services; Data Query found real issues: Elasticsearch index 404 (~144k failures) and MySQL-side InsufficientStockException. AI Brain merged both into a forwardable report.

P0 ES index down + P1 inventory business errors; Inspection confirms service health; HTML report ready to share

The report: P0 Elasticsearch indexes unavailable (my_index_1 / my_index_2 all 404, ~144k failures); P1 MySQL stock business errors (InsufficientStockException on service-b — business stock-out, not infra down); plus service self-health confirmed by Inspection. Full HTML you can preview or paste into an incident channel.

Takeaway: the “A” in AIOps is not a chat box — it is AI Brain splitting work across real experts, then synthesizing. One human sentence; a squad that can name and dispatch.

Scenario 3 · Inspection: one sentence → shareable HTML report

Runbooks hate manual inspection — check metrics, thresholds, stitch a report for half a day. In DataBuff I switched to Intelligent Inspection and asked: “Run an inspection on service-b and output a full HTML report.”

Intelligent Inspection on service-b — full HTML report requested

Eighty-one seconds, twenty-two steps — not a chat essay but formatted HTML. Overview: entry health 98, downstream MySQL 60, Redis 100, zero active alerts. Entry looks fine (~4 req/min, 0% errors, ~70ms avg), but error logs expose “false green” — 60 ERROR lines in 30 minutes, all InsufficientStockException:

Top of report: health cards + entry metrics + errors (0% HTTP errors vs InsufficientStockException)

Further down: evidence chain — downstream [mysql]demo_apm at 50% errors, traces show findInventory → stock query throwing; graded conclusion P0 system OK, P1 partial business impact , with fixes — repair stock data, alert on this business exception so HTTP 200 does not hide it. File at outputs/service-b-health-report.html, preview in-chat.

Bottom of report: dependencies, trace evidence, graded conclusion and actions

Inspection should end in an HTML report you can open and forward — green at the edge, red underneath, spelled out.

Scenario 4 · Root cause: one sentence with evidence

Incident triage is experience-heavy — charts, time alignment, hand-written PromQL, manual evidence chains. I asked bluntly: “Where is the bottleneck for service-a in the last hour — app, database, or downstream?”

Data Query pulls topology and ranks seven downstream calls by latency share

It delivered. Time range set, service-a topology — seven downstreams — then a table of call count, avg latency, and share: service-b HTTP 100ms and RPC 80ms on top; MySQL 20ms, ES 18ms, Redis 13ms, Kafka 8ms, remote payment 7ms all normal.

Bottleneck: downstream service-b (73.2% of outbound time); app and other deps cleared

One-line conclusion: bottleneck is downstream service-b — HTTP 100ms + RPC 80ms = 180ms per request, 73.2% of outbound time. It also listed who is not at fault — service-a 120 requests 0 errors 0 alerts, MySQL/ES/Redis/Kafka/remote payment all normal — then next steps on service-b HTTP/RPC for slow calls or thread blocking.

Takeaway: RCA here is not “here is a trend chart” — it is ranked downstream latency, attributed share, and a sentence you can paste into an incident doc.

Scenario 5 · Ops Expert: don’t just look — fix it

The first four scenarios are read-only intelligence. The next step for AIOps is hands-on — Ops Expert SSHs in, investigates, changes config, and repairs. Most tools stop at “here is what broke.”

Real case: demo container ai-apm-demo stuck in Restarting. I said: “Container keeps restarting — fix it.”

Plain-language request: container keeps restarting

Ops Expert SSH’d in, ran docker logs, docker inspect, free -m, found OOM kill (137) from too-low memory limit, applied the fix, restarted the container.

On-host investigation via docker logs / inspect / free -m

OOM from memory cap — parameter updated, docker ps healthy again

After docker ps, the container stayed up. Other AIOps tell you what broke; this one helped fix it. That is the line between “see” and “repair” in open-source AIOps.

Ops Expert does not only suggest commands — it investigates, concludes, and proposes the fix. DataBuff closes the loop.

Scenario 6 · Capacity: from reactive firefighting to foresight

Everything so far is after failure. Higher-level AIOps judges capacity before you scale. I asked: “This Redis averages 366ms — capacity bottleneck or not? Should we scale? Planning advice.”

Topology clarifies which Redis instance matters; capacity vs slow-operation guidance

First it clarified topology — service-a actually depends on [redis]redis:6379 (154 calls/hr, 13ms, healthy) ; the 366ms [redis]redis.test:6379 is not on service-a’s path. Easy to mix up under pressure.

On the slow instance: 352,807 calls/hour, ~98 QPS, 366ms avg. It did not say “scale because slow” — 98 QPS is far below what a single Redis can take; the bottleneck is likely operations (big keys, blocking commands like KEYS / wide SMEMBERS), with Top 3 causes and SLOWLOG GET / redis-cli --bigkeys. Advice: do not scale blindly — find the slow command first.

Good capacity analysis separates “under-provisioned” from “misused” — only the former deserves more machines.

Scenario 7 · Product Q&A: open source with built-in support

Hardest part of open source: “how do I configure this?” — docs, issues, waiting. DataBuff ships a Q&A expert. I asked the classic newbie question: “How do I wire the OpenTelemetry SDK? Where do I set alert thresholds? Step-by-step paths.”

Answers from product docs: OTLP ports and alert menu paths

Not a generic tutorial — it read this product’s docs: OTLP ingest, gRPC 4317 / HTTP 4318 , point any SDK exporter at Ingest for traces, metrics, and logs without a separate agent.

Env vars + Spring Boot javaagent one-liner + Python doc paths

It also pointed to language quick starts: Spring Boot with opentelemetry-javaagent.jar, one java -javaagent:... for zero-code instrumentation; Python OTLP docs too. Alerts under Configuration → Alerting → Detection rules → New rule — pick object, metric, threshold, severity; evaluates every minute on the last five minutes; plus convergence and silence policies.

Open source’s gap is often “someone to ask.” DataBuff embeds Q&A that reads its own docs — better than a random search result.

Worth saying: the UI is a solid APM on its own

DataBuff is not chat-only. Under the hood you get global topology, services, service map, databases, queues, caches, external deps — full-path drill-down from the big graph to metrics to a single trace. AI reads; the UI lets you verify.

Global topology — services and dependencies at a glance

Service list — health status in one scan

Will you actually run it?

Yes — quickly. One curl, three components, minimal deps. Open the Web UI, add an API key, start asking.

One command — three components up

AIOps should not be a luxury for a few big vendors — any team should spin it up in five minutes. DataBuff is open, self-hostable, data stays yours, code is inspectable, built on OpenTelemetry so most SDKs plug in without lock-in.

Open-source AIOps, finally built — you could be next

If these seven scenarios resonate, don’t stop at reading.

Star it on GitHub, run it in minutes, and ask it the question that annoys you most on-call.

https://github.com/databufflabs/databuff

Questions? Open DataBuff and ask the Q&A expert — it is already in the product.

Star DataBuff on GitHub → https://github.com/databufflabs/databuff

5 Ops Tasks That Take Half an Hour — Ops Expert Finishes in 3 Minutes

databufflabs — Thu, 16 Jul 2026 01:45:06 +0000

On-call pain is rarely that a problem is “too hard.” It’s that you already know what to check — and still have to SSH in and type every command yourself. These five show up in almost every test environment:

OTel won’t connect / no data — SDK is configured, the platform is blank; you guess endpoint, port, or a dead process.
Container keeps restarting — you see Restarting, dare not poke randomly, and crawling logs inside the container is slow.
Inspect real Java JVM flags — startup args, container limits, and effective values disagree; you bounce between jinfo / jcmd.
Port in use, service won’t start — lsof / ss once, then make sure you don’t kill the wrong process.
CPU hotspots — flame-graph tooling slips your mind; sampling configs take half a day before a chart appears.

Solo, these jobs often take 15–40 minutes (find the host, log in, recall commands, reconcile the conclusion). With DataBuff (Star on GitHub →) Ops Expert, the same work usually lands in 1–3 minutes: open AI Platform → AI Chat, pick Ops Expert, describe the symptom in plain language.

All screenshots below are from a live test environment.

Scenario · DIY SSH (typical) · Ops Expert (measured)

OTel won’t connect — endpoint / port / process ~15–25 min → ~1–2 min
Container restart loop — log diving ~20–40 min → ~2–3 min to find & fix
JVM flags — jinfo / jcmd ~10–20 min → ~1 min
Port conflict — ss / lsof ~5–15 min → ~1 min
Flame graph — tooling + sample ~20–40 min → ~2–3 min

Walkthrough: three shots per scenario — your prompt → Ops Expert at work → the conclusion.

1. OpenTelemetry won’t connect / no data

Example prompt:

Test env OTel won’t connect and the platform has no data. Check whether 4317/4318 are reachable and tell me in one sentence if the endpoint is correct.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

2. Container keeps restarting

Example prompt:

ai-apm-demo keeps Restarting — help me get it healthy.

① User prompt:

② Ops Expert process:

③ Conclusion (memory limit 10MB → OOM 137 → raised to 512MB):

3. Inspect Java runtime flags

Example prompt:

Show me the effective JVM flags for the ai-apm-web Java process — especially heap and GC.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

4. Port in use — service won’t start

Example prompt:

Who is holding port 27403? Tell me the process and command — do not kill anything.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

5. Capture a flame graph

Example prompt:

Take a short CPU flame graph for the ai-apm-web Java service on the test host and point out the hotspots.

① User prompt:

② Ops Expert process:

③ Conclusion (sample result + hotspot readout):

Same loop every time: describe the symptom → watch the process → read the conclusion. The timing list above is the DIY SSH vs Ops Expert gap.

Open source · one-line install

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Found this useful? Star DataBuff on GitHub → https://github.com/databufflabs/databuff

Live demo: https://demo.databuff.ai

SkyWalking Meets AI: Keep Your Agent, Ask Why Checkout Is Slow

databufflabs — Tue, 14 Jul 2026 01:25:51 +0000

Many Java teams run Apache SkyWalking in production. The Agent often stays in place for years: bytecode instrumentation, clear Segments, JVM + traces on one path.

DataBuff v0.1.3 layers AI on top of that same collection. Keep the Agent. Point ingest at SkyWalking gRPC :11800 (OAP’s default). Ask in plain English why checkout is slow — get traceId + bottleneck span, then verify on a flame graph. Logs still jump to traces. Same Segments. Faster answers.

All screenshots below are from a live demo environment.

0. First, SkyWalking

Typical strengths:

Java Agent with minimal app changes
Segments that keep multi-service spans readable
Traces, JVM metrics, and logs on one reporting path
Mature OAP + UI, strong docs and community

SkyWalking is solid at getting telemetry in.

Where on-call time goes is turning that data into a conclusion: alerts without traceId, hunting slow traces, summarizing span trees for Slack, hopping between topology, metrics, and logs. That is not a SkyWalking flaw — it is the next step: faster conclusions from the same telemetry.

Note: This article is about AI-assisted interpretation on SkyWalking ingestion — same Agent, same Segment data — focused on whether one pipeline can produce answers faster.

1. Highlight: slow checkout — ask once, get an answer

Demo: GET /demo/checkout P99 ≈ 240ms. Alerts often arrive without a traceId. With DataBuff, triage shifts from scrolling trace lists to asking AI.

① Ask directly (no traceId): “Why is service-a’s checkout endpoint slow lately?”

② AI breaks down the trace: typical slow trace, segment timings, bottleneck span — no manual row-by-row comparison.

③ Structured conclusion: actionable incident text, not just “maybe the DB is slow.”

④ One-click verify: jump to the flame graph with the returned traceId and confirm end-to-end spans.

What you gain: from “filter list → read span tree → write incident note” to ask → get traceId → verify in UI. Same Segment source; added AI readout.

2. Go deeper: topology, service flow, JVM

After AI answers, the same UI keeps going — no tool hopping.

3. Before vs after: how the path changes

For the checkout scenario:

Step	SkyWalking UI only	With DataBuff
1 · Find service	General → Service → pick service-a	Topology / service list → service-a
2 · Find slow trace	Trace page filter checkout → open rows one by one	Ask AI → `traceId` + bottleneck span returned
3 · See bottleneck	Read span tree manually for DB / RPC time	AI summary + flame graph jump
4 · Conclusion	Human writes “maybe DB query slow”	AI remediation hints; ops expert can SSH-check JVM
5 · Logs	Separate log system, match `traceId`	Log list “Trace · View” → call chain

Your SkyWalking Agent keeps reporting; you add AI query, unified UI, and log deep-links — same collection target.

4. How to connect: keep the Agent, point backend to DataBuff

v0.1.3 ingests native SkyWalking gRPC on 11800 (OAP default). No Agent jar swap — change the collector address:

# agent.config
agent.service_name=${SW_AGENT_NAME:your-service}
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:your-databuff-host:11800}

Two onboarding paths, both AI-capable:

Mode	Setup	Best for
Side-by-side trial	Keep SkyWalking OAP; read SkyWalking data via MCP	Cannot move Agents yet — try AI Q&A first
Native ingest	Point Agent to DataBuff `:11800`, Segments direct to ingest	Switch backend; traces / JVM / logs unified in DataBuff

5. FAQ

Do I replace the Agent?

No. Existing SkyWalking Java Agents work — usually only collector.backend_service changes.

Must OAP go away immediately?

No. Keep OAP for a trial; when you commit to DataBuff, point Agents over in batches and verify checkout-style Q&A works.

Can OpenTelemetry coexist?

Yes. Java on SkyWalking (11800), Go/Python on OTLP (4317), one DataBuff UI.

What does AI actually do?

Demo covers on-call staples: slow endpoint → traceId + bottleneck span, log line → trace, JVM curves for follow-up. You do not need to be a trace expert to get direction.

Who is this for?

SkyWalking in production, want AI Q&A on existing trace data
Java-heavy estates not ready to re-instrument with OpenTelemetry
Want traces, logs, and metrics in one troubleshooting flow

Suggested path: test env or one instance → change Agent address → hit checkout-like traffic → ask AI “why slow lately” → confirm traceId + flame graph → expand rollout.

SkyWalking ingestion + AI interpretation · Keep your Agent · Ask once for answers · Logs link to traces

Star DataBuff on GitHub →

AI Ops Squad Evolves: Ask ERROR Logs in Plain English

databufflabs — Tue, 07 Jul 2026 05:52:53 +0000

You can ask DataBuff about slow traces in plain English. That shipped in our last walkthrough.

Now ERROR logs join the party.

Same checkout demo, same InsufficientStockException on service-b — but this time we follow three real on-call paths: facet search in the Logs UI, trace-to-log deep links, and one-sentence AI queries that call log.queryLog* tools against live OTLP data.

All screenshots below are from a single incident window on a live demo environment.

Demo scenario: inventory checkout failure

The demo app hammers GET /demo/checkout. When stock runs out, service-b throws:

InsufficientStockException: inventory unavailable for skuId=…

OTLP logs land in Doris with trace_id and span_id attached. We walk the same failure three ways — the way a real shift would:

Path A — You know the exception name; search the global Logs page
Path B — You have a slow trace; read span logs and deep-link back
Path C — You delegate to the AI squad in one sentence

Path A: global log search (no LogQL required)

Menu: Application Performance → Log Analysis

No query language required — keyword + facets is enough.

Steps: search InsufficientStockException → filter ERROR + service-b → 95 matching lines with an ERROR spike in the histogram.

Each row has Trace · View on the right — jump straight to the call chain without copying IDs.

Path B: trace ↔ log deep links

B1 · Trace header: one click to logs

Open a slow GET /demo/checkout trace (~240 ms). The trace header shows TraceID; Log Analysis on the right pre-fills traceId — no copy-paste.

B2 · Span sidebar: flame graph + Logs tab

Spans marked Logs on the flame tree open a sidebar Logs tab: a timeline from Received checkout request through Delegating inventory check to service-b. Select the service-b span to see the ERROR stack.

B3 · Deep link: "View all in Log Analysis"

Click View all in Log Analysis at the bottom of the sidebar. The global page auto-fills traceId + spanId and shows only the four logs in that span's context.

Path C: ask the AI squad about logs

The UI is for precision. The AI is for one-sentence delegation.

The Smart Query expert registers a log tool family — visible under Tool Management:

Registered tools:

log.queryLogDetail — search by service, level, keyword
log.queryLogsByTraceId — all logs on a trace
log.queryLogsBySpanId — logs for one span
log.queryLogTrend — ERROR volume over time

Scenario 1: find ERROR logs by service + keyword

Find ERROR logs for service-b in the last hour related to
InsufficientStockException. List traceIds and key log summaries.

Scenario 2: known traceId → root cause

Given traceId edfa44615dcee4d6bdfeed46d84bfb20, list all ERROR-level
logs on this trace and explain why checkout failed.

The agent walks queryLogsByTraceId → 13-span call chain → ERROR lines → checkout failed due to insufficient inventory.

Scenario 3: ERROR volume trend

How has ERROR log volume for service-b looked in the last hour?
Any obvious spike periods?

Result: steady ~2 ERROR logs per minute — no spike, just a sustained inventory shortage.

Tool selection cheat sheet

Search by service/level/keyword → queryLogDetail
Known traceId → queryLogsByTraceId
One span's context → queryLogsBySpanId
Volume spikes → queryLogTrend

Where the data comes from

OTLP Logs (:4317 / :4318) → Ingest → Doris log_dc_record → POST /log/search.

Inject traceId via Java MDC and logs correlate automatically.

DataBuff is log exploration in an APM context — not a replacement for ELK or Loki. The win is sharing context with traces, metrics, and AI agents without hopping three systems.

Try it (5 minutes)

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash
curl -fsSL https://databuff.ai/databuff/ai-apm-demo-install.sh | bash

Open http://YOUR_HOST:27403 — login admin / Databuff@123 — add an LLM key under Settings → AI model.

Then try Path A, B, or C on the built-in checkout demo.

GitHub: https://github.com/databufflabs/databuff

OpenTelemetry Logs · Trace correlation · AI-native observability · Built in public