DEV Community: AIdevops2088

AI Agent Squad for Ops: One Complex Request, Multiple Agents in Action

AIdevops2088 — Thu, 06 Aug 2026 01:19:55 +0000

Complex incidents: 5 tabs, 30 minutes of stitching

2am alert: “Why is checkout slowing down?” You need P99, slow traces, root cause, and an incident report.

The old way: Grafana → Jaeger → topology → hand-written summary. Charts everywhere, root cause nowhere — 20–30 minutes per round.

Screenshot: hopping across systems, no evidence chain

Pain point: Monitoring gives data, not conclusions. Every tool is a separate hunt.

DataBuff: one goal, an AI squad wires the evidence

Open-source AI Native OTel APM. OTel ingest + Doris storage + multi-agent troubleshooting.

In one line: orchestrate an AI squad to turn metrics, traces, and topology into an incident-ready report.

Screenshot: Ingest → Doris → Platform

Screenshot: global topology — agents judge blast radius

Up and running in 5 minutes

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Screenshot: one command install, Web UI URL in output

Open http://localhost:27403 (admin / Databuff@123)

Screenshot: traffic-light service health overview

Settings → AI model — add your LLM API key:

Screenshot: natural-language troubleshooting after API key setup

Demo data: curl -fsSL https://databuff.ai/databuff/ai-apm-demo-install.sh | bash

Back to §1: multi-agent joint ops

Drop the §1 request into the AI chat — all 5 screenshots below are from one conversation (live demo).

①

State the goal — P99, traces, root cause, report in one message.

Screenshot: full incident goal in one prompt

②

Brain dispatches — dispatchExpertTask to metrics + inspection experts.

Screenshot: concurrent dispatch to metrics + inspection

③

Smart query — breaks down the 240ms entry span.

Screenshot: trace latency breakdown — 240ms entry

④

Health inspection — topology + slow traces + metrics to find the bottleneck.

Screenshot: inspect queryTraceList → queryMetricData

⑤

Deliver summary — remediation + incident report.

Screenshot: P0 actions + brain-merged report

vs §1: 5 tabs, 30 minutes → one message, root-cause report in minutes.

From manual stitching to commanding an AI squad

No more five browser tabs. Data stays local; code is open and auditable.

- **Module** — What you get - **📊 APM** — Topology, P99, slow-trace drill-down - **🤖 AI squad** — Multi-agent queries, synthesized reports - **🔌 OTel** — Unified trace / metric / log storage

⭐ Deploy your ops AI squad in 5 minutes

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Try: “Analyze why checkout slowed down and draft an incident report.”

View on GitHub →

Celebrating! This Open-Source CN Project Is Now an Official OpenTelemetry Vendor

AIdevops2088 — Thu, 06 Aug 2026 01:19:35 +0000

DataBuff is the second China-built open-source professional APM — after SkyWalking — to appear on the OpenTelemetry official Vendors page, marked as Native OTLP. Why did the community list it? What does it mean for teams still choosing an OTel backend? This article includes demo screenshots and walks through standards, architecture, and AI-powered troubleshooting.

If you're an SRE or ops engineer, read this through. Let's dive in.

OpenTelemetry: CNCF's Second-Largest Project After Kubernetes

OpenTelemetry (OTel or OTLP) is a CNCF-hosted, vendor-neutral observability standard. CNCF's 2025 project velocity report shows OTel commits up 39% year over year, contributors +35% — making it the second-largest CNCF project after Kubernetes. Contributors include AWS, Microsoft, Google, Splunk, Dynatrace, New Relic, and other industry leaders.

DataBuff Listed on OpenTelemetry Vendors Within Two Weeks of Launch

The OpenTelemetry website maintains a Vendors ecosystem list — observability backends that natively consume OTLP telemetry for end users. Think of it as a publicly verifiable whitelist.

In the Pure OSS group, DataBuff is marked Open Source Yes, Commercial No, Native OTLP Yes, with Learn more linking to OTLP integration docs.

What Makes DataBuff Stand Out

DataBuff Tagline: AI Native OpenTelemetry APM

The project's goal is to help enterprises move from application performance observation to performance governance, and ultimately to autonomous operations — emphasizing the technical value of AI-native capabilities in SRE workflows.

.1 Minimal Architecture

DataBuff uses a minimal three-component design: Ingest → Doris (storage) → Web (platform).

No traditional Elasticsearch + Kafka + microservices stack. One Docker command gets the demo running. The direct payoff: fewer ops components, runs on 8 GB RAM, and a single storage entry point for AI queries — the engineering foundation for an AI-native OpenTelemetry APM.

.2 OpenTelemetry-Native Capabilities

Fig. 1 · Service List & RED Dashboard

The Application Performance module directly consumes Traces and Metrics reported via OTLP from the OTel SDK, aggregating RED metrics and visualizing service dependencies on the platform — no proprietary agent protocol required. It shows request volume, response time, and error rate for each service, with trend curves for health comparison — the core troubleshooting entry point after OTel Metrics land in storage.

Fig. 2 · Global Topology

Automatically draws service and middleware dependency graphs from Span parent-child relationships in Traces. Node colors indicate health status — get a full system view without manually maintaining a CMDB.

.3 AI-Native Capabilities

Many tools treat AI as a generic chat window. The DataBuff AI Platform is different: Skills invoke platform tools to directly query OTel data stored in Doris. Below is a fault-diagnosis example of its AI-native capabilities.

Fig. 3-1 · Ask About a Service Failure

Fig. 3-2 · AI Fault Deduction Tree

Fig. 3-3 · AI Root Cause Analysis

Fig. 3-4 · AI Remediation Advice & Summary

Describe an anomaly in natural language in the AI chat (e.g., "A topology node turned red — help me diagnose the cause") to trigger multi-expert collaborative analysis. The AI brain automatically dispatches inspection and data-query experts, pulls service latency trends, cross-validates topology and traces, and outputs a structured diagnostic report with remediation advice — translating distributed tracing data into actionable troubleshooting paths.

Upcoming roadmap: OTLP Logs ingestion, AI application monitoring, eBPF collection

References

[1] https://www.cncf.io/blog/2026/02/09/what-cncf-project-velocity-in-2025-reveals-about-cloud-natives-future/

[2] https://opentelemetry.io/ecosystem/vendors/

[3] https://github.com/databufflabs/databuff

[4] https://github.com/databufflabs/databuff/blob/master/deploy/docker/README.md

[5] https://databuff.ai/databuff/ai-apm-install.sh

[6] https://github.com/databufflabs/databuff/blob/master/docs/产品介绍.md

[7] https://github.com/databufflabs/databuff/blob/master/docs/Roadmap.md

[8] https://demo.databuff.ai/

Open Source APM Feature Comparison: SkyWalking vs Databuff

AIdevops2088 — Thu, 06 Aug 2026 01:19:13 +0000

> Summary: When evaluating open-source APM, feature checklists often look alike — the real gap is troubleshooting paths and onboarding cost. This article skips architecture pitches and goes straight into the official SkyWalking and Databuff Demos, comparing service monitoring, distributed tracing, topology, alerts, and AI Q&A with live screenshots. After the comparison tables, you'll have a clear view of who to keep and who to try.

Comparison environment: demo.databuff.ai and demo.skywalking.apache.org[1][2]; SkyWalking UI is in Chinese (Simplified). Each chapter includes dual-product screenshots and a feature comparison table, with a selection summary at the end.

No concept dumping here — we open both products' real feature pages side by side in the Demo: from single-service monitoring drill-down, to Trace list and Span detail comparison, through topology and alerts, plus Databuff's unique AI Q&A — differences are in the screenshots, one chapter per capability.

Service Monitoring

Drill from the service list into single-service monitoring detail: metric trends, instance/API rankings, and service dependencies.

SkyWalking · General Service → rating Service Detail

SkyWalking Demo · Chinese UI

Figure 1-1 · rating service selected: RPM / Apdex / error rate, top 20 APIs, traffic and response-time percentiles, instance ranking[2]

After selecting a specific service (e.g. rating in the Demo) under General Service, SkyWalking shows a full application performance monitoring dashboard: Top APIs, traffic/error-rate/Apdex time series, response-time percentiles, instance load ranking — with further drill-down to instances, endpoints, Trace Profiling / eBPF, and more. Deep feature set for teams already invested in SkyWalking agents.

Service / instance / API / Trace multi-level menu linkage
Rich Profiling (Trace / eBPF / pprof) entry points
Fine-grained metrics, but deeper menu hierarchy

Databuff · Application Performance → service-a Detail

Databuff Demo

Figure 1-2 · service-a detail: health status, service relationship graph (Web → HTTP/RPC/DB/external calls), instance list, shortcuts to API analysis and service flow[1]

Databuff opens single-service detail from the service list with a service relationship graph + instance table by default: Web entry, downstream HTTP/RPC, MySQL/Redis dependencies at a glance, plus "API Analysis" and "Service Flow" shortcuts. Data comes from OTLP — no SkyWalking proprietary probe protocol.

Service relationship visualization on the detail landing page — shorter troubleshooting path
Unified OTLP ingestion, language/framework agnostic
Same page links alerts, JVM metrics, and other tabs

§1 Comparison summary: SkyWalking offers richer per-service metric charts and stronger Profiling; Databuff puts dependency relationships and service instances on the detail landing page — a more intuitive path for OTel teams from list → relationship graph → Trace. If you're already on OTel or planning a unified Collector, Databuff service detail needs zero extra probe protocols.

§1 Feature comparison · Service monitoring

- **Dimension** — SkyWalking · Databuff · Databuff advantage - **Detail page focus** — Top APIs, traffic/Apdex/error-rate time series, instance/API ranking · Service relationship graph + instance table on landing page, health at a glance · See full dependency picture on entry, fewer page switches - **Drill-down path** — Instance → API → Trace → Profiling multi-level menus · API analysis / service flow / JVM / alerts tabs on one page · Common capabilities aggregated — shorter troubleshooting chain - **Dependency view** — Jump to Topology / API dependency modules · Built-in Web→HTTP/RPC/DB/external relationship graph on detail page · Dependencies and metrics on one screen — faster impact assessment - **Data ingestion** — SkyWalking agents primary; OTLP needs extra config · OTLP 4317/4318 unified ingestion; point Exporter at Ingest · Zero proprietary agents for OTel teams, lower migration cost - **Onboarding** — Deep feature set, deeper menu hierarchy · Relationship graph landing + native Chinese UI, Demo out of the box · Newcomers build "service→dependency→Trace" mental model faster

Distributed Tracing

Two screenshots per product: Trace list search entry + single Trace detail (Span tree / waterfall).

SkyWalking · Distributed Tracing

SkyWalking Demo · Chinese UI

Figure 2-1 · Traces list: instance/endpoint/status filters, distribution scatter plot (success/error), Trace list after query[2]

Figure 2-2 · After clicking a Trace: TraceID, duration, Span tree (default/tree/statistics views), expandable Tags and Logs[2]

SkyWalking Trace supports multi-dimensional filters and a Distribution scatter plot; clicking a list row shows the Span tree and Tags. A mature distributed tracing solution for long-term troubleshooting in complex microservices.

Databuff · Distributed Tracing

Databuff Demo

Figure 2-3 · Trace list after chart time-point click: TraceID, API, duration, service, status code, left-side quick filters[1]

Figure 2-4 · Call chain detail: GET /demo/checkout waterfall, Redis/MySQL/remote calls/service-b full-chain Spans with execution share[1]

Databuff tracing flows distribution chart → list → waterfall seamlessly; Spans are color-coded by Web/DB/cache/MQ, with TraceID/SpanID and environment info on the right. Shares OTLP-ingested data with topology, service detail, and AI Q&A.

§2 Comparison summary: Both meet production-grade Trace search and drill-down. SkyWalking has stronger filter dimensions and Profiling linkage; Databuff's waterfall distinguishes middleware Span types more clearly, and Trace shares the OTLP pipeline with metrics/topology/AI — no second Trace format to maintain during migration.

§2 Feature comparison · Distributed tracing

- **Dimension** — SkyWalking · Databuff · Databuff advantage - **Trace list entry** — Instance/endpoint/status/tag filters + Distribution scatter · Chart point-select → list, chart and Trace on one screen · See trends before single-Trace lookup — more intuitive slow-request pinpointing - **List fields** — Endpoint, duration, TraceID, success/error status · TraceID, API name / status code / host, service, duration · More on-call-readable fields, less guessing Endpoint meaning - **Detail view** — Span tree (default/tree/statistics), expandable Tags and Logs · Waterfall + Web/DB/cache/MQ coloring + execution share · Middleware time contribution visible at a glance — faster root cause - **Search capability** — Trace ID, duration range, tag key=value — rich dimensions · Left-side quick filters (response time/status/service/API) · Common filters out of the box — no tag expressions required - **Protocol & data** — Segment native or OTLP, separate pipeline config · OTLP sole entry, same source as topology/metrics/AI Q&A · No second Trace format during dual-write or migration

Service Topology

Visualize service dependencies and middleware calls to quickly scope incident impact.

SkyWalking · Topology

SkyWalking Demo · Chinese UI

Figure 3-1 · Topology: rating / gateway / app / frontend nodes, RPM and latency on edges, Go/Spring/Node tech stack on nodes[2]

SkyWalking topology aggregates call edges by service; nodes show tech-stack icons with RPM/latency — the familiar "war map" for the community, linkable to alerts and Trace modules.

Databuff · Global Topology

Databuff Demo

Figure 3-2 · Global topology: service-a/b with Redis, Kafka, MySQL, ES, remote payment middleware dependencies[1]

Databuff derives topology from OTLP Traces automatically; middleware labeled as [redis]/[mysql]/[kafka], nodes drill down to service detail. Easy to run in parallel with OTel Collector or other OTel backends to verify dependencies match.

§3 Comparison summary: Both have mature topology. SkyWalking annotates edges with real-time RPM/latency more finely; Databuff middleware naming aligns with OTel semantics — lower cost to validate during migration.

§3 Feature comparison · Service topology

- **Dimension** — SkyWalking · Databuff · Databuff advantage - **Node presentation** — Service nodes + tech-stack icons (Go/Spring/Node, etc.) · Services + [redis]/[mysql]/[kafka] semantic middleware nodes · Middleware type readable, aligned with OTel resource attributes - **Edge metrics** — RPM, latency annotated on call edges in real time · Dependency arrows + node health color (abnormal services highlighted) · Faulty nodes visible at a glance — no need to read edge numbers first - **Drill-down linkage** — Click node → Service / Trace / alerts · Click node → service detail and Trace directly · Shorter topology→root-cause Span path - **Data source** — SkyWalking Segment aggregation · OTLP Trace auto-derived, consistent with Collector semantics · Same semantics as OTel ecosystem — simple parallel validation - **Migration validation** — Parallel OTel backend needs conversion or dual-write · Parallel validation against OTel stack on same workload · Low-cost topology consistency check during SkyWalking migration

Alerting Comparison

Alert rules, event lists, and timelines — critical for on-call and migration periods.

SkyWalking · Alert Center

SkyWalking Demo · Chinese UI

Figure 4-1 · Alerts: active/other stats, layer/service/instance filters, Timeline, Mesh/General category tabs[2]

SkyWalking alerts offer Active/Other categories, Layer/Service/Instance multi-dimensional filters and a Timeline; messages include SLA, response-time thresholds, etc. — for teams needing fine-grained alert policies and long-term rule history.

Databuff · Alert List

Databuff Demo

Figure 4-2 · Alert center → alert list: severity filters, alert frequency bar chart, alert ID/description/service/trigger time/event count[1]

Databuff alert list filters by critical/minor severity and service; bar chart shows alert frequency distribution; each alert gives readable text like "avg latency 240ms exceeds threshold 60ms", sharing the data foundation with global dashboard and AI inspection.

§4 Comparison summary: SkyWalking has finer alert rule engine and Layer dimensions with more community config samples; Databuff alert list Chinese descriptions + frequency visualization are closer to on-call reading habits, and can link to AI intelligent inspection for natural-language follow-up.

§4 Feature comparison · Alerting

- **Dimension** — SkyWalking · Databuff · Databuff advantage - **Alert view** — Active/other stats + Timeline brush selection · Alert list + frequency bar chart, severity distribution intuitive · On-call landing shows "which minute had most alerts" - **Filter dimensions** — Layer / Service / Instance / Endpoint / keywords · Critical/minor severity + service name — simple filters · Two-step filter for common on-call scenarios - **Alert description** — SLA, response-time threshold rule trigger messages · Plain-language "metric vs threshold" (e.g. 240ms > 60ms) · No need to reverse-engineer rule syntax — messages forwardable directly - **Classification** — Mesh / General Layer tabs · Aggregated by service + severity, event count column · Grouped by business service — matches SRE on-call habits - **Intelligent linkage** — AI Pipeline ML detection, separate from dashboards · Shares OTLP data with AI Q&A/intelligent inspection · Natural-language root-cause follow-up after alerts — no broken chain

AI Smart Q&A and Inspection

From "dashboard clicking" to "natural language questions" — Databuff's core differentiator.

SkyWalking · AI / Intelligence

SkyWalking

SkyWalking offers AI Pipeline and other ML detection capabilities, focused on anomaly detection models and pipeline configuration — a "in-platform ML module" path. Day-to-day troubleshooting still centers on Trace/Topology/Log dashboards; conversational Q&A is not the default interaction[3].

ML pipelines and metric anomaly detection
Mature alerting + event management
No built-in conversational APM main interface

Databuff · AI Platform Conversational Q&A

Databuff Demo

Figure 5-1 · AI platform → chat: ask "query upstream/downstream topology of the first service", returns upstream/downstream table and natural-language topology summary[1]

Databuff has built-in AI-native APM: natural-language questions query service lists, topology, metrics, and Traces; Agent answers from OTLP-ingested data. Supports intelligent inspection, MCP exposure, and external MCP integration — a unified entry for SRE and AI Agent workflows.

Q&A reads Trace/Metrics/Topology directly
One-click intelligent inspection for full-environment health checks
MCP / Skill extends IDE toolchains

§5 Comparison summary (biggest gap): SkyWalking excels at ML pipelines and traditional alerting; Databuff excels at conversational APM + MCP openness. If selection criteria include intelligent ops / agent monitoring, Databuff has a clear functional lead.

§5 Feature comparison · AI smart Q&A

- **Dimension** — SkyWalking · Databuff · Databuff advantage - **Intelligence path** — AI Pipeline / ML anomaly detection pipelines · Conversational Q&A + intelligent inspection, built-in AI platform · Default interaction for 2026 agent ops — not an add-on module - **Interaction** — Dashboard clicking + rule configuration · Natural language questions, table + text summary responses · SRE/devs query APM in plain language — lower learning curve - **Q&A scope** — ML pipeline independently configured metrics · Service list, topology, metrics, Trace same-source queries · One question spans multiple modules — no switching five dashboards - **Data consistency** — ML module separate from Trace dashboards · Shares OTLP ingestion with APM — answers verifiable · AI conclusions match dashboard data — avoids "chat hallucination" - **Extension integration** — Open API / plugin ecosystem · MCP Server exposure + bidirectional external MCP · Cursor/IDE Agents can call APM directly — DevOps chain connected

Full-Dimension Summary

Based on five chapters of live Demo experience — selection reference on architecture, UX, and intelligence (not scoring; focused on OTel unification and 2026 intelligent-ops needs).

§6 Full-dimension summary table

- **Dimension** — SkyWalking · Databuff · Selection hint - **Data ingestion** — SkyWalking agents + OTLP/Mesh, etc. · OTLP 4317/4318 unified entry · Already on OTel → Databuff zero extra protocol - **Deployment** — OAP + UI + storage (more components) · Ingest + Doris + Web three components · Lightweight self-host → Databuff stack simpler - **Service monitoring** — Deepest metrics/Profiling · Service relationship graph landing + instance table · Deep Profiling → SW; fast troubleshooting → DB - **Distributed tracing** — Multi-dimensional search + Distribution scatter · Distribution chart → list → waterfall · Dual-write migration → DB OTLP same source - **Topology** — Fine RPM/latency on edges · Clear OTel middleware semantics · Parallel OTel validation → DB low comparison cost - **Alerting** — Mature Layer rules + Timeline · Chinese descriptions + frequency chart + AI linkage · Long rule history → SW; fast on-call reading → DB - **AI / intelligence** — AI Pipeline ML detection · Conversational Q&A + inspection + MCP · Agent ops → Databuff clearly ahead - **UI language** — Chinese supported (some English terms) · Native Chinese · Domestic teams — both work; DB more unified - **Community & ecosystem** — ASF top-level project, vast case library · Emerging OTel-native stack, MCP open · Deep SW footprint → OTLP parallel validation of DB

Summary: SkyWalking is a mature full-stack observability platform with deep Profiling, alert rules, and large-scale experience; Databuff is closer to 2026's "OTel unification + intelligent ops" with OTLP-native ingestion, three-component lightweight deployment, service-relationship/waterfall troubleshooting paths, readable alerts, and conversational APM. SkyWalking users should OTLP dual-write to the Demo for parallel comparison, then evaluate whether AI Q&A and a lightweight stack are worth adding incrementally.

References

Learn more: github.com/databufflabs/databuff

Why Do IT Systems Need Observability?

AIdevops2088 — Thu, 06 Aug 2026 01:18:52 +0000

Summary: In the microservices and cloud-native era, "the server is still alive" cannot answer why the business failed or which segment is slow. Starting from Metrics / Logs / Traces, this article uses live screenshots from the official Databuff Demo to explain how observability differs from traditional monitoring, why it is production-critical, and an OTel adoption path plus self-check questions for selection.

Before debating "which product to buy," let's narrow the question: what does observability actually solve, and how is it different from monitoring? The sections below follow concept clarification → three signal types with demos → why it's more needed now → adoption path. Demo environment: demo.databuff.ai[3], with real UI screenshots at each step so you can see which piece your team is missing.

Monitoring ≠ Observability

Many equate "observability" with "we deployed Prometheus + ELK." They focus on different questions:

Traditional monitoring answers: Is CPU spiking? Is disk almost full? Is the process alive?
Observability answers: Why did this order fail? Which call segment pushed P99 from 200ms to 2s? Did the canary release introduce new errors?

Monitoring is threshold alerting; observability is drill-down and correlation across arbitrary dimensions — you can still "ask" what happened from existing telemetry[1]. During a flash-sale traffic spike, host monitoring may say "all Pods are fine"; with Traces and RED metrics, you see a downstream RPC timeout dragging down the whole chain.

Key takeaway: IT systems need observability because once architecture grows complex, you can no longer answer "why is the business broken, where, and who is affected?" with "machines are alive" alone. In the monolith era, tailing logs often sufficed; in microservices one request crosses a dozen services — error logs may not contain the root cause, and on-call engineers bounce between views, inflating MTTR.

Three Signals: Metrics, Logs, Traces

- **Signal** — Typical use · Troubleshooting role - **Metrics** — Request volume, error rate, latency percentiles · Find "which service is red" - **Logs** — What the process printed at a moment · Exception stacks, business context - **Traces** — Full path of one request across services · Pinpoint "which hop is slow"

They complement each other: metrics surface anomalies, traces narrow scope, logs fill in details. This is why application performance monitoring (APM) and distributed tracing keep coming up in microservices.

.1 Metrics: see "who is slow, who is wrong" first

Databuff Demo · Service list

Figure 2-1 · Service list RED metrics: service-a avg 240ms, service-b ~70ms, 0% error rate[3]

Open-source APM Databuff puts RED metrics (request rate, error rate, response time) on one screen — observability's "business-view health," not just CPU curves.

.2 Traces: one click into the full call chain

Databuff Demo · Distributed tracing

Figure 2-2 · Trace count, error stats, and P50–P99 response time; click the chart to drill into slow requests[3]

Metrics alone are not enough. The tracing page shows Trace distribution and latency percentiles; click any point on the chart to drill into a specific slow request — the jump from "metrics to evidence."

Figure 2-3 · Single Trace Span waterfall: pinpoint which SQL, RPC, or gateway hop slowed the whole request[3]

.3 Topology: understand "who depends on whom"

Databuff Demo · Global topology

Figure 2-4 · Global topology: service-a / service-b with MySQL, Redis, Kafka, Elasticsearch dependencies[3]

With many services, no one can draw the full call graph from memory. Global topology auto-draws service and middleware dependencies — during incidents, quickly tell whether the app or a downstream component is the bottleneck.

Why It's "More Needed" Now

Rising architecture complexity: More services mean call paths keep changing with releases and scaling.
Faster change cadence: Multiple CI/CD deploys per day — without trace-level evidence, hard to tell which change caused a regression.
UX tied to SLA: P99 latency and payment failure rate directly affect revenue.
SRE culture: On-call needs one evidence chain, not five disconnected tools.

The global dashboard shows per-service alerts and abnormal states on a minute-level timeline side by side — ideal for first-pass on-call inspection, and the foundation for observability moving from reactive firefighting to proactive discovery.

Databuff Demo · Global dashboard

Figure 3-1 · Global dashboard: per-minute alert and health timeline per service[3]

Adoption Path

Engineering teams usually take three steps:

Unified collection: OpenTelemetry is the de facto multi-language instrumentation standard[2], ingested via OTLP — common ports gRPC 4317, HTTP 4318.
Storage/compute separation: Traces and metrics go to time-series or OLAP storage; the query layer correlates them.
Troubleshooting loop: Topology for dependencies, Trace waterfall for slow points, alerts for proactive discovery; ideally one UI covers "alert → topology → Trace → log context."

Also watch sampling rate and TraceId propagation — full Trace storage is expensive; broken instrumentation makes even the prettiest topology misleading.

Many teams start with SkyWalking, Jaeger, or other open-source APM; others want OTel as the sole data plane, putting metrics, traces, and assisted analysis in one stack. Databuff uses native OTLP ingestion, supports topology and RED metrics in the UI, and offers natural-language Q&A for service lists, topology, and trends — less DSL to memorize for on-call engineers.

Databuff Demo · AI Q&A

Figure 4-1 · AI Q&A: natural-language queries for service list, topology, and metric trends[3]

If you already run SkyWalking or similar, new services can OTel-ize and ingest in parallel, comparing Traces for the same request to evaluate dual-stack cost. Not a replacement for every scenario — one more option to consider.

Summary

IT systems need observability because complexity exceeds what humans can hold in working memory. Whenever a system delivers business value through multiple cooperating components, observability shifts from "nice to have" to "production essential."

When prioritizing, ask three questions first: Do you have cross-service Traces? Can you drill from a slow request to a Span? How many tools do you switch after an alert? The more answers lean toward "no / many," the higher the build priority — more important than arguing "which vendor" first.

References

[1] https://opentelemetry.io/docs/concepts/observability-primer/

[2] https://opentelemetry.io/docs/languages/

[3] https://demo.databuff.ai/

Learn more: github.com/databufflabs/databuff

AI Ops Squad Evolves Again: Log Analysis Gets a Rescue

AIdevops2088 — Thu, 06 Aug 2026 01:18:31 +0000

Demo scenario: InsufficientStockException

The demo app keeps hitting GET /demo/checkout. When inventory runs out, service-b throws InsufficientStockException: inventory unavailable for skuId=…. OTLP logs land in Doris with trace_id / span_id.

Every screenshot below comes from the same incident window (last 1 hour), walked in on-call order three ways:

Path A — You know the exception class: facet search on the global Logs page
Path B — You have a slow trace: span logs on the flame graph + deep link back to global logs
Path C — One sentence to AI, dispatching log.queryLog* tools for ERROR logs

Path A: global Log Analysis search

Menu: Application Performance → Log Analysis. No LogQL required — keywords + facets are enough.

Steps: search InsufficientStockException → filter ERROR + service-b → 95 log lines, bar chart shows ERROR spike windows.

Scene A: keyword + ERROR + service-b — each row has “Trace · View” to jump to the call chain

Path B: span logs in Trace + bidirectional deep links

B1 · Trace header: Log Analysis next to TraceID

Open a slow GET /demo/checkout trace (240ms) from Trace Search. The header shows TraceID; Log Analysis on the right jumps to the global page with traceId prefilled.

Scene B1: TraceID + Log Analysis entry — no copy-paste of traceId

B2 · Span sidebar: flame graph + Logs tab

Spans marked Logs on the flame tree open a sidebar Logs tab: timeline shows Received checkout request → Delegating inventory check to service-b INFO lines; selecting service-b’s span surfaces the ERROR stack.

Scene B2: flame graph (SELECT demo_inventory highlighted) + span log timeline on the right

B3 · Deep link: “View all in Log Analysis”

Click the sidebar footer link — the global page auto-fills traceId + spanId and shows only the 4 logs in that span’s context.

Scene B3: URL carries traceId/spanId — checkout context logs in one view

Path C: ask AI about logs

The UI is for precision; AI is for one-sentence delegation. The smart-query expert registers log tool families visible under Tool Management:

Scene C0: log.queryLogDetail · queryLogsByTraceId · queryLogsBySpanId · queryLogTrend

Scenario 1 — search ERROR logs by service + keyword:

Find service-b ERROR logs in the last hour related to InsufficientStockException. List traceId and key log summaries.

Scene C1: AI brain → dispatch smart query → getCurrentTimeRange + queryLogDetail

Scene C2: summary table with traceId + InsufficientStockException: inventory unavailable

Scenario 2 — known traceId, ask for ERROR logs and root cause:

Given traceId edfa44615dcee4d6bdfeed46d84bfb20, list all ERROR logs on this trace and explain why checkout failed.

Scene C3: queryLogsByTraceId → 13-span chain + ERROR logs + insufficient-inventory conclusion

Scenario 3 — check for ERROR log volume spikes (queryLogTrend):

How does service-b ERROR log volume trend over the last hour? Any obvious spike windows?

Scene C4: queryLogTrend → steady 2/min · no spike · ongoing inventory shortage

Tool selection: search by service/level → queryLogDetail; known traceId → queryLogsByTraceId; specific span → queryLogsBySpanId; volume spikes → queryLogTrend.

Where the data comes from · vs ELK

OTLP Logs (:4317 / :4318) → Ingest → Doris log_dc_record → POST /log/search. Inject traceId via MDC on the Java side for correlation.

DataBuff is log exploration in an APM context, not a replacement for ELK/Loki. The win is Trace/Metrics/AI in one context — fewer system hops.

Companion to article ⑤: ⑤ covers T→M→T→L troubleshooting order; this article shows three entry points for L in DataBuff on the same checkout scenario.

Try it yourself

One-command deploy · demo includes checkout + inventory failure

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Star on GitHub →

OpenTelemetry LogsTrace correlationAI Native

OpenClaw Finally Understands OpenTelemetry APM

AIdevops2088 — Thu, 06 Aug 2026 01:18:10 +0000

Meet DataBuff: AI-native OpenTelemetry APM

DataBuff (github.com/databufflabs/databuff) is an open-source AI-native OpenTelemetry APM — deploy with one Docker command.

In one line: ingest telemetry via OTel standards first, then let AI read traces, metrics, topology, and alerts — not a chat box bolted onto dashboards.

📊 OpenTelemetry APM foundation

Native OTLP — traces, service metrics, topology, alerting
Rich data model derived from traces (services, APIs, flows, component metrics)
Three components: Ingest → Doris → Web, low ops overhead

🤖 AI-native capabilities

Multi-agent collaboration: AI brain + metrics / inspection experts
Smart query: natural language over metrics, traces, topology, alerts
Health inspection: threshold-free anomaly screening with evidence chains

- **Capability** — What it does - **In-platform AI chat** — Ask in the Web UI: “Why is this trace slow?” or “Inspect order-service” - **MCP for external agents (this article)** — Cursor, Claude Code, OpenClaw call the same APM tools - **Official Skills** — Constrain query semantics and inspection flows — agents don't invent metrics

Screenshot: Ingest → Doris → Platform — one command to run

Screenshot: global topology — AI and MCP tools read the same live data

Screenshot: in-platform AI brain dispatches experts — external agents use the same backend via MCP

This article: DataBuff exposes platform capabilities via MCP + Skills to OpenClaw (and Feishu / DingTalk channels). Below: 5-minute setup for one-sentence whole-cluster inspection.

am alert: five tabs, or one sentence?

Alert fires: “Checkout is slow — anything wrong cluster-wide?”

Old way: APM dashboard → traces → topology → alert list → hand-written summary. 20–30 minutes per round.

Screenshot: multi-tab evidence stitching — agents can do it in one prompt

With OpenClaw: send one message in CLI, WebChat, or Feishu / DingTalk. The agent calls DataBuff MCP against real OTel data in Doris, follows official Skills, and returns an evidence-backed report — not hallucinated numbers.

❌ Without MCP

The model invents “error rate 2.3%” — pure hallucination, not safe for incident channels.

✅ MCP + Skills

inspectService first → traces / topology for anomalies → Chinese or English report with evidence.

Screenshot: OpenClaw activity — Databuff-apm GetCurrentTimeRange / QueryServicesAll

MCP is the hands; Skills are the brain

DataBuff ships an agent integration pack (integrations/agent/). You need both layers for a real ops agent:

MCP = 11 APM tools (services, traces, metrics, alerts, inspection, charts)

Skills = query semantics + inspection workflow (when to inspect, time ranges, evidence chaining)

- **Layer** — Contents · Examples - **MCP tools** — Live data access · queryServicesAll, inspectService, queryTraceDetail - **Skills** — Behavior rules · skill.data.metrics, skill.inspection.health

External agents hit the same Spring beans as in-platform AI experts — not pasted APIs, real OTel data in Doris.

Screenshot: skill.data.metrics enabled on OpenClaw Skills page

-minute setup: MCP + Skills end-to-end

Prerequisite: DataBuff deployed; OpenClaw can reach the Web port.

Step 0 · Get the official integration pack — do not download Skills/MCP from third-party marketplaces:

git clone https://github.com/databufflabs/databuff.git cd databuff/integrations/agent/

Or browse integrations/agent/ on GitHub. Contains skills/ (2 official Skills) and mcp/ (client config examples).

Step 1 · Deploy DataBuff (if not installed):

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Screenshot: one-command install, note Web URL and port

Step 2 · Configure MCP — copy mcp/openclaw-amc-config.example.json, set mcp.url to your DataBuff host:

{ "name": "databuff-apm", "description": "DataBuff APM MCP — metrics, traces, topology, alerts, inspection", "mcp": { "transport": "streamable-http", "url": "http://YOUR\_DATABUFF\_HOST:PORT/mcp" }, "skills": [ "skill.data.metrics", "skill.inspection.health" ] }

Replace YOUR_DATABUFF_HOST:PORT. MVP needs no Authorization header. MCP endpoint: /mcp (Streamable HTTP).

Step 3 · Install Skills — copy integrations/agent/skills/ to your client Skills path:

- **Client** — Skills path - **OpenClaw** — OpenClaw workspace skills/ (Control UI → Skills should list both) - **Cursor** — ~/.cursor/skills/ or project .cursor/skills/ - **Claude Code** — ~/.claude/skills/

# From databuff repo root (OpenClaw example — adjust target path) cp -r integrations/agent/skills/skill.data.metrics /skills/ cp -r integrations/agent/skills/skill.inspection.health /skills/

Each Skill is a directory with SKILL.md. Synced with deploy/common/skills/ — re-copy after updates.

- **skillId** — Purpose - **skill.data.metrics** — APM metrics, trace, alert query semantics (time ranges, which tool) - **skill.inspection.health** — Health inspection flow (inspect first, gather evidence, conclude)

Step 4 · Verify — after restarting OpenClaw Gateway:

Control UI → Skills: skill.data.metrics, skill.inspection.health
Control UI → Instances: Gateway online
New chat → Activity: Databuff-apm tools (e.g. QueryServicesAll)

Screenshot: Gateway and Control UI connected (127.0.0.1:18789)

Security: MVP has no separate MCP token — use on private network / VPN; add a gateway before public exposure.

Three prompts to validate the full path

①

“List services from the last hour” — Skill parses time → getCurrentTimeRange → queryServicesAll

Screenshot: 7 services in the last hour (service-a / service-b / MySQL, etc.)

②

“Inspect service-a health” — inspectService → queryMetricData / queryServiceAlarms

Screenshot: Activity 3 tools — InspectService / QueryMetricData / QueryServiceAlarms

③

“Chart service-a error rate for the past hour” — queryMetricData + drawTrendCharts

Screenshot: 0% error rate trend — real data, not invented

Pass criteria: tool panel shows queryServicesAll / inspectService with JSON responses — not plain-text guesses.

vs in-platform AI and vs Cursor?

- **Scenario** — Path · Best for - **DataBuff Web UI** — AgentScope JAVA_BEAN + multi-expert · Day-to-day console troubleshooting - **OpenClaw / Feishu / DingTalk** — MCP + Skills · One-liner queries in alert channels - **Cursor / Claude Code** — Same MCP + Skills · Developers querying APM from the IDE

All three paths are independent. OpenClaw brings APM into your existing agent runtime — no extra browser tab for AI incidents.

Full config and FAQ: docs/manual/agent-integration.md and integrations/agent/README.md in the repo.

From guessing metrics to calling real APM

DataBuff opens APM to any agent runtime; OpenClaw / Feishu / DingTalk are just entry points — MCP for real data, Skills for semantics is the core.

⭐ Wire up DataBuff agent integration in 5 minutes

github.com/databufflabs/databuff/integrations/agent

Try: “Inspect service-a, then check if anything is wrong cluster-wide.”

View on GitHub →

DataBuff v0.1.3 Release Notes

AIdevops2088 — Thu, 06 Aug 2026 01:17:59 +0000

DataBuff v0.1.3 focuses on three things: native SkyWalking gRPC ingest (no Agent swap), Ops Expert install recovery and read-only SSH troubleshooting, and verified in-place upgrades — built for teams already on SkyWalking and for installs that exit non-zero while the UI still opens.

Highlights

Native SkyWalking ingest: ingest listens on 11800 (OAP default); Java only changes collector.backend_service.
Ask why checkout is slow: natural-language questions return traceId and bottleneck spans without browsing the Trace list first.
Logs link to traces: one click from a log row to the full call chain.
Ops Expert · troubleshooting mode: when install exits non-zero, Web still opens; configure an LLM and let Ops Expert recover Doris over read-only SSH.
Ops Expert · on-host checks: align container/host evidence with slow spans — complements telemetry-only AI.
In-place upgrade: upgrade scripts keep data/ intact; amd64/arm64 offline packages ship for 0.1.3.

Keep your SkyWalking Agent

Ingest listens on SkyWalking gRPC port 11800. No Agent jar swap — topology, traces, JVM metrics, and log correlation work the same as the OTLP path.

Fig. 1 · SW protocol demo: checkout dependencies in DataBuff topology

Fig. 2 · Ingest exposes OTLP (4317/4318) and SkyWalking (11800)

Checkout latency: ask once, get an answer

Demo scenario: GET /demo/checkout P99 around 240ms. After ingest, troubleshooting shifts from browsing traces to asking AI.

Fig. 3 · service-a checkout ~240ms avg — triggers AI and Ops Expert

Fig. 4 · Natural-language question over SkyWalking Agent segments

Fig. 5 · 240ms request split by service/middleware; bottleneck span highlighted

Fig. 6 · Jump to flame graph by traceId — DB/Cache/MQ/RPC expandable

Fig. 7 · Log analytics: each row links to the call chain

Ops Expert: install failed, UI still opens

When install.sh / start.sh exits non-zero and Doris FE/BE are not ready, Web enters troubleshooting mode: configure an LLM API key, pick the built-in Ops Expert, and recover containers over read-only SSH.

Fig. 8 · Start failed but troubleshooting guidance prints — configure LLM before Doris is healthy

Ops Expert: no OTel data — conclusion in minutes

For missing OTel data, container restarts, port conflicts, and similar chores, Ops Expert runs read-only Bash + SSH to align host evidence with what APM shows.

Fig. 9 · Describe missing OTel data to Ops Expert

Fig. 10 · Read-only Bash checks containers and ports

Fig. 11 · Evidence-backed conclusion ready for the incident channel

Full SkyWalking walkthrough: native ingest deep dive. Five Ops Expert scenarios: runtime troubleshooting playbook.

Install & upgrade

New install: Docker installation. Upgrade an existing deployment: Upgrade and uninstall.

Try it

GitHub Release v0.1.3 · Live demo · Star on GitHub

SkyWalking Meets AI-Native Troubleshooting

AIdevops2088 — Thu, 06 Aug 2026 01:17:38 +0000

First, SkyWalking

Many Java teams run SkyWalking in production — an Apache top-level project whose Agent often stays in place for years. Typical strengths:

Java Agent bytecode enhancement with minimal app changes
Segments bundle Spans and cross-service refs so traces stay readable
Traces, JVM metrics, and logs can share one reporting path
Strong docs and community; OAP + UI battle-tested at scale

Once Segments land in OAP, you get service topology, trace lists, span trees, and JVM charts — SkyWalking is solid at getting traces in.

SkyWalking UI (:18080): services, Apdex, latency, endpoint load — the daily ops view

Where teams spend time is turning that data into answers: alerts without traceId, hunting representative slow traces, summarizing span trees for incident channels, jumping between topology, metrics, and logs. That is not a SkyWalking flaw — it is the next step: faster conclusions from the same telemetry.

DataBuff v0.1.3 layers AI on top of mature SkyWalking collection: Agents keep reporting, Segments stay identical; you can ask why an endpoint is slow in natural language and view traces, metrics, and logs in one flow. Here is the most visible change.

Note: This article is about AI-assisted interpretation on SkyWalking ingestion — same Agent, same Segment data — focused on whether one pipeline can produce answers faster.

Highlight: slow checkout — ask once, get an answer

Demo: GET /demo/checkout P99 ~ 240ms. Alerts often ship without traceId. With DataBuff, triage shifts from scrolling trace lists to asking AI.

①

Ask directly (no traceId): “Why is service-a’s checkout endpoint slow lately?”

Natural-language question; backend reads Segment data from the SkyWalking Agent

②

AI breaks down the trace: typical slow trace, segment timings, bottleneck span — no manual row-by-row comparison.

240ms request split across services and middleware; bottleneck span is obvious

③

Structured conclusion: actionable incident text, not just “maybe the DB is slow.”

traceId + bottleneck + priority — ready to paste into Slack or war room

④

One-click verify: jump to the flame graph with the returned traceId and confirm end-to-end spans.

Flame graph from SkyWalking Segments: GET /demo/checkout 240ms, DB / cache / MQ / RPC expandable

What you gain: from “filter list → read span tree → write incident note” to ask → get traceId → verify in UI. Same Segment source; added AI readout.

💬

AI slow-path Q&A

No traceId upfront — endpoint-level question returns bottleneck span

🔗

Logs → trace

“Trace · View” on log lines jumps to the call chain for slow + ERROR context

🖥️

Ops expert review

After AI flags JVM risk, read-only SSH check on host/container

Go deeper: topology, service flow, JVM

After AI answers, the same UI keeps going — no tool hopping.

Global topology: checkout upstream/downstream in one graph

Service flow: 240ms entry, downstream contribution expandable

Log analysis: each row links to traces — slow requests and ERROR in one context

JVM charts from SkyWalking: threads, GC, heap — cross-check with AI and ops expert

Before vs after: how the path changes

For the checkout scenario, SkyWalking-only UI vs DataBuff overlay:

- **Step** — SkyWalking UI only · With DataBuff - **1 · Find service** — General → Service → pick service-a · Topology / service list → service-a - **2 · Find slow trace** — Trace page filter checkout → open rows one by one · Ask AI → traceId + bottleneck span returned - **3 · See bottleneck** — Read span tree manually for DB / RPC time · AI summary + flame graph jump - **4 · Conclusion** — Human writes “maybe DB query slow” · AI remediation hints; ops expert can SSH-check JVM - **5 · Logs** — Separate log system, match traceId · Log list “Trace · View” → call chain

Your SkyWalking Agent keeps reporting; you add AI query, unified UI, and log deep-links — same collection target.

How to connect: keep the Agent, point backend to DataBuff

v0.1.3 ingests native SkyWalking gRPC on 11800 (OAP default). No Agent jar swap — change the collector address:

Ingest exposes OpenTelemetry (4317/4318) and SkyWalking (11800) — polyglot services side by side

Key Agent setting: collector.backend_service → DataBuff host

# agent.config agent.service_name=${SW_AGENT_NAME:your-service} collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:your-databuff-host:11800}

SkyWalking gRPC and OpenTelemetry into one ingest → storage → AI + Web

Two onboarding paths, both AI-capable:

- **Mode** — Setup · Best for - **Side-by-side trial** — Keep SkyWalking OAP; read SkyWalking data via MCP · Cannot move Agents yet — try AI Q&A first - **Native ingest** — Point Agent to DataBuff :11800, Segments direct to ingest · Switch backend; traces / JVM / logs unified in DataBuff

FAQ

Do I replace the Agent?

No. Existing SkyWalking Java Agents work — usually only collector.backend_service changes.

Must OAP go away immediately?

No. Keep OAP for a trial; when you commit to DataBuff, point Agents over in batches and verify checkout-style Q&A works.

Can OpenTelemetry coexist?

Yes. Java on SkyWalking (11800), Go/Python on OTLP (4317), one DataBuff UI.

What does AI actually do?

Demo covers on-call staples: slow endpoint → traceId + bottleneck span, log line → trace, JVM curves for follow-up. You do not need to be a trace expert to get direction.

Who is this for?

SkyWalking in production, want AI Q&A on existing trace data
Java-heavy estates not ready to re-instrument with OpenTelemetry
Want traces, logs, and metrics in one troubleshooting flow

Suggested path: test env or one instance → change Agent address → hit checkout-like traffic → ask AI “why slow lately” → confirm traceId + flame graph → expand rollout.

SkyWalking ingestion + AI interpretation

Keep your Agent · Ask once for answers · Logs link to traces

Star on GitHub →

DataBuff v0.1.4 Release Notes

AIdevops2088 — Thu, 06 Aug 2026 01:17:17 +0000

DataBuff v0.1.4 is out: configurable seven AI ops entry points, multi-agent collaboration and deep inspection reports, plus SkyWalking/Trace data fidelity and Doris/upgrade hardening. Relative to v0.1.3: 45 commits / 274 files.

Highlights

Seven AI ops entry points: See, Squad, Inspect, Diagnose, Repair, Predict, Answer — names, suggested prompts, and expert bindings are configurable.
Multi-agent collaboration: parallel across experts, serial within one expert; HTML / Markdown / SVG reports previewable in the workspace.
Deep service inspection: synthesizes metrics, logs, alerts, dependencies, error traces, instances, and JVM signals into an evidence-backed HTML report.
Product Q&A expert: answers deploy/ingest/config questions from in-image docs and code.
SkyWalking MQ virtual services: MQ producers normalized; RPC/entry stats and Trace time alignment fixed.
Doris runtime recovery: AI troubleshooting entry stays available during outages; auto-hydrate after recovery without restarting Web.
Upgrade compatibility: legacy Docker Compose auto-selected; backs up data/ before upgrade with rollback on failure.

Seven AI ops entry points

Go to AI Platform → AI Chat. Pick a capability from the home surface; configure names, prompts, and expert bindings in the admin page.

Fig. 1 · AI chat home with seven configurable entry points

Multi-agent collaboration

Hand complex tasks to the AI brain — it dispatches specialists in parallel (e.g. data queries + health inspection) and merges an actionable report.

Fig. 2 · Parallel dispatchExpertTask calls to data and inspection experts

Fig. 3 · P0/P1 summary with evidence chain, previewable in chat

Deep service inspection

Go to AI Platform → Smart Inspection. One sentence triggers a full HTML report — entry metrics, error logs, downstream health, and graded conclusions.

Fig. 4 · Trigger inspection and request an HTML report

Fig. 5 · HTML report: health scores, entry metrics, errors, downstream deps

Product Q&A expert

Deployment, ingest, port, and alert questions are answered from in-image DataBuff docs and code — not model memory alone.

Fig. 6 · Product Q&A expert grounded in in-image docs

Data fidelity: MQ virtual services & topology

v0.1.4 normalizes SkyWalking/OTel MQ producers into virtual services and fixes RPC CLIENT/outbound peer mapping, entry stats, and Trace end-time alignment.

Fig. 7 · MQ page: producers and consumers tracked separately

Fig. 8 · Global topology: checkout dependencies at a glance

Fig. 9 · Service flow: per-downstream contribution for RCA

Install & upgrade

New install: Docker installation. Upgrade an existing deployment: Upgrade and uninstall.

Try it

GitHub Release v0.1.4 · Live demo · Star on GitHub

5 Ops Tasks That Take Half an Hour — Ops Expert Finishes in 3 Minutes

AIdevops2088 — Thu, 06 Aug 2026 01:16:56 +0000

On-call pain is rarely that a problem is “too hard.” It’s that you already know what to check — and still have to SSH in and type every command yourself. These five show up in almost every test environment:

OTel won’t connect / no data — SDK is configured, the platform is blank; you guess endpoint, port, or a dead process.
Container keeps restarting — you see Restarting, dare not poke randomly, and crawling logs inside the container is slow.
Inspect real Java JVM flags — startup args, container limits, and effective values disagree; you bounce between jinfo / jcmd.
Port in use, service won’t start — lsof / ss once, then make sure you don’t kill the wrong process.
CPU hotspots — flame-graph tooling slips your mind; sampling configs take half a day before a chart appears.

Solo, these jobs often take 15–40 minutes (find the host, log in, recall commands, reconcile the conclusion). With DataBuff Ops Expert, the same work usually lands in 1–3 minutes: open AI Platform → AI Chat, pick Ops Expert, describe the symptom in plain language.

- **Scenario** — DIY SSH (typical) · Ops Expert (measured) - **OTel won’t connect** — endpoint / port / process ~15–25 min · ~1–2 min to a conclusion - **Container restart loop** — log diving ~20–40 min · ~2–3 min to find & fix - **JVM flags** — jinfo / jcmd ~10–20 min · ~1 min - **Port conflict** — ss / lsof ~5–15 min · ~1 min - **Flame graph** — tooling + sample ~20–40 min · ~2–3 min

Walkthrough below: three shots per scenario — your prompt → Ops Expert at work → the conclusion.

1. OpenTelemetry won’t connect / no data

Example prompt:

Test env OTel won’t connect and the platform has no data. Check whether 4317/4318 are reachable and tell me in one sentence if the endpoint is correct.

1 Prompt2 Process3 Conclusion

① User prompt

② Ops Expert process

③ Ops Expert conclusion

2. Container keeps restarting

Example prompt:

ai-apm-demo keeps Restarting — help me get it healthy.

1 Prompt2 Process3 Conclusion

① User prompt

② Ops Expert process

③ Conclusion (memory limit 10MB → OOM 137 → raised to 512MB)

3. Inspect Java runtime flags

Example prompt:

Show me the effective JVM flags for the ai-apm-web Java process — especially heap and GC.

1 Prompt2 Process3 Conclusion

① User prompt

② Ops Expert process

③ Ops Expert conclusion

4. Port in use — service won’t start

Example prompt:

Who is holding port 27403? Tell me the process and command — do not kill anything.

1 Prompt2 Process3 Conclusion

① User prompt

② Ops Expert process

③ Ops Expert conclusion

5. Capture a flame graph

Example prompt:

Take a short CPU flame graph for the ai-apm-web Java service on the test host and point out the hotspots.

1 Prompt2 Process3 Conclusion

① User prompt

② Ops Expert process

③ Conclusion (sample result + hotspot readout)

Same loop every time: describe the symptom → watch the process → read the conclusion. The timing table at the top is the DIY SSH vs Ops Expert gap.

Open source · one-line install

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

GitHub (Star welcome):

https://github.com/databufflabs/databuff

Live demo:

https://demo.databuff.ai

Ops ExpertOTelCrashLoopFlame graph

Open-Source AIOps, Finally Someone Built It

AIdevops2088 — Thu, 06 Aug 2026 01:15:56 +0000

Gartner coined AIOps back in 2016. For years the gut feeling was “vaporware”: commercial tools are expensive, closed, and black-box; in open source you get alert-noise scripts or log analyzers — nothing that truly wires AI into live monitoring data and can fix problems. The moment I got DataBuff running, my first thought was — someone finally built it.

This post skips architecture slides and protocol names. I walk through seven scenarios I ran hands-on in a test environment. Each starts with one sentence; the AI pulls data, draws conclusions, and in some cases SSHs in to repair. After these, AIOps stops being a deck concept and becomes something you can use.

What it is: open source, AI-native, OpenTelemetry APM

One-line positioning: DataBuff is an open-source, AI-native APM built on OpenTelemetry. Plain English — it is a monitoring platform for metrics, traces, and logs; unlike classic APM, AI is not a bolt-on chat box on the side. It lives on the data: you ask a question, it reads live metrics, follows traces, queries logs, and returns evidence-backed answers instead of dumping charts on you.

Minimal stack: ingest + storage + AI platform — one command to start

The gap is “AI-native.” Many products are “APM + chat widget” where the model just tells you to look at a dashboard. DataBuff’s AI reads data, calls tools, and can reach hosts — the seven scenarios below are proof. The arc to keep in mind:

Glance the roadmap first, then walk each case

Scenario 1 · Ask your system in plain language

Start with the obvious. To find slow services you used to open Grafana, memorize PromQL, write queries, read charts, and sort yourself. In DataBuff I asked in plain English: “Which service was slowest in the last hour? List the top 3.”

One question — AI queried 20 services and returned a ranked table with latency and error rate

It did not flash a chart and leave me guessing. It actually scanned 20 services, computed average latency, and returned a table: slowest service-a at 240ms avg, then service-b 70ms, skyWalking-service-a 35.7ms, with volume and error rate. Twenty-three seconds, zero query language.

For newcomers, that is what AIOps should feel like — no query DSL, no metric math — just ask.

Takeaway: natural-language Q&A is not “search.” The model reads intent, hits live data, and answers with evidence. The more casual the question, the more concrete the reply.

Scenario 2 · Multi-agent: not one bot, a squad

If scenario one felt “nice,” this one reframes the product — DataBuff is not a single AI, it is a squad. The experts on the roster: AI Brain, Data Query, Intelligent Inspection, Ops Expert, Product Q&A. The right pattern is not inventing fake “metrics expert / trace expert” labels — leave the expert picker alone, throw the hard task at AI Brain, and let it dispatch real experts. I said one sentence:

“Any cluster anomalies in the last hour? Run a joint diagnosis with Data Query and Intelligent Inspection: Data Query checks latency, error rate, and slow traces; Inspection runs tiered health checks; summarize into an incident report I can forward.”

AI Brain calls dispatchExpertTask twice — Data Query (data) and Inspection in parallel

What followed is not typical APM. AI Brain said it would dispatch two tasks, then successfully called dispatchExpertTask twice — latency/errors/slow traces to Data Query, tiered health checks to Intelligent Inspection. Both worked in parallel. Inspection reported JVM/GC/thread metrics healthy across 34 services; Data Query found real issues: Elasticsearch index 404 (~144k failures) and MySQL-side InsufficientStockException. AI Brain merged both into a forwardable report.

P0 ES index down + P1 inventory business errors; Inspection confirms service health; HTML report ready to share

The report: P0 Elasticsearch indexes unavailable (my_index_1 / my_index_2 all 404, ~144k failures); P1 MySQL stock business errors (InsufficientStockException on service-b — business stock-out, not infra down); plus service self-health confirmed by Inspection. Full HTML you can preview or paste into an incident channel.

Takeaway: the “A” in AIOps is not a chat box — it is AI Brain splitting work across real experts, then synthesizing. One human sentence; a squad that can name and dispatch.

Scenario 3 · Inspection: one sentence → shareable HTML report

Runbooks hate manual inspection — check metrics, thresholds, stitch a report for half a day. In DataBuff I switched to Intelligent Inspection and asked: “Run an inspection on service-b and output a full HTML report.”

Intelligent Inspection on service-b — full HTML report requested

Eighty-one seconds, twenty-two steps — not a chat essay but formatted HTML. Overview: entry health 98, downstream MySQL 60, Redis 100, zero active alerts. Entry looks fine (~4 req/min, 0% errors, ~70ms avg), but error logs expose “false green” — 60 ERROR lines in 30 minutes, all InsufficientStockException:

Top of report: health cards + entry metrics + errors (0% HTTP errors vs InsufficientStockException)

Further down: evidence chain — downstream [mysql]demo_apm at 50% errors, traces show findInventory → stock query throwing; graded conclusion P0 system OK, P1 partial business impact, with fixes — repair stock data, alert on this business exception so HTTP 200 does not hide it. File at outputs/service-b-health-report.html, preview in-chat.

Bottom of report: dependencies, trace evidence, graded conclusion and actions

Inspection should end in an HTML report you can open and forward — green at the edge, red underneath, spelled out.

Scenario 4 · Root cause: one sentence with evidence

Incident triage is experience-heavy — charts, time alignment, hand-written PromQL, manual evidence chains. I asked bluntly: “Where is the bottleneck for service-a in the last hour — app, database, or downstream?”

Data Query pulls topology and ranks seven downstream calls by latency share

It delivered. Time range set, service-a topology — seven downstreams — then a table of call count, avg latency, and share: service-b HTTP 100ms and RPC 80ms on top; MySQL 20ms, ES 18ms, Redis 13ms, Kafka 8ms, remote payment 7ms all normal.

Bottleneck: downstream service-b (73.2% of outbound time); app and other deps cleared

One-line conclusion: bottleneck is downstream service-b — HTTP 100ms + RPC 80ms = 180ms per request, 73.2% of outbound time. It also listed who is not at fault — service-a 120 requests 0 errors 0 alerts, MySQL/ES/Redis/Kafka/remote payment all normal — then next steps on service-b HTTP/RPC for slow calls or thread blocking.

Takeaway: RCA here is not “here is a trend chart” — it is ranked downstream latency, attributed share, and a sentence you can paste into an incident doc.

Scenario 5 · Ops Expert: don’t just look — fix it

The first four scenarios are read-only intelligence. The next step for AIOps is hands-on — Ops Expert SSHs in, investigates, changes config, and repairs. Most tools stop at “here is what broke.”

Real case: demo container ai-apm-demo stuck in Restarting. I said: “Container keeps restarting — fix it.”

Plain-language request: container keeps restarting

Ops Expert SSH’d in, ran docker logs, docker inspect, free -m, found OOM kill (137) from too-low memory limit, applied the fix, restarted the container.

On-host investigation via docker logs / inspect / free -m

OOM from memory cap — parameter updated, docker ps healthy again

After docker ps, the container stayed up. Other AIOps tell you what broke; this one helped fix it. That is the line between “see” and “repair” in open-source AIOps.

Ops Expert does not only suggest commands — it investigates, concludes, and proposes the fix. DataBuff closes the loop.

Scenario 6 · Capacity: from reactive firefighting to foresight

Everything so far is after failure. Higher-level AIOps judges capacity before you scale. I asked: “This Redis averages 366ms — capacity bottleneck or not? Should we scale? Planning advice.”

Topology clarifies which Redis instance matters; capacity vs slow-operation guidance

First it clarified topology — service-a actually depends on [redis]redis:6379 (154 calls/hr, 13ms, healthy); the 366ms [redis]redis.test:6379 is not on service-a’s path. Easy to mix up under pressure.

On the slow instance: 352,807 calls/hour, ~98 QPS, 366ms avg. It did not say “scale because slow” — 98 QPS is far below what a single Redis can take; the bottleneck is likely operations (big keys, blocking commands like KEYS / wide SMEMBERS), with Top 3 causes and SLOWLOG GET / redis-cli --bigkeys. Advice: do not scale blindly — find the slow command first.

Good capacity analysis separates “under-provisioned” from “misused” — only the former deserves more machines.

Scenario 7 · Product Q&A: open source with built-in support

Hardest part of open source: “how do I configure this?” — docs, issues, waiting. DataBuff ships a Q&A expert. I asked the classic newbie question: “How do I wire the OpenTelemetry SDK? Where do I set alert thresholds? Step-by-step paths.”

Answers from product docs: OTLP ports and alert menu paths

Not a generic tutorial — it read this product’s docs: OTLP ingest, gRPC 4317 / HTTP 4318, point any SDK exporter at Ingest for traces, metrics, and logs without a separate agent.

Env vars + Spring Boot javaagent one-liner + Python doc paths

It also pointed to language quick starts: Spring Boot with opentelemetry-javaagent.jar, one java -javaagent:... for zero-code instrumentation; Python OTLP docs too. Alerts under Configuration → Alerting → Detection rules → New rule — pick object, metric, threshold, severity; evaluates every minute on the last five minutes; plus convergence and silence policies.

Open source’s gap is often “someone to ask.” DataBuff embeds Q&A that reads its own docs — better than a random search result.

Worth saying: the UI is a solid APM on its own

DataBuff is not chat-only. Under the hood you get global topology, services, service map, databases, queues, caches, external deps — full-path drill-down from the big graph to metrics to a single trace. AI reads; the UI lets you verify.

Global topology — services and dependencies at a glance

Service list — health status in one scan

Will you actually run it?

Yes — quickly. One curl, three components, minimal deps. Open the Web UI, add an API key, start asking.

One command — three components up

AIOps should not be a luxury for a few big vendors — any team should spin it up in five minutes. DataBuff is open, self-hostable, data stays yours, code is inspectable, built on OpenTelemetry so most SDKs plug in without lock-in.

Open-source AIOps, finally built — you could be next

If these seven scenarios resonate, don’t stop at reading.

Star it on GitHub, run it in minutes, and ask it the question that annoys you most on-call.

https://github.com/databufflabs/databuff

Questions? Open DataBuff and ask the Q&A expert — it is already in the product.

DataBuff vs OpenObserve: Same-Host Lab Comparison

AIdevops2088 — Thu, 06 Aug 2026 01:15:49 +0000

Same-host lab: DataBuff (OTLP :4318) and OpenObserve (OTLP HTTP :5080/api/default) on the same Demo (service-a / service-b). Host: 192.168.50.140 · DataBuff v0.1.4 · OpenObserve v0.91.0-rc1. Marks: ✅ verified in this lab · △ present but limited · ❌ no equivalent. Green bold cells are clear DataBuff leads.

Positioning: DataBuff = AI-native APM depth; OpenObserve = unified observability platform (Logs / Metrics / Traces / RUM), strong on log search and object-storage cost.

1. Capability matrix

Seven AI capabilities (v0.1.4: See → Squad → Inspect → Diagnose → Repair → Predict → Answer)

- **Capability** — OpenObserve v0.91.0-rc1 · DataBuff v0.1.4 - **① See · natural-language questions** — ❌ · ✅ Ask about services / topology / trends; AI reads telemetry - **② Squad · multi-agent collaboration** — ❌ · ✅ Parallel evidence gathering; serial context preservation; reusable task orchestration - **③ Inspect · service inspection + report** — ❌ · ✅ One-shot inspection with evidence and recommended actions - **④ Diagnose · bottleneck / RCA evidence** — ❌ · ✅ Trace / metrics / topology evidence (not a black-box “root cause”) - **⑤ Repair · Ops Expert actions** — ❌ · ✅ Repair under policy + human approval; dangerous-command denylist - **⑥ Predict · capacity / trends** — ❌ · ✅ Capacity and trend analysis — from after-the-fact to ahead-of-time - **⑦ Answer · product Q&A** — ❌ · ✅ Answers deploy / ingest / config from docs and code - **Extend · MCP / Skill / custom experts** — ❌ · ✅ External MCP / Skill and custom digital experts

Largest gap: OpenObserve has no equivalent AI platform (Traces has an LLM Insights entry, not validated here as APM triage); DataBuff exposes the seven capabilities as configurable home entries with APM as AI context.

APM

- **Capability** — OpenObserve v0.91.0-rc1 · DataBuff v0.1.4 - **1. Global topology** — ❌ No service dependency topology · ✅ Global topology + health colors + node drill-down - **2. Service list & golden metrics** — ✅ Service Catalog (Requests / Error Rate / P99, etc.) · ✅ Service list + charts; same demo shows service-a / b - **3. Service-level topology** — ❌ · ✅ Dedicated service topology - **4. Service call analysis (up/downstream + Trace)** — ❌ · ✅ Upstream/downstream structure, latency/contribution; drill to Trace - **5. Instance golden metrics** — ❌ · ✅ Instance golden-metric charts / list - **6. Instance topology** — ❌ · ✅ Dedicated instance topology - **7. Instance call analysis (up/downstream + Trace)** — ❌ · ✅ Per-instance up/downstream + Trace - **8. Endpoint topology** — ❌ · ✅ Dedicated endpoint topology - **9. Endpoint call analysis (up/downstream + Trace)** — ❌ · ✅ Per-endpoint caller/callee + Trace - **10. Service flow (service / endpoint Trace contribution)** — ❌ · ✅ Response contribution from entry; service / endpoint Trace view - **11. Middleware / external pages (DB / cache / MQ / external)** — ❌ db/http visible on Span fields; no dedicated pages · ✅ Dedicated pages: DB / cache / MQ / external - **12. Error analysis (stats + endpoint)** — △ Can filter ERROR spans / logs · ✅ Error stats + endpoint drill-down - **13. Trace list / search** — ✅ Spans/Traces + flexible query; this lab shows service-a · GET /demo/checkout · ✅ Charts + list, multi-dimension filters - **14. Trace detail** — ✅ Waterfall / Flame Graph / Trace Graph · ✅ Call-order waterfall + Span attributes - **15. Trace Span → logs** — ✅ Trace / Span can link to logs · ✅ Top “Log analysis” + Span Logs / Logs tab - **16. Log list / search** — ✅ Strength: SQL / full-text + histogram; hundreds of events in this lab · ✅ - **17. Log detail** — ✅ · ✅ - **18. Log → Trace** — ✅ Log → Trace (down to Span) · ✅ Log → Trace, down to Span - **19. Flexible Metrics query (SQL / PromQL)** — ✅ Metrics page: SQL / PromQL / Builder · △ Internal SQL; no public PromQL entry - **20. Custom dashboards** — ✅ Dashboards can be created (list may be empty in this lab; capability present) · ❌ Not yet - **21. Unified storage cost (object store + compression)** — ✅ Home shows Ingested / Compressed (~96MB → 10.5MB in this lab) · △ Doris columnar; not an object-storage cost story - **22. RUM** — ✅ Built-in RUM (Real User Monitoring) · ❌ Not yet - **23. Pipelines** — ✅ Realtime / Scheduled: transform / enrich / filter / route after ingest (VRL); logs→metrics, etc. · ❌ Not yet - **24. Reports** — ✅ Scheduled / Cached reports; timed generate & distribute · ❌ Not yet

Shared base: service list & golden metrics, Trace list/waterfall, logs, and Span↔log links. DataBuff leads on topology / service·instance·endpoint call analysis / service flow / middleware pages. OpenObserve leads on log search & cost, SQL/PromQL, custom dashboards, Pipelines, Reports, unified L/M/T + RUM.

Alerting

- **Capability** — OpenObserve v0.91.0-rc1 · DataBuff v0.1.4 - **How rules are configured** — ✅ Alerts UI (needs Destination / Template first) · ✅ Alert center in product - **Threshold alerts** — ✅ Scheduled / Realtime · ✅ Managed in platform - **Smart alerts** — ❌ No equivalent smart-alert product · ✅ Smart alerts linked with APM metrics - **Alert event list** — ✅ Alerts UI for triggered alerts / rules · ✅ Alert list (severity / service / time) - **Alerts linked to service / middleware** — △ Stream-oriented alerts; APM context must be stitched manually · ✅ List links service / middleware back into APM

Both can configure alerts in UI; difference is smart alerts and alert → APM service context. OpenObserve alerts lean Logs/Metrics streams; DataBuff leans APM triage loop.

When to pick which

- **Scenario** — Better fit · Note - **Already on OTLP; want AI / APM depth first** — DataBuff · Point exporters at DataBuff; no need to migrate off OpenObserve first - **Need the seven AI capabilities** — DataBuff · OpenObserve has no equivalent AI platform - **Need MCP / Skill or custom digital experts** — DataBuff · AI platform is extensible; OO has no such layer - **Need global topology + health colors at a glance** — DataBuff · OO has no service dependency topology - **Need “who slowed the response” from entry service** — DataBuff · Service flow + contribution; OO has no equivalent page - **Need service / instance / endpoint call analysis → Trace** — DataBuff · Three-level call analysis all link to Trace; OO has no path - **Need instance golden metrics / instance topology** — DataBuff · OO has no equivalent instance pages - **Need slow SQL / cache / MQ / external service pages** — DataBuff · OO mostly Span fields; no dedicated pages - **Need dedicated error analysis** — DataBuff · OO requires manual ERROR filtering - **Need smart alerts tied to service / middleware** — DataBuff · OO alerts are stream-oriented; no smart-alert APM loop - **Huge log volume; need object-storage cost control** — OpenObserve · Compression / storage story is a strength - **Need SQL / PromQL Metrics + custom dashboards** — OpenObserve · DataBuff has no custom dashboards yet - **Need post-ingest transform / enrich / filter / route** — OpenObserve · Pipelines (Realtime / Scheduled + VRL) - **Need scheduled / cached reports** — OpenObserve · Reports (Scheduled / Cached) - **Need unified Logs + Metrics + Traces + RUM** — OpenObserve · DataBuff focuses on APM depth - **Only need the same Demo Trace waterfall** — Either · No need to migrate for branding

Boundary: Staying on OpenObserve is reasonable when deeply tied to its log pipelines / dashboards / Pipelines / Reports / cost story. DataBuff fits OTLP + seven AI + topology / call analysis / service flow / dedicated pages / smart alerts. Dashboards, Pipelines, Reports, and large-scale log cost are not peer capabilities yet.

2. Screenshot evidence (explains the tables)

Screenshots from the same lab (192.168.50.140). Captions map to capability rows. Focus on DataBuff AI / topology / dedicated pages / alerts, plus OpenObserve logs / Trace / Metrics / dashboard entry points.

Seven AI capabilities (no OpenObserve equivalent UI; DataBuff evidence)

DataBuff AI chat home and seven capability entries: See / Squad / Inspect / Diagnose / Repair / Predict / Answer

DataBuff ① See: real question on service-a checkout / service-b; AI reads telemetry

DataBuff ② Squad: digital expert / multi-agent entries (no OpenObserve equivalent)

Overview & data plane

OpenObserve Home: Streams≈38 · Events≈205K · Ingested 96MB → Compressed 10.5MB (matrix row 21)

OpenObserve Streams: unified Logs / Metrics / Traces data plane

Services & topology

DataBuff Service list + golden-metric charts (peers with OO Service Catalog)

DataBuff Global topology + health colors (mysql can show red); OpenObserve has no topology view

Service / endpoint call analysis + service flow (matrix rows 4 / 9 / 10)

OpenObserve can list Trace / Span, but has no service / instance / endpoint call analysis, and no service-flow contribution view. DataBuff goes from “who connects” to “who slows, then into Trace”.

DataBuff Service call analysis: service-a → service-b (drill to Trace)

DataBuff Endpoint call analysis for /demo/checkout: volume / latency / error rate

DataBuff Service flow: entry service-a → downstream response contribution; no OpenObserve equivalent

Trace

OpenObserve Traces: service-a · GET /demo/checkout · 240ms · 13 Spans

DataBuff Trace list: charts + table

OpenObserve Trace Waterfall: service-a → service-b · includes SQL / Dubbo Spans

DataBuff Call-order waterfall + Span attributes; can link to logs in-product

Log / Metric

OpenObserve Logs: histogram + event table; Trace / Span correlation (matrix rows 15–18)

DataBuff Log analysis: Log → Trace down to Span

OpenObserve Metrics: SQL / PromQL / Builder (matrix row 19; DataBuff has no public PromQL)

Dashboards / Pipelines / Reports (matrix rows 20 / 23 / 24; OpenObserve strengths)

OpenObserve Dashboards: create custom boards

OpenObserve Pipelines: Realtime / Scheduled data pipelines

OpenObserve Reports: Scheduled / Cached

DataBuff dedicated pages (matrix rows 11 / 12; no OO equivalents)

DataBuff Database page

DataBuff Cache page

DataBuff Message queue page

DataBuff External service page

DataBuff Endpoint analysis list

DataBuff Error analysis (stats + endpoint)

These pages are depth beyond “middleware Spans in Trace” — the APM differences most worth validating against OpenObserve.

Alerting

OpenObserve Alerts UI: configure and view rules / events

DataBuff Alert center: rules in-product; list links service context; non-empty in this lab

If this helped, give us a Star:

GitHub: https://github.com/databufflabs/databuff

Online Demo: https://demo.databuff.ai