DEV Community: databufflabs

5 Ops Tasks That Take Half an Hour — Ops Expert Finishes in 3 Minutes

databufflabs — Thu, 16 Jul 2026 01:45:06 +0000

On-call pain is rarely that a problem is “too hard.” It’s that you already know what to check — and still have to SSH in and type every command yourself. These five show up in almost every test environment:

OTel won’t connect / no data — SDK is configured, the platform is blank; you guess endpoint, port, or a dead process.
Container keeps restarting — you see Restarting, dare not poke randomly, and crawling logs inside the container is slow.
Inspect real Java JVM flags — startup args, container limits, and effective values disagree; you bounce between jinfo / jcmd.
Port in use, service won’t start — lsof / ss once, then make sure you don’t kill the wrong process.
CPU hotspots — flame-graph tooling slips your mind; sampling configs take half a day before a chart appears.

Solo, these jobs often take 15–40 minutes (find the host, log in, recall commands, reconcile the conclusion). With DataBuff (Star on GitHub →) Ops Expert, the same work usually lands in 1–3 minutes: open AI Platform → AI Chat, pick Ops Expert, describe the symptom in plain language.

All screenshots below are from a live test environment.

Scenario · DIY SSH (typical) · Ops Expert (measured)

OTel won’t connect — endpoint / port / process ~15–25 min → ~1–2 min
Container restart loop — log diving ~20–40 min → ~2–3 min to find & fix
JVM flags — jinfo / jcmd ~10–20 min → ~1 min
Port conflict — ss / lsof ~5–15 min → ~1 min
Flame graph — tooling + sample ~20–40 min → ~2–3 min

Walkthrough: three shots per scenario — your prompt → Ops Expert at work → the conclusion.

1. OpenTelemetry won’t connect / no data

Example prompt:

Test env OTel won’t connect and the platform has no data. Check whether 4317/4318 are reachable and tell me in one sentence if the endpoint is correct.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

2. Container keeps restarting

Example prompt:

ai-apm-demo keeps Restarting — help me get it healthy.

① User prompt:

② Ops Expert process:

③ Conclusion (memory limit 10MB → OOM 137 → raised to 512MB):

3. Inspect Java runtime flags

Example prompt:

Show me the effective JVM flags for the ai-apm-web Java process — especially heap and GC.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

4. Port in use — service won’t start

Example prompt:

Who is holding port 27403? Tell me the process and command — do not kill anything.

① User prompt:

② Ops Expert process:

③ Ops Expert conclusion:

5. Capture a flame graph

Example prompt:

Take a short CPU flame graph for the ai-apm-web Java service on the test host and point out the hotspots.

① User prompt:

② Ops Expert process:

③ Conclusion (sample result + hotspot readout):

Same loop every time: describe the symptom → watch the process → read the conclusion. The timing list above is the DIY SSH vs Ops Expert gap.

Open source · one-line install

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash

Found this useful? Star DataBuff on GitHub → https://github.com/databufflabs/databuff

Live demo: https://demo.databuff.ai

SkyWalking Meets AI: Keep Your Agent, Ask Why Checkout Is Slow

databufflabs — Tue, 14 Jul 2026 01:25:51 +0000

Many Java teams run Apache SkyWalking in production. The Agent often stays in place for years: bytecode instrumentation, clear Segments, JVM + traces on one path.

DataBuff v0.1.3 layers AI on top of that same collection. Keep the Agent. Point ingest at SkyWalking gRPC :11800 (OAP’s default). Ask in plain English why checkout is slow — get traceId + bottleneck span, then verify on a flame graph. Logs still jump to traces. Same Segments. Faster answers.

All screenshots below are from a live demo environment.

0. First, SkyWalking

Typical strengths:

Java Agent with minimal app changes
Segments that keep multi-service spans readable
Traces, JVM metrics, and logs on one reporting path
Mature OAP + UI, strong docs and community

SkyWalking is solid at getting telemetry in.

Where on-call time goes is turning that data into a conclusion: alerts without traceId, hunting slow traces, summarizing span trees for Slack, hopping between topology, metrics, and logs. That is not a SkyWalking flaw — it is the next step: faster conclusions from the same telemetry.

Note: This article is about AI-assisted interpretation on SkyWalking ingestion — same Agent, same Segment data — focused on whether one pipeline can produce answers faster.

1. Highlight: slow checkout — ask once, get an answer

Demo: GET /demo/checkout P99 ≈ 240ms. Alerts often arrive without a traceId. With DataBuff, triage shifts from scrolling trace lists to asking AI.

① Ask directly (no traceId): “Why is service-a’s checkout endpoint slow lately?”

② AI breaks down the trace: typical slow trace, segment timings, bottleneck span — no manual row-by-row comparison.

③ Structured conclusion: actionable incident text, not just “maybe the DB is slow.”

④ One-click verify: jump to the flame graph with the returned traceId and confirm end-to-end spans.

What you gain: from “filter list → read span tree → write incident note” to ask → get traceId → verify in UI. Same Segment source; added AI readout.

2. Go deeper: topology, service flow, JVM

After AI answers, the same UI keeps going — no tool hopping.

3. Before vs after: how the path changes

For the checkout scenario:

Step	SkyWalking UI only	With DataBuff
1 · Find service	General → Service → pick service-a	Topology / service list → service-a
2 · Find slow trace	Trace page filter checkout → open rows one by one	Ask AI → `traceId` + bottleneck span returned
3 · See bottleneck	Read span tree manually for DB / RPC time	AI summary + flame graph jump
4 · Conclusion	Human writes “maybe DB query slow”	AI remediation hints; ops expert can SSH-check JVM
5 · Logs	Separate log system, match `traceId`	Log list “Trace · View” → call chain

Your SkyWalking Agent keeps reporting; you add AI query, unified UI, and log deep-links — same collection target.

4. How to connect: keep the Agent, point backend to DataBuff

v0.1.3 ingests native SkyWalking gRPC on 11800 (OAP default). No Agent jar swap — change the collector address:

# agent.config
agent.service_name=${SW_AGENT_NAME:your-service}
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:your-databuff-host:11800}

Two onboarding paths, both AI-capable:

Mode	Setup	Best for
Side-by-side trial	Keep SkyWalking OAP; read SkyWalking data via MCP	Cannot move Agents yet — try AI Q&A first
Native ingest	Point Agent to DataBuff `:11800`, Segments direct to ingest	Switch backend; traces / JVM / logs unified in DataBuff

5. FAQ

Do I replace the Agent?

No. Existing SkyWalking Java Agents work — usually only collector.backend_service changes.

Must OAP go away immediately?

No. Keep OAP for a trial; when you commit to DataBuff, point Agents over in batches and verify checkout-style Q&A works.

Can OpenTelemetry coexist?

Yes. Java on SkyWalking (11800), Go/Python on OTLP (4317), one DataBuff UI.

What does AI actually do?

Demo covers on-call staples: slow endpoint → traceId + bottleneck span, log line → trace, JVM curves for follow-up. You do not need to be a trace expert to get direction.

Who is this for?

SkyWalking in production, want AI Q&A on existing trace data
Java-heavy estates not ready to re-instrument with OpenTelemetry
Want traces, logs, and metrics in one troubleshooting flow

Suggested path: test env or one instance → change Agent address → hit checkout-like traffic → ask AI “why slow lately” → confirm traceId + flame graph → expand rollout.

SkyWalking ingestion + AI interpretation · Keep your Agent · Ask once for answers · Logs link to traces

Star DataBuff on GitHub →

AI Ops Squad Evolves: Ask ERROR Logs in Plain English

databufflabs — Tue, 07 Jul 2026 05:52:53 +0000

You can ask DataBuff about slow traces in plain English. That shipped in our last walkthrough.

Now ERROR logs join the party.

Same checkout demo, same InsufficientStockException on service-b — but this time we follow three real on-call paths: facet search in the Logs UI, trace-to-log deep links, and one-sentence AI queries that call log.queryLog* tools against live OTLP data.

All screenshots below are from a single incident window on a live demo environment.

Demo scenario: inventory checkout failure

The demo app hammers GET /demo/checkout. When stock runs out, service-b throws:

InsufficientStockException: inventory unavailable for skuId=…

OTLP logs land in Doris with trace_id and span_id attached. We walk the same failure three ways — the way a real shift would:

Path A — You know the exception name; search the global Logs page
Path B — You have a slow trace; read span logs and deep-link back
Path C — You delegate to the AI squad in one sentence

Path A: global log search (no LogQL required)

Menu: Application Performance → Log Analysis

No query language required — keyword + facets is enough.

Steps: search InsufficientStockException → filter ERROR + service-b → 95 matching lines with an ERROR spike in the histogram.

Each row has Trace · View on the right — jump straight to the call chain without copying IDs.

Path B: trace ↔ log deep links

B1 · Trace header: one click to logs

Open a slow GET /demo/checkout trace (~240 ms). The trace header shows TraceID; Log Analysis on the right pre-fills traceId — no copy-paste.

B2 · Span sidebar: flame graph + Logs tab

Spans marked Logs on the flame tree open a sidebar Logs tab: a timeline from Received checkout request through Delegating inventory check to service-b. Select the service-b span to see the ERROR stack.

B3 · Deep link: "View all in Log Analysis"

Click View all in Log Analysis at the bottom of the sidebar. The global page auto-fills traceId + spanId and shows only the four logs in that span's context.

Path C: ask the AI squad about logs

The UI is for precision. The AI is for one-sentence delegation.

The Smart Query expert registers a log tool family — visible under Tool Management:

Registered tools:

log.queryLogDetail — search by service, level, keyword
log.queryLogsByTraceId — all logs on a trace
log.queryLogsBySpanId — logs for one span
log.queryLogTrend — ERROR volume over time

Scenario 1: find ERROR logs by service + keyword

Find ERROR logs for service-b in the last hour related to
InsufficientStockException. List traceIds and key log summaries.

Scenario 2: known traceId → root cause

Given traceId edfa44615dcee4d6bdfeed46d84bfb20, list all ERROR-level
logs on this trace and explain why checkout failed.

The agent walks queryLogsByTraceId → 13-span call chain → ERROR lines → checkout failed due to insufficient inventory.

Scenario 3: ERROR volume trend

How has ERROR log volume for service-b looked in the last hour?
Any obvious spike periods?

Result: steady ~2 ERROR logs per minute — no spike, just a sustained inventory shortage.

Tool selection cheat sheet

Search by service/level/keyword → queryLogDetail
Known traceId → queryLogsByTraceId
One span's context → queryLogsBySpanId
Volume spikes → queryLogTrend

Where the data comes from

OTLP Logs (:4317 / :4318) → Ingest → Doris log_dc_record → POST /log/search.

Inject traceId via Java MDC and logs correlate automatically.

DataBuff is log exploration in an APM context — not a replacement for ELK or Loki. The win is sharing context with traces, metrics, and AI agents without hopping three systems.

Try it (5 minutes)

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash
curl -fsSL https://databuff.ai/databuff/ai-apm-demo-install.sh | bash

Open http://YOUR_HOST:27403 — login admin / Databuff@123 — add an LLM key under Settings → AI model.

Then try Path A, B, or C on the built-in checkout demo.

GitHub: https://github.com/databufflabs/databuff

OpenTelemetry Logs · Trace correlation · AI-native observability · Built in public

Show DEV: AI Ops Squad Evolves — Ask ERROR Logs in Plain English

databufflabs — Tue, 07 Jul 2026 05:51:10 +0000

You can already ask AI about traces. What about ERROR logs?

In our launch post we showed DataBuff — open-source, OpenTelemetry-native APM with an AI agent squad — answering one question across metrics, traces, and topology.

This follow-up uses the same checkout inventory failure (InsufficientStockException on service-b) and walks three real paths: search logs in the UI, jump between Trace and logs, and delegate ERROR log queries to AI.

All screenshots come from one incident window on a live demo environment.

Demo scenario: InsufficientStockException

The demo hammers GET /demo/checkout. service-b throws:

InsufficientStockException: inventory unavailable for skuId=…

OTLP logs land in Apache Doris with trace_id / span_id. Three investigation paths:

Path A — You know the exception name → global log search with facets
Path B — You have a slow trace → span logs + deep links back to global search
Path C — You ask in plain English → AI dispatches log.queryLog* tools

Path A: Global log search

Menu: Application Performance → Log Analysis. No LogQL required — keyword + facets are enough.

Search InsufficientStockException → filter ERROR + service-b → each row has Trace · View to jump to the call chain.

Path B: Span logs inside Trace + bidirectional deep links

B1 · Trace header: one click to log analysis

Open a slow checkout trace. The trace header shows TraceID; Log Analysis on the right pre-fills traceId on the global page.

B2 · Span sidebar: flame graph + Logs tab

Spans marked Logs on the flame tree open a sidebar Logs tab. Select the service-b span to see the ERROR stack.

B3 · Deep link: "View all in log analysis"

Click at the bottom of the sidebar — the global page auto-fills traceId + spanId and shows only logs in that span context.

Path C: Ask the AI platform about logs

The UI is for precision. AI is for one-sentence delegation. The smart-query expert registers a log tool family:

Scenario 1: Search ERROR logs by service + keyword

Find ERROR logs for service-b in the last hour related to InsufficientStockException.
List traceIds and key log summaries.

Scenario 2: Known traceId → root cause

Given traceId edfa44615dcee4d6bdfeed46d84bfb20, list all ERROR logs on this trace
and explain why checkout failed.

Scenario 3: ERROR log volume spike

How did ERROR log volume for service-b trend over the last hour? Any obvious spikes?

Tool routing: service/level search → queryLogDetail; known traceId → queryLogsByTraceId; specific span → queryLogsBySpanId; volume spikes → queryLogTrend.

Where the data comes from · vs ELK

OTLP Logs (:4317 / :4318) → Ingest → Doris log_dc_record → POST /log/search. Inject traceId via MDC on the Java side and correlation is automatic.

DataBuff is log exploration in an APM context — not a replacement for ELK/Loki. The win is same context as traces, metrics, and AI without tab-hopping across three systems.

Try it in 5 minutes

curl -fsSL https://databuff.ai/install.sh | bash

Web UI: http://localhost:27403 · default admin / Databuff@123

GitHub: https://github.com/databufflabs/databuff

Show DEV: Stop Tab-Hopping at 2am — Open-Source APM With an AI Agent Squad

databufflabs — Sat, 20 Jun 2026 13:15:51 +0000

2:14am. Checkout P99 is on fire. Slack wants a root cause in 10 minutes.

You open Grafana. Jaeger. Logs. Topology. Notes doc. Twenty minutes later you think you know — but you're still not sure if it's order-service, MySQL, or that sketchy downstream RPC.

Metrics didn't fail you. The workflow did.

We built DataBuff — open-source, OpenTelemetry-native APM — to fix that: one question in, evidence-backed root cause out. Not a chat box glued to dashboards. An AI Brain that dispatches specialists (metrics, traces, topology, inspection) and merges real telemetry into an incident-ready answer.

Try it first (5 minutes)

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash
curl -fsSL https://databuff.ai/databuff/ai-apm-demo-install.sh | bash

Open http://YOUR_HOST:27403 — login admin / Databuff@123 — add LLM key under Settings → AI model.

Paste this:

Which services had errors in the last hour? For the slowest one,
show me a typical trace, the slowest span, and what I should do first.

That's the product. Below: architecture, agent squad, and a real demo output.

Before vs after

Before (tab safari)

5–6 tools, zero shared context
You translate the question into 12 queries
Senior engineer stitches the story by hand
Slack gets guesses and screenshots

After (DataBuff)

One UI, unified Doris storage
You ask in plain English
AI Brain + agents query live metrics, traces, topology
Slack gets root cause, TraceId, P0/P1 actions

You already pay for observability. You're still paying in human attention at 2am.

AI-native ≠ ChatGPT on Grafana

Most "AI observability" in 2024:

Only sees what you paste
Cannot query your trace store
Guesses under incident pressure

DataBuff agents call real tools. Every claim should trace to evidence.

The agent squad

AI Brain — plans, dispatches, synthesizes
Smart query — P99, error rate, QPS from Doris
Trace analyst — slow traces, hottest spans
Topology — upstream/downstream blast radius
Inspection — sustained pain vs one-off spikes
Report — incident summary you can forward

Apache 2.0 · self-hosted · data stays on your network.

Architecture: 3 components, not 13

Data flow: OTLP apps → Ingest (4317/4318) → Apache Doris → Web UI (:27403) + AI Brain

Legacy stack vs DataBuff

Components: 10+ → 3
RAM: 16 GB+ → ~8 GB
First deploy: 1–2 days → ~5 min (one install script)
Storage: siloed → unified Doris
AI: bolt-on chat → multi-agent native

OpenTelemetry in. Unified storage. Agents on top.

Product at a glance

Demo walkthrough

After demo install you get service-a → service-b with real traces.

1. Service health

2. Ask the hard question

Analyze why service-a calls slowed in the last 30 minutes.
Find highest-P99 endpoint, a typical slow trace, slowest span,
root cause (app vs DB vs downstream), impact, and P0 fixes.

3. Watch agents work in parallel

4. Real demo output

Smart query
  service-a GET /demo/checkout ~240ms avg (17 reqs)
  service-b HTTP 100ms + Dubbo 80ms ~ 75% of latency

Inspection
  Sustained slowness, not a spike; service-a JVM/errors normal

Trace 12e3a078bdbe183d567a2f7e888fe7b3
  Slowest span: service-b -> MySQL SELECT demo_order (~45ms)

Root cause
  Downstream service-b + slow SQL (not service-a)

P0: dedupe service-b double calls; fix MySQL slow queries
P1: fix InsufficientStockException on inventory path

5. Topology proof

6. Call graph

vs SigNoz / Datadog

DataBuff wins on: multi-agent AI RCA built-in, OTel-native, self-host in minutes, you own the data.

SigNoz wins on: mature classic OSS APM, huge community.

Datadog wins on: SaaS polish, enterprise integrations.

We're betting the next moat is orchestrated agents on OTel data — for teams without a 24/7 SRE bench.

Who should try this?

Running OTel but still living in 5 tabs during incidents
Self-hosting (finance, gov, privacy-sensitive)
Want agents that use tools, not vibes
Apache 2.0 you can audit

Your turn

curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash
curl -fsSL https://databuff.ai/databuff/ai-apm-demo-install.sh | bash

GitHub: https://github.com/databufflabs/databuff

What would you ask the agent squad first? (comment below)
What OTel signals are we missing?
Star the repo if 2am tab-hopping should die.