DEV Community: databufflabs

A Different Way to Handle Everyday Operations: A Preview of OpenOcta

databufflabs — Mon, 14 Sep 2026 11:35:07 +0000

Let's start with four familiar operations headaches, then look at four ways OpenOcta addresses them.

1. Four everyday problems

Problem 1/4: Investigating one issue means switching between several systems

Finding out why a service has slowed down often starts with opening several different systems.

First, you check the CMDB to find the hosts running the service and the team responsible for it. Then you open Prometheus or Zabbix to check metrics and host status.

Next comes APM to find the slow part of a request, Elasticsearch to search logs from the same period, and your change management system to check for recent releases.

Each switch means finding the service again, selecting a time range, and setting filters. One system expects a service name; another needs a host IP.

You still have to bring the results together yourself. When a new clue appears, you return to an earlier system and keep searching.

Problem illustration: information about the same issue is spread across several systems.

What if I could ask in one place, have it query those systems, and continue the investigation using the results?

Problem 2/4: Alerts arrive everywhere, and you have to identify duplicates

Prometheus sends an alert. Zabbix reports something too. You open both pages and compare the service, host, and time to work out whether they are related. Before you finish, the same abnormal condition triggers another notification.

Without a shared view, you first have to work out which notifications represent new problems and which are repeats. Once you find the alert that needs attention, there are still metrics to check, logs to search, and possible remedies to investigate.

Problem illustration: several systems raise alerts; someone must identify duplicates before investigating.

What if I could see these alerts together, group repeated notifications, and continue straight into investigation and handling?

Problem 3/4: Daily, weekly, and monthly work still involves repetitive manual steps

Even when no alert interrupts you, routine work keeps coming.

Daily health checks mean reviewing the status and metrics of core services one by one. If something looks wrong, you gather more information. Then you write up the results.

Scripts can do part of the work, but someone often still has to decide what to run first and what to check next.

What if I could describe the services and checks, have the steps run, investigate abnormalities, and receive a report?

Weekly reviews of unresolved issues raise another set of questions: which alerts from last week remain unresolved, and which problems have returned? With records spread across systems, you have to compare them by business service and time.

What if last week's records could give me an initial list of unresolved and recurring issues, instead of making me find everything again?

Monthly operational summaries require collecting alert, health-check, and handling records, then calculating statistics and reviewing trends for the services you own. The report format barely changes, but next month you repeat the collection and organization with a new set of data.

What if I could specify the services and period, and have that summary prepared for me?

Problem illustration: check status each day, follow up on issues each week, and compile a summary each month.

Problem 4/4: A fixed dashboard may not fit the work you are doing

While on call, you want to know whether your services are healthy, which alerts need attention, and what recent checks found.

When preparing a change, you care about the affected systems and their check results. During a monthly review, you need trends over a longer period. You are the same person, but the information you need changes with the job.

A fixed dashboard often meets only part of that need. You filter and combine the remaining information yourself. Adding another service or changing a calculation may require another request for page development.

Problem illustration: when the work changes, the view needs to change too.

What if I could describe the services I own and the work I need to do, generate a suitable operations view, and keep refining it through conversation?

2. Four corresponding approaches

Approach 1/4: Connect systems and investigate one question across them

Your CMDB, Prometheus, Zabbix, APM, Elasticsearch, and your team's change management system can connect to OpenOcta through the interfaces and tools each provides.

OpenOcta supports MCP, API, and CLI integrations, allowing existing query capabilities to be used together.

Once connections, access permissions, and tool instructions are configured, you can ask:

Find out why the payment service has slowed down in the last half hour, and check whether there were any releases during that period.

Based on the question, the AI queries the CMDB for related resources, monitoring and APM for abnormal metrics and slow calls, then combines those findings with logs and recent change records to investigate possible causes.

You continue asking questions in the same conversation. It calls the systems it needs and uses the returned data to continue the investigation.

Approach illustration: connected systems provide inventory, monitoring, tracing, logs, and change information for the same question.

This can reduce copying service names, repeatedly selecting time ranges, and moving query results between systems. The same integrations can also support workflows and operations views.

Approach 2/4: Bring alerts together, group them by rules, and connect handling workflows

Alerts from systems such as Prometheus and Zabbix appear in a shared alert dashboard, with their sources, affected objects, and status visible together.

Repeated alerts of the same kind for the same object can be grouped using configured labels and time windows. Original events remain available when you need to inspect individual records.

Approach illustration: view alerts together, group them by rules, then use an associated workflow to investigate and handle them.

You can then associate a handling workflow with that type of alert: query the connected monitoring, APM, and log systems, have the AI analyze the findings, and proceed through predefined handling steps.

The workflow runs with the alert's context. When another alert of that type arrives, investigation can follow the existing process.

Your configuration determines which steps run automatically and which require human confirmation. You can also see how far the process has progressed and what it found.

Approach 3/4: Generate workflows to carry out repetitive steps

Workflows can also handle daily checks, weekly reviews, and monthly summaries. Each has its own process, which can be generated from your requirements.

For example, describe a health check:

Find the services I own in the CMDB, check their operating status, query related metrics and logs if anything is abnormal, and compile a health-check report.

The AI generates a workflow from that request. You can inspect each step, adjust the scope and handling steps, and give it a trial run.

Teaser illustration: describe the work, generate a workflow, then inspect and adjust its steps.

Once the necessary systems are connected and you have confirmed the process, the workflow can use tools to query, check, analyze, and summarize.

Steps that previously required someone to pass one result to the next can run in sequence. Human confirmation can remain wherever your judgment is needed.

A weekly review can query alerts and handling records for a specified period and summarize unresolved issues. A monthly summary can collect data by business service, calculate statistics, and produce a report.

Save the workflow, and you can run the same steps again with a different time range.

Approach 4/4: Generate operations views around your work

Now consider the fixed dashboard that does not quite fit. You can describe the view you need:

Create an on-call view for the payment service, bringing together service health, priority alerts, recent changes, and the health-check workflow.

The AI generates a page from that request, drawing data from connected systems. Inventory, monitoring, alerts, and change information can appear together around the business service you care about.

Teaser illustration: generate an operations view for the work you do each day.

The generated view supports further queries and changes to the time range. You can ask for another trend or replace information you rarely use.

For monthly reviews, you can generate a separate view focused on trends and statistics. Choose the view that suits the work you are doing.

3. OpenOcta launch preview

This is a first look at OpenOcta, ahead of its launch. These are a few glimpses; fuller demonstrations of the actual workflows will follow.

OpenOcta launches on September 15, 2026, at 7:00 p.m. China Standard Time (UTC+8).

Where Does an Ops Topology Actually Come From?

databufflabs — Tue, 08 Sep 2026 02:00:43 +0000

A checkout hits which services? OpenTelemetry records the app handling a request, calling downstream, and talking to the database. One full request is a Trace. Each hop inside it is a Span.

Look at three checkouts. The entry is A (service-a). It sometimes calls B (service-b) and also queries MySQL directly. B uses the same database. The numbers below are simplified. All three requests succeed.

Trace ① · A entry 100 ms

B · 80 ms
- MySQL · 40 ms
MySQL · 10 ms

Trace ② · A entry 200 ms

B · 120 ms
- MySQL · 60 ms
MySQL · 20 ms

Trace ③ · A entry 100 ms

MySQL · 10 ms

This one never called B. Indentation is who called whom: ① and ② each hit the database twice; ③ once. Internal work is omitted. IDs and numbers are teaching examples.

These calls wait for the downstream reply. If B spent 80 ms, that already includes 40 ms waiting on MySQL — do not add them to 120 ms. A's entry time is the whole wait from request in to response out.

One trace shows one request. When traffic grows, you also want: who depends on whom across the system? From A, where does the time go? Reading traces one by one gets painful.

DataBuff is an AI-native backend for OpenTelemetry: it receives what apps report, serves query pages, and lets AI experts query the same data through tools. The project is open source: https://github.com/databufflabs/databuff. Follow these three traces to see how it builds a global topology and a service flow.

What you get	What it keeps
Global topology	Merge calls into one graph: who depends on whom.
Service flow	Keep the entry and the paths: who A went through, and each path's share of time.

Architecture first: who processes, who stores, who queries.

Collector is an optional forwarder. OTLP is the OpenTelemetry export protocol. Both graphs are computed on the backend by fixed rules; the UI draws them. AI experts can query the same data.

Ingest first groups fragments of the same request.

Records from A and B may arrive in batches. Every request has a trace_id. Ingest groups fragments by that id. Each fragment also has its own id and a parent id parent_span_id: this hop started during that parent's work.

Trace ①: same trace_id, joined by parent

s1 · A receives the checkout · 100 ms (no parent: this is the entry)
- s2 · A calls B · 80 ms (parent = s1)
- s3 · B receives the request · 80 ms (parent = s2 → s2 belongs to A → B's upstream is A)
  - s4 · B queries MySQL · 40 ms (parent = s3)
- s5 · A queries MySQL directly · 10 ms (parent = s1 → this query belongs to A)

s1–s5 are short labels. s2 is A waiting on B; s3 is B handling the request. In the example both are 80 ms. Real ends can differ; it is still one A→B call.

When Ingest sees s3's parent is s2, and s2 belongs to A, it fills B's upstream as A and A's downstream as B. s1 and s2 both belong to A, so it does not draw A→A. Ingest can wait for late fragments, but it must continue on timeout — it cannot always wait until every piece arrives.

Then it turns database spans into database nodes.

s4 is recorded by the probe on B. It carries the database type and name. Ingest can name the component and remember B's edge to it.

B's DB span s4: db.system = mysql, db.name = demo_apm
Component node: [mysql]demo_apm
Edge: B → MySQL, 40 ms, success

s5 fills A → the same MySQL. Later we just say MySQL. Whether the fields are enough depends on what was collected.

Redis and message queues use their own component fields. You do not need an agent on every middleware to draw the edge. That is not the same as having all of their internal metrics.

Only then are many calls rolled into stats.

Count the right records. How many times A was entered: A's entry spans. How many times B queried the DB: B's DB spans. Trace ① lists five spans; A still received the checkout once.

Pick records by use: A's request count from s1; B's DB count from s4
Extract numbers: call count +1, error count +1 on errors, duration = end − start
Roll up by minute: add like with like, keep counts and total duration

That is the second kind of aggregation: statistics over many calls. Grouping by trace id was the first: putting one request's fragments together.

Assume the three requests finished in the same minute, and A's call to B matches B's receive time. The rollup looks like this:

After the three example requests	Count	Total duration	Average
A entry	3	400 ms	~133 ms
A → B	2	200 ms	100 ms
B → MySQL	2	100 ms	50 ms
A → MySQL directly	3	40 ms	~13 ms

Average duration is total duration ÷ call count. Error rate is errors ÷ call count. In the example every error rate is 0%. Across several minutes, add counts and durations first — do not average the averages.

Service flow also keeps the entry and the path. For A→B→DB the parent path is A→B; for A→DB the parent is A. Store those two DB paths separately so the tree knows where each hangs.

Data lands in Doris; the page queries by time.

What is stored is detail and stats, not a pre-drawn picture:

Data and example tables	What it stores	Where it is used
Trace detail `trace_dc_span`	Spans with relations filled in	Open one request
Service stats `metric_service`	Request counts, duration, …	Service metrics; services with traffic but no edges still show
Pair stats `metric_service_http`, `metric_service_db`, …	Minute stats for A→B, A→DB, B→DB	Topology edges and numbers
Entry-path stats `metric_service_flow`	Entry, parent/child path, counts and duration	Build the service-flow tree

The UI sends a time range; service flow also picks an entry
Web queries Doris: merge stats in the window, assemble nodes and edges
The UI lays out nodes, draws lines, shows numbers and status

Each graph queries its own stats. Opening the page does not scan raw spans.

Same data, why two graphs?

Global topology: who depends on whom

The backend merges call stats for the selected window. One identity is one node. Arrows go from caller to callee. A node is a service or component, not a machine.

When both A and B counted A→B, the current implementation prefers A's side so the two copies are not added. Edges carry count, error rate, and average duration. Node alerts are queried and joined separately.

Three example requests become three edges. Both callers connect to one MySQL node. This graph summarizes many requests; it does not keep each request's full path.

DataBuff global topology (demo data)

service-a and service-b both connect to [mysql]demo_apm. Red comes from node alerts in that window — not automatically the root cause.

Service flow: which paths from the entry

The backend loads path stats with A as entry: A→B hangs under A, A→B→DB hangs under B, A→DB hangs under A. Repeat calls on the same path merge. The two DB paths stay separate.

After the tree is built, each path node's total duration divided by the entry's total duration is response contribution. The UI uses that ratio to order branches. It is the time mix of many requests in the window, not the order of one checkout.

All numbers come from the three examples. MySQL appears on two paths. Vertical position is not execution order.

Response contribution = that path node's total duration ÷ entry total duration

B: 200 ÷ 400 = 50% | MySQL under B: 100 ÷ 400 = 25%

A's direct DB: 40 ÷ 400 = 10%

MySQL twice means two access paths, not two databases. 25% is vs entry A, not “25% of B”. B's 50% already includes waiting on that query; do not add 50% and 25%. Parallel calls can overlap in time.

The shares also need not add to 100%. The entry still spends time on compute and other waits. This graph does not split the whole response into non-overlapping slices.

DataBuff service flow (demo data)

The live page also splits the two DB paths. The screenshot is another batch (449 calls, 58% / 31% / 8%) for the UI only — not used in the example math.

Open-source APM products draw these relations differently. SkyWalking and SigNoz have dependency maps. Jaeger's deep dependency graph can show paths through a chosen service from search results.

DataBuff is more complete here: the open-source build ships both a global dependency view and a minute-aggregated service flow from an entry, with response contribution. One graph shows the system; the other narrows where to look.

After new calls are processed and written, refresh or change the time range and the graph is rebuilt from what was queried. Missing instrumentation, or async calls that did not propagate context, can drop an edge. Filters and display limits can too.

On-call, global topology shows dependencies; service flow shows paths worth checking. High contribution or a red node is a clue: still compare latency and errors to a healthy window, then confirm the slow hop on a trace.

DataBuff

AI-native OpenTelemetry APM. Metrics, traces, logs, and AI troubleshooting on one stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

There's Now an Open-Source Fault Localization Project in Ops

databufflabs — Thu, 03 Sep 2026 02:06:23 +0000

Ops rarely lacks metrics, topology, traces, or logs. What it lacks is lining those four up when something breaks.

Take an entry service that returns 200 with a 0% error rate. It looks fine, so the investigation often stops on that page. The real problem is downstream — SQL already threw — and this page never shows it.

Classic monitoring · four pages, each does one job, a human joins them

Metric — who is slow, who is erroring
Topology — which upstream/downstream node went red
Trace — which hop ate the time
Log — search the original line by call ID

You line up the time window yourself, copy the call ID yourself, search again on the next page — the four pages only talk through you.

Open-source DataBuff · one chat runs the same chain

Metrics pick the target
Topology looks downstream
Trace splits that hop
Logs match the original line
→ inspection report

It’s called DataBuff. Repo: https://github.com/databufflabs/databuff. Below is that chat on the official demo, watching it line the four pages up.

1 · Find the service with the highest response time

In the official demo, service-a / service-b are hitting GET /demo/checkout. The chat starts with:

找出最近 1 小时平均响应时间最高的服务，对它进行故障定位

Find the service with the highest average response time in the last hour, then localize the fault on it.

Step 1 · The AI brain hands the job to metric query

It splits the sentence into two steps: find the slowest service in the last hour, then localize the fault on it.
The first cut goes to metric query, time window pinned to the last hour, pull the service list first.

Step 1 · The brain sets the time window, then metric query pulls the service list.

Step 2 · Metric query names service-a

On the table, service-a averages 240ms, the highest in this batch.
service-b is 70ms; service-a is about 3.4× that. Fault localization stays on service-a.
Both Java entries show 0% error rate — looks fine at a glance. MySQL error rate is 33.33%, which is high, but it isn’t the slowest, so it isn’t the main target yet.

Step 2 · Metric query names service-a; the rest of the localization stays on it.

2 · Inspect its own health, then downstream and the call chain

Step 3 · Hand fault localization to the inspection expert

service-a itself: entry error rate 0%, no ERROR, no failed traces, 240ms is a flat line, not a sudden blast.
The red dot is downstream [mysql]demo_apm, with 60 “entry error rate too high” alerts.
Itself is green, the database keeps ringing, the two pages don’t match.

Step 3 · Itself is still green; downstream MySQL keeps alerting.

Step 4 · Read traces, then check the alerts

Sampled traces are all GET /demo/checkout, status 200, duration pinned at 240ms. The entry hop shows nothing wrong.
The database still has 60 alerts, but dependency stats write MySQL errors as 0.
The two sentences don’t match. The only move left is to split one chain.

Step 4 · checkout is all 200; the database keeps ringing; the two sentences don’t match.

3 · Split one call, match the logs, write the report

Step 5 · Split one trace, pull the same log line

The hop that blew up is SELECT demo_inventory, throwing InsufficientStockException.
The entry still returns 200. The same log line matches SKU DEMO-10001, inventory left at 2.
service-a recorded that service-b returned 200, so the entry stayed green and never went red.

Step 5 · Insufficient stock swallowed into a 200; the entry looks fine.

Step 6 · Write the inspection report

The chat ends with an HTML inspection report, not just “done.”
Verdict, trends, call tree, advice — the next section reads the report’s own chapters. This step doesn’t copy them first.

Step 6 · The report is out; the next section reads it by its own chapters.

4 · Open the report: verdict, trends, call tree

Start with the verdict. The report says “watch,” meaning this chain is worth looking at, not that the service is already down. 240ms is written as serial inherent latency, not a sudden spike.

You can preview this HTML report at the bottom of the chat.

Verdict “watch.” 240ms, 75%, 60 alerts, and the SKU sit on one screen.

Then the trends. All three charts are flat lines. Response time is pinned at 240ms; request rate is about 7–8/min, also flat; MySQL error rate 33.33%, also flat. This is not a one-off jitter — it was slow for the whole hour, and the alerts kept firing.

Left: 240ms flat. Right: request rate steady. Bottom: MySQL 33.33% flat.

Then the call tree. HTTP to service-b is about 100ms, the Dubbo hop about 80ms; together they are about 75% of the 240ms. The red dot lands on SELECT demo_inventory. Sum the hops and you get about 246ms, which matches the 240ms at the entry — nothing else is missing from the picture.

240ms is a serial sum; the red dot is the inventory SQL.

Then the advice. On latency, the bottleneck is two calls to service-b plus two slow SELECTs, 45ms and 30ms. On alerts, the cause is insufficient stock swallowed: SKU DEMO-10001, inventory 2. The entry returned 200 and hid it.

P0: look at the slow queries and inventory first. P1: consider parallel or merge. P2: don’t silently turn this into a 200.

Priority	What the report wrote
P0	Check why two SELECTs take 45ms / 30ms; whether inventory 2 is demo data or a real stockout
P1	Consider parallelizing or merging the two calls to service-b. This is advice; nothing was measured on how much faster it would be
P2	Don’t silently turn insufficient stock into a 200

5 · Same job: how long a human takes vs DataBuff’s AI analysis

The comparison is the same job: find the slowest service in the last hour, then localize the insufficient stock swallowed into a 200. Not two different incidents — one job, done twice.

The “about 3 minutes” on the right is wall-clock from this chat’s question to the report, measured. The “about 40 minutes” on the left is an estimate for a practiced human chaining the five items. There was no second stopwatch on a human run.

Item	What a human does	Practiced estimate	This AI chat
Metric	Open the service list, sort by latency, pick the slowest	~5 min	~30 sec
Topology	Open the relationship map, line downstream alerts to this service	~5 min	~30 sec
Call chain	Entry is already 200; open the waterfall and find the stuck hop	~10 min	~1 min
Logs	Copy the call ID, search the original line on another page	~8 min	~30 sec
Write the report	Turn what the four pages lined up into a conclusion	~12 min	~20 sec
Total	Chain the five, still lining up the time window	~40 min	~3 min

About 40 minutes vs about 3 minutes — roughly 13× faster.

6 · After the AI analysis, inventory, 200, and slow queries still need a human call

This chat took about 3 minutes. The slowest service is service-a, the red dot is on the database, insufficient stock was swallowed into a 200. Metrics, topology, traces, and logs lined up. Nobody had to copy a call ID across four pages.

Lined up is not closed. Whether to restock the 2 units, whether that 200 should keep swallowing this, whether those two SELECTs need a change — the report wrote advice. The call is still a human’s.

DataBuff

AI-native OpenTelemetry APM. Metrics, traces, logs, and AI troubleshooting on one stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

There's a Hidden Gem in Ops You Probably Missed

databufflabs — Mon, 31 Aug 2026 02:21:48 +0000

If you work in ops, you rarely lack a stack for monitoring, APM, logs, or alerts. When something breaks and you ask “who’s on fire,” you still stitch those stacks together yourself.

You can just ask what’s broken.

It’s called DataBuff, and it’s open source. Repo: https://github.com/databufflabs/databuff. Once data is in, dashboards, alerts, and questions share one stack. The UI goes down to endpoints and slow SQL.

This is not a how-to for one integration. It’s a walk through what you can use today.

Ingest: where the data comes from, which collectors, which backend.
Store: where it lives, how many components you keep.
See: how far the UI can drill.
Alert: how it fires, where you write the rule.
Analyze: whether you can ask instead of flipping charts.
Operate: what you do when the platform itself is stuck or dropping data.
Help: who answers when you don’t know how to use it or wire it up.

Start with how data gets in.

Ingest: language agents, eBPF, RUM, SkyWalking

Change language or collector, keep the same backend. Four paths into one Ingest.

Language agents cover Java, Python, Node, Go, .NET, PHP, Ruby. Point the official exporter at 4317 (gRPC) or 4318 (HTTP). The Nginx module uses the same OTLP path. Don’t want a process probe? Use eBPF — Grafana’s OBI. Browser OTel traffic is accepted too. If you still run SkyWalking, the report port is 11800. You don’t have to swap old probes right away.

Language agent (OpenTelemetry) Java / Python / Node / Go and more
eBPF (OpenTelemetry) OBI · no in-process probe
RUM (browser) page OTLP in
SkyWalking (legacy probes) port 11800

All four land in DataBuff Ingest: 4317 gRPC · 4318 HTTP · SkyWalking 11800 · traces / metrics / logs.

The install “data access” page already has the official start commands per language. Point the endpoint over.

Store: three containers, where others often assemble a kit

Data is in. Next question: where it lives, and how many pieces you maintain. DataBuff is Ingest to receive, Doris to store, Web to see and ask. No extra Kafka or Elasticsearch in the middle. Topology, golden metrics, alerts, and AI queries read the same store.

A common Grafana LGTM kit

Collect · Alloy / Agent
Metrics · Prometheus
Traces · Tempo
Logs · Loki
Dashboards · Grafana

DataBuff’s three core pieces

Ingest receives
Doris stores · one copy of the data
Web sees / alerts / AI

We’ve installed and used that Grafana kit on the same machine. The kit is fine. You just switch entries to look things up, and you keep more moving parts. With a small team that’s louder: one less stack is one less late night. 8G RAM is enough to stand it up on a single box.

See: the UI goes down to endpoints and SQL

Open global topology. Besides apps you also get bracket nodes — [mysql]…, [kafka]…. Most of those middlewares never got a probe; they come from outbound spans on the caller.

When the entry turns red, look downstream. The cause often isn’t the first red dot. From a service you can reach endpoints, a single trace, slow SQL. Trace detail has a flame graph. Logs line up on the same Trace ID.

Global topology. Red means unhealthy, not decoration.

Endpoint calls, then down into a trace.

One trace: call order and time.

Slow SQL on the database page.

Alerts: rules on the numbers you’re already looking at

You don’t bolt on another monitoring product. Golden metrics are already in the store. In the alert center you set a threshold or a spike. Rules look at recent minutes; over the line becomes an event, back to normal is recovery. Events hang on a service or middleware; you can jump back into APM. Related rules also show on the service page. No config-file hunt.

Rules live in the product. A lot of SkyWalking setups still edit OAP config files.

Analyze: ask, inspect, capacity calls

When you can’t flip another chart and only want to ask “who’s on fire,” open the chat. It queries the same data above. Ask which services showed up in the last hour, who talks to whom, how error rates moved — it looks it up and answers.

Ask it to inspect a service and it walks a checklist: entry, OOM / timeout in logs, alerts, downstream, whether instances are still there, and it can produce a report with evidence. Not the model guessing. Behind the chat it hands work to ask and inspect experts that query the store.

Ask: services, topology, trends.

Inspection report, not small talk.

The home page also has capacity. Ask it to look at historical QPS and latency, whether you need to scale next — it draws a chart and gives a call.

Capacity calls from historical QPS / latency.

You bring your own model key. Kimi, DeepSeek, OpenAI, Ollama are on the list.

Operate: when the platform itself is the problem

If ingest is stuck, writes are dropping, or disk is full, open deploy status. Processes and SSH on business hosts are a different expert. Not this page.

The overview is four cards: inbound, write failures, Doris disk, query failures. Click a chart title for a note: how the number is computed, when it counts as bad, which env var to touch. When writes start dropping, look at the queue and timeouts first. Don’t blame the app for not reporting.

Check whether the platform itself is stuck.

Click the title. The note sits beside it.

You can also tell the AI: inspect the DataBuff platform, find why data is dropping. It reads the platform’s own metrics. Say “fix” and it changes INGEST_*-style settings, restarts in-stack services, then checks again.

Inspect the platform, find drops; it only changes settings when you say fix.

Help: how to use it, how to connect, how to wire MCP

When you’re stuck, ask: how the product works, how to connect OpenTelemetry, how to set up MCP. The Q&A expert reads the install docs and access notes. It can walk OTLP endpoints, hanging a Java agent, then checking the service list.

Ask “how do I connect OpenTelemetry” and it gives the endpoint and the check steps.

To hang an external MCP on an expert: AI platform → tool admin → new MCP, fill the URL and SSE / Streamable HTTP, then tick that tool on the digital expert. Chat can call it after that.

New MCP: URL + transport.

The expert only calls it after you tick it.

The other way works too: an external agent like Cursor talks to DataBuff over POST /mcp. Point it at the URL and ask.

Put them on one table

Grafana’s kit, SkyWalking, and DataBuff. Differences show up when they sit side by side.

	Grafana kit	SkyWalking	DataBuff
Components you keep	Collect + Prom + Tempo + Loki + Grafana	OAP + store + UI	Ingest + Doris + Web
Single-box standup	Five pieces stacked, heavy	OAP cluster	Runs on 8G RAM
Looking up data	Metrics, traces, logs on different entries	OAP queries	One UI
Alert rules	Hand-written PromQL	Edit OAP config files	Point at this data in the product
AI troubleshooting	Another plugin	—	Ask, inspect, capacity, same data
When the platform itself breaks	Wire your own monitoring	Figure it out	Self-monitor + notes; it only “fixes” when you say so

On a small team, fewer stacks and fewer entry switches is something you feel.

Don’t turn everything on at once. Hang an agent on one non-core service, see it on topology, add one alert, then ask “how has this service been lately.” One command brings the stack up.

DataBuff

AI-native OpenTelemetry APM. Metrics, traces, logs, and AI troubleshooting on one stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

LangGraph: Orchestrate a Swarm of AI Agents with One Graph

databufflabs — Mon, 24 Aug 2026 02:18:12 +0000

A single AI assistant is already standard in many teams: ask a question, get an answer, like chatting with a seasoned hand. But the moment you want several AIs to work together — one checks metrics, one digs through logs, one draws the conclusion — it falls apart: who hands off to whom? Can they run in parallel? Should it stop and ask you midway? If it crashes halfway, is everything before it wasted?

LangGraph exists to answer those questions. It's LangChain's open-source multi-agent orchestration framework (MIT). But the essence first: it is, before anything else, a general-purpose graph execution engine — nodes, edges, conditional edges, parallelism, checkpointing, waiting on a human are all generic graph capabilities, with no inherent tie to AI; it just so happens that a node can hold an AI, and once it does, you're "orchestrating multiple AI agents." The core is simply: you draw the graph, it runs the graph.

This article uses a single case — a 2 a.m. alert: "order-service error rate spiking" — to explain it end to end: the graph first, then the code, then the execution model and context passing, and finally a comparison with Dify and Claude Code's dynamic workflow.

A case: 2 a.m., order-service error rate spikes

Your on-call phone buzzes. The alert says order-service error rate just hit 12%, and your AI assistant has to walk the whole investigation flow itself. This graph is the entire logic it will execute:

Fig 1 · A late-night investigation: one graph exercises nodes, edges, conditional edges, Send, interrupt, and checkpoint.

A single run goes roughly like this: the AI diagnoses first, finds it's serious, and fans out 12 parallel subtasks to check instances one by one; after checking, it summarizes "these 3 are bad"; then it stops and asks whether to restart — you approve, and only then does it execute the fix, finally re-checking and reporting.

This graph uses almost every core capability LangGraph has: nodes (each box), edges (arrows), conditional edges (serious?), Send (batch checks), interrupt (await your approval), checkpoint (persist each step). Let's draw it in code.

Drawing this graph in code

Build a graph, register each box as a node, connect arrows as edges:

from langgraph.graph import StateGraph, START, END

def diagnose(state):   # 1 diagnose: check error rate, pull Trace, return severity
    error_rate = query_error_rate(state["service"])
    return {"abnormal": error_rate > 5}

def fan_out(state):    # 3 batch check: Send dispatches 12 subtasks at once
    return [Send("check_host", {"host": h}) for h in state["hosts"]]

def check_host(state): # 4 check one instance (each Send is its own small task)
    return {"results": [f"{state['host']}: ok"]}

def ask_human(state):  # 5 await your approval
    decision = interrupt("3 bad instances found — auto-restart?")
    return {"decision": decision}

def fix(state):        # 6 execute the fix
    restart(state["hosts"])
    return {}

def report(state):     # 7 produce the report
    return {"report": "handled"}

g = StateGraph(dict)
g.add_node("diagnose", diagnose)          # register nodes
g.add_node("fan_out", fan_out)
g.add_node("check_host", check_host)
g.add_node("ask_human", ask_human)
g.add_node("fix", fix)
g.add_node("report", report)

g.add_edge(START, "diagnose")             # start -> diagnose
g.add_conditional_edges("diagnose", route)# 2 conditional edge: serious? -> fan_out or report
g.add_conditional_edges("fan_out", fan_out)  # 3 Send fan-out
g.add_edge("check_host", "ask_human")     # checked -> await approval
g.add_edge("ask_human", "fix")            # approved -> fix
g.add_edge("fix", "report")               # fixed -> report
g.add_edge("report", END)

graph = g.compile(checkpointer=InMemorySaver())   # compile + enable persistence

A few key points, one by one:

1 · A node is just a plain function. diagnose, check_host, fix are ordinary Python functions. Whether to call an LLM is entirely up to the function body — diagnose can call a model, check_host can be pure computation.

2 · A conditional edge picks the path. route returns "fan_out" or "report", and the graph walks to the matching node:

def route(state):
    return "fan_out" if state.get("abnormal") else "report"

3 · Send = dispatch in bulk. fan_out returns 12 Send("check_host", {...}), executed in parallel within the same superstep, results auto-merged:

def fan_out(state):
    return [Send("check_host", {"host": h}) for h in state["hosts"]]

Fig 2 · Send zoomed: dispatch N independent subtasks in one superstep, then merge.

4 · interrupt = stop and wait for a human. Call interrupt() inside a node and the graph halts, surfacing the question to you; after you approve and resume with an answer, the graph resumes from its checkpoint:

decision = interrupt("3 bad instances found — auto-restart?")
# after you approve:
graph.invoke(Command(resume="yes"),
             {"configurable": {"thread_id": "t1"}})

5 · checkpoint = persist every step. The graph stores progress after each step (the thread_id acts as a ticket id). If it crashes midway, re-invoke with the same thread_id and it resumes from the breakpoint — officially called durable execution.

How does LangGraph actually execute this graph?

Running the graph isn't a headlong dash from start to end; it moves forward round by round. The official term is superstep, and each round does four things:

Fig 3 · One superstep: figure out who runs -> run in parallel -> refresh the board -> persist.

Nodes don't pass messages to each other. Everyone faces the same whiteboard: this round only reads what's already on the board, and writes are set aside — only after the whole round finishes does the board refresh. The source comment is one line: what step N writes, step N+1 sees.

Fig 4 · How context passes: no talking between nodes, just read/write the same whiteboard.

Walk the Fig 1 on-call case through it, and the rhythm is:

Fig 5 · The on-call case unrolled by superstep: writes land on the board next round.

Chat history works the same way: put messages on the board, and the next step sees them.

Compared to other options — where's the difference?

Dify and Claude Code are both often called "multi-agent," but set beside LangGraph, one table each is enough.

Dify: visual platform vs code library. Dify is a web canvas where you drag nodes; LangGraph is a graph you draw in Python code.

Dimension	Dify	LangGraph
Form	Web canvas, drag nodes	Draw the graph in Python
Who it's for	Non-developers / quick prototypes	Developers / fine control
License	Apache 2.0 modified (commercial conditions apply)	MIT (commercial use OK)

Claude Code dynamic workflow: the AI writes a LangGraph on the spot. You just say what to do; in the background it auto-decomposes the task and spins up tens to hundreds of agents in parallel.

Dimension	Claude Code dynamic workflow	LangGraph
Who orchestrates	The AI itself: reads the request, decomposes dynamically	You: draw a fixed graph in code
Orchestration artifact	An agent tree in memory, gone after the run	A compilable, replayable graph definition
Predictability	Same input may produce different graphs	Graph fixed, behavior predictable

One line to remember: Dify hands you a ready-made car, Claude Code has the AI write the orchestration on the spot, LangGraph hands you the engine and the blueprint — the most freedom, and the most work.

Our open-source DataBuff — how do its agents collaborate?

Our open-source DataBuff (AI-native APM, GitHub: https://github.com/databufflabs/databuff) also does multi-agent collaboration. You talk to a single entry point; an AI brain dispatches the work to experts — query, inspection, ops, Q&A — in parallel, then assembles their findings into a conclusion with an evidence chain:

Fig 6 · You face one entry point; the complex collaboration happens behind it.

The core purpose

LangGraph's core purpose isn't "to hand you a ready-made multi-agent solution," but to provide a controllable runtime for agent flows that need to run long, hold state, persist, and loop in a human — it turns the low-level dirty work — parallelism, checkpointing, waiting, recovery — into primitives, so you only worry about the business flow itself.

Takeaway: copy those 6 functions and the graph structure from the case and run them locally, and you'll understand what multi-agent orchestration is about.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs, and AI troubleshooting in one.

GitHub: https://github.com/databufflabs/databuff

Live demo: https://demo.databuff.ai

Grafana's LGTM Stack — Tempo and Loki Are Starting to Feel Dated

databufflabs — Wed, 19 Aug 2026 02:45:11 +0000

Same slow checkout: four hops from metrics to Tempo to Loki. DataBuff drills down from the service list, then you can ask the AI and check the platform itself.

DataBuff is an AI-native APM. It ingests OpenTelemetry and SkyWalking, and ships seven AI capabilities. GitHub: https://github.com/databufflabs/databuff

Same slow checkout: Grafana needs three systems. DataBuff finishes the path in one.

What you check	Grafana: 3 systems / 3 separate pages	DataBuff: 1 system / 3 linked pages
Metric: who is slow	Grafana: Service Map	Service list
Trace: which hop	Tempo: trace list, waterfall	Tracing: drill down from the service page
Log: what happened then	Loki: query by traceId	Logs: open Trace and jump back

On-call, whether those three sit on one path matters more than whether you have all three pillars.

This LGTM: Metric / Trace / Log on three lines

The scene is service-a GET /demo/checkout getting slow. Find who is slow in Metric, then the request and span in Trace, then context in Log. In this stack the three entries are different datasources, with different filters.

Metric · ① Who is slow: metrics-generator → metrics store → Service Map

Tempo stores spans. To draw a service map, metrics-generator aggregates spans into time series, writes them to a separate metrics store, then Grafana can paint the Service Map.

Metric · Service Map (from metrics-generator, not Tempo itself)

Trace · ② Find the request: Explore → Tempo, search the list

Leave Service Map, switch Explore to the Tempo datasource, filter by service / operation for the slow checkout, and copy the traceId.

Trace · Explore → Tempo: different query box, different syntax

Trace · ③ Waterfall: where the time went

Open the trace. See which hop of service-a → service-b → service-c is slow. Still in Tempo. Logs not touched yet.

Trace · Tempo waterfall: find the slow span

Log · ④ Logs: Explore → Loki, paste the traceId

Log is another line: Explore → Loki, put the traceId into LogQL. If tracesToLogs is not wired, this hop is manual.

Log · Explore → Loki: LogQL, paste the traceId yourself

In this stack, Metric, Trace, and Log are separate datasources

This run: four hops, and logs still needed a hand-copied traceId.

DataBuff: same incident, drill down from the service list

Same slow checkout. After OTLP lands in DataBuff, Metric / Trace / Log are already joined. Stay in one product. You do not paste a traceId into a new query language to open logs.

Metric · ① App performance → service list

The service list shows service-a latency, errors, and traffic. Click the name. Metrics and traces share one store — no extra extract from spans.

Metric · service health: latency / errors / traffic

Trace · ② Tracing: open the slow waterfall

From the service page, open the 240ms checkout. Middleware spans sit on the same picture. No system switch.

Trace · waterfall + call chain

Log · ③ Logs: Trace on the row, one click back

Each log row has a Trace button. That is the on-call step you skip: copy the id, change the query.

Log · logs → trace, same product

Topology, service health, and waterfall share one dataset — no extra metrics-generator layer

By here the gap is no longer “can you see the data.” DataBuff still has ask-the-AI and platform health. This LGTM stack did not expose those two entries.

After that path: AI and self-monitoring

After services, traces, and logs, two more things sit on the same menu: ask the data in natural language, and see the platform's own health. Those were the biggest feel differences on this run.

AI · Q&A / inspect / docs — same data you just drilled

Open the AI platform. You do not pick Tempo or Loki first, and you do not write TraceQL / LogQL. The page already has ask-data, inspect, product Q&A, and an ops expert. Sample prompts are concrete: last-hour service list, service-b upstream/downstream, request and error trends.

You still bring your own model. This lab used DeepSeek. The difference is the experts and tools are already in the product, so you do not assemble three datasource contexts first.

AI platform · ask, inspect, Q&A, ops expert — same menu as Metric / Trace / Log

This Grafana 11.5 does not ship that as a chat entry. To ask the same way you add an LLM plugin, then wire Tempo, Loki, and the metrics store into the model context.

Self-monitoring · deploy status: treat the platform as a business system

Install → Deploy status. One page: ingest TPS, write failures, Doris disk, query failures; legend split by trace / metric / log. Inbound requests, bytes, latency, and outbound drops share one timeline.

Deploy status · ingest overview: 32.9/s in, Doris disk 49%; trace / metric / log on one page

At 2 a.m., “is the app slow, or is the platform stuck?”, this page knocks out a batch: are the three lines still rising, did writes fail, is Doris disk full. In this LGTM stack those checks still mean Tempo, Loki, the metrics store, and Grafana, each on its own page.

LGTM: stop when you can see the data. DataBuff: after that you can still ask, and still see the platform.

What you want	Grafana LGTM	DataBuff
Ask “who is slow, what does the topology look like”	Write the query yourself; or add an LLM plugin plus data context	AI chat: sample prompts are service list / topology / traffic
An inspect report you can forward	Stitch dashboards and alert rules	Inspect: the expert reads the same metrics and writes the report
Is the platform itself stuck	Open each component health page	One deploy-status page: ingest + Doris
Component bill	Tempo + metrics-generator + metrics store + Loki + Grafana	ingest + one engine + Web (AI and deploy status included)

How to try: OTLP dual-write, no rip-and-replace

The collector already speaks OTLP. Add a DataBuff exporter in Alloy / Collector, write to Tempo in parallel for a few days, confirm the same spans line up, then drop the Tempo path if you want. The whole thing rolls back.

Walk it once: service health → trace → logs → ask the AI for a service list or topology → deploy status, three ingest lines. LGTM can stay. Just see if this on-call path feels shorter.

DataBuff

AI-native APM. Ingests OpenTelemetry and SkyWalking. Seven AI capabilities on the same stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

DeepSeek Harness Hit 126k Stars — What Self-Evolution Means for Software (and Observability)

databufflabs — Mon, 17 Aug 2026 01:50:53 +0000

A repo on GitHub has been exploding lately. DeepSeek open-sourced an agent framework called deepseek-harness (often shortened to dsh). It went public on August 13; when I took this screenshot, it was already at 126k stars.

GitHub repo page, captured 2026-08-16. Top-right shows 126k stars; About lists 125.9k.

Repo: https://github.com/deepseek-ai/deepseek-harness

Its one-line pitch: Everything is a Plugin. You're not stuck with whatever shipped in the box. The AI can see what it already has, and when something is missing it can add it to itself. They call that self-evolution.

With stars climbing that fast, I wanted to see what it actually does. So I opened the web UI and typed one request.

I asked it to add a sidebar item

Roughly: build a plugin, inject it into yourself. Change the left menu — add a Workflow item above Workspace. Click it and show a complex flow chart; click a node and open a drawer on the right.

It actually did it. Bottom-left showed Cordis Plugin: 1 running. The three screenshots below are from the live page.

Red box is the original prompt. It inspected its own sidebar first — didn't jump straight to editing files.

The left **Workflow* item is live. This ticket canvas wasn't built-in — the plugin it just wrote drew it.*

Clicking the red **External API Query* node: timeout and retry 3/5 in the drawer.*

That extra Workflow entry wasn't in a release. It wrote a plugin on the spot and mounted it.

That made me think about on-call pages

The troubleshooting pages you open on shift are usually fixed at ship time. How topology switches, which metrics sit where, what the drawer shows first — often one team's playbook baked into the UI. Banks want ledger latency; e-commerce wants checkout funnels; one layout rarely fits both.

Changing that in traditional software is hard: tweak a page, wait for the next release. Low-code is better — drag and drop — but you're still picking from the vendor's blocks; step outside and you're stuck.

dsh is different. You say what you need; it adds a piece to itself right then. The block is fabricated on site, not chosen from a catalog.

Fig 1 · Traditional software, low-code, and dsh in one glance

Three capabilities — we've shipped two

We've been thinking about this for a while. Software that takes care of itself, in my view, comes down to three things: self-ops, self-Q&A, and self-evolution. DataBuff already has the first two in product.

Project on GitHub: https://github.com/databufflabs/databuff

Self-ops isn't just flashing a red chart. In our demo, logs were dropping — thousands to fifteen thousand per minute. A human said two things: let it investigate, then allow it on the box. It checked its own self-monitoring, found the write queue capped at 16 batches, bumped it to 32, restarted ingest. After that, write drops went to zero.

Self-ops · SSH to change params and restart ingest; write drops back to zero. It fixed itself.

Self-Q&A: installed but don't know where to click? You used to hunt external docs. In DataBuff, switch to product support and ask: how do I wire OpenTelemetry, where do I set alert thresholds? It answers with menu paths and fields from in-product docs.

Self-Q&A · Ask how to onboard OTel and configure alerts — get paths, not a link dump.

Both are visible the moment you open the product. Self-evolution we haven't built yet. dsh just demonstrated the third.

Fig 2 · Three capabilities — DataBuff has the first two

Self-evolution in observability

Observability isn't a chat window. Collection, storage, query — touch the wrong layer and your data is dirty. dsh can rewrite sidebars and canvases because almost everything lives in plugins. Observability can't copy that wholesale. The ingest/store/query pipeline must not be edited live.

What can grow on demand is the human-facing layer. "Show me UnionPay transactions first" — add a troubleshooting panel or drawer on the spot. The pipeline underneath stays the same.

Fig 3 · Observability: lock the base, extend the pages

Fig 4 · dsh can change the whole building; observability only the room users see

Same telemetry as before. Pages can be tailored per customer on the spot.

Beyond the hype — still those three

Most of dsh's star rush is probably about "AI adding features to itself."

Self-ops — when it breaks, it fixes itself. Not just a red dashboard.
Self-Q&A — don't know how to use it? Ask; get menu paths.
Self-evolution — not enough? Add a piece for your request on the spot.

We've shipped the first two in DataBuff. The third — for observability — we're not there yet.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too

databufflabs — Wed, 12 Aug 2026 01:42:13 +0000

The page is slow, APIs time out randomly, and you've already checked Java logs and slow SQL — still nothing. The gap is often the Nginx in front: it proxy-passes the request, but classic access logs won't tell you whether the hop succeeded, how long it took, or where it stalled.

OpenTelemetry turns that into "load a module, write a few lines of config." This post walks a Demo we actually ran: instrument Nginx and Java with OpenTelemetry, then open one Trace in DataBuff that spans Nginx → Java → Redis.

What OpenTelemetry is doing

OpenTelemetry stitches each service hop into one call chain. A full request is a Trace; each processing segment is a Span. Nginx uses the official module, Java uses the official agent — each records its hop, exports with the same TraceID, and the platform merges the fragments.

curl / browser
   │   GET /hello
   ▼
[ Nginx  nginx:1.27-alpine-otel ]   ngx_otel_module   → OTLP gRPC 4317
   │   reverse proxy proxy_pass
   ▼
[ Java   JDK HttpServer + Jedis ]   opentelemetry-javaagent → OTLP HTTP 4318
   │   set + get
   ▼
[ Redis  192.168.50.120:16379 ]     Jedis auto-instrumented
   ▼
[ DataBuff ]  ← gRPC 4317 + HTTP 4318, merged under one TraceID

Hop 1 · Nginx — two install paths

The official OTel module needs a recent Nginx (1.21+). Our host still runs 1.20.1, so path A uses the official image that already ships the module; path B installs the module package on an existing Nginx and adds one load_module line. OTel directives are the same either way.

Path A — official image:

docker run -d --name nginx-otel \
  -p 8090:80 \
  --add-host host.docker.internal:host-gateway \
  -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro \
  nginx:1.27-alpine-otel

Path B — Alpine example: apk add --repository https://nginx.org/packages/mainline/alpine/v3.21/main nginx-module-otel, then load_module /usr/lib/nginx/modules/ngx_otel_module.so; and nginx -s reload.

Config points that matter:

Export over gRPC host.docker.internal:4317 (HTTP 4318 fails with OTel export failure ... Socket closed)
otel_service_name nginx-otel-demo
otel_trace on

The line people miss: otel_trace_context propagate;. By default the module receives upstream context but does not inject it downstream. Without this line Nginx reports its own Span, but Java never sees the TraceID — you get two unrelated Traces. After adding propagate, they joined immediately.

otel_exporter {
    endpoint host.docker.internal:4317;   # gRPC
}
otel_service_name nginx-otel-demo;
otel_trace_context propagate;             # required: inject traceparent

Hop 2 · Java agent, zero code changes

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=java-redis-demo \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318 \
  -cp "libs/*:classes" OtelDemoServer

Java uses HTTP OTLP (4318), Nginx uses gRPC (4317) — two protocols, one DataBuff, one Trace later.

Hop 3 · Two exporters, one Trace

Both nginx-otel-demo and java-redis-demo show up with volume, error rate, and latency.

Topology draws nginx-otel-demo → java-redis-demo → Redis.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/epov100r6v9hxqllxlv9.png

Waterfall: /hello (nginx) → GET /hello (java) → AUTH/SET/GET (redis), total 975ms, TraceID 7dff87e865b7fe975877a506db26a2ab. The Demo sleeps about one second in Java on purpose — nginx Span (975ms) and Java Span (974.75ms) rise together, so Nginx duration really covers the whole request. Redis even records AUTH as its own Span when the connection uses a password.

Don't trust the UI alone — check Doris

SELECT service, name, span_id, parent_id, is_parent, duration
FROM trace_dc_span
WHERE trace_id='7dff87e865b7fe975877a506db26a2ab'
ORDER BY startTime;

nginx-otel-demo  /hello      (empty, root)  975000000
java-redis-demo  GET /hello  <nginx span>   974745285
[redis]…:16379   AUTH        <java span>     638593
[redis]…:16379   SET         <java span>     361212
[redis]…:16379   GET         <java span>     164941

Filter the trace list by nginx / java when you're hunting edge issues.

Try it tonight — three steps

docker run nginx:1.27-alpine-otel with otel_trace_context propagate;, expose http://<host>:8090/hello.
Start Java with -javaagent:opentelemetry-javaagent.jar pointing at http://127.0.0.1:4318.
curl http://192.168.50.140:8090/hello a few times, wait ~30s for batch export, open DataBuff Traces, filter nginx-otel-demo.

Self-check: nginx root Span present; Java under nginx (not a sibling Trace); Redis SET/GET under Java. If you see two Traces, you almost certainly forgot propagate.

Fifteen minutes on the Nginx module and the Java agent turns the first hop from a blind spot into a span you can scroll.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

What Does a Self-Caring Software Look Like? (DataBuff Self-Monitoring & Self-Troubleshooting)

databufflabs — Mon, 10 Aug 2026 01:22:18 +0000

When you're on call, the worst thing isn't that your business went down — it's that your monitoring platform itself broke and you had no idea. Log volume suddenly drops, a chunk of data silently disappears — and after hours of searching you realize it was never the business at fault: the collection pipeline choked on itself. To make it worse, traditional monitoring software is always watching "everything else," while being a black box itself: CPU spikes, memory full, queues blocked — it never says a single word.

First, what is DataBuff: an open-source AI-native APM where the AI lives directly on top of your OpenTelemetry data — asking questions and troubleshooting are done by AI experts calling tools against real data. Project on GitHub (github.com/databufflabs/databuff). Its approach is different: it treats itself as a business system being monitored. Once installed, you can see whether it's healthy, why it's abnormal, which parameter to tune — you can even say one sentence and it will run an inspection on itself. This article walks through a real case.

1. It watches itself: the platform's self-monitoring page

DataBuff's "Deployment Status" page exposes metrics for all three core components: ingest (collects data), web (queries data), and Doris (stores data). The overview opens with four cards that state the conclusion up front: inbound events per second, any write failures, Doris disk remaining, any query errors.

Scrolling down gets more granular: every signal path in ingest — trace / metric / log — is tracked individually with "how many, how big, how slow, any dropped"; Doris disk and CPU are broken out too.

2. It understands itself: every metric ships with its own manual

More metrics isn't the point — every metric comes with an explanation and its tuning parameters. Click the title of any chart and a drawer pops up: how this metric is computed, whether you should worry, which env var to tune when it goes wrong, and the default value. All explained at once.

Take the "Write Drop" metric: the drawer tells you it only occurs in two situations — the queue is full and a whole batch is dropped, or writes fail consecutively and are discarded; it reminds you that "any sustained drop means data loss — check queue depth, write failures, and Doris liveness first"; and finally it lists the tunable parameters directly.

3. It does health checkups: one sentence, AI inspects the whole platform

Don't want to flip through page after page? Just tell the AI platform "inspect the DataBuff platform and produce an HTML inspection report". The product support expert reads the metric catalog, queries the platform's own self-monitoring metrics, picks out the anomalies, and produces an HTML report you can forward directly.

We ran it live: within minutes it automatically checked ingest, writes, pipeline, query domain, Doris, and process resources. The conclusion matched manual troubleshooting — one query-domain failure was still ongoing, and it flagged it as-is.

4. It diagnoses: logs are dropping, and it finds the cause itself

This data loss isn't made up — it actually happened and was actually fixed in our demo environment. The "Write Drop" metric on the Deployment Status page went red — only the log signal was dropping continuously, thousands to fifteen thousand per minute:

At this point, no one needs to flip through pages. Just say the word in the AI platform and let the expert investigate. It first rules out two "non-issues": the business side is normal — not the app's fault; Doris storage is alive and writes are fast — not the storage's fault. The real cause lands in ingest. Logs are batched and pushed into a write queue — this environment's queue holds only 16 batches, normally 32. Sudden bursts don't fit, and the overflow gets dropped as whole batches. The log shows Doris ready queue full (16/16) every minute — confirmed.

It packages "why it dropped, where it dropped, which param to tune, and to what value" into a conclusion with evidence, and hands over the fix suggestion:

5. It fixes itself: changing params and restarting, all on its own

The suggested fix is exactly this group of INGEST_DORIS_* parameters — the core problem is a too-small queue, just make it bigger:

Parameter	Before	After	Effect
`INGEST_DORIS_MAX_READY_BATCHES`	16	32	Write queue too small for bursts; double it
`INGEST_DORIS_FLUSH_TIMEOUT_MS`	30s	60s	Restore default write timeout
`INGEST_DORIS_FLUSH_BATCH_BYTES`	50 MiB	50 MiB (unchanged)	Not touched this time

And you don't change the params or restart by hand. The product support expert SSHs in, edits the config, and restarts ingest per the suggestion, then re-checks to confirm:

After the restart takes effect, drops return to zero — before the fix, thousands dropped per minute; after the restart, several consecutive minutes at 0, the curve back to normal:

All a human did: ask it to diagnose, then ask it to fix.

Getting started — you can use it the same way

After install, check Deployment Config → Deployment Status; scan the four overview cards for health at a glance
To confirm whether data is being lost: ingest page → "Write Drop", click the title for the explanation
Add "how to fix dropping data" to your on-call runbook: trigger troubleshooting in the AI platform for a fix suggestion, let the product support expert SSH in to change INGEST_DORIS_* and restart ingest
Periodically have the support expert run a platform inspection and forward the report to the team

Closing — the self-discipline software should have

Software used to be done the moment it shipped; problems were handled by people watching, investigating, and restarting. What DataBuff does isn't complicated: it builds the operational work into the software itself, using the same methods you already use to watch your business.

This data loss — from discovery to on-box fix — nobody opened a runbook or guessed a direction even once.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

The OpenTelemetry Ecosystem Is Strong — DataBuff Can Plug Into Its eBPF Trace Path

databufflabs — Thu, 06 Aug 2026 01:24:44 +0000

The OpenTelemetry community packaged eBPF trace collection as OBI (OpenTelemetry eBPF Instrumentation). On Kubernetes you run a DaemonSet per node; eBPF watches HTTP calls between local services, assembles traces, and exports them — app Pods stay unchanged. Image: otel/ebpf-instrument. Project: opentelemetry-ebpf-instrumentation.

Where do traces land? DataBuff is an open-source APM platform for services, topology, and call chains. Point OBI at DataBuff and you're done.

One DaemonSet, one collector Pod per node — ns obi, app Pods untouched, eBPF on local HTTP → export to DataBuff.

Step 1 · Install DataBuff

curl -fsSL https://databuff.ai/databuff/ai-apm-k8s-install.sh | bash
kubectl -n databuff get pods

All Pods Running and the UI opens — you're good.

Step 2 · Check eBPF readiness on app nodes

uname -r
ls /sys/kernel/btf/vmlinux

Kernel 5.8+ recommended; the second command must list a file. Without BTF, collector Pods won't capture.

Step 3 · Image

Use otel/ebpf-instrument:latest (worked in our test; pin a version in production). Skip if the cluster can pull; offline clusters need docker load on nodes.

Step 4 · Apply the DaemonSet

Replace YOUR_APP_NAMESPACE and YOUR_DATABUFF_HOST (ai-apm-ingest.databuff.svc). hostPID + privileged are required.

Key config:

discovery.instrument.k8s_namespace
ebpf.context_propagation: headers
otel_traces_export.endpoint: http://YOUR_DATABUFF_HOST:4318

kubectl apply -f obi.yaml
kubectl -n obi get ds,pods -o wide

DESIRED / READY should match node count.

Step 5 · Logs

kubectl -n obi logs -l app=obi --tail=80 | grep -iE "instrumenting|process|error" | head -30

Look for instrumenting process.

Step 6 · Traffic

for i in $(seq 1 80); do
  curl -sS -m 2 "http://your-app-url/" >/dev/null || true
  sleep 0.2
done

Step 7 · Verify in DataBuff

Open APM → Services, then topology and trace detail.

eBPF vs language Agent

Approach	Better when
eBPF + DaemonSet	No injection/restart; HTTP-first; polyglot quick layer
Language Agent	Dubbo, slow SQL, method stacks; kernel < 5.8 / no BTF

Limitations: no Dubbo RPC yet; no method stacks or custom business spans; privileged Pod + BTF required.

How multi-hop traces connect

With context_propagation: headers and no app code changes:

Ingress — read Traceparent: on incoming HTTP; attach or create trace context.
Local correlate — match outbound HTTP to the inbound request.
Egress — sockmap sk_msg inserts this hop's Traceparent after the request line.

HTTPS uses a separate TCP Option path upstream; this walkthrough is plain HTTP + headers.

Full bilingual post (with more YAML detail): databuff.ai/blog/en/databuff-obi-ebpf-traces

Palantir Made Ontology Hot — Can APM Troubleshooting Stop at Ontology Alone?

databufflabs — Fri, 31 Jul 2026 08:25:28 +0000

"Ontology" is back in fashion. Palantir Foundry is not pitched as yet another data warehouse — it builds an ontology for the enterprise: tables, streams, and sensors become "Customer," "Order," and "Device," with relationships and actions defined so data becomes a world model programs can reason over. LLMs made the pitch louder — models still need to know what exists in the world.

For our field: what is the APM ontology? And if you build it, does incident triage just work? Two separate questions.

Q1 · How to build an APM ontology

An APM ontology is two steps: extract entities and relationships from traces, then materialize them into metric tables.

Setup: service A has instances A-1 and A-2; two endpoints; both call database DB and run the same SQL1. Four traces:

#	service	instance	endpoint	call	sql
T1	A	A-1	ep1	DB	SQL1
T2	A	A-1	ep2	DB	SQL1
T3	A	A-2	ep1	DB	SQL1
T4	A	A-2	ep2	DB	SQL1

Step 1 — extract. From T1: entities are service A, instance A-1, endpoint ep1, database DB, SQL1. Relationships: A has instance A-1; A exposes ep1; A-1 serves requests on ep1; ep1 calls DB; ep1 executes SQL1; DB has statement SQL1. P99, latency, and call count are not entities — they are observations attached to entities.

All four traces together:

Service hasInstance Instance: A → A-1, A-2
Service exposes Endpoint: A → ep1, ep2
Instance serves request on Endpoint: A-1 / A-2 both serve ep1 and ep2
Endpoint calls DB: ep1 → DB; ep2 → DB
Endpoint executes SQL: ep1 → SQL1; ep2 → SQL1
DB hasStatement SQL: DB → SQL1

Step 2 — materialize. Extracted entities become queryable metric tables. In DataBuff's Doris layer (schema is open source), dimension columns are entities; call_count / resp_time are observations:

Table	Dimensions	Observations	Entities
metric_service	service	call_count / resp_time / error_count	Service
metric_service_instance	service + instance	same	Service, Instance
metric_service_http	service + instance + endpoint	same	Service, Instance, Endpoint
metric_service_db	service + instance + db + sql	db calls / db latency	Service, Instance, DB, SQL

Aggregating the four traces:

metric_service: service A → 1 row, 4 entry calls
metric_service_instance: by instance → 2 rows
metric_service_http: by instance + endpoint → 4 rows
metric_service_db: by instance + DB + SQL → 2 rows

Ontology done: traces tell you what exists and what connects; metric tables materialize those entities.

Q2 · Is ontology enough?

Entities and relationships complete — is troubleshooting solved? Consider how a connection pool actually behaves:

Even if you add ConnectionPool as Service → ConnectionPool → Database, you only label the big boxes. The branches and states in between cannot be covered by adding one more entity — that layer is logic:

a) Acquire: idle → return; can grow → create; pool full → lock wait; timeout → error
b) Use / return: run SQL → return → wake waiters
c) Background maintenance: heartbeat, evict stale connections

Entity-only reasoning fails when an endpoint slows down:

	Ontology only	With logic
Reasoning	endpoint → DB, slow ⇒ DB slow	stuck in lock wait, SQL never ran
Symptom	endpoint latency up	DB execution may be fine
Conclusion	root cause is DB	root cause is waiting for a connection

First principles: program = data + logic.

	Ontology	Logic
Manages	what exists, who connects	how requests flow, where they stall
Locks onto	target entities	last-mile root cause
Without it	no drill-down target	stuck at "something is wrong"

Connection pool is one slice — it shows entity-only is not enough. See how DataBuff breaks down latency for one endpoint:

Entry average ~240ms; breakdown shows HTTP service-b ~100ms, RPC service-b ~80ms, remainder in MySQL, ES, Redis, Kafka. That is logic modeling for one entry — track entry latency and state, then each operation type after the entry.

Q3 · How to model logic

Scale to a real app: track latency and state for every entry, and for each operation type after every entry. Entries include RPC, HTTP, MQ consumers, scheduled jobs, and more.

After a request enters, it typically lands in 5 operation types + 1 behavior:

Type	Key dimensions
DB remote operation	dal group / table / operation / sql
Redis remote operation	command
MQ remote operation	exchange / routingKey / vhost
RPC remote operation	downstream service / remote method
Local operation	no extra attributes yet
Thrown exception	exception name

For remote operations, split latency into three parts:

a) Client: connect, send, receive
b) Network: transport
c) Server: peer execution

Pool-full lock wait accrues on the client — waiting for a connection ≠ slow SQL.

Incidents then follow one chain: which entries → local ops → remote ops (down to SQL/method) → network vs server → exceptions.

Ontology locks entities; logic locks root cause. Logic models entry → operation → remote three-part latency → exception, with latency and state at each layer. For AI to separate "waiting for a connection" from "slow SQL," it must land on entities via ontology, then walk this logic chain to the actual reason.

Install Failed but the Page Still Opens — Ops Expert Restores Doris

databufflabs — Tue, 28 Jul 2026 01:27:27 +0000

Almost every product hits two kinds of user pain:

Can't use it: install fails, won't start, or dies in production — errors only, then humans grep logs and guess commands.
Don't know how: docs elsewhere, menus buried — ask a helper and get "see the official documentation."

Self-healing ops fixes the first: when storage is down the product stays usable, SSH for evidence, change config, restart, verify.

In-product Q&A fixes the second: ask the product in natural language; answers come from in-product docs and your deployment.

DataBuff is an open-source, AI-native OpenTelemetry APM — metrics, traces, logs first; AI on the same telemetry. Three pieces: Ingest, Doris, AI platform / Web.

curl -fsSL https://databuff.ai/install.sh | bash

Self-healing in practice: Doris breaks during install. Many products go dark; you SSH and guess. We inject mem_limit: 256m on Doris BE so start.sh exits non-zero. Troubleshooting mode keeps Web up while Doris is unhealthy.

Configure an LLM, open AI chat, pick Ops Expert, grant SSH, ask for locate-and-fix:

I installed DataBuff on 192.168.50.140 (dir /opt/databuff-ai-apm-failover).
install/start failed but Web opens.
SSH to root@192.168.50.140 (password Databuff@123)
Find why Doris FE/BE is not ready; fix root cause and report results.
Install dir: /opt/databuff-ai-apm-failover

On the host: BE Restarting → mem_limit: 256m OOM → memory raised to 4g, config persisted, ingest restarted, full-stack Healthy.

Terminal matches: four containers healthy, Doris SELECT 1 passes. Troubleshooting mode exits automatically.

In-product Q&A on the same AI entry — OTel ingest and alert thresholds:

Then LLM setup — menu paths, required fields, common pitfalls:

In one line: self-healing fixes "can't use it"; in-product Q&A fixes "don't know how."