Where Does an Ops Topology Actually Come From?

A checkout hits which services? OpenTelemetry records the app handling a request, calling downstream, and talking to the database. One full request is a Trace. Each hop inside it is a Span.

Look at three checkouts. The entry is A (service-a). It sometimes calls B (service-b) and also queries MySQL directly. B uses the same database. The numbers below are simplified. All three requests succeed.

Trace ① · A entry 100 ms

• B · 80 ms
  • MySQL · 40 ms
• MySQL · 10 ms

Trace ② · A entry 200 ms

• B · 120 ms
  • MySQL · 60 ms
• MySQL · 20 ms

Trace ③ · A entry 100 ms

• MySQL · 10 ms

This one never called B. Indentation is who called whom: ① and ② each hit the database twice; ③ once. Internal work is omitted. IDs and numbers are teaching examples.

These calls wait for the downstream reply. If B spent 80 ms, that already includes 40 ms waiting on MySQL — do not add them to 120 ms. A's entry time is the whole wait from request in to response out.

One trace shows one request. When traffic grows, you also want: who depends on whom across the system? From A, where does the time go? Reading traces one by one gets painful.

DataBuff is an AI-native backend for OpenTelemetry: it receives what apps report, serves query pages, and lets AI experts query the same data through tools. The project is open source: https://github.com/databufflabs/databuff. Follow these three traces to see how it builds a global topology and a service flow.

What you get	What it keeps
Global topology	Merge calls into one graph: who depends on whom.
Service flow	Keep the entry and the paths: who A went through, and each path's share of time.

Architecture first: who processes, who stores, who queries.

Collector is an optional forwarder. OTLP is the OpenTelemetry export protocol. Both graphs are computed on the backend by fixed rules; the UI draws them. AI experts can query the same data.

Ingest first groups fragments of the same request.

Records from A and B may arrive in batches. Every request has a trace_id. Ingest groups fragments by that id. Each fragment also has its own id and a parent id parent_span_id: this hop started during that parent's work.

Trace ①: same trace_id, joined by parent

• s1 · A receives the checkout · 100 ms (no parent: this is the entry)
  • s2 · A calls B · 80 ms (parent = s1)
  • s3 · B receives the request · 80 ms (parent = s2 → s2 belongs to A → B's upstream is A)
    • s4 · B queries MySQL · 40 ms (parent = s3)
  • s5 · A queries MySQL directly · 10 ms (parent = s1 → this query belongs to A)

s1–s5 are short labels. s2 is A waiting on B; s3 is B handling the request. In the example both are 80 ms. Real ends can differ; it is still one A→B call.

When Ingest sees s3's parent is s2, and s2 belongs to A, it fills B's upstream as A and A's downstream as B. s1 and s2 both belong to A, so it does not draw A→A. Ingest can wait for late fragments, but it must continue on timeout — it cannot always wait until every piece arrives.

Then it turns database spans into database nodes.

s4 is recorded by the probe on B. It carries the database type and name. Ingest can name the component and remember B's edge to it.

1. B's DB span s4: db.system = mysql, db.name = demo_apm
2. Component node: [mysql]demo_apm
3. Edge: B → MySQL, 40 ms, success

s5 fills A → the same MySQL. Later we just say MySQL. Whether the fields are enough depends on what was collected.

Redis and message queues use their own component fields. You do not need an agent on every middleware to draw the edge. That is not the same as having all of their internal metrics.

Only then are many calls rolled into stats.

Count the right records. How many times A was entered: A's entry spans. How many times B queried the DB: B's DB spans. Trace ① lists five spans; A still received the checkout once.

1. Pick records by use: A's request count from s1; B's DB count from s4
2. Extract numbers: call count +1, error count +1 on errors, duration = end − start
3. Roll up by minute: add like with like, keep counts and total duration

That is the second kind of aggregation: statistics over many calls. Grouping by trace id was the first: putting one request's fragments together.

Assume the three requests finished in the same minute, and A's call to B matches B's receive time. The rollup looks like this:

After the three example requests	Count	Total duration	Average
A entry	3	400 ms	~133 ms
A → B	2	200 ms	100 ms
B → MySQL	2	100 ms	50 ms
A → MySQL directly	3	40 ms	~13 ms

Average duration is total duration ÷ call count. Error rate is errors ÷ call count. In the example every error rate is 0%. Across several minutes, add counts and durations first — do not average the averages.

Service flow also keeps the entry and the path. For A→B→DB the parent path is A→B; for A→DB the parent is A. Store those two DB paths separately so the tree knows where each hangs.

Data lands in Doris; the page queries by time.

What is stored is detail and stats, not a pre-drawn picture:

Data and example tables	What it stores	Where it is used
Trace detail trace_dc_span	Spans with relations filled in	Open one request
Service stats metric_service	Request counts, duration, …	Service metrics; services with traffic but no edges still show
Pair stats metric_service_http, metric_service_db, …	Minute stats for A→B, A→DB, B→DB	Topology edges and numbers
Entry-path stats metric_service_flow	Entry, parent/child path, counts and duration	Build the service-flow tree

1. The UI sends a time range; service flow also picks an entry
2. Web queries Doris: merge stats in the window, assemble nodes and edges
3. The UI lays out nodes, draws lines, shows numbers and status

Each graph queries its own stats. Opening the page does not scan raw spans.

Same data, why two graphs?

Global topology: who depends on whom

The backend merges call stats for the selected window. One identity is one node. Arrows go from caller to callee. A node is a service or component, not a machine.

When both A and B counted A→B, the current implementation prefers A's side so the two copies are not added. Edges carry count, error rate, and average duration. Node alerts are queried and joined separately.

Three example requests become three edges. Both callers connect to one MySQL node. This graph summarizes many requests; it does not keep each request's full path.

DataBuff global topology (demo data)

service-a and service-b both connect to [mysql]demo_apm. Red comes from node alerts in that window — not automatically the root cause.

Service flow: which paths from the entry

The backend loads path stats with A as entry: A→B hangs under A, A→B→DB hangs under B, A→DB hangs under A. Repeat calls on the same path merge. The two DB paths stay separate.

After the tree is built, each path node's total duration divided by the entry's total duration is response contribution. The UI uses that ratio to order branches. It is the time mix of many requests in the window, not the order of one checkout.

All numbers come from the three examples. MySQL appears on two paths. Vertical position is not execution order.

Response contribution = that path node's total duration ÷ entry total duration

B: 200 ÷ 400 = 50% | MySQL under B: 100 ÷ 400 = 25%

A's direct DB: 40 ÷ 400 = 10%

MySQL twice means two access paths, not two databases. 25% is vs entry A, not “25% of B”. B's 50% already includes waiting on that query; do not add 50% and 25%. Parallel calls can overlap in time.

The shares also need not add to 100%. The entry still spends time on compute and other waits. This graph does not split the whole response into non-overlapping slices.

DataBuff service flow (demo data)

The live page also splits the two DB paths. The screenshot is another batch (449 calls, 58% / 31% / 8%) for the UI only — not used in the example math.

Open-source APM products draw these relations differently. SkyWalking and SigNoz have dependency maps. Jaeger's deep dependency graph can show paths through a chosen service from search results.

DataBuff is more complete here: the open-source build ships both a global dependency view and a minute-aggregated service flow from an entry, with response contribution. One graph shows the system; the other narrows where to look.

After new calls are processed and written, refresh or change the time range and the graph is rebuilt from what was queried. Missing instrumentation, or async calls that did not propagate context, can drop an edge. Filters and display limits can too.

On-call, global topology shows dependencies; service flow shows paths worth checking. High contribution or a red node is a clue: still compare latency and errors to a healthy window, then confirm the slow hop on a trace.

---

DataBuff

AI-native OpenTelemetry APM. Metrics, traces, logs, and AI troubleshooting on one stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai