DEV Community: Oscar Kandir

Your ownership records are already out of date: building Drift on FalkorDB

Oscar Kandir — Sun, 18 Oct 2026 09:18:47 +0000

I spent four days, solo, asking Kubernetes one simple question. Who actually owns this code? It turns out the answer isn't what its files say.

TL;DR

Kubernetes lists 157 approvers in its OWNERS files. 48 of them haven't touched the main repo in a year.

I built Drift on FalkorDB. It finds the right reviewer for a change 78% of the time, compared with 37% if you pick from the OWNERS files (tested on 400 real pull requests). It also drafts fixes to those files and tests them on a copy of the graph first.

Along the way I hit a FalkorDB bug that made my first result look better than it was, and I dropped a claim that didn't hold up.

The problem

Every big engineering team writes down who owns what. GitHub has CODEOWNERS, and Kubernetes has an OWNERS file in almost every folder, listing the people who can approve changes there. That's how pull requests get routed, and it's how a newcomer figures out who to ask.

The trouble is that these files go stale. People change teams, take a break or just get busy, and nobody goes back to edit the file. Meanwhile someone new quietly becomes the person who actually knows that code, and they never get added. Nothing compares the record with what's really happening, so nobody notices.

I built Drift for Graph Hacks (WeMakeDevs × FalkorDB) to do exactly that comparison, on a real organisation. It's live at https://drift-egny.onrender.com and the code is at https://github.com/oscarosk/drift.

The first number

Once everything was in a graph, my first query was the obvious one. Which listed approvers have done nothing in the last year? No reviews, no /lgtm, no /approve, no merged pull requests in kubernetes/kubernetes.

The answer was 48 of 157, and every one of them is still listed. (I try to say "inactive here" rather than "gone", because plenty of them are probably busy in other Kubernetes repos.)

Everything else in this post is what I built on top of that number, and how I tried to make sure I wasn't fooling myself.

The data, and why it had to be a graph

All of it is public. I used the team list (sigs.yaml, 24 teams), every OWNERS and OWNERS_ALIASES file, about 680 enhancement proposals (the decision records Kubernetes calls KEPs), every merged pull request since October 2024 with the files it changed, and the reviews and /lgtm and /approve comments on those pull requests from the GitHub API. I only store GitHub handles. No names, emails or comment text.

The whole product is about comparing two sides of the same graph:

(:Person)-[:APPROVER_OF]->(:Dir)                                                what the records say
(:Person)-[:REVIEWED|LGTM|APPROVED_PR|AUTHORED_PR]->(:PR)-[:TOUCHES]->(:Dir)    what people actually do
(:Dir)-[:INSIDE]->(:Dir)                                                        OWNERS inheritance

The reason it needs a graph is inheritance. If a folder has no approvers of its own, approval falls back to the nearest parent folder that has some, and so on up the tree. In Cypher, "everyone who reviewed anything at or below this folder" is a single pattern:

MATCH (p:Person)-[:REVIEWED|LGTM|APPROVED_PR]->(pr:PR)-[:TOUCHES]->(:Dir)-[:INSIDE*0..10]->(d:Dir {path: $p})

In SQL that's a recursive query I really wouldn't want to maintain.

The part people would actually use

Nobody wakes up wanting to know how stale ownership is. What they want to know is who should review their change.

So that's the main feature. You paste the files you changed, and Drift ranks people by their reviews and approvals in those folders over the last six months, preferring people who know more of your change. It also shows why it picked them (this person reviewed this pull request, which changed this folder), and draws the evidence as a graph, with the inactive listed approvers in plum.

[IMAGE: upload evidence-graph.png here]

Measuring it, and the bug

Screenshots can look convincing without proving anything, so I tested it against what actually happened.

I took 400 pull requests merged between July and October 2026. For each one, Drift only got to see activity from before that pull request (never including the author), and I checked whether its top three included someone who really typed /approve on it. The comparison is what a contributor does today, which is pick from the OWNERS files.

My first run said 86%. That felt too good, so I wrote a quick leak check. For a sample of pull requests, was any of the evidence from after the merge? Some of it was.

It took a while to find the cause, and it turned out to be FalkorDB itself. If you put a range index on a string property, a range filter ignores its upper bound:

CREATE INDEX FOR (p:PR) ON (p.s);
MATCH (p:PR) WHERE p.s >= '2026-09-05' AND p.s < '2026-09-10' RETURN count(p)   // 15, expected 5

The same query on an integer property is fine. I'd been filtering on date strings, so "before this pull request" was quietly letting the future in. I checked it on FalkorDB 4.18.3 and 4.22.1, wrote a short script that reproduces it, and switched every date filter to integers (PR.day = 20260910). There's a test now that fails if the leak ever comes back.

Here's the honest result:

Who was in the top 3?	Right person first	Right person in top 3
Drift	53%	78%
Three people from the OWNERS files	13%	37%
The three busiest approvers overall	n/a	46%

So it's about twice as good as the records. I kept the 86% story in the README on purpose. Honestly, the wrong number taught me more than the right one did.

Drafting the fix, and testing it on a copy

Pointing at problems only gets you so far, so Drift also drafts fixes. For the folders that have drifted most, it moves inactive approvers to emeritus_approvers (Kubernetes' own way of saying someone stepped back) and adds people who actually review there. It edits the real file text, keeps the comments, and exports a patch that passes git apply --check, with the evidence next to each line.

But removing approvers can change where approval requests end up, and I didn't want Drift suggesting something that quietly breaks a folder. This is where GRAPH.COPY became my favourite FalkorDB feature. Before you download anything, Drift copies the whole graph, applies the patch to the copy, runs the same "where does approval go?" query on both, compares them and deletes the copy. In code it's pretty small:

def with_copy(tools, edit):
    before = coverage(tools)               # where does every approval go today?
    name = f"whatif_{uuid.uuid4().hex[:8]}"
    tools.g.copy(name)                     # GRAPH.COPY: a full, isolated copy
    try:
        copy = Tools(name)
        edit(copy)                         # e.g. move inactive approvers to emeritus
        after = coverage(copy)             # the same query, on the copy
    finally:
        graph(name).delete()               # the copy never outlives the request
    return diff(before, after)

And coverage is one query that walks each folder up its ownership chain:

MATCH (d:Dir {repo: 'kubernetes'}) WHERE d.has_owners
MATCH chain = (d)-[:INSIDE*0..10]->(o:Dir)
OPTIONAL MATCH (p:Person)-[:APPROVER_OF]->(o)
RETURN d.path, length(chain), o.path, o.no_parent_owners, collect(DISTINCT p.handle)

For the 40 drafted fixes, 12 areas whose approval requests only reached inactive people get unstuck, and none get stuck. It takes a couple of seconds and never touches the real graph.

The same trick answers "what if?" questions. If one particular approver stepped back, 6 areas would be left waiting on inactive people. For 50 of the 60 busiest approvers, nothing would get stuck at all. Kubernetes turned out to be more resilient than I expected, with a few sharp edges.

The real teams, and a claim I dropped

The official labels say which team owns each folder. I wanted to know which folders are really worked on by the same group of people, so I linked two folders whenever the same people work in both, and ran FalkorDB's algo.labelPropagation on that.

It found 19 working groups, and 13 of them cross the official team lines. To check they weren't just noise, I built the groups from activity before July and tested them on later pull requests. 47% of folder pairs changed together in a pull request landed in the same group, compared with 15% by chance.

I also tested a second idea, that pull requests crossing groups need more approvers. The first time I ran it, it looked like three times as many. Then I re-ran it on a fresh download of the data and the effect disappeared. So I dropped it, and the README says so. It was the most exciting finding I had, which is exactly why I didn't trust it.

The agent never writes Cypher

Drift has an Ask box and an MCP server, so Claude Desktop or Cursor can use the same tools. I followed the approach from the organisers' guide on agents that reason over graphs. The model picks one of eight fixed queries and fills in the arguments, but never writes a query itself. Every folder, person or KEP it mentions is looked up in the graph before anything runs, so a typo gets "did you mean…" instead of a confident wrong answer. Answers are saved back into the graph as (:Finding) nodes, linked to what they're about, so next time Drift can tell you what's changed.

The piece I'd copy into any project is the check on the model's sentence. It's just a few regular expressions and set differences:

HANDLE = re.compile(r"@([A-Za-z0-9][A-Za-z0-9-]{0,38})")

def check(text, answer):
    invented = {h.lower() for h in HANDLE.findall(text)} - known_handles(answer)
    if invented:
        return False, f"named people not in the result: {sorted(invented)}"
    # ...the same for folders, KEP numbers, and any number of two digits or more
    return True, "ok"

If a sentence fails, the reader gets a plainer one built straight from the query results instead. I'd rather Drift sound a bit flat than say something wrong.

Things that went wrong (the less glamorous ones)

Not everything was a clever bug. My very first build failed with "connection refused" because Docker wasn't running. FalkorDB Cloud doesn't allow the KEYS command, which one of my tests used, so I switched it to GRAPH.LIST. My first deployment landed in a US region and felt slow from India, so I rebuilt it in Singapore, next to the database in Mumbai. And the CI badge was red for a whole day before I noticed that GitHub runs pytest slightly differently from how I ran it locally, so it couldn't even find my code. It was one missing line in pytest.ini.

What I learned

Compare against what people actually do today, not against a strawman. The honest question was never "graph vs RAG". It was "is this better than what a contributor already does?"

Be suspicious of good numbers. The 86% was a bug, and the leak check that caught it took about ten minutes to write.

Throw out results that don't reproduce, especially the exciting ones.

And graph databases earn their place through the shape of the problem as much as speed. Inheritance chains, two snapshots of the organisation in one database, and copy-edit-compare what-ifs are what made this a graph problem for me.

Five things you can reuse on your own graph project

Keep the past as a second graph. FalkorDB holds many graphs in one instance, so a snapshot from a year ago can sit next to today's. Run the same queries on both and you can test predictions against what really happened.
Copy, edit, compare. GRAPH.COPY turns any "what if?" into the same query run twice. Delete the copy in a finally block.
Test with a time split, then try to break it. Only give the system what it would have known at the time, and write a leak check. Mine caught a real bug.
Filter date ranges on integers. Until the string-index bug is fixed, store 20260910 rather than "2026-09-10" for anything you filter by range.
Let the model choose queries, never write them. Use fixed, parameterised queries, look names up in the graph first, and check every sentence against the result.

Try it

Live: https://drift-egny.onrender.com (it's on free hosting, so the first load can take a minute). Click Find a reviewer, then Try an example, or type What if @deads2k stepped back? into the Ask box.
Code: https://github.com/oscarosk/drift. It runs locally with Docker, and the README has the full graph model and every Cypher query.

If you look after an OWNERS or CODEOWNERS file, I'd love to know whether yours has drifted too. Let me know in the comments.

Built for Graph Hacks: Context for AI Agents by WeMakeDevs and FalkorDB. I used Claude as an AI coding assistant while building it; there are details in the README.

Stroll Call: a screen-free voice walk companion for older adults, on a local open model

Oscar Kandir — Mon, 12 Oct 2026 06:56:34 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Older people in my family don't use apps, and walk less every year. Step counters don't move them. Someone saying, warmly and specifically, "go to the park gate, rest under the big tree, say hello to the tea seller" does.

Stroll Call is a screen-free voice walk companion for older adults. Every morning a local open-weight model writes one short, personal walk suggestion and reads it aloud. The person walking never touches a screen; the screen time for them is zero.

A family member fills in a small profile once: mobility, real nearby places, real people, likes, and things to avoid (the main road, broken steps, midday heat).
Each morning the model writes a 40–70 word spoken message using only those places and people, and never anything on the avoid list.
It remembers how yesterday's walk went: shorter after a tired day, gentle after skipped days, a new route after a good streak.
After the walk, three fields are logged: minutes, how it felt, and one note.

Demo

Honest status: the full pipeline (message → voice → memory → next-day adaptation, the setup page, and the eval) was tested end to end with a stand-in model. I have not yet run it on real Gemma with my family member. My connection ran at about 30 kB/s and the model download didn't finish before the deadline. I'll post the real walk and eval results in the comments.

Code

oscarosk / stroll-call

Stroll Call

A screen-free voice walk companion for older adults. Every morning a local open-weight model writes a short, personal walk suggestion, remembers how yesterday's walk went, and reads it aloud. The person walking never touches a screen.

Runs on an ordinary laptop with Ollama and Gemma. No cloud model, no subscription.
Their routine, places, and health notes stay at home.
Works in any language the model and voice support: change "language" in profile.json.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass (Oct 5–11, 2026).

Setup (Windows 11, 8 GB RAM)

Install Ollama from https://ollama.com, then in a terminal
```
ollama pull gemma3:4b
ollama run gemma3:4b "hello"
```
If that is slow on your machine, use gemma3:1b and set OLLAMA_MODEL=gemma3:1b in .env.

Install Python 3.10+, then in this folder

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
copy .env.example .env
copy profile.example.json profile.json

Edit…

View on GitHub

How I Built It

Model: Gemma, running locally through Ollama. No cloud model and no API key.
App: Python. nudge.py builds the prompt from the profile and the last 5 walks; memory.py stores walks (a local JSON file, or MongoDB Atlas if configured); speak.py turns the text into audio (an offline Windows voice, or ElevenLabs if configured); app.py is a one-button Flask setup page.
Safety: the prompt allows only listed places and people, bans the avoid list, gives no medical advice, and never pushes after a tired day. Markdown is stripped before speaking so the voice never reads "asterisk asterisk".
Evaluation: eval.py generates messages for four situations (first day, good streak, tired yesterday, two skipped days) and checks spoken length, real places only, avoid-list mentions (target 0%), memory of the last walk, latency and tokens.
Tracing: optional Sentry agent tracing wraps each run (load walks → model call with tokens and latency → voice → save). If the voice fails, the text is still saved.

Why Does Open Innovation Matter?

Privacy: an elderly person's daily movements and "knee hurt today" notes stay at home, never on someone else's server.
Cost: free to run, with no subscription. That matters for families with elders.
Offline: it runs on an ordinary 8 GB laptop.
Any language: change one field in the profile and everything she hears changes.
Control: I can read and change every rule, swap models, and re-run the eval myself, instead of hoping a vendor changes behaviour.

My Agent Session

I built this with an AI coding agent: idea selection after reviewing the other entries, the code, and an end-to-end test with a stand-in model.

My AI agent was waiting in line: how I traced and fixed hidden sequential LLM calls with SigNoz on Windows

Oscar Kandir — Sun, 19 Jul 2026 16:39:13 +0000

What I built

The agent is small on purpose. You give it a topic, and it does three things: it asks an LLM to break the topic into three sub-questions, answers each of those questions with a separate LLM call, then makes one final call to combine the answers into a short summary. It's built with FastAPI and uses OpenAI's gpt-4o-mini.

That's five LLM calls for a single request. It always returned a sensible answer, so functionally it was "done." But a normal HTTP log only tells me the whole request took ten seconds, not which of those five calls ate the time, how many tokens each one burned, or whether any of them could have run at the same time. That's the gap I wanted to close, and it's exactly what tracing is for: seeing inside one request, step by step.

Setting up SigNoz on Windows (WSL 2)

What you'll need to follow along: Windows with WSL 2 (I used Ubuntu), Docker, Python 3, and an OpenAI API key.

I self-hosted SigNoz rather than using a cloud version, because I wanted the whole stack on my own machine. On Windows that means working inside WSL 2, and the first thing I learned is that the Docker you think you have isn't always the Docker you can use.

I had Docker Desktop installed, but running docker inside my Ubuntu WSL distro gave me this:

The command 'docker' could not be found in this WSL 2 distro.

Docker Desktop keeps its engine in its own separate WSL distro, and mine wasn't wired into Ubuntu. Rather than fight the Docker Desktop integration, I installed Docker natively inside Ubuntu instead:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

After a wsl --shutdown and reopening Ubuntu, docker ps finally returned a clean, empty table. That was the sign the daemon was actually reachable.

Then SigNoz itself, using their Foundry CLI (the current install method; the older install.sh script is deprecated):

curl -fsSL https://signoz.io/foundry.sh | bash

I created a casting.yaml describing the install, ran foundryctl cast -f casting.yaml, and after a few minutes of image pulls, docker ps showed the full stack healthy: SigNoz on port 8080, the OTel collector on 4317/4318. One quirk: the signoz-mcp container shows as unhealthy, but that's a cosmetic healthcheck issue, not a real failure.

Here's how the whole thing fits together:

   POST /research
        │
        ▼
┌──────────────────┐
│  FastAPI agent   │   plan → answer ×3 → combine  (5 LLM calls)
│  (WSL 2 host)    │
└────────┬─────────┘
         │  OpenTelemetry SDK
         │  OTLP → localhost:4318
         ▼
┌──────────────────────────┐
│  SigNoz OTel Collector   │   (Docker, port 4317/4318)
└────────┬─────────────────┘
         ▼
┌──────────────┐      ┌──────────────────┐
│  ClickHouse  │◄─────│   SigNoz UI      │  traces · metrics
│  (storage)   │      │   :8080          │  dashboards · alerts
└──────────────┘      └──────────────────┘

Testing the pipe before trusting it

Before I instrumented my real app, I wanted to know the telemetry pipeline actually worked. So I fired a handful of test traces straight at the collector with telemetrygen, over the Docker network:

docker run --rm --network signoz-network \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
  traces --otlp-endpoint signoz-ingester-1:4317 --otlp-insecure --traces 5

Within about thirty seconds, a telemetrygen service showed up in SigNoz with five traces. That confirmed the whole path was working: collector to ClickHouse to the UI. It paid off later. When my own app's data was slow to appear, I already knew the pipeline was fine and the problem was my config.

[SCREENSHOT: telemetrygen traces in SigNoz (optional)]

Instrumenting the agent

With SigNoz running, I needed the agent to actually send it data. OpenTelemetry has two levels of this, and I used both.

First, auto-instrumentation, the low-effort win. A couple of installs and a wrapper command, no code changes:

pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi
opentelemetry-bootstrap -a install

Then I ran the app through the OpenTelemetry wrapper instead of plain uvicorn:

opentelemetry-instrument uvicorn app:app --port 8001

One detail that trips people up: my app runs on the WSL host, not inside Docker, so the telemetry endpoint is localhost:4318, not the container name signoz-ingester-1 you'd use from inside the Docker network. Getting that wrong is a silent failure: data just never shows up, with no error.

That alone got the incoming FastAPI request and the OpenAI calls into SigNoz. But auto-instrumentation gives you generic spans, "chat gpt-4o-mini" five times. It can't tell you which step of my pipeline each call belongs to. So I added manual spans to label the stages and attach the numbers I actually cared about:

def ask_llm(step_name: str, prompt: str) -> str:
    with tracer.start_as_current_span(step_name) as span:
        resp = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": prompt}],
        )
        u = resp.usage
        span.set_attribute("llm.total_tokens", u.total_tokens)
        span.set_attribute("llm.cost_usd", u.prompt_tokens * PRICE_IN + u.completion_tokens * PRICE_OUT)
        return resp.choices[0].message.content

Now every LLM span carries its real token count and dollar cost, and the pipeline stages (plan_questions, answer_questions, combine_summary) show up as named spans in the trace instead of anonymous HTTP calls. (In my runs, a single research request cost about $[FILL IN from your llm.cost_usd spans] in tokens.)

Looking inside: the staircase

Once the custom spans were flowing, I opened one /research trace in SigNoz and finally saw where those ten seconds went. The flame graph made it obvious in a way no log ever could.

The five LLM calls weren't overlapping. They were running one after another, in a perfect staircase. plan_questions finished, then the first answer call started, then the second, then the third, then the combine step. Each step waited for the one before it.

The numbers told the story:

Total request: 10.48s
answer_questions alone: 6.54s, more than half the entire request
Inside it, three separate answer calls running back to back (roughly 2.76s, 1.33s, and 2.45s)

And here's the thing that made it click: those three answer calls are completely independent. Answering one sub-question doesn't depend on the answer to another. They were only running one at a time because I'd written a plain for loop. That's the most natural way to write it, and the most wasteful. My agent was making the LLM wait in line for no reason.

I hadn't planned this bug. I didn't even know it was there until the trace drew it for me. That's the part that sold me on tracing: the inefficiency wasn't in my code review or my logs. It was sitting in the shape of the waterfall, plain as day.

The fix: stop waiting in line

The fix followed directly from what the trace showed. If the three answer calls are independent, they shouldn't run one at a time. They should run at once. I swapped the sequential for loop for a thread pool so all three fire in parallel:

with ThreadPoolExecutor(max_workers=3) as pool:
    futures = [
        pool.submit(ask_llm_ctx, ctx, f"llm.answer_{i+1}", f"Answer in 2 sentences: {question}")
        for i, question in enumerate(questions)
    ]
    answers = [f.result() for f in futures]

But this is where I hit a real gotcha, and it's one worth knowing if you trace anything threaded: OpenTelemetry's trace context doesn't automatically cross into new threads. The first time I ran the parallel version, the three answer spans detached and floated off as their own separate traces instead of nesting under the request, and the waterfall fell apart.

The fix was to capture the current context before spawning the threads and re-attach it inside each worker:

def ask_llm_ctx(ctx, step_name, prompt):
    token = attach(ctx)
    try:
        return ask_llm(step_name, prompt)
    finally:
        detach(token)

With that in place, I re-ran the same request and opened the new trace. The staircase was gone. The three answer calls now started together and overlapped. answer_questions collapsed from 6.54s down to about 1.65s, roughly the length of its single slowest call instead of the sum of all three.

The whole request dropped from 10.48s to 5.62s, about a 46% cut, from a change that only worked because a trace showed me where to make it.

Here's the full endpoint after the fix, so you can reproduce it (the only real change from the original is that step 2 runs in a thread pool instead of a plain for loop):

from concurrent.futures import ThreadPoolExecutor
from opentelemetry import trace
from opentelemetry.context import attach, detach, get_current

tracer = trace.get_tracer("ai-research-agent")

@app.post("/research")
def research(q: Query):
    with tracer.start_as_current_span("research_pipeline") as root:
        root.set_attribute("research.topic", q.topic)

        # 1. plan: break the topic into 3 sub-questions
        with tracer.start_as_current_span("plan_questions"):
            plan = ask_llm("llm.plan",
                f"Give 3 short sub-questions about: {q.topic}. One per line, no numbering.")
            questions = [l.strip() for l in plan.splitlines() if l.strip()][:3]

        # 2. answer: independent calls, run in parallel (the fix)
        with tracer.start_as_current_span("answer_questions"):
            ctx = get_current()
            with ThreadPoolExecutor(max_workers=3) as pool:
                futures = [
                    pool.submit(ask_llm_ctx, ctx, f"llm.answer_{i+1}",
                                f"Answer in 2 sentences: {question}")
                    for i, question in enumerate(questions)
                ]
                answers = [f.result() for f in futures]

        # 3. combine into a summary
        with tracer.start_as_current_span("combine_summary"):
            summary = ask_llm("llm.combine",
                "Combine these into a short paragraph:\n" + "\n".join(answers))

        return {"topic": q.topic, "questions": questions, "summary": summary}

Beyond a single trace: a dashboard and an alert

One trace showed me the problem, but a trace is a single request. To watch behavior over time, I built a dashboard panel plotting the p95 latency of POST /research from my trace data (p95(durationNano), filtered to the service).

Because I'd run the slow sequential version and the fast parallel version, the panel captured the whole story in one line: latency sitting up around 10 seconds, then dropping and settling near 5 after the fix. The optimization I'd just made, visible as a trend instead of a one-off.

Then I added a guardrail: a trace-based alert that fires if p95 latency goes above 8 seconds at least once in a 5-minute window. The idea is simple: if the sequential slowdown ever creeps back in (a bad refactor, a new blocking call), I want to know before a user does. I wired it to a test webhook and fired a test notification to confirm it actually delivers.

One small thing worth flagging for anyone following along: SigNoz won't let you save an alert rule without attaching a notification channel, so create the channel first, then the rule.

What I'd tell myself before starting

A few things I actually took away from this:

Auto-instrumentation shows you the shape; custom spans give you the answers. The wrapper got data flowing in minutes, but it was generic: five anonymous LLM calls. The insight only appeared once I added my own named spans and attached token and cost numbers. The low-effort version is worth doing first, but it's not where the value is.

A trace can show you things a code review won't. I'd read that for loop plenty of times and never thought twice about it. It took seeing the staircase in the flame graph to realize those calls were independent and could run in parallel. The bug wasn't hidden in the code. It was hidden in the timing, and timing is exactly what a trace makes visible.

On Windows, install Docker inside WSL, not around it. Fighting Docker Desktop's WSL integration wasn't worth it. A native install inside Ubuntu was simpler and just worked.

Prove your telemetry pipeline before you build on it. Before instrumenting anything real, I sent a handful of test traces with telemetrygen and confirmed they landed in SigNoz. That five-minute check meant that when my app's data didn't show up later, I knew the pipeline was fine and the problem was my config, not the other way around.

Wrapping up

I started the night unable to run docker in my terminal and ended it watching a trace tell me exactly why my AI agent was twice as slow as it needed to be, then watching that same tool confirm the fix worked. The agent's output never changed; what changed is that I could finally see what it was doing. That's the whole point: you can't fix what you can't see, and for a multi-step AI agent, a trace is how you see it.

If you want to try the same thing, everything I used is open source and self-hostable: SigNoz, OpenTelemetry, and the OpenAI Python SDK. Self-host SigNoz, instrument one real app you already have, and open a single trace. I'd bet you find something you didn't expect, too.