DEV Community

Manju Vasanth
Manju Vasanth

Posted on

My API Was 4 Seconds Slow. The Fix Was 9 Characters.

#signoz #opentelemetry #observability #debugging

I signed up for the Agents of SigNoz hackathon planning to warm up with the pre-event blog challenge: self-host SigNoz, send some data, write about a feature I liked. A quiet Saturday afternoon task.

Instead, SigNoz found a real performance bug in my own code within 30 minutes of sending it traffic. This is that story.

The App: A Mock Workday Tenant

To practice Workday integration patterns without touching a real tenant, I built a Python/Flask simulator that mimics a Workday environment: SOAP endpoints, RaaS reports, an Extend-style orchestration canvas, the works.

The flow I care about is a supervisory org provisioning orchestration. One button click runs a config-driven, multi-step pipeline: WQL lookups, widget value extraction, and SOAP calls against the simulator's own endpoints. It worked. It just always felt a little... sluggish. I assumed that was the cost of a chatty multi-step flow and moved on.

Spoiler: it was not.

Self-Hosting SigNoz (Note: Docker Compose Is Deprecated)

First surprise: if you follow older tutorials and clone the SigNoz repo looking for deploy/docker, it's gone. SigNoz recently moved installation to Foundry, a new CLI (foundryctl) that generates and deploys your whole stack from one declarative YAML file. It's new enough that barely anyone has blogged about it yet.

The entire install was three steps.

1. Install foundryctl (I'm on Windows, I grabbed the release binary and added it to PATH; on Mac/Linux there's a one-line install script).

2. Create casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
Enter fullscreen mode Exit fullscreen mode

3. Deploy:

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

Foundry validated my Docker setup, generated Compose files into a pours/ directory, and started everything: SigNoz UI, OTel ingester, ClickHouse, ClickHouse Keeper, and Postgres.

docker compose -f pours/deployment/compose.yaml ps
Enter fullscreen mode Exit fullscreen mode

Five services, all healthy. UI on http://localhost:8080, OTLP ingestion on 4317/4318. From zero to a running observability stack in under 15 minutes, most of which was image pulls.

Instrumenting Flask With Zero Code Changes

I wanted a baseline before writing any custom spans, so I used OpenTelemetry auto-instrumentation:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
Enter fullscreen mode Exit fullscreen mode

Then launched my app through the OTel wrapper:

OTEL_RESOURCE_ATTRIBUTES=service.name=mock-workday-orchestrate \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument python workday_ui.py
Enter fullscreen mode Exit fullscreen mode

No code edits. The wrapper auto-instruments Flask and the requests library, so every incoming route and every outgoing HTTP call becomes a span.

I clicked through my orchestration flow 10-15 times to generate traffic, then opened SigNoz.

The First Look: Something Is Wrong

The Services page populated immediately: latency percentiles, request rate, Apdex, and a key operations table.

And it did not look good. My orchestration endpoint showed p99 latency of 4.21 seconds, and the Apdex score was sliding downhill with every run. For a local app calling itself on the same machine, that's absurd.

Time to open a trace.

The Waterfall: 99.5% of the Time Was Missing

I clicked into a 4.14 second trace for POST /extend/screen/create-sup-org. The flame graph and waterfall showed 5 spans:


POST /extend/screen/create-sup-org   4.14s   ← root
 ├─ GET  (client span)               2.05s
 │   └─ GET /tenant/soap/...         3.7ms
 └─ POST (client span)               2.08s
     └─ POST /tenant/soap/...        20.35ms
Enter fullscreen mode Exit fullscreen mode

Two things jumped out.

First, the two internal calls ran sequentially and each took almost exactly 2 seconds. Suspiciously round, suspiciously identical.

Second, and this is the part that made me sit up: when I clicked the inner POST server span, the Span details panel showed it took 20.35ms, just 0.49% of total execution time. The actual handler doing the SOAP work was fast. The client span wrapping it took 2.08 seconds.

So roughly 2 seconds per call was being spent somewhere between "my code initiated the request" and "my server started handling it." On localhost. Twice per flow.

I didn't have any time.sleep() in the orchestration path (I grepped to be sure). This was real, invisible overhead.

The Smoking Gun Was a Span Attribute

The answer was sitting in the Span details panel the whole time, in the request attributes:

http.url: "http://localhost:8443/tenant/soap/human_resources"
Enter fullscreen mode Exit fullscreen mode

That word, localhost, is the entire bug.

Here's what happens on Windows: when Python's requests library connects to localhost, the name resolves to both the IPv6 address (::1) and the IPv4 address (127.0.0.1), and IPv6 gets tried first. My Flask dev server was bound to IPv4 only. So every single internal call:

  1. Attempts a connection to ::1 (IPv6)
  2. Waits for that attempt to fail (roughly 2 seconds)
  3. Falls back to 127.0.0.1 (IPv4)
  4. Connects instantly, handler responds in milliseconds

A fixed 2-second tax on every internal HTTP call, completely invisible in application logs, because the application code was never slow. My "chatty flow is just sluggish" assumption had been this bug all along.

The 9-Character Fix

Replace localhost with 127.0.0.1 in the URLs my server calls at runtime.

One wrinkle worth sharing: my orchestration is config-driven, so the step URLs don't only live in Python source. They also live in a saved flow store (orchestrate_store.json). My first fix edited the Python defaults, restarted, and changed nothing, because the running flow loaded its URLs from the JSON store. Tracing the actual http.url attribute in SigNoz is what proved the old value was still live. One sed across the config and source files later:

sed -i.bak 's|http://localhost:8443|http://127.0.0.1:8443|g' orchestrate_store.json *.py
Enter fullscreen mode Exit fullscreen mode

Restart, rerun the flow, open the newest trace.

Before and After 


                     Before      After
Root span            4.18s   →   45ms
Internal GET         2.06s   →   11.4ms
Internal POST        2.11s   →   26.1ms
Enter fullscreen mode Exit fullscreen mode

4.18 seconds down to 45 milliseconds. A 93x improvement, from changing 9 characters, found by reading one span attribute.

The new waterfall is beautiful: the client spans now hug their server spans instead of dwarfing them. And the Span details confirm the fix took: http.host: "127.0.0.1:8443".

My Favorite Feature: The Trace Waterfall + Span Attributes

The blog prompt asks for the feature I liked most, and after this experience it's not close: the trace detail view, specifically the combination of the waterfall and the Span details attribute panel.

Here's why. Logs told me nothing about this bug. My handler logs showed fast, healthy requests, because the handler was fast and healthy. Metrics alone would have told me "p99 is 4 seconds" without telling me where the time went.

The waterfall answered where: not in my handler (20ms), but in the client-side gap around it (2s). The span attributes answered why: the http.url showed exactly which hostname the live code was calling, which both exposed the IPv6 fallback and caught my incomplete first fix when the config store was still serving the old URL. Observation, localization, root cause, and verification, all in one screen, with zero custom instrumentation.

Honorable mention to Foundry. Coming from the Workday world where I think in config-driven deployments, "describe your stack in one YAML, run one command" felt immediately familiar, and it worked first try on Windows.

What's Next

This was supposed to be a warm-up, and it accidentally became a genuine debugging war story. For the main hackathon (Track 01, AI & Agent Observability), I'm building on this exact foundation: manual OpenTelemetry spans for every step of the orchestration pipeline, config-driven failure injection (401s, 403s, timeouts), and an AI diagnostic agent that reads SigNoz traces to explain why an integration step failed, not just that it did.

If a single auto-instrumented afternoon found a 93x slowdown, I'm very curious what step-level tracing will turn up.

Built for the Agents of SigNoz hackathon by WeMakeDevs and SigNoz. SigNoz is open source and OpenTelemetry-native; the self-host docs are here.

AI disclosure: I used Claude (Anthropic) as an assistant for debugging guidance and drafting this post. The setup, investigation, fix, and all screenshots are from my own environment.

Top comments (0)