DEV Community: Anurag Sati

My k6 run said 70% failed. SigNoz told me why — then my agent wrote the fix

Anurag Sati — Sat, 25 Jul 2026 18:40:58 +0000

I was staring at a load-test number that looked decisive: ~70% of requests failed. Cool. Except that number alone is almost useless at 2am.

Was it my new endpoint? Rate limiting? A bad client script? A flaky network? Or a real server bug?

For Agents of SigNoz, I built omen — an agent that load-tests a code change with k6, correlates the same window against live SigNoz traces, names the root cause, and hands back a remediation diff. This post is what I actually learned wiring that loop on Windows with Gemini, Foundry, and the official SigNoz MCP server.

Repo: github.com/Satianurag/omen

The problem I kept hitting

k6 is great at client-side truth: request count, p95, failure rate. It is terrible at answering “what broke inside the service?”

I planted a small FastAPI demo (petclinic) with a bad write path:

POST /api/visits
SQLite with BEGIN IMMEDIATE
tiny timeout
a fake sleep under lock

Under concurrent load it fails hard. Client-side you see 5xx. Server-side, SigNoz sees the real error string: database is locked.

That gap — client metrics vs server spans — is what omen exists for.

What omen is (short version)

omen is a governed Burr state machine served over MCP by Theodosia. The driver sees one tool: step. Illegal transitions get refused. Every step lands on a hash-chained ledger you can verify later.

Rough walk:

Read a git diff (or an intent)
Scaffold + generate a k6 script
Validate and run it through Grafana’s k6 MCP
Preflight the SigNoz service
Correlate traces (signoz_aggregate_traces, signoz_search_traces, signoz_get_trace_details)
Detect latency anomalies over the test window
Analyze → groundedness screen → report + remediation

Figure: omen render — 16 actions, legal edges only. Note signoz_preflight after run_test.

How I used SigNoz (not “I used observability”)

1. Self-hosted via Foundry

I deployed SigNoz with Foundry from the repo’s casting.yaml:

foundryctl cast -f casting.yaml

That gave me:

Piece	URL
UI	`http://localhost:8080`
OTLP gRPC	`http://localhost:4317`
SigNoz MCP	`http://localhost:8000/mcp`

API key went into .env as OMEN_SIGNOZ_API_KEY. No inline comments on that line — that bit me once and silently broke auth.

2. Official OpenTelemetry on the target app

Demo apps run under stock opentelemetry-instrument. No custom shipper:

$env:OTEL_SERVICE_NAME='petclinic'
$env:OTEL_EXPORTER_OTLP_ENDPOINT='http://localhost:4317'
$env:OTEL_EXPORTER_OTLP_PROTOCOL='grpc'
$env:OTEL_TRACES_EXPORTER='otlp'
uv run opentelemetry-instrument python examples/petclinic/app.py serve

After a few omen runs, the SigNoz home page stopped looking empty:

Figure: Traces + metrics ingestion active. Services widget shows petclinic with ~70% error rate and ~596ms p99 over the last 30 minutes.

That services row is the “something is on fire” view:

And the service overview shows the load spikes from the k6 runs as clear humps in rate and latency:

Figure: Two load bursts on petclinic — same windows omen used for correlation.

3. Correlation through the SigNoz MCP (not hand-clicked UI)

This was the interesting part for Track 01. omen does not scrape the UI. It talks to the official SigNoz MCP server during the run.

On a real petclinic run (Gemini backend, local k6 MCP, SigNoz correlate), the phase stream looked like this:

Figure: Terminal output from a full omen run — cards for each phase, including SigNoz tools.

The numbers from that run (not invented for this blog):

k6: 3897 requests, p95 547.59ms, ~72% failed
SigNoz preflight: service petclinic, exists=True
Correlate: aggregate_traces ×9 + search_traces + get_trace_details ×3
Worst path: /api/visits 72.1% 5xx, top error database is locked
Guardian screen: grounded
Verdict: server-side regression on /api/visits

Client p95 (~548ms) and server-side p95 (~543ms) lined up. That consistency mattered to me — if those numbers diverged wildly, I’d distrust the join.

4. The agent publishes itself back into SigNoz

After report, omen emits omen.run and omen.step.* spans over OTLP (when OTEL_EXPORTER_OTLP_ENDPOINT is set). So the agent walk is visible in the same backend that observed the app. That self-observability loop is half the point of this track.

The fix it proposed (and why it wasn’t hand-wavy)

omen didn’t stop at “SQLite is sad.” It returned a validated unified diff that:

raised the connect timeout (0.25 → 5.0)
enabled PRAGMA journal_mode=WAL
removed BEGIN IMMEDIATE
removed the artificial time.sleep(0.015) under the lock

That’s exactly the planted fault class. The Guardian check marked the analysis grounded against the cited telemetry before the verdict was sealed.

Also: every stepped phase went through Theodosia’s mounted MCP step tool, so the session wrote a real ledger.jsonl. omen verify came back HMAC-keyed and intact — important if you’re going to claim “audited by construction” in a demo.

Gotchas I only learned by breaking things

These are the details that bit me on a fresh clone:

Windows + Docker k6 is a trap.

OMEN_K6_DOCKER=1 with --network host cannot reach 127.0.0.1 demo apps on Windows. Local k6 x mcp worked. Put OMEN_K6_CMD=k6 x mcp in .env and leave Docker k6 off.

uv run omen, not bare omen.

After uv sync, the omen entrypoint lives in .venv. Use uv run omen … (or activate the venv). Same for pytest and scripts.

opentelemetry-bootstrap needs pip in a uv venv.

uv run opentelemetry-bootstrap --action=install failed here with No module named pip. Fix with uv run python -m ensurepip --upgrade, or skip it — uv sync already installs FastAPI + OTLP packages the demos need.

Runnable demos are on 8400–8404, not :8000 / petstore.

examples/petclinic serves http://localhost:8400. examples/petstore is OpenAPI-only (unit tests). SigNoz MCP itself is on :8000/mcp — don’t confuse that with the app port.

omen pilot ≠ Gemini by default (until I fixed it).

Phase workers already respected OMEN_LLM=gemini. The pilot driver still assumed Ollama/Granite. I fixed the non-Ollama path to walk the same mounted MCP server deterministically so Gemini runs still seal the ledger. Banner now says Gemini is reading. (honest: Burr/MCP walks the graph; Gemini authors the work inside phases).

Bypassing mount() drops your ledger.

Calling build_application().arun() (what verify_scenario.py uses for a fast live check) can leave you with a thin or empty multi-step ledger listing. Driving through MCP step via omen pilot writes a real ledger.jsonl that omen verify can HMAC-check. Use verify_scenario to prove k6+SigNoz; use pilot when you need the audited session story.

SigNoz API keys hate trailing comments.

OMEN_SIGNOZ_API_KEY=.... # note broke auth. Key alone on the line.

Filename must be .env, not .env.txt.

Windows “Save as text” bit me once. omen only loads .env.

What I’d tell myself on day one

Instrument the target first. If SigNoz Services doesn’t show your app with a nasty error rate after a load test, stop debugging the agent.
Correlate on a wall-clock window from the k6 run — not “last 1 hour of everything.”
Prefer MCP tools that return structured aggregates (signoz_aggregate_traces) before you drown in individual spans.
Keep the agent’s own walk on OTLP. Judges (and future-you) can see whether the FSM actually ran.
Demo a failure k6 alone can’t explain cleanly. database is locked from traces beats “high error rate” from a client report.
On Windows, never start from OMEN_K6_DOCKER=1 — local k6 first.

Try the same path

cd omen
foundryctl cast -f casting.yaml
Copy-Item .env.example .env
# Edit .env: OMEN_SIGNOZ_API_KEY, GEMINI_API_KEY, OMEN_LLM=gemini, OMEN_K6_CMD=k6 x mcp
# Do NOT set OMEN_K6_DOCKER=1 on Windows
uv sync
uv run omen warm-k6
# optional (needs pip in the venv):
# uv run python -m ensurepip --upgrade
# uv run opentelemetry-bootstrap --action=install

# Fastest live proof (starts petclinic :8400 for you):
uv run python scripts/verify_scenario.py petclinic

# Audited pilot path (full ledger) — terminal 1:
$env:OTEL_SERVICE_NAME='petclinic'
$env:OTEL_EXPORTER_OTLP_ENDPOINT='http://localhost:4317'
$env:OTEL_EXPORTER_OTLP_PROTOCOL='grpc'
$env:OTEL_TRACES_EXPORTER='otlp'
uv run opentelemetry-instrument python examples/petclinic/app.py serve

# terminal 2 — intent mode (or --ref HEAD~1 against a two-commit demo repo)
uv run omen pilot `
  --intent "load test recording a new visit" `
  --repo-path examples/petclinic `
  --target-base-url http://127.0.0.1:8400 `
  --signoz-service petclinic

uv run omen sessions ls
uv run omen verify <app-id>

Open http://localhost:8080 → Services → petclinic. If you see a fat error rate after the run, the loop is alive.

Closing

I didn’t learn that “observability is important.” I learned that a 70% k6 failure rate is a question, and SigNoz traces are where the answer lives — especially when an agent can query those traces through MCP in the same window as the load test and come back with a grounded fix.

That’s omen. Divines disaster, crafts the cure — but only because SigNoz made the server side legible.

Links

Project: github.com/Satianurag/omen
Hackathon: Agents of SigNoz
SigNoz MCP docs: signoz.io/docs/ai/signoz-mcp-server
Blog guide I followed: WeMakeDevs Blog Guide

Immich promised OpenTelemetry traces. SigNoz showed me metrics instead — and that's what found the 750ms spike

Anurag Sati — Sun, 19 Jul 2026 18:22:03 +0000

I wanted one screenshot for a blog: Immich showing up under SigNoz Services, with a nice flamegraph. I never got that screenshot.

What I got instead was more useful — Immich's real Prometheus histograms in SigNoz, a dashboard that showed album list p99 drop from ~25ms to ~7ms after warmup, and a thumbnail-generation spike at ~750ms p95. This is the path that actually works today, and the one people keep getting wrong.

If you're trying to connect Immich → SigNoz right now, start here. Don't start with the NestJS auto-instrumentation guide.

Why Immich + SigNoz?

Immich is the self-hosted Google Photos alternative a lot of us already run. It already uses OpenTelemetry (nestjs-otel). SigNoz is the place I wanted those signals.

Immich's own docs say Prometheus metrics are first-class, and that exporting traces is also possible because of OpenTelemetry — then they document only Prometheus. That gap is exactly why people open issues asking how to point Immich at SigNoz.

I decided to stop reading and just run both stacks on one machine.

My lab:

Piece	Version / URL
SigNoz	v0.133.0 (self-hosted, Foundry) → `http://localhost:3301`
Immich	`release` image (v3.0.3) → `http://localhost:2283`
Bridge	`otel/opentelemetry-collector-contrib:0.120.0` scraping Immich `/metrics`

Mistake #1: treating Immich like a blank NestJS app

SigNoz's NestJS docs (and half of Reddit) say: set OTEL_EXPORTER_OTLP_ENDPOINT, then:

NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"

On Immich that is a trap. Immich already boots its own OpenTelemetry SDK for Prometheus. Adding auto-instrumentation gives you:

Error: @opentelemetry/api: Attempted duplicate registration of API: trace

That's the same crash people hit in immich#14045. I reproduced the shape of the problem by inspecting the running container: Immich ships @opentelemetry/exporter-prometheus. It does not ship an OTLP trace exporter. So env vars like OTEL_TRACES_EXPORTER=otlp look correct and do almost nothing for traces.

Rule I wish I'd known on day one: if the app initializes OTel itself, don't also --require the Node auto-instrumentation pack.

What actually works: scrape Immich, export OTLP to SigNoz

Immich exposes Prometheus on port 8081 when you enable telemetry:

IMMICH_TELEMETRY_INCLUDE=all
IMMICH_API_METRICS_PORT=8081

I put Immich and SigNoz on the same Docker network (signoz-network), then ran a tiny collector that scrapes Immich and forwards OTLP to SigNoz's ingester:

# configs/otel-collector.yaml (trimmed)
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: immich-api-metrics
          scrape_interval: 15s
          metrics_path: /metrics
          static_configs:
            - targets: ["immich-server:8081"]  # service name, NOT container name

exporters:
  otlp:
    endpoint: signoz-ingester-1:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp]

Gotcha that wasted an evening

My first scrape target was immich_server:8081 (container name). Collector logs said Failed to scrape Prometheus endpoint forever. Docker DNS resolves the Compose service name (immich-server), not the container_name. One hyphen later, the collector started printing:

Metrics ... resource metrics: 1, metrics: 77, data points: 122

Then SigNoz Metrics Explorer lit up when I searched immich.

Figure 1 — SigNoz Metrics Explorer autocomplete for `immich_` histograms.*

The part that still doesn't work (be honest)

I opened Services. Empty. Traces. Empty. Service map? Nothing.

That isn't SigNoz being broken. Services/APM in SigNoz are built from spans. Immich's stock image only exports Prometheus metrics from its NodeSDK. No spans → no APM row.

Figure 2 — SigNoz Services with no Immich traces. This is expected today.

So if your blog (or your weekend) depends on a pretty Immich flamegraph from the official image alone: change the goal. Instrument a sidecar, fork Immich's telemetry bootstrap, or wait for native OTLP traces. Don't pretend env vars fixed it.

Feature-rich SigNoz without traces: dashboard + Metrics Explorer

I built a dashboard called Immich Observability with six panels:

Album list p99
Thumbnail generation p95
CLIP encode p95
Face detection p95
Event bus emit p99
Thumbnail jobs / second

Figure 3 — Immich Observability dashboard in SigNoz.

The album panel was the first “okay, this is worth it” moment. Fresh after restart, album list p99 sat around ~25ms. After a few dozen API hits it settled near ~7ms. Same endpoint, same machine — warmup is not a myth when you can see it.

Then thumbnail generation showed a p95 near 750ms on a real upload path. Client-side curl for a cached thumbnail was ~6ms. Server-side generation is a different animal. That distinction is exactly why repo-level histograms beat “the API felt slow.”

Numbers from the load I actually ran

I hammered the Immich API with a script (warmup + repeated reads + photo upload). Client-side timings:

Endpoint	n	p50	p95	max
`GET /api/server/ping`	60	2.1ms	3.2ms	4.6ms
`GET /api/users/me`	30	4.6ms	6.6ms	9.8ms
`GET /api/albums`	20	6.1ms	12.6ms	28.3ms
`GET /api/assets/{id}/thumbnail`	25	6.2ms	9.0ms	15.3ms
`GET /api/assets/{id}`	10	16.2ms	31.2ms	56.6ms
`POST /api/assets` (upload)	1	—	—	44.6ms

Server-side histograms after a second load batch (from /metrics):

Operation	avg	count
`immich_event_repository_emit`	38.8ms	64
`immich_asset_repository_create`	21.9ms	61
`immich_asset_repository_upsert_exif`	22.8ms	61
`immich_machine_learning_repository_check`	27.2ms	101

Early in the session, before more traffic diluted the average, event_repository_emit showed ~252ms on a tiny sample — the kind of spike you'd never notice from curl alone if you only look at the happy path.

None of these numbers are “production SLOs.” They're what my laptop + Docker setup produced on 2026-07-19. Your SSD and CPU will disagree. The method is what transfers.

Minimal Immich override (copy-paste)

# docker-compose.override.yml
services:
  immich-server:
    networks: [default, signoz-network]
    ports:
      - "2283:2283"
      - "8081:8081"
    environment:
      IMMICH_TELEMETRY_INCLUDE: all
      IMMICH_API_METRICS_PORT: 8081

  immich-prometheus-scraper:
    image: otel/opentelemetry-collector-contrib:0.120.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./configs/otel-collector.yaml:/etc/otelcol/config.yaml:ro
    networks: [default, signoz-network]
    depends_on: [immich-server]

networks:
  signoz-network:
    external: true

Bring SigNoz up first so signoz-network and signoz-ingester-1:4317 exist. Then docker compose up -d in the Immich folder.

Working configs from this write-up live under configs/ in the companion repo (dashboard JSON export included).

What I'd tell myself last week

Don't chase the Services page first. For stock Immich, Metrics Explorer + a custom dashboard is the win.
Never stack two OTel SDKs. Skip NODE_OPTIONS=--require @opentelemetry/... on Immich.
Name DNS correctly. immich-server, not immich_server.
Look at histograms, not vibes. Album list warmup (~25ms → ~7ms) and thumbnail generation (~750ms p95) are the kind of findings that change how you size a homelab box.
Document the failure. The missing traces story is half the value. Someone else will burn a weekend on the same env vars.

Closing

Immich's docs weren't lying when they said OpenTelemetry is in the tree. They just didn't tell you the exporter story. SigNoz didn't need me to invent a new photo app — it needed an honest scrape pipeline and a dashboard that charts the metrics Immich already emits.

I still want native Immich → OTLP traces. Until then, this is the setup I'll leave running: Immich for photos, SigNoz for the slow bits that aren't the thumbnails I already cached.

f

Anurag Sati — Sun, 15 Feb 2026 07:18:31 +0000

This is a submission for the GitHub Copilot CLI Challenge

What I Built

Demo

My Experience with GitHub Copilot CLI

t

Anurag Sati — Sun, 08 Feb 2026 13:46:12 +0000

This is a submission for the Algolia Agent Studio Challenge: Consumer-Facing Conversational Experiences

What I Built

Demo

How I Used Algolia Agent Studio

Why Fast Retrieval Matters

Fg

Anurag Sati — Sat, 07 Feb 2026 17:21:26 +0000

This is a submission for the Algolia Agent Studio Challenge: Consumer-Facing Conversational Experiences

What I Built

Demo

How I Used Algolia Agent Studio

Why Fast Retrieval Matters

h

Anurag Sati — Sun, 01 Feb 2026 13:02:15 +0000

This is a submission for the New Year, New You Portfolio Challenge Presented by Google AI

About Me

Portfolio

How I Built It

What I'm Most Proud Of

a

Anurag Sati — Fri, 02 Jan 2026 07:41:18 +0000

This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux

What I Built

My Pitch Video

Demo

The Story Behind It

Technical Highlights

Use of Mux (Additional Prize Category Participants Only)

g

Anurag Sati — Mon, 15 Dec 2025 06:50:56 +0000

This is a submission for the Xano AI-Powered Backend Challenge: Production-Ready Public API

What I Built

API Documentation

Demo

The AI Prompt I Used

How I Refined the AI-Generated Code

My Experience with Xano

f

Anurag Sati — Sun, 14 Dec 2025 08:27:55 +0000

This is a submission for the Xano AI-Powered Backend Challenge: Full-Stack, AI-First Application

What I Built

Demo

The AI Prompt I Used

How I Refined the AI-Generated Code

My Experience with Xano

an

Anurag Sati — Mon, 08 Dec 2025 07:11:15 +0000

This is a submission for the AI Challenge for Cross-Platform Apps - AI Acceleration

DEV Community: Anurag Sati

My k6 run said 70% failed. SigNoz told me *why* — then my agent wrote the fix

The problem I kept hitting

What omen is (short version)

How I used SigNoz (not “I used observability”)

1. Self-hosted via Foundry

2. Official OpenTelemetry on the target app

3. Correlation through the SigNoz MCP (not hand-clicked UI)

4. The agent publishes itself back into SigNoz

The fix it proposed (and why it wasn’t hand-wavy)

Gotchas I only learned by breaking things

What I’d tell myself on day one

Try the same path

Closing

Immich promised OpenTelemetry traces. SigNoz showed me metrics instead — and that's what found the 750ms spike

Why Immich + SigNoz?

Mistake #1: treating Immich like a blank NestJS app

What actually works: scrape Immich, export OTLP to SigNoz

Gotcha that wasted an evening

The part that still doesn't work (be honest)

Feature-rich SigNoz without traces: dashboard + Metrics Explorer

Numbers from the load I actually ran

Minimal Immich override (copy-paste)

What I'd tell myself last week

Closing

Links

f

What I Built

Demo

My Experience with GitHub Copilot CLI

t

What I Built

Demo

How I Used Algolia Agent Studio

Why Fast Retrieval Matters

Fg

What I Built

Demo

How I Used Algolia Agent Studio

Why Fast Retrieval Matters

h

About Me

Portfolio

How I Built It

What I'm Most Proud Of

a

What I Built

My Pitch Video

Demo

The Story Behind It

Technical Highlights

Use of Mux (Additional Prize Category Participants Only)

g

What I Built

API Documentation

Demo

The AI Prompt I Used

How I Refined the AI-Generated Code

My Experience with Xano

f

What I Built

Demo

The AI Prompt I Used

How I Refined the AI-Generated Code

My Experience with Xano

an

What I Built

Demo

AI Tooling in Action

Using App MCP and Uno Platform MCP

Cross-Platform Reach

Development Experience

My k6 run said 70% failed. SigNoz told me why — then my agent wrote the fix