Anurag Sati

Posted on Jul 19

Immich promised OpenTelemetry traces. SigNoz showed me metrics instead — and that's what found the 750ms spike

#signoz #opentelemetry #immich #selfhosted

I wanted one screenshot for a blog: Immich showing up under SigNoz Services, with a nice flamegraph. I never got that screenshot.

What I got instead was more useful — Immich's real Prometheus histograms in SigNoz, a dashboard that showed album list p99 drop from ~25ms to ~7ms after warmup, and a thumbnail-generation spike at ~750ms p95. This is the path that actually works today, and the one people keep getting wrong.

If you're trying to connect Immich → SigNoz right now, start here. Don't start with the NestJS auto-instrumentation guide.

Why Immich + SigNoz?

Immich is the self-hosted Google Photos alternative a lot of us already run. It already uses OpenTelemetry (nestjs-otel). SigNoz is the place I wanted those signals.

Immich's own docs say Prometheus metrics are first-class, and that exporting traces is also possible because of OpenTelemetry — then they document only Prometheus. That gap is exactly why people open issues asking how to point Immich at SigNoz.

I decided to stop reading and just run both stacks on one machine.

My lab:

Piece	Version / URL
SigNoz	v0.133.0 (self-hosted, Foundry) → `http://localhost:3301`
Immich	`release` image (v3.0.3) → `http://localhost:2283`
Bridge	`otel/opentelemetry-collector-contrib:0.120.0` scraping Immich `/metrics`

Mistake #1: treating Immich like a blank NestJS app

SigNoz's NestJS docs (and half of Reddit) say: set OTEL_EXPORTER_OTLP_ENDPOINT, then:

NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"

On Immich that is a trap. Immich already boots its own OpenTelemetry SDK for Prometheus. Adding auto-instrumentation gives you:

Error: @opentelemetry/api: Attempted duplicate registration of API: trace

That's the same crash people hit in immich#14045. I reproduced the shape of the problem by inspecting the running container: Immich ships @opentelemetry/exporter-prometheus. It does not ship an OTLP trace exporter. So env vars like OTEL_TRACES_EXPORTER=otlp look correct and do almost nothing for traces.

Rule I wish I'd known on day one: if the app initializes OTel itself, don't also --require the Node auto-instrumentation pack.

What actually works: scrape Immich, export OTLP to SigNoz

Immich exposes Prometheus on port 8081 when you enable telemetry:

IMMICH_TELEMETRY_INCLUDE=all
IMMICH_API_METRICS_PORT=8081

I put Immich and SigNoz on the same Docker network (signoz-network), then ran a tiny collector that scrapes Immich and forwards OTLP to SigNoz's ingester:

# configs/otel-collector.yaml (trimmed)
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: immich-api-metrics
          scrape_interval: 15s
          metrics_path: /metrics
          static_configs:
            - targets: ["immich-server:8081"]  # service name, NOT container name

exporters:
  otlp:
    endpoint: signoz-ingester-1:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp]

Gotcha that wasted an evening

My first scrape target was immich_server:8081 (container name). Collector logs said Failed to scrape Prometheus endpoint forever. Docker DNS resolves the Compose service name (immich-server), not the container_name. One hyphen later, the collector started printing:

Metrics ... resource metrics: 1, metrics: 77, data points: 122

Then SigNoz Metrics Explorer lit up when I searched immich.

Figure 1 — SigNoz Metrics Explorer autocomplete for `immich_` histograms.*

The part that still doesn't work (be honest)

I opened Services. Empty. Traces. Empty. Service map? Nothing.

That isn't SigNoz being broken. Services/APM in SigNoz are built from spans. Immich's stock image only exports Prometheus metrics from its NodeSDK. No spans → no APM row.

Figure 2 — SigNoz Services with no Immich traces. This is expected today.

So if your blog (or your weekend) depends on a pretty Immich flamegraph from the official image alone: change the goal. Instrument a sidecar, fork Immich's telemetry bootstrap, or wait for native OTLP traces. Don't pretend env vars fixed it.

Feature-rich SigNoz without traces: dashboard + Metrics Explorer

I built a dashboard called Immich Observability with six panels:

Album list p99
Thumbnail generation p95
CLIP encode p95
Face detection p95
Event bus emit p99
Thumbnail jobs / second

Figure 3 — Immich Observability dashboard in SigNoz.

The album panel was the first “okay, this is worth it” moment. Fresh after restart, album list p99 sat around ~25ms. After a few dozen API hits it settled near ~7ms. Same endpoint, same machine — warmup is not a myth when you can see it.

Then thumbnail generation showed a p95 near 750ms on a real upload path. Client-side curl for a cached thumbnail was ~6ms. Server-side generation is a different animal. That distinction is exactly why repo-level histograms beat “the API felt slow.”

Numbers from the load I actually ran

I hammered the Immich API with a script (warmup + repeated reads + photo upload). Client-side timings:

Endpoint	n	p50	p95	max
`GET /api/server/ping`	60	2.1ms	3.2ms	4.6ms
`GET /api/users/me`	30	4.6ms	6.6ms	9.8ms
`GET /api/albums`	20	6.1ms	12.6ms	28.3ms
`GET /api/assets/{id}/thumbnail`	25	6.2ms	9.0ms	15.3ms
`GET /api/assets/{id}`	10	16.2ms	31.2ms	56.6ms
`POST /api/assets` (upload)	1	—	—	44.6ms

Server-side histograms after a second load batch (from /metrics):

Operation	avg	count
`immich_event_repository_emit`	38.8ms	64
`immich_asset_repository_create`	21.9ms	61
`immich_asset_repository_upsert_exif`	22.8ms	61
`immich_machine_learning_repository_check`	27.2ms	101

Early in the session, before more traffic diluted the average, event_repository_emit showed ~252ms on a tiny sample — the kind of spike you'd never notice from curl alone if you only look at the happy path.

None of these numbers are “production SLOs.” They're what my laptop + Docker setup produced on 2026-07-19. Your SSD and CPU will disagree. The method is what transfers.

Minimal Immich override (copy-paste)

# docker-compose.override.yml
services:
  immich-server:
    networks: [default, signoz-network]
    ports:
      - "2283:2283"
      - "8081:8081"
    environment:
      IMMICH_TELEMETRY_INCLUDE: all
      IMMICH_API_METRICS_PORT: 8081

  immich-prometheus-scraper:
    image: otel/opentelemetry-collector-contrib:0.120.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./configs/otel-collector.yaml:/etc/otelcol/config.yaml:ro
    networks: [default, signoz-network]
    depends_on: [immich-server]

networks:
  signoz-network:
    external: true

Bring SigNoz up first so signoz-network and signoz-ingester-1:4317 exist. Then docker compose up -d in the Immich folder.

Working configs from this write-up live under configs/ in the companion repo (dashboard JSON export included).

What I'd tell myself last week

Don't chase the Services page first. For stock Immich, Metrics Explorer + a custom dashboard is the win.
Never stack two OTel SDKs. Skip NODE_OPTIONS=--require @opentelemetry/... on Immich.
Name DNS correctly. immich-server, not immich_server.
Look at histograms, not vibes. Album list warmup (~25ms → ~7ms) and thumbnail generation (~750ms p95) are the kind of findings that change how you size a homelab box.
Document the failure. The missing traces story is half the value. Someone else will burn a weekend on the same env vars.

Closing

Immich's docs weren't lying when they said OpenTelemetry is in the tree. They just didn't tell you the exporter story. SigNoz didn't need me to invent a new photo app — it needed an honest scrape pipeline and a dashboard that charts the metrics Immich already emits.

I still want native Immich → OTLP traces. Until then, this is the setup I'll leave running: Immich for photos, SigNoz for the slow bits that aren't the thumbnails I already cached.

DEV Community