DEV Community

Jay Bamroliya
Jay Bamroliya

Posted on

My registration endpoint took 5.3 seconds. The INSERT took 38 ms.

#signoz #opentelemetry #wemakedevs #observability

Instrumenting a five-year-old Spring Boot app with zero code changes, and everything that broke on the way to self-hosting SigNoz on a 6 GB Windows laptop.

My Spring Boot registration endpoint took 5.3 seconds. The SQL INSERT inside it took 38 milliseconds. I only learned that after pointing SigNoz at a five-year-old side project — without changing a single line of its code. This post is the story of that evening: what broke on the way (a lot), and what the traces taught me about my own app.

Around 2020, when I was learning Java, I built Smart Contact Manager — a Spring Boot 2.3 app with Spring Security, JPA, and Thymeleaf. I made it purely for my personal learning, and it shows: it has System.out.println debugging all over it, because that's what I knew back then.

Full disclosure before we start: I did this whole setup pair-working with an AI coding agent — it drove the terminal while I made the decisions, clicked the UAC prompts, and created the accounts. For a hackathon literally called Agents of SigNoz, that felt fitting. Every number and error in this post is from my machine, and I watched all of it happen.

For the Agents of SigNoz pre-event challenge, I asked a different question than "can I instrument a fresh demo app": can modern observability tooling light up an app I wrote five years ago, without me changing a single line of its code?

The answer turned out to be yes — and the traces immediately showed me two things about my own app that I never knew. Along the way, almost every step of the setup broke in an instructive way, so this post is both a "how to" and an honest list of everything that went wrong.

Setup: Windows 11 laptop, 6 GB RAM (yes, really), Java 19, no Docker installed, and — as I discovered — a broken WSL.

Step 0: The setup fought back

SigNoz self-hosting needs Docker. My machine had neither Docker nor a working WSL — wsl --version died with:

Class not registered
Error code: Wsl/CallMsi/Install/REGDB_E_CLASSNOTREG
Enter fullscreen mode Exit fullscreen mode

Re-registering the Store package didn't help. What fixed it was installing the official WSL MSI directly from the microsoft/WSL GitHub releases (wsl.2.7.10.0.x64.msi), enabling the Virtual Machine Platform feature, and rebooting.

Then came the first genuinely useful discovery, straight from the SigNoz Docker install docs: on Windows, don't run SigNoz under Docker Desktop. ClickHouse Keeper is known to crash in a restart loop with segmentation faults under Docker Desktop's virtualization. The docs recommend native Docker Engine inside WSL 2 instead:

# inside Ubuntu on WSL2
curl -fsSL https://get.docker.com | sh
Enter fullscreen mode Exit fullscreen mode

With 6 GB of total RAM, I also capped the WSL VM so ClickHouse, the JVM, and my browser could coexist — C:\Users\<me>\.wslconfig:

[wsl2]
memory=4GB
swap=6GB
processors=4
Enter fullscreen mode Exit fullscreen mode

This worked. ClickHouse Keeper reported healthy on the first boot and stayed that way.

Step 1: Casting SigNoz with Foundry

If you last looked at SigNoz a while ago, the install has changed: the docker-compose manifests and install.sh are deprecated. SigNoz now deploys through Foundry, a CLI that treats your observability stack as declarative config:

curl -fsSL https://signoz.io/foundry.sh | bash

cat > casting.yaml <<'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

cast validates Docker, generates Compose files into pours/deployment/, and starts five containers: ClickHouse, ClickHouse Keeper, Postgres, the SigNoz server, and the OTel collector ("ingester"). UI on :8080, OTLP ingestion on :4317 (gRPC) and :4318 (HTTP).

Two real-world notes from my run:

  1. On a slow connection, the image pull step can get killed mid-way. Pre-pulling with docker compose pull inside pours/deployment/ and re-running foundryctl cast (it's idempotent) got me through.
  2. A killed first run left a stale migration lock. The SigNoz container crash-looped with migrations table is already locked ... duplicate key value violates unique constraint "migration_lock_table_name_key". One DELETE FROM migration_lock; against the bundled Postgres and it healed itself. If your signoz container restarts endlessly after an interrupted install, check this first.

And a genuinely non-obvious gotcha: telemetry ingestion doesn't start until you create the admin account. The collector registers with the SigNoz server via OpAMP, and until the first user/org exists, the server rejects it (cannot create agent without orgId in the logs) and the collector never opens ports 4317/4318. I spent a while staring at Failed to export errors from my app before realizing the fix was simply… finishing the signup form at localhost:8080.

Step 2: Zero-code instrumentation

The app didn't need MySQL for this experiment, so I pointed it at an in-memory H2 database (a dependency swap in pom.xml plus datasource properties — config, not code). Then the entire instrumentation was one JVM flag and a few environment variables:

$env:OTEL_SERVICE_NAME = "smartcontactmanager"
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318"
$env:OTEL_EXPORTER_OTLP_PROTOCOL = "http/protobuf"

java -javaagent:opentelemetry-javaagent.jar `
     -jar smartcontactmanager-0.0.1-SNAPSHOT.jar
Enter fullscreen mode Exit fullscreen mode

That's it. The OpenTelemetry Java agent (v2.29.0) attached to my Spring Boot 2.3.4 app running on Java 19 — a framework version from 2020 — and instrumented Tomcat, Spring MVC, Spring Data, Hibernate, JDBC, and Spring Security without complaint.

I generated traffic with a shell script: user registration, failed logins, successful logins, adding contacts, paginating through them, plus some deliberate errors (a bad path variable, 404s, unauthenticated access). Within seconds, 277 spans had landed in ClickHouse.

Logs Explorer full of Hibernate SQL statements

My five-year-old app, alive in the APM view.

Log detail panel showing trace_id correlation

Latency percentiles, request rate, Apdex — all from one JVM flag.

What the traces showed me

1. The 5.3-second registration

The Services view showed smartcontactmanager with a P99 latency of 3.75 seconds. Filtering the Traces Explorer by durationNano >= 2000000000 surfaced the culprit: POST /do_register, 5.29 s.

Service overview with latency percentiles, request rate and Apdex charts

Every request slower than 2 seconds — including an 11.5 s cold-start GET /.

The flame graph broke it down mercilessly:

Span Duration
POST /do_register 5,292 ms
└ UserRepository.save 2,761 ms
  └ Session.persist 834 ms
  └ INSERT smartcontact.user 38 ms

SigNoz Services page showing smartcontactmanager with P99 latency

The flame graph: 5.29 s total, and the actual INSERT is the tiny bar at the end.

My first reaction was: wait, 5 seconds just for a registration? Do my other APIs also take this long and I never knew? That question — which I could never have answered before tonight — is the entire reason observability exists.

The database was innocent. The actual INSERT took 38 milliseconds — 0.7% of the request. The rest was BCrypt password hashing and Hibernate initializing its machinery on the first write of the session. Without the waterfall I'd have "optimized" the database, which was never the problem.

(The traces also caught my app restart red-handed: the first GET / after a restart took 11.5 seconds — a JVM cold start on a memory-starved machine, visible as two lonely 11,587 ms rows in the explorer. Every restart of a Java app has a story like this; traces just make it impossible to ignore.)

2. The exception my app was swallowing

This one I didn't stage. My traffic script re-registers the same test user every cycle, and the Exceptions tab lit up with JdbcSQLIntegrityConstraintViolationException → DataIntegrityViolationException → RollbackException, three per cycle.

Here's the thing: the endpoint returns HTTP 200 for those requests. My 2020-era controller catches the exception, prints a stack trace to stdout, and renders the signup page with a friendly error banner. Status-code monitoring would tell you this endpoint never fails. The exceptions tab, populated automatically from span error events, tells you the truth: the database is rolling back a transaction on every duplicate signup, and the "success" metric is lying to you.

Traces Explorer filtered to requests slower than 2 seconds

Constraint violations my app has been silently swallowing — while returning HTTP 200.

That contrast — success metrics green, exceptions tab red — was the single most useful thing SigNoz showed me.

3. My logs pillar was empty, and the reason was my own code

Traces and metrics flowed instantly, but the Logs Explorer sat at "No logs yet". The OTel agent captures logs by hooking the logging framework (Logback here) — and my student self logged everything with System.out.println, which no logging framework ever sees.

The zero-code fix: turn on framework logging via startup args, no rebuild required:

--logging.level.org.hibernate.SQL=DEBUG
Enter fullscreen mode Exit fullscreen mode

Suddenly every SQL statement Hibernate executes arrives in SigNoz as a log record, automatically stamped with the trace_id of the request that caused it — so you can click from a slow span straight to the exact SQL it ran. My own println "logs" remain invisible forever. Consider this a message from your future self: use a logger.

Flame graph of the 5.29 second POST do_register trace with 38 ms INSERT

Real Hibernate SQL flowing into the Logs Explorer.

Exceptions tab showing swallowed constraint violation exceptions

Every log record carries the trace_id of the request that produced it — one click from log to trace.

Alerts, and an honest note about 6 GB of RAM

To round things out I created an alert rule in the UI: request rate above 0.4 ops/s over the last 5 minutes. The query-builder-with-live-chart flow made this pleasantly concrete — you see the exact line your threshold will cut across before you save.

Alert rule in firing state

The alert rule, genuinely firing on my traffic generator's waves.

One honest limitation of my potato-spec setup: when I tried to build the alert on P90 latency from the http.server.request.duration histogram, the percentile query kept failing with an internal error — the ClickHouse window-function query behind percentile aggregation didn't survive my 4 GB WSL memory cap. Switching the alert to the request-count metric worked instantly. On a machine with the recommended RAM I'd expect the percentile alert to work; on 6 GB, know your limits.

Quick reference: every error I hit, and the fix

Error / symptom Fix
Wsl/CallMsi/Install/REGDB_E_CLASSNOTREG on any wsl command Install the official WSL MSI from microsoft/WSL releases, reboot
ClickHouse Keeper segfault restart-loop on Windows Don't use Docker Desktop — install Docker Engine natively inside WSL2 (per SigNoz docs)
foundryctl cast dies with signal: killed mid-pull Pre-pull with docker compose pull inside pours/deployment/, then re-run cast (it's idempotent)
signoz container restart-loops: migrations table is already locked ... migration_lock_table_name_key Stale lock from an interrupted install: docker exec <postgres-container> psql -U signoz -d signoz -c "DELETE FROM migration_lock;"
App logs Failed to export forever; ports 4317/4318 closed Create the first admin account at localhost:8080 — the collector can't register via OpAMP until an org exists (cannot create agent without orgId in server logs)
SigNoz dies when you close your WSL terminal The WSL VM shuts down with its last client — keep a WSL shell open, or configure vmIdleTimeout
Percentile (P90/P99) alert queries fail with internal error ClickHouse ran out of memory under my 4 GB WSL cap — use count/rate metrics, or give the VM more RAM

Takeaways

  • Zero-code instrumentation is real, even for old apps. A 2020 Spring Boot 2.3 app on Java 19 got full traces — HTTP, JPA, JDBC, security filters — from one -javaagent flag.
  • Traces answer "where did the time go" in a way metrics can't. P99 said "slow"; only the waterfall said "BCrypt + Hibernate warm-up, not the database".
  • Exceptions ≠ error responses. My app returned 200 while rolling back transactions. Watch both.
  • Logs are only as good as your logging discipline. The agent can ship your logs, but not your printlns.
  • Read the platform-specific docs. Native Docker-in-WSL instead of Docker Desktop (ClickHouse Keeper segfaults), the account-creation-before-ingestion dependency, and the Foundry migration were all things I'd never have guessed.
  • 6 GB of RAM is enough for SigNoz + ClickHouse + a JVM + a browser, if you cap the WSL VM. Tight, but enough.

I won't pretend this was a smooth evening. It took many more hours than I expected, most of them spent on things that had nothing to do with SigNoz itself — a broken WSL, a forced reboot at midnight, a laptop with barely enough RAM. But that's exactly why I'm writing it all down.

I don't have a grand plan for the hackathon on the 20th yet. What I do have is something most teams won't: a self-hosted SigNoz that already survived everything my machine threw at it, and an app already streaming traces into it. The warm-up did its job.

If you're sitting on an old side project, don't build a fresh demo to try observability — instrument the old thing. Its skeletons make much better screenshots.

SigNoz docs: signoz.io/docs · OpenTelemetry Java agent: opentelemetry.io/docs/zero-code/java/agent

Top comments (0)