The 500 That Told Me Nothing — Until I Self-Hosted SigNoz

Jesutofunmi — Thu, 16 Jul 2026 20:08:04 +0000

A few days ago, a route in one of my side projects started returning this:

{"status":500,"unhandled":true,"message":"HTTPError"}

That's it. That's the whole error. No stack trace in the response, nothing useful in the handler — my own try/catch never even fired. The browser got a 500, my code swore it did nothing wrong, and I spent longer than I'd like to admit staring at both.

I eventually found the cause (more on that later — it's a genuinely sneaky one), but the experience exposed a bigger problem: I could only see what happened inside my handler. Once the request left my code and passed back through the framework, I was blind. If an error can happen outside my own code, console.log will never be enough.

What I needed was a way to follow the entire request — through my handler, the framework, and every operation in between — and see exactly where it failed. That is what distributed tracing is built for. To try it properly, I chose SigNoz: an open-source, OpenTelemetry-native observability platform I could self-host for free. Then I built a tiny reproduction app, instrumented it, and pointed it at my local SigNoz instance.

This post is what actually happened — including the parts where things didn't work, because those turned out to be the most instructive bits.

The bug, revisited — what tracing could and couldn't see

Once the reproduction app was sending traces, I went straight back to the problem that started all of this: what was my opaque 500? In TanStack Start, this innocent-looking handler is broken:

GET: async ({ request }) => {
  setCookie('demo_session', crypto.randomUUID(), { httpOnly: true, path: '/' })
  return Response.redirect(new URL('/', new URL(request.url).origin), 302)
}

Response.redirect() returns a Response whose headers are immutable — that's the Fetch spec, not a framework quirk. The framework then tries to append the Set-Cookie header to the returned response after the handler has returned. Appending to immutable headers throws TypeError: immutable, the framework converts it to a generic 500, and your try/catch never sees a thing, because the explosion happens outside your code. (The fix: build the redirect by hand with new Response(null, { status: 302, headers: { location } }).)

Here's the part that taught me something about observability. In SigNoz, the /bug requests show up as error spans — they're why the error rate chart works. But the Exceptions tab stayed empty. Why? Because nothing ever called recordException. The framework swallowed the TypeError and returned a 500; the span was marked as an error by its status code, but no exception event was ever attached to it.

Status-code errors and recorded exceptions are different signals. If your code never sees the error, neither does your exception tracking — only the trace does.

To prove the contrast, I added /crash, a failure that is instrumented the way failures should be:

try {
  throw new TypeError("Cannot read properties of undefined (reading 'amount')")
} catch (error) {
  span.recordException(error)
  span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
  return new Response(JSON.stringify({ error: 'payment processing failed' }), { status: 500 })
}

The trace view shows it as a red, failed span:

And this one shows up in the Exceptions tab, grouped by type, with a count and timeline:

Same status code as /bug. Completely different visibility. That contrast — an invisible failure next to an instrumented one, in the same service, on the same screen — is the clearest argument for tracing I've come across.

What else SigNoz showed me

Once I understood the original bug, I explored what the same instrumentation revealed across the rest of the app. A few minutes of clicking around, and the Services page already had opinions:

P99 latency, throughput, and an error rate — computed entirely from auto-instrumented spans. I never wrote separate metrics code. The 500s from /bug were already there as a percentage on a chart, which made the service-level impact of the failure visible before I opened an individual trace.

The service detail page also has a Key Operations table that ranks every span by latency — including my hand-written query-database and load-user-session spans, sitting alongside the auto-captured HTTP and TLS spans, with P50/P95/P99 for each. Zero dashboard configuration.

Following a slow request

Opening a single /slow request shows the anatomy of the whole 3.56 seconds:

The GET request contains load-user-session (120ms), then query-database (340ms), then the TCP + TLS handshake and the outbound GitHub API call. Every "why is this page slow?" conversation I've ever had would have been shorter with this view.

A detail I didn't expect: each trace contained two GET server spans. Vite's dev mode runs application code in a worker process behind a small internal proxy — and OpenTelemetry's context propagation stitched the proxy process and the worker process into a single coherent trace, automatically, across process boundaries. That's the machinery working exactly as designed, visible on a laptop.

The feature I like most

After a few hours of poking at dashboards, alerts, logs, and explorers, the feature I keep coming back to is the trace detail view — the flamegraph plus span waterfall.

Everything else in SigNoz is aggregate: charts, percentiles, rates. All useful, all things other tools also do. But the trace view answers the question aggregates can't: what happened to this one request? It turned my invisible framework bug into a red span with a timestamp and a duration. It showed me my dev server was secretly two processes. It put my hand-written spans, the framework's HTTP handling, and a third-party API call on one timeline with no effort beyond starting the SDK.

And because SigNoz is OpenTelemetry-native, getting there required nothing proprietary — one exporter URL. If I ever want to point the same app somewhere else, it's a one-line change. That's the right kind of dependency.

What I'd tell you if you're trying this

Create the SigNoz account before debugging "broken" OTLP ports. The collector's receivers don't start until an org exists.
Filter dev-server noise with ignoreIncomingRequestHook, or your traces are unreadable.
recordException is not optional if you want the Exceptions view to mean anything. Span status alone won't do it.
Windows works fine via the Docker Compose files, even though the shiny new installer says otherwise.

The whole demo app is a handful of files — an instrumentation.mjs, four routes, and a docker-compose you didn't have to write. For a free, self-hosted stack, the distance between "nothing" and "I can see every request my app makes" turned out to be about an afternoon — most of which was me learning these pitfalls so you don't have to.

Those are the lessons I would carry into the next project. If you want to reproduce the experiment yourself, here is the exact setup that produced them.

The setup

Windows 11, Docker Desktop (WSL2 backend)
A minimal TanStack Start app (React + file-based routing with server routes), Node 24
SigNoz v0.99.0, self-hosted with Docker Compose

The demo app has four routes, each designed to produce a different kind of telemetry:

/slow — a deliberately multi-step request: two fake I/O waits wrapped in manual spans, plus a real outbound call to the GitHub API
/bug — a faithful reproduction of my opaque 500
/fixed — the same logic as /bug, done correctly
/crash — a failure that is properly instrumented

Self-hosting SigNoz (the Windows detour)

SigNoz's current docs push a new installer called Foundry — which plainly states that Windows is not officially supported. The older Docker Compose files still ship in the repo though, and the last compose-based release is the current release, so on Windows this works fine:

git clone --depth 1 --filter=blob:none --sparse https://github.com/SigNoz/signoz.git
cd signoz
git sparse-checkout set deploy
git checkout v0.99.0
cd deploy/docker
docker compose up -d

First run pulls a few gigabytes (ClickHouse, ZooKeeper, the SigNoz server, and the OTel collector), then the stack boots in order: ClickHouse → schema migrator → SigNoz server → collector. A few minutes later the UI is up at http://localhost:8080.

One honest note from the logs: the server warns loudly if SIGNOZ_JWT_SECRET isn't set. Fine for a laptop demo, but set it if you deploy this anywhere real.

Setup pitfall: the OTLP ports are dead until you create an account

This one cost me actual debugging time, so I'm writing it down for the next person.

After the stack came up, the collector logged Everything is ready. Begin running and processing data. — and yet nothing could connect to the OTLP endpoints on ports 4317/4318. Connection refused, even container-to-container. The ports were mapped, the config said 0.0.0.0:4318, the collector claimed to be alive.

The answer was in the SigNoz server's logs, not the collector's:

Failed to find or create agent ... cannot create agent without orgId

SigNoz's collector runs in managed mode: it fetches its live configuration from the SigNoz server over OpAMP, and it won't start its receivers until it registers. Registration requires an organization to exist — and the organization is only created when you set up the first admin account in the UI.

So the fix for "my OTLP ports won't open" was: open localhost:8080, create the login. Thirty seconds later the collector registered and ports 4317/4318 came alive. Every span my app had sent before that moment was silently dropped.

Instrumenting a TanStack Start app

With SigNoz running, the last step is connecting the application to it. The Node OpenTelemetry SDK gets preloaded into the dev server via NODE_OPTIONS. The whole setup is one file:

// instrumentation.mjs
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'

const NOISY_PATHS = [/^\/@/, /node_modules/, /\.vite/, /favicon/, /\.css/, /\.js\b/]

const sdk = new NodeSDK({
  serviceName: 'otel-playground',
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (request) =>
          NOISY_PATHS.some((pattern) => pattern.test(request.url ?? '')),
      },
    }),
  ],
})

sdk.start()

// package.json
"dev": "cross-env NODE_OPTIONS=--import=./instrumentation.mjs vite dev --port 3001"

Two practical notes:

The ignoreIncomingRequestHook matters in dev. Vite serves a storm of module and asset requests; without the filter, your traces drown in /@vite/client noise.
Custom spans are just a few lines inside a server function:

const tracer = trace.getTracer('otel-playground')

await tracer.startActiveSpan('query-database', async (span) => {
  await simulateDbCall()
  span.end()
})

That's the entire integration. No agent daemon, no vendor SDK — the OTLP endpoint is the only SigNoz-specific thing in the app, which means none of this is lock-in.

Links

Demo code (the app, all four routes, and the instrumentation file): github.com/devtofunmi/otel-playground
SigNoz self-hosting docs: signoz.io/docs/install/self-host
OpenTelemetry JS getting started: opentelemetry.io/docs/languages/js
Written for the Agents of SigNoz hackathon early-blog challenge.

Transparency note: I used an AI assistant (Claude) to help draft and edit this post. Every install step, error message, trace, and screenshot is from my own machine — including the hours those pitfalls cost me.

DEV Community: Jesutofunmi