DEV Community

Cover image for What a Corrupted ClickHouse Binary Taught Me About Self-Hosting SigNoz
Samuel Dahunsi
Samuel Dahunsi

Posted on

What a Corrupted ClickHouse Binary Taught Me About Self-Hosting SigNoz

#programming #signoz #agents

Three crash-looping containers, one corrupted Docker image, and a trace waterfall that made the entire detour worthwhile.

My SigNoz containers kept crash-looping with CANNOT_PARSE_ELF and Segmentation fault. The install guide doesn't mention either error, because neither is SigNoz's fault. Both are what happens when your machine dies mid docker pull. This is the full trail: what broke, why, and how I got traces flowing anyway.

Why I was doing this

I wanted to actually understand observability, not just read about it. The plan was simple: self-host SigNoz, point a real app at it, and poke at traces, logs, and dashboards until the concepts clicked.

Quick vocabulary, since these words get thrown around a lot:

  • Telemetry is the data an app emits about what it's doing while it runs, essentially a flight recorder for software.
  • It comes in three flavors: traces (the journey of one request through your system), metrics (numbers over time, like requests/sec), and logs (the text your app prints).
  • OpenTelemetry (OTel) is the vendor-neutral standard for formatting and shipping that data.
  • SigNoz receives all of it, stores it, and gives you dashboards, search, and alerts so you can make sense of it.

My setup: a Mac, Docker Desktop, and SigNoz's newer installer, Foundry.

The clean install (this part is easy)

SigNoz has moved off the old docker-compose instructions to a CLI called Foundry. Three steps:

# 1. Install the Foundry CLI (I downloaded and read it before running it)
curl -fsSL https://signoz.io/foundry.sh -o foundry.sh
bash foundry.sh
Enter fullscreen mode Exit fullscreen mode 
# 2. A tiny config file, casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
Enter fullscreen mode Exit fullscreen mode 
# 3. Deploy
foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

That generates a real Compose file under pours/deployment/ and starts every container SigNoz needs: ClickHouse (the database), ClickHouse Keeper (coordination), Postgres (metadata), an OTel Collector (the thing your app sends data to), and the SigNoz app itself.

That part is in the docs. Here's the part that isn't.

Where it broke

My first deploy got interrupted while Docker was still pulling images. When I brought the stack back up, three containers went into a crash-loop. The logs:

Code: 464. DB::Exception: The ELF '/usr/bin/clickhouse' doesn't have
string table with section names. (CANNOT_PARSE_ELF)
Enter fullscreen mode Exit fullscreen mode 
Segmentation fault
Segmentation fault
Segmentation fault
Enter fullscreen mode Exit fullscreen mode 
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
Enter fullscreen mode Exit fullscreen mode

Three different services, three different errors. It looked like SigNoz was badly broken. It wasn't.

The diagnosis

The common thread is that all three failures are about half-written files.

Docker Desktop on a Mac doesn't run containers directly on macOS. It runs a hidden Linux VM, and every image, container, and database volume lives on one big virtual disk inside that VM. When Docker (or the whole machine) is killed abruptly instead of shut down cleanly, whatever was mid-write gets left truncated, the same way a file can end up corrupted if you yank a USB stick out while it's still saving.

Mapping the errors back:

Error What actually happened
CANNOT_PARSE_ELF The ClickHouse binary inside the image was written only partway, so Linux can't parse it as an ELF file
Segmentation fault (collector) Same story: an incomplete binary crashing the instant it runs
Postgres directory exists but is not empty Postgres got interrupted mid-initialization and left a half-built data folder

None of this is a SigNoz bug. It's what an interrupted write looks like at three different layers of the stack.

The fix

The recovery recipe is the same every time: delete the corrupted artifact and recreate it from scratch.

# Stop everything
docker compose down

# Remove the corrupted images so they re-pull cleanly
docker rmi -f clickhouse/clickhouse-server:25.12.5
docker rmi -f signoz/signoz-otel-collector:latest

# Remove the half-initialized Postgres volume
docker volume rm signoz-metastore-postgres-0-data

# Re-pull and verify the binary is actually intact this time
docker compose pull
docker run --rm --entrypoint clickhouse \
  clickhouse/clickhouse-server:25.12.5 local --version
# -> ClickHouse local version 25.12.5.44 (official build)   ✅

# Bring it back up
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

That last verify step is the one I'd skip in a hurry, and the one I'd regret skipping. Checking the binary before starting the stack turns a 20-minute crash-loop mystery into a five-second yes/no.

After that, every container came up healthy:

signoz-signoz-0                            Up (healthy)
signoz-telemetrystore-clickhouse-0-0       Up (healthy)
signoz-metastore-postgres-0                Up (healthy)
signoz-telemetrykeeper-clickhousekeeper-0  Up (healthy)
signoz-ingester-1                          Up
Enter fullscreen mode Exit fullscreen mode

SigNoz UI came up at http://localhost:8080.

the SigNoz home dashboard showing traces and metrics ingestion both active

home dashboard

Getting real data in

An empty observability tool is useless, so I grabbed SigNoz's Node.js sample app, which is already OpenTelemetry-instrumented. It has one endpoint, /data, that calls three public APIs in parallel: a cat fact, a dog photo, and a joke.

The only change needed was pointing it at my local collector instead of SigNoz Cloud, in tracing.js. Hitting the endpoint directly confirms what it returns:

the /data endpoint's JSON response showing catFact, dogImage, and joke fields

data endpoint response

const exporterOptions = {
  url: 'http://host.docker.internal:4318/v1/traces',
};
Enter fullscreen mode Exit fullscreen mode

host.docker.internal is how a container reaches a service running on the host machine, in this case the OTel Collector listening on port 4318.

One gotcha worth saving you the hour I lost: on my network, the app's outbound calls kept failing with connect ENETUNREACH because Node was resolving the APIs to IPv6 addresses my router wasn't routing. The fix was one line in the Compose file to prefer IPv4:

environment:
  - NODE_OPTIONS=--dns-result-order=ipv4first
Enter fullscreen mode Exit fullscreen mode

Then a build, and a burst of traffic:

docker compose up -d --build
for i in $(seq 1 20); do curl -s http://localhost:3000/data; done
Enter fullscreen mode Exit fullscreen mode

The feature I liked most: the trace waterfall

This is where tracing earned its keep. A burst of traffic against /data lands as a flat list first:

the Traces Explorer list view showing spans from a burst of traffic against /data

traces list

Click into one trace and the /data endpoint fires three API calls at once with Promise.all, and the trace waterfall shows exactly that: a parent GET /data span with three child spans running side by side.

the /data trace waterfall showing a parent span with three parallel child spans

trace waterfall

The insight lands in one glance. Because the three calls are parallel, the total response time isn't their sum, it's whichever single call was slowest. When a request was slow, I could see which of the three APIs dragged it, not guess. That's the whole pitch of distributed tracing, made concrete in one screen.

The part that genuinely surprised me: I wrote zero tracing code to get this. No manual spans, no timers. getNodeAutoInstrumentations() wrapped Express and axios automatically and produced the entire waterfall, including each outbound call's URL, method, and status code as span attributes.

a single GET span expanded, showing auto-captured HTTP attributes like http.method, http.url, and http.status_code

span attributes

What worked, what didn't

  • Worked: traces, immediately and impressively. 218 spans from one burst of traffic, waterfalls rendering with parallel child spans and full HTTP attributes, no custom instrumentation code.
  • Also worked: metrics. http.client.duration.bucket and http.server.duration.bucket both showed up in the Metrics Explorer with thousands of samples, no extra config beyond what Foundry sets up by default.

[the Metrics Explorer showing http.client.duration.bucket and http.server.duration.bucket with 12.1K+ and 3.2K+ samples]

metrics explorer

  • Didn't (yet): logs. I wired up an OTel log exporter too, but it didn't land in SigNoz on the first pass while traces and metrics did. Traces, logs, and metrics are three separate export pipelines, and getting two working doesn't mean the third is, a lesson I'd rather learn on a toy app than in production. That's my next debugging session.

  • The big one: an abrupt shutdown can corrupt Docker's VM disk, and the fix is always the same: delete the broken image or volume and recreate it. If your machine has ever hung and forced you to hard-quit, this is worth filing away for next time.

Conclusion

The install is genuinely three commands. The real learning was in the failure: reading CANNOT_PARSE_ELF, a segfault, and a Postgres init error, and realizing they were all the same problem wearing different masks. Traces made my toy app's behavior obvious at a glance, and I got there without writing a line of instrumentation. Metrics came through cleanly too. Logs are the unfinished piece, which feels like the right note to end a first-week-with-a-new-tool post on.

Links:

Top comments (2)

Collapse
 
youngancient profile image
Okoye Jude •

This is well explanatory man!

Collapse
 
biliqis_onikoyi_5e0e948ce profile image
Biliqis Onikoyi •

hmm, i know the intention is to share your experience about bad actors but i ended up learning a whole lot about signoz. I should check it out for my applications that keeps going down without due explanation.