DEV Community: Samuel Dahunsi

The Commitment Has to Come First: What My Rust KZG Prototype Taught Me About Binding

Samuel Dahunsi — Tue, 25 Aug 2026 05:06:54 +0000

A polynomial-opening proof can pass every pairing check and still fail to prove the statement you care about.

I ran into that distinction while implementing a multilinear KZG-style commitment scheme in Rust as part of my zero-knowledge cryptography study repository. The prototype could commit to a multilinear polynomial, open it at a point, construct quotient witnesses, and verify the resulting pairing equation over BLS12-381.

The tests passed. The algebra checked out. But the first version of the API left an important question unanswered:

Which commitment did the verifier intend to check?

That question sounds administrative. It is actually the boundary between an opening proof and a self-consistent collection of prover-chosen values.

This article walks through the implementation, the equation behind it, the mistake, and the design changes I would make before treating the code as anything more than a learning prototype.

The argument in one sentence

A polynomial commitment opening is meaningful only relative to a commitment that was fixed before the opening proof was produced.

If the prover is allowed to supply the commitment inside the proof—and the verifier has no independently trusted copy—then verification only establishes:

“This value and these witnesses are consistent with this commitment.”

It does not establish:

“This is the value of the polynomial that was committed to earlier.”

The pairing equation cannot recover missing protocol context. The API has to preserve it.

What I implemented

The repository builds the relevant pieces from first principles:

multilinear polynomials represented by their evaluations on the Boolean hypercube;

partial evaluation and interpolation;

a structured reference string evaluated at a secret point;

commitment computation;

quotient-witness generation for an opening point; and

pairing-based verification using Arkworks and BLS12-381.

This is intentionally educational code. It favors making the algebra visible over hiding it behind an optimized polynomial-commitment interface.

One naming detail matters: the Rust field is called coefficients, but the stored vector behaves as an evaluation table. For an n-variable multilinear polynomial, it contains the values

{ f(b) | b ∈ {0,1}ⁿ }

Calling these values “coefficients” is convenient in code, but misleading in an explanation. They are not monomial-basis coefficients.

From a Boolean evaluation table to any field point

Let f be a multilinear polynomial in n variables. Its multilinear extension is

f̃(x) = Σb ∈ {0,1}ⁿ f(b) χb(x)

where

χb(x) = ∏j=1ⁿ [bjxj + (1 − bj)(1 − xj)]

The implementation evaluates the polynomial one variable at a time. If two entries differ only in the variable being fixed, partial evaluation at r is just linear interpolation:

y(r) = y0 + r(y1 − y0)

The Rust implementation is the same interpolation written directly:

fn interpolate(y0: F, y1: F, r: F) -> F {
y0 + r * (y1 - y0)
}

Repeating that fold over all coordinates reduces the full hypercube table to the single value f(r₁, …, rₙ).

This representation is especially useful for studying sumcheck and GKR because those protocols repeatedly fix variables of multilinear extensions. It also makes the multilinear commitment construction easier to inspect.

Committing in the multilinear Lagrange basis

Classic KZG is often introduced for a univariate polynomial in the monomial basis. This prototype instead commits to a multilinear evaluation table in the Lagrange basis over the Boolean hypercube.

For a secret setup point

τ = (τ₁, …, τₙ)

the setup computes group elements corresponding to the basis evaluations

[χb(τ)]1

for every Boolean vector b. The commitment is then

C = Σb ∈ {0,1}ⁿ f(b)[χb(τ)]1
= [f̃(τ)]1

The commitment code mirrors that inner product. Each evaluation scales its
matching basis element, and the results are accumulated in G₁:

fn commit(g1_basis: &[P::G1], evaluations: &[F]) -> P::G1 {
    let mut commitment = P::G1::zero();

    for (i, basis) in g1_basis.iter().enumerate() {
        let scalar = evaluations[i].into_bigint();
        commitment += basis.mul_bigint(scalar);
    }

    commitment
}

This is one of the things I like about implementing cryptography in Rust with Arkworks: group and field operations stay explicit, while the type system prevents accidentally multiplying unrelated algebraic objects.

It also exposed a small but real engineering hazard. This loop indexes one slice using the length of another, while the quotient-commitment path uses zip, which stops at the shorter iterator. Without explicit length checks, malformed dimensions can either panic or be silently truncated. The mathematical definition assumes matching dimensions; the implementation must enforce them.

Opening the polynomial

Suppose the prover claims

v = f̃(r)

at the point r = (r₁, …, rₙ).

For multilinear polynomials, the difference from the claimed value can be decomposed as

f̃(X) − v = Σi=1ⁿ (Xi − ri)Qi(X)

where each Qᵢ is a quotient polynomial produced as variables are fixed one by one.

The prototype constructs each quotient from pairs in the current evaluation table. For a variable Xᵢ, the difference between the upper and lower halves gives the slope along that coordinate. After recording that quotient, the current polynomial is partially evaluated at rᵢ, and the process continues.

At the end, the remainder should be zero. That is the computational form of the identity above evaluated at the opening point.

The proof contains the claimed value and commitments to the quotient polynomials. The verifier checks a pairing relation equivalent to

e(C − v[G1], [G2]) = ∏i=1ⁿ e([Qi(τ)]1, [τi − ri]2)

Arkworks represents the target-group wrapper with additive operators. That is
why the implementation uses + even though the conventional equation shows a
product:

let claimed_value = g1.mul_bigint(value.into_bigint());
let lhs = P::pairing(commitment - claimed_value, g2);

let mut rhs = PairingOutput::<P>::ZERO;

for (i, tau_i) in g2_taus.iter().enumerate() {
    let r_i = g2.mul_bigint(point[i].into_bigint());
    let tau_minus_r = *tau_i - r_i;
    let quotient = quotient_commitments[i];

    rhs += P::pairing(quotient, tau_minus_r);
}

lhs == rhs

At this point, my tests could generate an opening and verify it successfully. That was useful—but not sufficient.

The API mistake: the proof carried its own commitment

My first proof type looked conceptually like this. The highlighted design problem
is that the proof carries the commitment it is supposed to be checked against:

pub struct KzgProof<G1, F> {
    pub commitment: G1, // Prover-controlled reference
    pub value: F,
    pub quotient_commitments: Vec<G1>,
}

The verifier read commitment from the proof and checked the equation against it.

The equation can be perfectly valid. The problem is that the prover selected every important input to it: the polynomial, the commitment, the claimed value, and the quotient witnesses.

Imagine a database commits to a polynomial encoding a table at time t₀. Later, a client asks for the value at point r. The client needs proof that the answer opens the commitment stored at t₀. If the server can return a fresh commitment alongside the answer, it can choose a different polynomial that gives any convenient value at r and generate a valid opening for that new polynomial.

No pairing check detects this, because the new tuple may be internally correct.

The issue is not broken elliptic-curve arithmetic. It is a broken statement boundary.

The original KZG work defines a polynomial commitment as a short value against which later evaluation claims can be checked. That earlier commitment is the reference point; without it, there is no binding claim to verify. See Kate, Zaverucha, and Goldberg’s original construction.

A better interface

The commitment should not be part of the opening proof’s authority. The verifier should receive the expected commitment from trusted application state, a transcript, a signed object, a consensus layer, or some other previously fixed source.

A better separation removes that authority from the proof:

pub struct OpeningProof<G1, F> {
    pub value: F,
    pub quotient_commitments: Vec<G1>,
}

The verifier receives the expected commitment independently:

pub fn verify_opening<P: Pairing>(
    vk: &VerifierKey<P>,
    expected_commitment: P::G1,
    point: &[P::ScalarField],
    proof: &OpeningProof<P::G1, P::ScalarField>,
) -> Result<bool, VerificationError> {
    validate_dimensions(vk, point, proof)?;
    verify_pairing(vk, expected_commitment, point, proof)
}

This API says more than a comment could:

the proof supplies an opening, not the identity of the committed object;

the caller supplies the commitment it expects;

the verifier only needs a verifier key, not the polynomial or prover state; and

malformed input is distinct from a well-formed but invalid proof.

For a transcript-based protocol, I would also absorb the commitment, evaluation point, claimed value, and protocol/domain identifier into the transcript before deriving any Fiat–Shamir challenges. For an application protocol, I would bind the commitment to the relevant record ID, chain, version, and context.

“The math checks” and “the proof is bound to this application statement” are separate properties. Both need to be designed.

Other changes needed before production

The commitment-boundary issue was the most instructive finding, but it was not the only gap.

Separate prover and verifier state

The current KZG object owns the polynomial and the complete setup. A real verifier should not need the witness polynomial. I would split the code into ProverKey, VerifierKey, commitment, and opening-proof types, with the smallest possible verifier surface.

Treat setup generation as a security protocol

The prototype initializes its setup from explicit, locally known τᵢ values. That is useful for deterministic tests and completely inappropriate as a trusted setup. If the trapdoor is known, binding can fail because forged openings become possible.

A production design needs a secure ceremony, a setup inherited from a correctly generated system, or a transparent commitment scheme with different assumptions. The setup’s origin, size, validation, and update story are protocol requirements—not deployment details.

Validate dimensions before doing algebra

The implementation should reject:

non-power-of-two evaluation tables;

a point whose length differs from the polynomial’s variable count;

setup vectors of the wrong size;

the wrong number of quotient commitments; and

invalid or non-canonical group encodings during deserialization.

Assertions are helpful during development, but public verification APIs should return structured errors and must not panic on adversarial input.

Add serialization and transcript rules

Cryptographic types are not a wire protocol by themselves. A production version needs canonical serialization, subgroup checks, versioning, domain separation, and explicit transcript ordering. Two implementations that perform the same abstract operations can still disagree—or become vulnerable—if they encode or hash them differently.

Test failures, not only successes

My happy-path tests use small explicit polynomials and known opening points. The next useful test suite should mutate one component at a time:

expected commitment;

opening point;

claimed value;

one quotient commitment;

proof length; and

serialized group elements.

Property tests can compare multilinear evaluation against a slow reference implementation. Fuzzing should target parsers and verification entry points. Benchmarks should separate commitment time, witness generation, and verification.

Be precise about what is—and is not—zero knowledge

A polynomial commitment and an evaluation proof are not automatically zero knowledge. Depending on the scheme and how it is used, the commitment or openings may leak information. Hiding requires additional randomization and a security argument for the complete protocol.

This prototype studies commitment and opening mechanics; it does not yet provide a production zero-knowledge polynomial-commitment layer.

Why this lesson matters beyond KZG

The same design error appears in many forms:

verifying a Merkle path against a root supplied by the prover instead of the root stored by the application;

checking a signature while letting the request choose the public key with no identity binding;

validating a SNARK against a verification key or public input that is not tied to the intended circuit or state;

accepting an attestation whose issuer identifier is not anchored to a trusted registry.

Cryptographic verification proves a precise relation over its inputs. It does not decide whether those inputs are the ones your application meant to trust.

That responsibility lives at the boundary between cryptographic code and protocol code—which is why the types and function signatures matter so much.

What I learned

Implementing the scheme from the evaluation table upward made the quotient identity and pairing equation much more concrete. But the more valuable lesson came after the equation worked:

A verifier does not merely need valid algebra. It needs a valid statement, anchored to prior state.

For my next iteration, I would start with the protocol roles and trusted inputs, then make the Rust types encode them. Only after that would I optimize multi-scalar multiplication, batch openings, or benchmark curves.

That ordering is less exciting than making a proof verify for the first time. It is also what turns a cryptographic experiment into dependable infrastructure.

The implementation discussed here is in the kzg module of my ZK cryptography study repository. The repository is educational and exploratory; it should not be used as production cryptographic software.

For more of my implementation notes and protocol work, visit samueldahunsi.me/writing.

Building a Drop-in SLO & Error-Budget Pack for SigNoz Across Traces, Metrics, and Logs

Samuel Dahunsi — Sun, 26 Jul 2026 10:44:58 +0000

How I built a production-grade observability pack with multi-window burn rate alerts, custom ClickHouse dashboards, and automated fault-injection drills for the Agents of SigNoz Hackathon.

The Hook: Why Most SLO Setup Fails in Practice

Setting up Service Level Objectives (SLOs) sounds simple on paper: pick a target like 99.9% availability, track errors, and stop shipping features when your budget burns out.

In reality, most teams hit a wall. Raw APM tools show request counts and average latencies, but they don’t tell you how fast you are consuming your monthly error budget or whether a 5-minute fault requires waking up an engineer on call. Worse, configuring multi-window burn rate alerts across three separate telemetry signals (Traces, Metrics, and Logs) usually takes days of custom ClickHouse query tuning and YAML wrangling.

For the Agents of SigNoz Hackathon (Track 02: Signals & Dashboards), I wanted to solve this permanently by building the SigNoz SLO & Error-Budget Pack: an all-in-one, drop-in observability suite that instruments Node.js/Express applications across all three OpenTelemetry signals and automatically ships three production-grade dashboards and multi-window burn alerts into SigNoz.

Here is how I built it, the mathematical decisions behind the error budget burn rates, and how to test it end-to-end with live-fire fault injection drills.

Architecture: The 3-Signal Telemetry Pipeline

To make SLO calculation accurate, relying on just one signal is a mistake.

Metrics provide high-frequency RED (Rate, Errors, Duration) counters without sampling overhead.
Traces expose p99 latency and child-span performance across downstream API calls.
Logs provide human-readable diagnostic context when an incident occurs.

Here is how data flows through the stack:

                  ┌─────────────────────────────────────────┐
                  │          demo-service (Express)         │
                  │  - tracing.js (OTel SDK)                │
                  │  - RED Middleware (http_requests_total) │
                  │  - Winston Logger (JSON + trace_id)     │
                  └────────────────────┬────────────────────┘
                                       │ OTLP / gRPC & HTTP
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │       SigNoz OpenTelemetry Collector    │
                  └────────────────────┬────────────────────┘
                                       │ ClickHouse Tables
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             SigNoz Engine                                   │
│  ┌───────────────────────┐ ┌────────────────────────┐ ┌──────────────────┐  │
│  │ Service Health        │ │ SLO & Error Budget     │ │ Cost Efficiency  │  │
│  │ Dashboard             │ │ Dashboard              │ │ Dashboard        │  │
│  └───────────────────────┘ └────────────────────────┘ └──────────────────┘  │
│                                      │                                      │
│                                      ▼                                      │
│                    Multi-Window Alert Rules (API v2)                        │
│                                      │ Webhook Payload                      │
│                                      ▼                                      │
│                     Webhook Catcher Service (:3002)                         │
└─────────────────────────────────────────────────────────────────────────────┘

1. Instrumenting the Three Signals

A. RED Metrics Middleware

Rather than relying solely on trace sampling for request rates, I implemented custom RED metrics in metrics.js using @opentelemetry/api-metrics. Every HTTP request is captured by an Express middleware:

// demo-service/index.js
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    recordRequest({
      route: req.path,
      method: req.method,
      statusCode: res.statusCode,
      durationMs: Date.now() - start,
    });
  });
  next();
});

This records three crucial series into SigNoz’s signoz_metrics.distributed_time_series_v4 table:

http_requests_total: Counter partitioned by route, method, and status_class (2xx, 4xx, 5xx).
http_request_duration: Histogram tracking bucketed latency distributions.
downstream_errors_total: Counter tracking third-party API dependencies (catfact, dogimg, joke).

B. Nested Traces with Downstream Attributes

For distributed tracing, the /data endpoint fans out requests to multiple external APIs in parallel. To observe downstream degradation without breaking root trace spans, each downstream call is wrapped in a explicit child span:

async function fetchOne({ name, url }) {
  const span = trace.getTracer('slo-pack-demo').startSpan(`downstream ${name}`);
  const t0 = Date.now();
  try {
    const resp = await axios.get(url, { timeout: 5000 });
    span.setAttributes({
      'downstream.name': name,
      'http.response.status_code': resp.status,
      'downstream.duration_ms': Date.now() - t0,
    });
    return { name, status: resp.status };
  } catch (err) {
    recordDownstreamError(name);
    span.setAttributes({
      'downstream.name': name,
      'error': true,
    });
    throw err;
  } finally {
    span.end();
  }
}

C. 100% Trace-Correlated Logs

One of the biggest friction points in debugging alerts is jumping from a metrics spike to log output. By configuring winston alongside @opentelemetry/instrumentation-winston, trace context is automatically injected into every stdout log line:

{
  "level": "error",
  "message": "handling /error -> 500",
  "service": "slo-pack-demo",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

In the SigNoz UI, clicking any log entry instantly renders the exact visual trace waterfall.

2. Deriving the SLO & Burn Rate Mathematics

Google’s SRE Handbook recommends multi-window, multi-threshold alerting to avoid alert fatigue. Instead of triggering an alert the second error rate hits 0.1%, we measure how fast the error budget is burning.

For a 30-day SLO window with a 99.9% Availability Target:
$$\text{Allowed Unreliability } (U) = 1 - 0.999 = 0.001 \quad (0.1\%)$$

Fast-Burn Alert (Critical)

Goal: Catch massive outages that would consume 2% of the total monthly error budget in 1 hour.
Burn Rate Formula: $$\text{Burn Rate} = \frac{\text{Budget Consumed}}{\text{Time Ratio}} = \frac{0.02}{1 \text{ hour} / 720 \text{ hours}} = 14.4$$
Condition: If the current 1-hour burn rate exceeds 14.4x, trigger a Critical Alert immediately.

Slow-Burn Alert (Warning)

Goal: Catch persistent, subtle error leaks that consume 5% of the error budget over 6 hours.
Burn Rate Formula: $$\text{Burn Rate} = \frac{0.05}{6 \text{ hours} / 720 \text{ hours}} = 6.0$$
Condition: If the 6-hour burn rate exceeds 6.0x, trigger a Warning Alert.

These exact mathematical expressions were converted into SigNoz API v2 alert definitions under alerts/fast-burn.json, alerts/slow-burn.json, and alerts/latency-slo.json.

3. The 3 Custom Dashboards

The pack includes three pre-configured dashboards in dashboards/:

1. Service Health Dashboard

Request Throughput: Real-time req/sec grouped by status class (2xx vs 5xx).
Latency Percentiles: p50, p90, and p99 derived directly from trace span durations.
Downstream Error Tracker: Isolates third-party API failures from internal app errors.

2. SLO / Error Budget Dashboard

Availability SLI %: Computed in real-time as (1 - (sum(5xx) / sum(total))) * 100.
Remaining Error Budget %: Gauge chart highlighting budget degradation before breach.
Burn Rate Gauges: Live 14.4x fast-burn and 6.0x slow-burn indicators.
Latency SLI %: Percentage of requests completing under the 6-second p99 ceiling.

3. Cost-Efficiency & Waste Dashboard

Waste Ratio %: Quantifies wasted compute spent servicing HTTP 5xx responses.
Waste Index: A hybrid metric ($Latency \times ErrorRatio$) pointing out high-latency, error-prone endpoints that consume disproportionate server resources.

4. Automated Verification & Live-Fire Alert Drills

Automated Test Harness

To ensure the pack remains reliable across environments, I wrote a verification script (scripts/verify-p1.sh) that queries both the SigNoz REST API and ClickHouse database:

SIGNOZ_EMAIL="dahunsisamuel1st@gmail.com" SIGNOZ_PASSWORD="..." ./scripts/verify-p1.sh

Output:

== A. Stack & reproducibility (PRD §7.6)
PASS  A1   SigNoz UI + API healthy
PASS  A2   OTLP HTTP ingest endpoint accepting
PASS  A3   demo service /healthz responds
PASS  A4   webhook catcher up
PASS  A5   casting.yaml + lock committed
== B. All three signals w/ required shape
PASS  B1   traces landing (fresh <2m)
PASS  B2   downstream child spans carry custom attrs
PASS  B3   http_requests_total with route/status_class labels
PASS  B4   http_request_duration histogram series present
PASS  B5   downstream_errors_total with downstream label
PASS  B6   metric samples fresh (<2m)
PASS  B7   logs landing (fresh <2m)
PASS  B8   >=95% of app logs trace-correlated
== C. Alert pack
PASS  C1   webhook notification channel registered
PASS  C2   rule 'SLO fast burn' exists w/ severity=critical
PASS  C3   rule 'SLO slow burn' exists w/ severity=warning
PASS  C4   rule 'Latency SLO breach' exists w/ severity=warning
PASS  C5   no rule stuck without state
== D. Paper requirements
PASS  D1   AI_USAGE.md present
PASS  D2   alert payloads committed
PASS  D3   SLO math documented

passed: 21  failed: 0  skipped-sections: 0

Live-Fire Fault Injection

To test alerting end-to-end without waiting for natural outages, I built environment knobs into the containerized service:

# Inject 90% error rate — forces burn rate to ~18x
cd demo-service
ERROR_RATE=0.9 docker compose up -d demo-service

Within minutes:

The Availability SLI dropped on the SLO Dashboard.
The SLO Fast-Burn Rule turned red (FIRING) in SigNoz.
The standalone Webhook Catcher (http://localhost:3002/) logged the incoming incident JSON payload:

{
  "status": "firing",
  "alerts": [{
    "labels": {
      "alertname": "SLO fast burn — availability error budget",
      "severity": "critical",
      "service": "slo-pack-demo"
    },
    "annotations": {
      "summary": "High availability error budget burn rate detected"
    }
  }]
}

Resetting the container restores normal operation immediately:

ERROR_RATE=0.1 SLOW_MAX_MS=3000 docker compose up -d demo-service

Key Lessons Learned & Gotchas

ClickHouse JSON Ingestion Behavior: When querying signoz_logs.distributed_logs_v2, resources_string['service.name'] must match exact string casing. Using LOWER() or un-indexed metadata queries can degrade query times during dashboard renders.
Bash Execution & curl -sf: When automating API provisioning scripts under set -e, avoid using curl -sf on authentication endpoints. Standard HTTP 401 responses will cause curl to exit non-zero silently without outputting SigNoz's JSON error response.
Trace-Log Context Ingestion: OpenTelemetry Winston instrumentation requires standard stdout JSON formatting to reliably extract trace_id and span_id.

Conclusion & Links

Building this pack demonstrated how powerful OpenTelemetry and SigNoz are when all three signals work in unison. Instead of staring at disconnected graphs during an incident, engineering teams get a single pane of glass showing budget remaining, exact firing alerts, and trace-correlated logs within seconds.

GitHub Repository: signoz-slo-pack
SigNoz Documentation: https://signoz.io/docs
OpenTelemetry Documentation: https://opentelemetry.io/docs

What a Corrupted ClickHouse Binary Taught Me About Self-Hosting SigNoz

Samuel Dahunsi — Thu, 16 Jul 2026 15:58:33 +0000

Three crash-looping containers, one corrupted Docker image, and a trace waterfall that made the entire detour worthwhile.

My SigNoz containers kept crash-looping with CANNOT_PARSE_ELF and Segmentation fault. The install guide doesn't mention either error, because neither is SigNoz's fault. Both are what happens when your machine dies mid docker pull. This is the full trail: what broke, why, and how I got traces flowing anyway.

Why I was doing this

I wanted to actually understand observability, not just read about it. The plan was simple: self-host SigNoz, point a real app at it, and poke at traces, logs, and dashboards until the concepts clicked.

Quick vocabulary, since these words get thrown around a lot:

Telemetry is the data an app emits about what it's doing while it runs, essentially a flight recorder for software.
It comes in three flavors: traces (the journey of one request through your system), metrics (numbers over time, like requests/sec), and logs (the text your app prints).
OpenTelemetry (OTel) is the vendor-neutral standard for formatting and shipping that data.
SigNoz receives all of it, stores it, and gives you dashboards, search, and alerts so you can make sense of it.

My setup: a Mac, Docker Desktop, and SigNoz's newer installer, Foundry.

The clean install (this part is easy)

SigNoz has moved off the old docker-compose instructions to a CLI called Foundry. Three steps:

# 1. Install the Foundry CLI (I downloaded and read it before running it)
curl -fsSL https://signoz.io/foundry.sh -o foundry.sh
bash foundry.sh

# 2. A tiny config file, casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

# 3. Deploy
foundryctl cast -f casting.yaml

That generates a real Compose file under pours/deployment/ and starts every container SigNoz needs: ClickHouse (the database), ClickHouse Keeper (coordination), Postgres (metadata), an OTel Collector (the thing your app sends data to), and the SigNoz app itself.

That part is in the docs. Here's the part that isn't.

Where it broke

My first deploy got interrupted while Docker was still pulling images. When I brought the stack back up, three containers went into a crash-loop. The logs:

Code: 464. DB::Exception: The ELF '/usr/bin/clickhouse' doesn't have
string table with section names. (CANNOT_PARSE_ELF)

Segmentation fault
Segmentation fault
Segmentation fault

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty

Three different services, three different errors. It looked like SigNoz was badly broken. It wasn't.

The diagnosis

The common thread is that all three failures are about half-written files.

Docker Desktop on a Mac doesn't run containers directly on macOS. It runs a hidden Linux VM, and every image, container, and database volume lives on one big virtual disk inside that VM. When Docker (or the whole machine) is killed abruptly instead of shut down cleanly, whatever was mid-write gets left truncated, the same way a file can end up corrupted if you yank a USB stick out while it's still saving.

Mapping the errors back:

Error	What actually happened
`CANNOT_PARSE_ELF`	The ClickHouse binary inside the image was written only partway, so Linux can't parse it as an ELF file
`Segmentation fault` (collector)	Same story: an incomplete binary crashing the instant it runs
Postgres `directory exists but is not empty`	Postgres got interrupted mid-initialization and left a half-built data folder

None of this is a SigNoz bug. It's what an interrupted write looks like at three different layers of the stack.

The fix

The recovery recipe is the same every time: delete the corrupted artifact and recreate it from scratch.

# Stop everything
docker compose down

# Remove the corrupted images so they re-pull cleanly
docker rmi -f clickhouse/clickhouse-server:25.12.5
docker rmi -f signoz/signoz-otel-collector:latest

# Remove the half-initialized Postgres volume
docker volume rm signoz-metastore-postgres-0-data

# Re-pull and verify the binary is actually intact this time
docker compose pull
docker run --rm --entrypoint clickhouse \
  clickhouse/clickhouse-server:25.12.5 local --version
# -> ClickHouse local version 25.12.5.44 (official build)   ✅

# Bring it back up
docker compose up -d

That last verify step is the one I'd skip in a hurry, and the one I'd regret skipping. Checking the binary before starting the stack turns a 20-minute crash-loop mystery into a five-second yes/no.

After that, every container came up healthy:

signoz-signoz-0                            Up (healthy)
signoz-telemetrystore-clickhouse-0-0       Up (healthy)
signoz-metastore-postgres-0                Up (healthy)
signoz-telemetrykeeper-clickhousekeeper-0  Up (healthy)
signoz-ingester-1                          Up

SigNoz UI came up at http://localhost:8080.

the SigNoz home dashboard showing traces and metrics ingestion both active

Getting real data in

An empty observability tool is useless, so I grabbed SigNoz's Node.js sample app, which is already OpenTelemetry-instrumented. It has one endpoint, /data, that calls three public APIs in parallel: a cat fact, a dog photo, and a joke.

The only change needed was pointing it at my local collector instead of SigNoz Cloud, in tracing.js. Hitting the endpoint directly confirms what it returns:

the /data endpoint's JSON response showing catFact, dogImage, and joke fields

const exporterOptions = {
  url: 'http://host.docker.internal:4318/v1/traces',
};

host.docker.internal is how a container reaches a service running on the host machine, in this case the OTel Collector listening on port 4318.

One gotcha worth saving you the hour I lost: on my network, the app's outbound calls kept failing with connect ENETUNREACH because Node was resolving the APIs to IPv6 addresses my router wasn't routing. The fix was one line in the Compose file to prefer IPv4:

environment:
  - NODE_OPTIONS=--dns-result-order=ipv4first

Then a build, and a burst of traffic:

docker compose up -d --build
for i in $(seq 1 20); do curl -s http://localhost:3000/data; done

The feature I liked most: the trace waterfall

This is where tracing earned its keep. A burst of traffic against /data lands as a flat list first:

the Traces Explorer list view showing spans from a burst of traffic against /data

Click into one trace and the /data endpoint fires three API calls at once with Promise.all, and the trace waterfall shows exactly that: a parent GET /data span with three child spans running side by side.

the /data trace waterfall showing a parent span with three parallel child spans

The insight lands in one glance. Because the three calls are parallel, the total response time isn't their sum, it's whichever single call was slowest. When a request was slow, I could see which of the three APIs dragged it, not guess. That's the whole pitch of distributed tracing, made concrete in one screen.

The part that genuinely surprised me: I wrote zero tracing code to get this. No manual spans, no timers. getNodeAutoInstrumentations() wrapped Express and axios automatically and produced the entire waterfall, including each outbound call's URL, method, and status code as span attributes.

a single GET span expanded, showing auto-captured HTTP attributes like http.method, http.url, and http.status_code

What worked, what didn't

Worked: traces, immediately and impressively. 218 spans from one burst of traffic, waterfalls rendering with parallel child spans and full HTTP attributes, no custom instrumentation code.
Also worked: metrics. http.client.duration.bucket and http.server.duration.bucket both showed up in the Metrics Explorer with thousands of samples, no extra config beyond what Foundry sets up by default.

[the Metrics Explorer showing http.client.duration.bucket and http.server.duration.bucket with 12.1K+ and 3.2K+ samples]

Didn't (yet): logs. I wired up an OTel log exporter too, but it didn't land in SigNoz on the first pass while traces and metrics did. Traces, logs, and metrics are three separate export pipelines, and getting two working doesn't mean the third is, a lesson I'd rather learn on a toy app than in production. That's my next debugging session.
The big one: an abrupt shutdown can corrupt Docker's VM disk, and the fix is always the same: delete the broken image or volume and recreate it. If your machine has ever hung and forced you to hard-quit, this is worth filing away for next time.

Conclusion

The install is genuinely three commands. The real learning was in the failure: reading CANNOT_PARSE_ELF, a segfault, and a Postgres init error, and realizing they were all the same problem wearing different masks. Traces made my toy app's behavior obvious at a glance, and I got there without writing a line of instrumentation. Metrics came through cleanly too. Logs are the unfinished piece, which feels like the right note to end a first-week-with-a-new-tool post on.

Links:

SigNoz docs: https://signoz.io/docs
Foundry install guide: https://signoz.io/docs/install/docker/
OpenTelemetry: https://opentelemetry.io
SigNoz Node.js sample app: https://github.com/SigNoz/sample-nodejs-app