I Ran Foundry and Got 6 Docker Containers. So I Broke Into All of Them.

Three minutes after running foundryctl cast, I had 6 Docker containers on my machine and no clue what most of them did. The SigNoz docs explain how to install. They don't tell you what's actually running.

So I opened every one of them. Read the configs, poked the databases, watched the logs. Here's what I found: a JWT security hole, an OpAMP error loop, and a ClickHouse table with 80+ columns.

Step 1: The YAML

Foundry needs one file:

# casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose
  mcp:
    spec:
      enabled: true

Run it:

foundryctl cast -f casting.yaml

Six containers show up:

signoz-signoz-0                         signoz/signoz:latest                   Up (healthy)
signoz-ingester-1                       signoz/signoz-otel-collector:latest    Up
signoz-telemetrystore-clickhouse-0-0    clickhouse/clickhouse-server:25.12.5   Up (healthy)
signoz-telemetrykeeper-clickhousekeeper-0 clickhouse/clickhouse-keeper:25.12.5 Up (healthy)
signoz-metastore-postgres-0             postgres:16                            Up (healthy)
signoz-mcp                              signoz/signoz-mcp-server:latest        Up (unhealthy)

All on the same Docker network with static IPs:

172.19.0.2 — ClickHouse Keeper
172.19.0.3 — ClickHouse
172.19.0.4 — Ingester (OTel collector)
172.19.0.5 — PostgreSQL
172.19.0.6 — SigNoz frontend + API
172.19.0.7 — MCP server (unhealthy, stays that way)

Foundry also writes a lock file called casting.yaml.lock that's 668 lines long. I didn't find it until day two. It has every config, every env var, every IP address. Would have saved me hours.

The First Thing That Went Wrong

I started reading logs immediately. The signoz container couldn't reach PostgreSQL:

ERROR  failed to connect to user=signoz database=signoz ... connection refused

It retried 4 times over 16 seconds. PostgreSQL just wasn't ready yet. Same thing happened with the ingester and ClickHouse. The ingester kept trying to connect and failing for about 30 seconds. I watched it spam this:

Error occurred while checking for sync migrations to complete, retrying
dial tcp: lookup signoz-telemetrystore-clickhouse-0-0: no such host

Everything worked eventually. But those first 30 seconds are noisy. If you deploy Foundry and see errors, just wait.

What's Actually Inside signoz-signoz-0

This is the main SigNoz binary. ./signoz server. It starts 12 internal services. I counted them from the logs:

instrumentation, pprof, analytics, alertmanager, ruler,
licensing, auditor, meterreporter, authz, statsreporter,
tokenizer, user

Each one is a mini-service inside the same process. Some depend on others: user waits for authz, meterreporter waits for licensing. I saw these dependency waits in the logs too.

It connects to two databases:

SIGNOZ_SQLSTORE_POSTGRES_DSN=postgres://signoz:signoz@postgres:5432/signoz
SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000

PostgreSQL for metadata (users, dashboards, alerts). ClickHouse for telemetry (traces, metrics, logs).

The Security Warning That Made Me Pause

I almost missed this. About 30 seconds into the signoz logs:

CRITICAL SECURITY ISSUE: No JWT secret key specified!
Your user sessions are vulnerable to tampering and unauthorized access.

Foundry does not generate a JWT secret. Your deployment runs with unsigned tokens by default. I checked the env vars. Nothing. Checked casting.yaml.lock. Nothing. It's just not there.

The fix:

spec:
  signoz:
    spec:
      env:
        SIGNOZ_TOKENIZER_JWT_SECRET: "put-a-64-char-random-string-here"

Add that to your casting.yaml and re-deploy. Without it, anyone who sees a session token can forge one. There's a GitHub issue open for this at #8400.

The Ingester Is the Interesting One

The ingester is an OpenTelemetry Collector. It receives traces (OTLP on ports 4317/4318) and writes them to ClickHouse.

I read the full config. Four pipelines:

traces:   OTLP → span-metrics → batch → ClickHouse
metrics:  OTLP → batch → ClickHouse
logs:     OTLP → batch → ClickHouse
meter:    internal → batch → ClickHouse

The trace pipeline is the interesting one. Every incoming span gets turned into latency histogram metrics before it's stored. Those histograms have 17 buckets, from 100 microseconds to 60 seconds.

The batch size surprised me:

batch:
  send_batch_size: 50000
  send_batch_max_size: 55000
  timeout: 5s

50,000 spans per batch. Every 5 seconds. And there's no retry queue at all. The config explicitly disables it:

sending_queue:
  enabled: false

If ClickHouse is slow or goes down, the ingester drops data immediately. I confirmed this across all four exporters. It's intentional. The trade-off is lower latency at the cost of some data loss.

The OpAMP Thing

The ingester also connects to the signoz container via WebSocket:

ws://signoz-signoz-0:4320/v1/opamp

This is the Open Agent Management Protocol. The ingester sends status updates every 30 seconds and receives config changes. But before you register a user, there's no organization in the database. So every 30 seconds, this shows up:

ERROR  cannot create agent without orgId

I checked the organizations table in PostgreSQL. Empty. Of course. The fix is simple: register your admin user, then restart the ingester:

docker restart signoz-ingester-1

After that, the errors stop. Well, most of them. There's still a SQL bug in the "delete old agents" query that shows up sometimes.

Inside ClickHouse

The trace data lives in signoz_traces.signoz_index_v3. I queried it to make sure my test spans made it through:

SELECT count(), serviceName FROM signoz_traces.signoz_index_v3
GROUP BY serviceName

9   otel-test
18  ai-learning-agent

They were there. This is the table with 80+ columns. The key design choice: attributes are stored as Map columns, not individual columns.

attributes_string  Map(LowCardinality(String), String)
attributes_number  Map(LowCardinality(String), Float64)
attributes_bool    Map(LowCardinality(String), Bool)

You can set any key-value pair on a span and it goes into the map automatically. No schema changes needed. My Flask app sets student.name and recommendations.count — they all appear in the map.

You can query these span attributes from ClickHouse directly too. The keys live inside the map, so you access them like this:

SELECT 
    attributes_string['student.name'] AS student_name,
    attributes_number['recommendations.count'] AS rec_count,
    count() AS total_spans
FROM signoz_traces.signoz_index_v3
WHERE has(attributes_string, 'student.name')
GROUP BY student_name, rec_count
ORDER BY total_spans DESC

There are 8 materialized views that pre-compute the service dependency graphs, trace summaries, and latency distributions. One of them does a self-join on the trace table to find parent-child span relationships across different services. That's how the service map is built.

The table uses a ReplicatedMergeTree engine even with a single node. Partitioned by day. 15-day TTL. 20+ skip indices for fast lookups.

What This Looks Like in SigNoz

All this data becomes useful when you open the SigNoz UI at http://localhost:8080. Log in with your admin account, and you can see your traces as flamegraphs, build dashboards with latency metrics, set up alerts when error rates spike, and correlate logs with traces.

My Flask app's traces showed up in the Trace Explorer right after I sent the first request:

Three traces, each one a student recommendation request. I clicked into one to see the flamegraph:

The trace had four spans: the HTTP request, the Flask route handler, the LLM call, and the database lookup. The total duration was 2.1 seconds. The LLM call alone took 1.8 seconds. The database lookup was 50 milliseconds.

That tells me where to optimize. If I want to make this app faster, I look at the LLM provider, not the database. Without the trace, I would have guessed they were equally slow. This is the kind of thing I mentioned earlier about the attributes Map columns in ClickHouse — every student.name and recommendations.count attribute I set on the span ended up in attributes_string and attributes_number automatically, no schema migration needed.

The Query Builder also lets you filter by any attribute. I set up a simple dashboard showing requests by student grade level. Took about 30 seconds.

Inside PostgreSQL

52 tables. I listed them all:

users, organizations, auth_token, auth_domain, factor_password,
dashboard, dashboard_view, alertmanager_config, alerts, rules,
service_account, pipelines, ttl_setting, span_mapper, ...

The users table:

id, display_name, email, org_id, updated_at, created_at, is_root, status

One user in my deployment:

019f5f5c-2611-7683-8ae5-99e83df8c05b  [e-mail]  2026-07-14

The auth_token table stores session tokens with a rotation mechanism using access_token, refresh_token, and prev_access_token/prev_refresh_token columns. When you issue a new token, the old one moves to prev_*.

The service_account table was empty. That's the table for API keys. You create service accounts through the UI, but I couldn't get past the login page programmatically.

The API Wall

I tried to call the SigNoz API directly:

curl -X POST http://localhost:8080/api/v1/login \
  -H "Content-Type: application/json" \
  -d '{"email":"[e-mail]","password":"Admin@123Signoz!"}'

It returned the HTML page. The entire SPA. Not a JSON token.

The browser version works fine though. Open http://localhost:8080, enter your credentials, and the SigNoz dashboard loads.

Turns out SigNoz uses session-based auth for the browser. The login endpoint returns session cookies for the SPA, not a JSON token. The logs told the same story:

/api/v1/user → 401 unauthenticated
/api/v2/sessions/rotate → 400 refresh token required

So I was stuck. The service_account table in PostgreSQL was sitting there ready — that's where API keys live. But I couldn't create one through the API because I couldn't log in through the API. Circular problem.

The way around it: log into the UI, go to Settings > Service Accounts, create a token, and use it as a header:

curl http://localhost:8080/api/v1/dashboards \
  -H "signoz-access-token: <your-service-account-token>"

No session cookie needed.

What I'd Tell My Past Self

If I were doing this again, here's what I'd do differently:

1. Read casting.yaml.lock on day one. It has everything: configs, env vars, IPs. Find it in your project directory and read it before touching anything else.

2. Set the JWT secret before registering users. Add SIGNOZ_TOKENIZER_JWT_SECRET to your casting.yaml. Do it now, not after you have users in the system.

3. Wait 30 seconds after foundryctl cast. The containers need time to settle. The startup errors look scary but they're normal.

4. Restart the ingester after registration. The OpAMP errors go away.

5. Query ClickHouse directly when the UI blocks you. docker exec ... clickhouse-client works without any authentication. All the data is there.

6. Use the lock file as your source of truth. The ingester config, the ClickHouse config, the PostgreSQL config. It's all in casting.yaml.lock.

Six containers, two databases, and a missing JWT secret. Not bad for one YAML file. If you try Foundry yourself, read the lock file on day one, not day two like I did.

---

This was my entry for the Agents of SigNoz pre-event blog contest. Foundry is SigNoz's one-command deployment tool — the casting.yaml and lock file are in the repo if you want to try it.