DEV Community

Ifeoluwa Afolabi (Afoxcute)
Ifeoluwa Afolabi (Afoxcute)

Posted on

I Self-Hosted SigNoz on AWS EC2 From Scratch — Every Wall I Hit and How I Fixed It

#signoz #opentelemetry #observability #devops

The WeMakeDevs "Agents of SigNoz" hackathon said: self-host SigNoz, instrument a real app, send data. Simple enough on paper. Two hours and seven distinct failure modes later, I had traces flowing. Here's everything that happened in the order it happened.

What I Set Out to Do

  1. Self-host the full SigNoz stack on a raw AWS EC2 Ubuntu instance using the Linux binary install
  2. Deploy a real Node.js Express backend with OpenTelemetry auto-instrumentation
  3. Generate real traffic — GETs, POSTs, 404s, 400s, intentional errors, slow requests
  4. Watch all 65 requests appear as traces in SigNoz

The machine: Ubuntu 22.04, t3.medium, ip-172-**-**-*** on AWS EC2.

The stack under the hood:

Component Role
ClickHouse Columnar database storing all traces, logs, and metrics
ZooKeeper Coordination layer ClickHouse needs for distributed operation
ClickHouse Keeper ZooKeeper-compatible alternative built into ClickHouse
SigNoz The observability application layer
SigNoz OTel Collector The OpenTelemetry ingestion pipeline
Node.js 20 + Express The demo backend being observed

Part 1: Installing the Stack

Wall 1 — ZooKeeper Download Returned 196 Bytes

The SigNoz docs say to run:

bash
curl -L https://dlcdn.apache.org/zookeeper/zookeeper-3.8.5/apache-zookeeper-3.8.5-bin.tar.gz -o zookeeper.tar.gz
tar -xzf zookeeper.tar.gz
Enter fullscreen mode Exit fullscreen mode

What I got:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Enter fullscreen mode Exit fullscreen mode

The Apache CDN returned an HTML redirect page of 196 bytes instead of the binary. Version 3.8.5 had moved off the active mirror. Fix:

wget https://archive.apache.org/dist/zookeeper/zookeeper-3.8.5/apache-zookeeper-3.8.5-bin.tar.gz -O zookeeper.tar.gz
Enter fullscreen mode Exit fullscreen mode

Always verify before extracting:

# Must say: gzip compressed data and NOT "HTML document"
file zookeeper.tar.gz

# Must be ~12-15MB and NOT 196 bytes
ls -lh zookeeper.tar.gz
Enter fullscreen mode Exit fullscreen mode

Wall 2 — Permission Errors Setting Up ZooKeeper

Running the manual setup commands without sudo caused a cascade of permission failures:

cp: cannot create regular file '/opt/zookeeper/conf/zoo.cfg': Permission denied
sed: can't read /opt/zookeeper/conf/zoo.cfg: No such file or directory
FAILED TO WRITE PID
Enter fullscreen mode Exit fullscreen mode

The correct sequence is using sudo bash -c to write configs and creating the ZooKeeper user before handing over ownership:

sudo mkdir -p /opt/zookeeper /var/lib/zookeeper /var/log/zookeeper
sudo cp -r apache-zookeeper-3.8.5-bin/* /opt/zookeeper

sudo bash -c 'cat <<EOF > /opt/zookeeper/conf/zoo.cfg
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
admin.serverPort=3181
EOF'

sudo getent passwd zookeeper >/dev/null || \
  sudo useradd --system --home /opt/zookeeper --no-create-home --user-group --shell /sbin/nologin zookeeper

sudo chown -R zookeeper:zookeeper /opt/zookeeper
sudo chown -R zookeeper:zookeeper /var/lib/zookeeper
sudo chown -R zookeeper:zookeeper /var/log/zookeeper
Enter fullscreen mode Exit fullscreen mode

Wall 3 — ClickHouse Was Running But Not via systemd

After installing ClickHouse, the service had been started manually rather than through systemd. This caused:

Failed to start clickhouse-server.service: Unit clickhouse-server.service not found.
Enter fullscreen mode Exit fullscreen mode

But ps aux | grep clickhouse showed it was actually running (PID 2338). The systemd unit file either didn't exist or pointed to the wrong binary path (status=203/EXEC).

The fix was creating a proper service file pointing to the correct binary and creating the missing runtime directory:

sudo mkdir -p /run/clickhouse-server
sudo chown clickhouse:clickhouse /run/clickhouse-server
sudo chown -R clickhouse:clickhouse /var/lib/clickhouse
Enter fullscreen mode Exit fullscreen mode

Then create the systemd unit file:

sudo bash -c 'cat <<EOF > /etc/systemd/system/clickhouse-server.service
[Unit]
Description=ClickHouse Server
After=network-online.target clickhouse-keeper.service

[Service]
Type=simple
User=clickhouse
Group=clickhouse
Restart=on-failure
RuntimeDirectory=clickhouse-server
ExecStart=/usr/bin/clickhouse server \
  --config=/etc/clickhouse-server/config.xml \
  --pid-file=/run/clickhouse-server/clickhouse-server.pid
LimitNOFILE=500000
LimitNPROC=500000
LimitCORE=infinity
TasksMax=infinity
OOMScoreAdjust=-1000

[Install]
WantedBy=multi-user.target
EOF'

sudo systemctl daemon-reload
sudo systemctl start clickhouse-server
Enter fullscreen mode Exit fullscreen mode

Wall 4 — Multiple Authentication Methods Conflict

Setting the ClickHouse password should be simple, create a file in users.d/ with a SHA256 hash. But ClickHouse 26.x is strict: if users.xml already defines an auth method (even an empty password), and your users.d/ file adds another, it refuses to start:

Cannot specify multiple authentication methods for user default.
Specify only one authentication method.
Enter fullscreen mode Exit fullscreen mode

The fix is to explicitly remove the existing password before adding the new one using the remove="1" attribute:

sudo bash -c 'cat <<EOF > /etc/clickhouse-server/users.d/default-password.xml
<clickhouse>
    <users>
        <default>
            <password remove="1"/>
            <password_sha256_hex>5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8</password_sha256_hex>
        </default>
    </users>
</clickhouse>
EOF'

sudo chown clickhouse:clickhouse /etc/clickhouse-server/users.d/default-password.xml
sudo chmod 640 /etc/clickhouse-server/users.d/default-password.xml
sudo systemctl restart clickhouse-server
sleep 5

# Confirm it works
clickhouse-client --password password --query "SELECT 1"
# Expected output: 1
Enter fullscreen mode Exit fullscreen mode

The hash 5e884898... is the SHA256 of the string password. Verify it yourself:

echo -n "password" | sha256sum
Enter fullscreen mode Exit fullscreen mode

Important: The password appears in four places. All four must match or something will silently fail:

  1. ClickHouse users.d/default-password.xml
  2. Migration DSN: tcp://localhost:9000?password=password
  3. SigNoz systemd.env: SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://localhost:9000?password=password
  4. OTel Collector config.yaml and all four exporter DSNs

Wall 5 — ClickHouse Keeper vs ZooKeeper

Here's something the docs gloss over: ClickHouse ships with ClickHouse Keeper which is a ZooKeeper-compatible replacement built in. It was already running on my machine:

● clickhouse-keeper.service
   Active: active (running) since Sun 2026-07-19 04:10
   Main PID: 3295 (/usr/bin/clickhouse-keeper)
Enter fullscreen mode Exit fullscreen mode

On a single node, you don't need both. ClickHouse Keeper uses the same port 2181 as ZooKeeper, so the cluster.xml config works with either. This is one less process to install and debug.

What the Migration Commands Actually Do

Once ClickHouse was running and the password was correct, the three migration commands ran clean:

ARCH=$(uname -m | sed 's/x86_64/amd64/g' | sed 's/aarch64/arm64/g')

# Step 1 — Creates the databases
./signoz-otel-collector_linux_${ARCH}/bin/signoz-otel-collector migrate bootstrap \
  --clickhouse-dsn="tcp://localhost:9000?password=password" \
  --clickhouse-replication=true

# Step 2 — Creates tables and indexes (synchronous, wait for prompt)
./signoz-otel-collector_linux_${ARCH}/bin/signoz-otel-collector migrate sync up \
  --clickhouse-dsn="tcp://localhost:9000?password=password" \
  --clickhouse-replication=true

# Step 3 — Runs background data migrations (async, takes longest)
./signoz-otel-collector_linux_${ARCH}/bin/signoz-otel-collector migrate async up \
  --clickhouse-dsn="tcp://localhost:9000?password=password" \
  --clickhouse-replication=true
Enter fullscreen mode Exit fullscreen mode

Run each one and wait for the $ prompt to return before running the next. Migrations are successful when you see only info level log lines with no error lines.

SigNoz Health Check: Green

After migrations, installing SigNoz and the OTel Collector, and starting all services:

curl -X GET http://localhost:8080/api/v1/health
# Expected: {"status":"ok"}
Enter fullscreen mode Exit fullscreen mode




Screenshot 1 — Terminal showing the health check response and all three systemd services active and running

Part 2: Wiring a Real Backend to SigNoz

Installing SigNoz is one thing. Sending real telemetry to it is where it gets interesting.

The App

A Node.js Express backend with routes that deliberately generate different kinds of traffic:

GET  /health           → health check (instant)
GET  /api/users        → returns all users (50ms delay)
GET  /api/users/:id    → single user, 404 if not found
POST /api/users        → create user, 400 if fields missing
GET  /api/orders       → returns all orders (100ms delay)
POST /api/orders       → create order
GET  /api/error        → intentional 500 error
GET  /api/slow?ms=N    → artificial delay, tests latency traces
Enter fullscreen mode Exit fullscreen mode

The slow and error routes exist specifically to generate interesting data in SigNoz latency distributions and error traces are only useful if you have something to look at.

OpenTelemetry Instrumentation

The key file is tracing.js, loaded before everything else via --require:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');

const OTLP_ENDPOINT = `${process.env.SIGNOZ_HOST || 'http://localhost'}:4318`;

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: `${OTLP_ENDPOINT}/v1/traces`,
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${OTLP_ENDPOINT}/v1/metrics`,
    }),
    exportIntervalMillis: 10000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
Enter fullscreen mode Exit fullscreen mode

The start script in package.json:

{
  "scripts": {
    "start": "node --require ./tracing.js server.js"
  }
}
Enter fullscreen mode Exit fullscreen mode

Zero changes to server.js OpenTelemetry instruments Express, HTTP, and all outgoing requests automatically. The --require flag guarantees tracing initializes before any other module loads. If you import it inside server.js instead, you may miss early spans.

Running as a systemd Service 

sudo bash -c 'cat <<EOF > /etc/systemd/system/demo-backend.service
[Unit]
Description=Demo Backend App
After=signoz-otel-collector.service

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/demo-backend
EnvironmentFile=/home/ubuntu/demo-backend/.env
ExecStart=/usr/bin/node --require ./tracing.js server.js
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF'

sudo systemctl daemon-reload
sudo systemctl start demo-backend
Enter fullscreen mode Exit fullscreen mode

Part 3: Generating 65 Real Requests

A load test script ran 5 rounds hitting every route healthy requests, 404s, 400s, intentional errors, and slow responses:

🔄 Round 5/5
[GET]  /health           → 200
[GET]  /api/users        → 200
[GET]  /api/users/1      → 200
[GET]  /api/users/2      → 200
[GET]  /api/users/999    → 404
[POST] /api/users        → 201
[POST] /api/users        → 400
[GET]  /api/orders       → 200
[GET]  /api/orders/1     → 200
[GET]  /api/orders/999   → 404
[POST] /api/orders       → 201
[GET]  /api/error        → 500
[GET]  /api/slow?ms=1500 → 200

──────────────────────────────────────
✅ Load test completed!
📊 Total requests : 65
✅ Successful     : 45
❌ Errors         : 20
Enter fullscreen mode Exit fullscreen mode

20 out of 65 requests are errors which is a mix of 404s, 400s, and intentional 500s. This is intentional: SigNoz's error rate and latency charts are only meaningful when you have both healthy and unhealthy traffic flowing.

Part 4: What SigNoz Showed

Of everything I saw, the Traces Explorer is the feature I'd tell someone about first. Before this, debugging a 500 error meant checking server logs, adding more logging, redeploying, reproducing the error, checking again. With the Traces Explorer, I clicked the 500 trace from /api/error, and the full span tree opened. I saw request in, handler hit, exception thrown, response out with the exact error message sitting in the attributes panel. No log file. No grep. No redeploy. The entire sequence of what happened was just there. That single click changed how I think about debugging.

Services View

demo-backend appears with request rate, error rate, and p99 latency. The /api/slow route pushed p99 latency above 1500ms, immediately visible as a red flag on the service overview.


Screenshot 2 — SigNoz Services page showing demo-backend with p99 latency, error rate, and RPS

Traces Explorer

Every one of the 65 requests appears as an individual trace. Each trace shows the full span: incoming HTTP request → Express route handler → response time.


Screenshot 3 — Traces Explorer filtered to demo-backend showing a mix of 200 and 500/404 status codes

Error Trace Lists

Clicking on one of the 500 trace from /api/error will show the exception message inline with the full span tree.


Screenshot 4 — All 500 traces for /api/error showing the errors span in red

Latency Distribution

The /api/slow?ms=1500 traces cluster visibly at the 1.5 second mark in the latency histogram, completely separate from the healthy sub-100ms requests.


Screenshot 5 — Latency histogram for demo-backend showing the p99 spike from the slow endpoint

Query Builder

Filtering by service.name = demo-backend and http.status_code = 500 returns exactly the 5 error traces, one per round:


Screenshot 6 — Query Builder filtered to demo-backend and status 500 returning exactly 5 error traces

Alert Setup

With real error data flowing, I set up an alert in under 3 minutes:

  1. Alerts → New Alert → Trace-based Alert
  2. Query: service.name = 'demo-backend' with count()
  3. Condition: ABOVE threshold, AT LEAST ONCE, during Last 5 minutes
  4. Severity: critical, value > 0
  5. Notification message: High error rate detected on demo-backend. Error count exceeded threshold. Check /api/error traces in SigNoz.




Screenshot 7 — Alert rule page showing the configured trace-based alert with query, threshold, and notification message

What I'd Tell You Before You Start

Set your ClickHouse password before anything else. It appears in four places: the ClickHouse user config, the SigNoz systemd env file, the OTel Collector config, and the migration DSN. All four must match. Decide your password before you install anything.

Use <password remove="1"/> when setting ClickHouse passwords via config file. Without it, ClickHouse 26.x will crash on startup complaining about multiple auth methods, even if the existing password is empty.

Use ClickHouse Keeper instead of standalone ZooKeeper. It's included with ClickHouse, uses the same protocol and port, and is one less process to install and debug.

Verify every download before extracting. file yourfile.tar.gz takes two seconds and saves you a confusing error that looks like a tar bug but is actually an HTML redirect page.

Create the runtime directory before starting ClickHouse. /run/clickhouse-server/ must exist and be owned by the clickhouse user or the server won't start and the error message (CANNOT_OPEN_FILE for the PID file) doesn't make this obvious.

Open your EC2 security group ports early. You need: 8080 (SigNoz UI), 4317/4318 (OTLP), 9000 (ClickHouse), 2181 (Keeper/ZooKeeper), 5000 (your app). I spent 20 minutes debugging a connectivity issue that was just a missing inbound rule.

Load tracing.js with --require, not import. OpenTelemetry must initialize before any other module loads. node --require ./tracing.js server.js guarantees this.

Run each migration command and wait for the $ prompt before running the next one. The async migration in particular can take 1–2 minutes.

Final Thoughts

Self-hosting SigNoz on Linux is worth doing if you want to understand what observability infrastructure actually is and not just what it looks like behind a cloud dashboard. Running 65 real requests through a real backend and watching every one appear as a searchable, filterable trace, with latency distributions and error details attached, makes the value of OpenTelemetry concrete in a way that reading docs never does.

The walls I hit stale mirror URLs, permission errors, password conflicts, missing runtime directories, wrong binary paths were all fixable. None of them were fundamental flaws in SigNoz. They were the kind of friction that only shows up when you go off the happy path of Docker Compose and actually install a distributed system from scratch.

If you're joining the Agents of SigNoz hackathon and want a foundation you understand from the ground up, the binary install is the way to get there.

Self-hosted on AWS EC2 Ubuntu 22.04 · ClickHouse 26.6.1 · SigNoz OTel Collector latest · Node.js 20 · July 2026

Part of the WeMakeDevs "Agents of SigNoz" hackathon — signoz.io · wemakedevs.org

Top comments (0)