DEV Community

Anh Hai Newton
Anh Hai Newton

Posted on

SigNoz Setup Guide - Monitor Server

#signoz #devops

Environment

  • OS: Ubuntu (snap docker)
  • Docker path: /var/snap/docker/common/var-lib-docker/
  • SigNoz path: /root/signoz/
  • SigNoz version: v0.117.1
  • ClickHouse version: 25.5.6
  • CPU: No AVX2 support (important — affects configuration)

1. Install SigNoz 

cd /root
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/docker
Enter fullscreen mode Exit fullscreen mode

2. Fix ClickHouse — Disable simdjson (REQUIRED for CPUs without AVX2)

CPUs without AVX2 will encounter CANNOT_ALLOCATE_MEMORY errors when SigNoz queries metrics via JSONExtractString.

Check if your CPU supports AVX2:

grep -o 'avx2' /proc/cpuinfo | head -1
# No output = no AVX2 support → apply the fix below
Enter fullscreen mode Exit fullscreen mode

Edit /root/signoz/deploy/common/clickhouse/users.xml and add <allow_simdjson>0</allow_simdjson> to the default profile:

<profiles>
    <default>
        <max_memory_usage>10000000000</max_memory_usage>
        <allow_simdjson>0</allow_simdjson>   <!-- Fix for CPUs without AVX2 -->
        <load_balancing>random</load_balancing>
    </default>
    ...
</profiles>
Enter fullscreen mode Exit fullscreen mode

Why this happens

ClickHouse uses the simdjson library for JSON parsing functions (JSONExtractString, etc.). Starting from ClickHouse 24.1+, simdjson requires AVX2 CPU instructions. On CPUs without AVX2, any query using JSONExtractString on the distributed_time_series_v4 table throws:

Code: 173. DB::Exception: Couldn't allocate N bytes when parsing JSON
Enter fullscreen mode Exit fullscreen mode

Setting allow_simdjson=0 forces ClickHouse to fall back to rapidjson, which does not require AVX2.

This affects:

  • Infrastructure → Hosts tab (returns 500 error)
  • Any dashboard querying metrics with label filters

3. Disable ClickHouse System Logs (REQUIRED — prevent disk exhaustion)

By default, ClickHouse writes internal diagnostic logs (trace_log, metric_log, etc.) that can grow to 70GB+ over time.

Create /root/signoz/deploy/common/clickhouse/z_log_disable.xml:

<?xml version="1.0"?>
<clickhouse>
    <asynchronous_metric_log remove="1"/>
    <trace_log remove="1"/>
    <metric_log remove="1"/>
    <query_views_log remove="1"/>
    <query_thread_log remove="1"/>
    <part_log remove="1"/>
    <processors_profile_log remove="1"/>
    <opentelemetry_span_log remove="1"/>
</clickhouse>
Enter fullscreen mode Exit fullscreen mode

The z_ prefix ensures this file is loaded last (configs are applied alphabetically), overriding any previous settings.

Mount this file into the ClickHouse container in /root/signoz/deploy/docker/docker-compose.yaml:

  clickhouse:
    volumes:
      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
      - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      - ../common/clickhouse/ttl.xml:/etc/clickhouse-server/config.d/ttl.xml
      - ../common/clickhouse/z_log_disable.xml:/etc/clickhouse-server/config.d/z_log_disable.xml  # ADD THIS LINE
      - clickhouse:/var/lib/clickhouse/
Enter fullscreen mode Exit fullscreen mode

4. Configure SMTP for Email Alerts

Add the following environment variables to the signoz service in docker-compose.yaml:

  signoz:
    environment:
      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
      # SMTP configuration
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=your-email@gmail.com
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=smtp.gmail.com:587
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=your-email@gmail.com
      - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=your-16-char-app-password
Enter fullscreen mode Exit fullscreen mode

Gmail note: You must use an App Password, not your regular Gmail password.
Generate one at: https://myaccount.google.com/apppasswords
The App Password is 16 characters — remove all spaces before using it.

Important: The correct variable names use the SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__ prefix with double underscores. Using SIGNOZ_ALERTMANAGER_SMTP_* (without SIGNOZ_GLOBAL) will not work.

5. Start SigNoz 

cd /root/signoz/deploy/docker
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Verify all containers are healthy:

docker compose ps
Enter fullscreen mode Exit fullscreen mode

Verify ClickHouse system logs are disabled:

docker exec signoz-clickhouse clickhouse-client --query "
SELECT name FROM system.tables
WHERE database='system' AND name LIKE '%log%'
ORDER BY name"
# Only query_log should remain
Enter fullscreen mode Exit fullscreen mode

6. Setup Host Metrics Monitoring (Infrastructure Tab)

Run an otel-collector agent on the main application server:

6.1 Create config file on the main server 

mkdir -p /opt/otel-agent
cat > /opt/otel-agent/config.yaml << 'EOF'
receivers:
  hostmetrics:
    root_path: /hostfs
    collection_interval: 60s
    scrapers:
      cpu: {}
      disk: {}
      load: {}
      filesystem:
        exclude_mount_points:
          mount_points: [/snap/*, /boot/efi]
          match_type: regexp
      memory: {}
      network: {}
      paging: {}
      processes: {}

processors:
  batch:
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  resource/env:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp:
    endpoint: "<SIGNOZ_MONITOR_SERVER_IP>:4317"
    tls:
      insecure: true

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/env, batch]
      exporters: [otlp]
EOF
Enter fullscreen mode Exit fullscreen mode

Replace <SIGNOZ_MONITOR_SERVER_IP> with your SigNoz monitor server IP.

6.2 Run the container 

docker run -d \
  --name otel-agent \
  --restart unless-stopped \
  --hostname navio-main-server \
  -v /opt/otel-agent/config.yaml:/etc/otelcol-contrib/config.yaml \
  -v /:/hostfs:ro \
  -e HOST_PROC=/hostfs/proc \
  -e HOST_SYS=/hostfs/sys \
  -e HOST_ETC=/hostfs/etc \
  -e HOST_VAR=/hostfs/var \
  -e HOST_RUN=/hostfs/run \
  -e HOST_DEV=/hostfs/dev \
  otel/opentelemetry-collector-contrib:latest
Enter fullscreen mode Exit fullscreen mode

After a few minutes, the host should appear in SigNoz UI → Infrastructure → Hosts.

7. Create Disk Usage Alert (> 80%)

Go to SigNoz UI → Alerts → New Alert → Metric Based Alert:

Query A (used disk):

  • Metric: system.filesystem.usage
  • Filter: state = used AND host.name = <your-hostname>
  • Time aggregation: avg, Space aggregation: sum

Query B (total disk — all states):

  • Metric: system.filesystem.usage
  • Filter: host.name = <your-hostname>
  • Time aggregation: avg, Space aggregation: sum

Formula F1: A / B * 100

Condition:

  • Warning: above 80
  • Critical: above 90
  • Evaluation window: last 5 min

Annotations:

Alert: Disk usage has exceeded {{$threshold}}% (current: {{$value}}%)
Enter fullscreen mode Exit fullscreen mode

8. Routine Maintenance 

# Check disk usage on monitor server
df -h /

# Verify ClickHouse system logs are not growing
docker exec signoz-clickhouse clickhouse-client --query "
SELECT table, formatReadableSize(sum(bytes_on_disk)) as size
FROM system.parts
WHERE database = 'system'
GROUP BY table
ORDER BY sum(bytes_on_disk) DESC"

# Clean up unused Docker images on main server
docker image prune -af

# Check Docker disk usage
docker system df
Enter fullscreen mode Exit fullscreen mode

9. Troubleshooting

ClickHouse disk full due to system logs 

# Enter clickhouse-client
docker exec -it signoz-clickhouse clickhouse-client

# Create force drop flag (run in a separate terminal)
docker exec signoz-clickhouse bash -c \
  "touch /var/lib/clickhouse/flags/force_drop_table && \
   chmod 666 /var/lib/clickhouse/flags/force_drop_table"

# Truncate all large log tables
TRUNCATE TABLE system.trace_log;
TRUNCATE TABLE system.metric_log;
TRUNCATE TABLE system.query_log;
TRUNCATE TABLE system.processors_profile_log;
TRUNCATE TABLE system.part_log;
TRUNCATE TABLE system.asynchronous_metric_log;
TRUNCATE TABLE system.query_views_log;
Enter fullscreen mode Exit fullscreen mode

Infrastructure Hosts tab returns 500 error

Root cause: CPU does not support AVX2 instructions.

# Check AVX2 support
grep -o 'avx2' /proc/cpuinfo | head -1
# No output = no AVX2
Enter fullscreen mode Exit fullscreen mode

Fix: Add <allow_simdjson>0</allow_simdjson> to ClickHouse users.xml as described in Section 2.

otel-agent not sending data 

docker logs otel-agent 2>&1 | tail -20
# Check that host.name is detected correctly
# Check that the endpoint IP is correct
Enter fullscreen mode Exit fullscreen mode

Top comments (0)