DEV Community

Cover image for From Broken Images to Blazing Fast Traces: How I Debugged and Connected the OpenTelemetry Demo to SigNoz
MATHAVAN E
MATHAVAN E

Posted on

From Broken Images to Blazing Fast Traces: How I Debugged and Connected the OpenTelemetry Demo to SigNoz

#signoz #wemakesdev #hackathon #ai

I recently signed up for a local observability hackathon, eager to explore distributed tracing. The goal of my project was straightforward: deploy the official OpenTelemetry Astronomy Shop Demo on my machine, connect it to a local instance of SigNoz for telemetry visualization, and analyze how microservices interact.

But within minutes of running the setup, my enthusiasm hit a wall. When I opened the storefront in my browser, it looked completely broken. All the product images were missing, showing only ugly broken image placeholders. The console was filled with 403 Forbidden and 404 Not Found errors. Behind the scenes, the container logs for image-provider were screaming with backpressure and memory failure events.

Observability is a fantastic practice, but you can’t observe an application if the UI is broken and the telemetry pipeline is dropping data. In this post, I will share the step-by-step story of how I prepared my local system, set up the containers, resolved Nginx config routing gaps, solved a silent Envoy port collision, tuned OTel Collector limits, and successfully visualized the shop's trace, log, and metric data inside SigNoz.

1. Preparing the Local Environment

Before starting, I verified that my development environment was up to date and Docker was running correctly.

First, I updated the local package database:

sudo apt-get update
Enter fullscreen mode Exit fullscreen mode

Updating Packages

Next, I verified the active Docker and Docker Desktop versions to ensure containerization compatibility:

docker -v
docker desktop version
Enter fullscreen mode Exit fullscreen mode

2. Spinning Up SigNoz

With the system environment verified, I navigated to my SigNoz deployment folder to start the application performance monitoring stack:

cd ~/signoz/pours/deployment
ls
Enter fullscreen mode Exit fullscreen mode


I spun up the SigNoz backend containers in detached mode:

docker compose up -d
Enter fullscreen mode Exit fullscreen mode


I checked the state of the containers using docker ps to ensure that the ClickHouse storage, PostgreSQL metastore, SigNoz query service, and OTel ingester were healthy:

docker ps --filter "name=signoz"
Enter fullscreen mode Exit fullscreen mode


SigNoz was successfully up and listening on port 8080.

3. Starting the OpenTelemetry Demo

Next, I navigated to the OpenTelemetry Astronomy Shop Demo directory:

cd ~/opentelemetry-demo/
ls
Enter fullscreen mode Exit fullscreen mode


I started the 20+ microservices in the demo:

docker compose up -d
Enter fullscreen mode Exit fullscreen mode


To verify how the storefront was exposed, I checked the frontend container:

docker compose ps frontend
Enter fullscreen mode Exit fullscreen mode


The direct frontend port was dynamically mapped to port 34753. At this point, the application was running, but three major issues prevented a functional demo.

4. Debugging and Fixing the System

Issue A: Nginx 403 Forbidden for Image Assets

I checked if the image-provider container could serve static images locally using curl:

curl -I http://localhost:39749/Banner.png
Enter fullscreen mode Exit fullscreen mode

This returned an immediate HTTP error:

HTTP/1.1 403 Forbidden
Server: nginx
Enter fullscreen mode Exit fullscreen mode

I checked the configuration template (src/image-provider/nginx.conf.template) and saw Nginx lacked a generic location block:

    server {
        listen ${IMAGE_PROVIDER_PORT};
        listen [::]:${IMAGE_PROVIDER_PORT};

        resolver 127.0.0.11;
        autoindex off;

        server_name _;
        server_tokens off;

        root /static;
        gzip_static on;

        location /status {
            stub_status on;
            access_log  on;
            allow all;
        }
    }
Enter fullscreen mode Exit fullscreen mode

Because autoindex was off and no fallback location / route was specified, Nginx refused all direct file requests under the root /static directory.

The Fix:

I added a simple location / fallback block to route static files safely:

        location / {
            try_files $uri $uri/ =404;
        }
Enter fullscreen mode Exit fullscreen mode

I rebuilt and restarted the service:

docker compose build image-provider
docker compose up -d --force-recreate image-provider
Enter fullscreen mode Exit fullscreen mode

This resolved the 403 error.

Issue B: The Silent Envoy Port Collision

Although Nginx was serving images, the browser console showed 404 Not Found for product images because I was accessing the application on port 38411/34753 (direct React frontend).

Static resources like product images are mapped relative to the proxy (e.g., /images/Banner.png). Envoy (acting as the proxy) is configured on port 8080 to intercept /images/ and route them to Nginx.

However, SigNoz was already running on host port 8080. This conflict prevented Envoy from binding to port 8080 on the host, causing it to fail silently.

The Fix:

I remapped Envoy's host port to 8090 in compose.yaml:

  frontend-proxy:
    image: ${IMAGE_NAME}:${DEMO_VERSION}-frontend-proxy
    ...
    ports:
      - "8090:${ENVOY_PORT}" # Mapped to host port 8090
      - "${ENVOY_ADMIN_PORT}:${ENVOY_ADMIN_PORT}"
Enter fullscreen mode Exit fullscreen mode

After updating the port and restarting Envoy, accessing the site on http://localhost:8090 loaded the store correctly with all images rendering as expected!

Issue C: OTel Collector Data Drops

With the UI fixed, I noticed the following warnings looping in the image-provider logs:

image-provider  | [error] OTel export failure: data refused due to high memory usage
image-provider  | [error] OTel export failure: sending queue is full
Enter fullscreen mode Exit fullscreen mode

Checking the resources via docker stats revealed the otel-collector container was consuming 176MB of its 200MB limit.

In otel-config.yml, the collector's memory_limiter processor starts dropping metrics and span exports at 80% memory saturation (160MB) to prevent an out-of-memory container crash:

processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
Enter fullscreen mode Exit fullscreen mode

The Fix:

I increased the memory resources allocated to the collector in compose.yaml:

  otel-collector:
    image: ${COLLECTOR_CONTRIB_IMAGE}
    container_name: otel-collector
    deploy:
      resources:
        limits:
          memory: 300M # Bumped from 200M
    ...
    environment:
      - GOMEMLIMIT=240MiB # Bumped from 160MiB
Enter fullscreen mode Exit fullscreen mode

I restarted the collector:

docker compose up -d --no-deps --force-recreate otel-collector
Enter fullscreen mode Exit fullscreen mode

The collector memory stabilized around 33% (100MB / 300MB), eliminating the backpressure log errors.

5. Visualizing Telemetry in SigNoz

With the pipelines cleared, I logged into my local SigNoz instance to verify telemetry data.

Collector Metrics Dashboard

The OpenTelemetry Collector dashboard shows active spans, metrics, and logs streaming in with 0 refused or failed spans:

Log Analysis & System Metrics

I navigated to the Logs explorer in SigNoz, which successfully captured structured logs from various demo containers (e.g. ad category requests, task completions) alongside real-time container CPU and Memory metrics:

Traces Explorer

In the Traces panel, I traced exact span transactions, including direct GET /status hits from the image-provider service:

Metrics Explorer

Finally, using the Metrics panel, I plotted active time-series data for system attributes like total CPU time:

Key Lessons Learned

  1. Keep Ports Clean: Avoid hardcoding port mappings like :8080 if you plan on deploying multiple observability tools concurrently on the same host. Remap proxy routes to distinct ports.
  2. Account for Limiter Headroom: Set up appropriate resource limits. An OTel collector running on default parameters will drop client data quickly if it hits backpressure limits.
  3. Verify Route Configuration: When setting up unprivileged static image servers like Nginx, ensure index and generic location mappings are declared so request fallbacks don't trigger 403 permissions issues.

Conclusion

By troubleshooting web server route defaults, remapping port bindings, and upgrading Collector system resource limits, we transformed a failing sandbox deployment into a fully operational observability pipeline.
Now, the entire shop works smoothly, and every microservice transaction is observable in real-time. Happy tracing!

By Mathavan & Suresh Krishna — WeMakeDevs — SigNoZ Hackathon

Top comments (0)