DEV Community

Cover image for Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too
databufflabs
databufflabs

Posted on Originally published at databuff.ai

Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too

#opentelemetry #nginx #devops #observability

The page is slow, APIs time out randomly, and you've already checked Java logs and slow SQL — still nothing. The gap is often the Nginx in front: it proxy-passes the request, but classic access logs won't tell you whether the hop succeeded, how long it took, or where it stalled.

OpenTelemetry turns that into "load a module, write a few lines of config." This post walks a Demo we actually ran: instrument Nginx and Java with OpenTelemetry, then open one Trace in DataBuff that spans Nginx → Java → Redis.

What OpenTelemetry is doing

OpenTelemetry stitches each service hop into one call chain. A full request is a Trace; each processing segment is a Span. Nginx uses the official module, Java uses the official agent — each records its hop, exports with the same TraceID, and the platform merges the fragments.

curl / browser
   │   GET /hello
   ▼
[ Nginx  nginx:1.27-alpine-otel ]   ngx_otel_module   → OTLP gRPC 4317
   │   reverse proxy proxy_pass
   ▼
[ Java   JDK HttpServer + Jedis ]   opentelemetry-javaagent → OTLP HTTP 4318
   │   set + get
   ▼
[ Redis  192.168.50.120:16379 ]     Jedis auto-instrumented
   ▼
[ DataBuff ]  ← gRPC 4317 + HTTP 4318, merged under one TraceID
Enter fullscreen mode Exit fullscreen mode

Hop 1 · Nginx — two install paths

The official OTel module needs a recent Nginx (1.21+). Our host still runs 1.20.1, so path A uses the official image that already ships the module; path B installs the module package on an existing Nginx and adds one load_module line. OTel directives are the same either way.

Path A — official image:

docker run -d --name nginx-otel \
  -p 8090:80 \
  --add-host host.docker.internal:host-gateway \
  -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro \
  nginx:1.27-alpine-otel
Enter fullscreen mode Exit fullscreen mode

Path B — Alpine example: apk add --repository https://nginx.org/packages/mainline/alpine/v3.21/main nginx-module-otel, then load_module /usr/lib/nginx/modules/ngx_otel_module.so; and nginx -s reload.

Config points that matter:

  • Export over gRPC host.docker.internal:4317 (HTTP 4318 fails with OTel export failure ... Socket closed)
  • otel_service_name nginx-otel-demo
  • otel_trace on

The line people miss: otel_trace_context propagate;. By default the module receives upstream context but does not inject it downstream. Without this line Nginx reports its own Span, but Java never sees the TraceID — you get two unrelated Traces. After adding propagate, they joined immediately.

otel_exporter {
    endpoint host.docker.internal:4317;   # gRPC
}
otel_service_name nginx-otel-demo;
otel_trace_context propagate;             # required: inject traceparent
Enter fullscreen mode Exit fullscreen mode

Hop 2 · Java agent, zero code changes

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=java-redis-demo \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318 \
  -cp "libs/*:classes" OtelDemoServer
Enter fullscreen mode Exit fullscreen mode

Java uses HTTP OTLP (4318), Nginx uses gRPC (4317) — two protocols, one DataBuff, one Trace later.

Hop 3 · Two exporters, one Trace

service-list

Both nginx-otel-demo and java-redis-demo show up with volume, error rate, and latency.

topology

Topology draws nginx-otel-demo → java-redis-demo → Redis.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/epov100r6v9hxqllxlv9.png

Waterfall: /hello (nginx) → GET /hello (java) → AUTH/SET/GET (redis), total 975ms, TraceID 7dff87e865b7fe975877a506db26a2ab. The Demo sleeps about one second in Java on purpose — nginx Span (975ms) and Java Span (974.75ms) rise together, so Nginx duration really covers the whole request. Redis even records AUTH as its own Span when the connection uses a password.

Don't trust the UI alone — check Doris

SELECT service, name, span_id, parent_id, is_parent, duration
FROM trace_dc_span
WHERE trace_id='7dff87e865b7fe975877a506db26a2ab'
ORDER BY startTime;
Enter fullscreen mode Exit fullscreen mode 
nginx-otel-demo  /hello      (empty, root)  975000000
java-redis-demo  GET /hello  <nginx span>   974745285
[redis]…:16379   AUTH        <java span>     638593
[redis]…:16379   SET         <java span>     361212
[redis]…:16379   GET         <java span>     164941
Enter fullscreen mode Exit fullscreen mode

trace-list

Filter the trace list by nginx / java when you're hunting edge issues.

Try it tonight — three steps

  1. docker run nginx:1.27-alpine-otel with otel_trace_context propagate;, expose http://<host>:8090/hello.
  2. Start Java with -javaagent:opentelemetry-javaagent.jar pointing at http://127.0.0.1:4318.
  3. curl http://192.168.50.140:8090/hello a few times, wait ~30s for batch export, open DataBuff Traces, filter nginx-otel-demo.

login-page

Self-check: nginx root Span present; Java under nginx (not a sibling Trace); Redis SET/GET under Java. If you see two Traces, you almost certainly forgot propagate.

Fifteen minutes on the Nginx module and the Java agent turns the first hop from a blind spot into a span you can scroll.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

Top comments (0)