DEV Community

Cover image for Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too
databufflabs
databufflabs

Posted on • Originally published at databuff.ai

Troubleshooting Stuck at Nginx? Instrument It with OpenTelemetry Too

#opentelemetry #nginx #devops #observability

The page is slow, APIs time out randomly, and you've already checked Java logs and slow SQL — still nothing. The gap is often the Nginx in front: it proxy-passes the request, but classic access logs won't tell you whether the hop succeeded, how long it took, or where it stalled.

OpenTelemetry turns that into "load a module, write a few lines of config." This post walks a Demo we actually ran: instrument Nginx and Java with OpenTelemetry, then open one Trace in DataBuff that spans Nginx → Java → Redis.

What OpenTelemetry is doing

OpenTelemetry stitches each service hop into one call chain. A full request is a Trace; each processing segment is a Span. Nginx uses the official module, Java uses the official agent — each records its hop, exports with the same TraceID, and the platform merges the fragments.

curl / browser
   │   GET /hello
   ▼
[ Nginx  nginx:1.27-alpine-otel ]   ngx_otel_module   → OTLP gRPC 4317
   │   reverse proxy proxy_pass
   ▼
[ Java   JDK HttpServer + Jedis ]   opentelemetry-javaagent → OTLP HTTP 4318
   │   set + get
   ▼
[ Redis  192.168.50.120:16379 ]     Jedis auto-instrumented
   ▼
[ DataBuff ]  ← gRPC 4317 + HTTP 4318, merged under one TraceID
Enter fullscreen mode Exit fullscreen mode

Hop 1 · Nginx — two install paths

The official OTel module needs a recent Nginx (1.21+). Our host still runs 1.20.1, so path A uses the official image that already ships the module; path B installs the module package on an existing Nginx and adds one load_module line. OTel directives are the same either way.

Path A — official image:

docker run -d --name nginx-otel \
  -p 8090:80 \
  --add-host host.docker.internal:host-gateway \
  -v /path/to/nginx.conf:/etc/nginx/nginx.conf:ro \
  nginx:1.27-alpine-otel
Enter fullscreen mode Exit fullscreen mode

Path B — Alpine example: apk add --repository https://nginx.org/packages/mainline/alpine/v3.21/main nginx-module-otel, then load_module /usr/lib/nginx/modules/ngx_otel_module.so; and nginx -s reload.

Config points that matter:

  • Export over gRPC host.docker.internal:4317 (HTTP 4318 fails with OTel export failure ... Socket closed)
  • otel_service_name nginx-otel-demo
  • otel_trace on

The line people miss: otel_trace_context propagate;. By default the module receives upstream context but does not inject it downstream. Without this line Nginx reports its own Span, but Java never sees the TraceID — you get two unrelated Traces. After adding propagate, they joined immediately.

otel_exporter {
    endpoint host.docker.internal:4317;   # gRPC
}
otel_service_name nginx-otel-demo;
otel_trace_context propagate;             # required: inject traceparent
Enter fullscreen mode Exit fullscreen mode

Hop 2 · Java agent, zero code changes

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=java-redis-demo \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318 \
  -cp "libs/*:classes" OtelDemoServer
Enter fullscreen mode Exit fullscreen mode

Java uses HTTP OTLP (4318), Nginx uses gRPC (4317) — two protocols, one DataBuff, one Trace later.

Hop 3 · Two exporters, one Trace

service-list

Both nginx-otel-demo and java-redis-demo show up with volume, error rate, and latency.

topology

Topology draws nginx-otel-demo → java-redis-demo → Redis.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/epov100r6v9hxqllxlv9.png

Waterfall: /hello (nginx) → GET /hello (java) → AUTH/SET/GET (redis), total 975ms, TraceID 7dff87e865b7fe975877a506db26a2ab. The Demo sleeps about one second in Java on purpose — nginx Span (975ms) and Java Span (974.75ms) rise together, so Nginx duration really covers the whole request. Redis even records AUTH as its own Span when the connection uses a password.

Don't trust the UI alone — check Doris

SELECT service, name, span_id, parent_id, is_parent, duration
FROM trace_dc_span
WHERE trace_id='7dff87e865b7fe975877a506db26a2ab'
ORDER BY startTime;
Enter fullscreen mode Exit fullscreen mode 
nginx-otel-demo  /hello      (empty, root)  975000000
java-redis-demo  GET /hello  <nginx span>   974745285
[redis]…:16379   AUTH        <java span>     638593
[redis]…:16379   SET         <java span>     361212
[redis]…:16379   GET         <java span>     164941
Enter fullscreen mode Exit fullscreen mode

trace-list

Filter the trace list by nginx / java when you're hunting edge issues.

Try it tonight — three steps

  1. docker run nginx:1.27-alpine-otel with otel_trace_context propagate;, expose http://<host>:8090/hello.
  2. Start Java with -javaagent:opentelemetry-javaagent.jar pointing at http://127.0.0.1:4318.
  3. curl http://192.168.50.140:8090/hello a few times, wait ~30s for batch export, open DataBuff Traces, filter nginx-otel-demo.

login-page

Self-check: nginx root Span present; Java under nginx (not a sibling Trace); Redis SET/GET under Java. If you see two Traces, you almost certainly forgot propagate.

Fifteen minutes on the Nginx module and the Java agent turns the first hop from a blind spot into a span you can scroll.

DataBuff

Open-source AI-native OpenTelemetry APM — metrics, traces, logs and AI troubleshooting in one platform.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai

Top comments (0)