How I Cut Authz p99 Latency from 4.2s to 120ms Using OpenTelemetry

It was 3:14 AM on a Thursday. PagerDuty was firing alerts about API timeouts. My Grafana dashboards showed CPU at 30%, memory stable, no database swapping, no network saturation. But p99 latency on the core authentication flow had jumped from 120ms to 4.2 seconds.

I had no visibility into why.

Grepping Kibana for a session ID returned logs scattered across fifteen pods with no causal links:

[2026-07-18T03:15:12Z] INFO [auth-svc] msg="Incoming login request" user_id="u_9812a"
[2026-07-18T03:15:13Z] DEBUG [user-svc] msg="Fetching user metadata" user_id="u_9812a"
[2026-07-18T03:15:16Z] ERROR [auth-svc] msg="context deadline exceeded" component="grpc_client"

I spent four hours blaming PostgreSQL—locked tables, missing indexes. I added logger.Info() statements throughout the codebase and tried reconstructing execution timelines by subtracting timestamps in a spreadsheet:

startTime := time.Now()
logger.Info("Starting role resolution", zap.String("user_id", req.UserID))

roles, err := resolveUserRoles(ctx, req.UserID)
if err != nil {
    logger.Error("Failed to resolve roles", zap.Error(err))
    return nil, err
}

logger.Info("Finished role resolution", 
    zap.String("user_id", req.UserID), 
    zap.Duration("elapsed", time.Since(startTime)),
)

In a highly concurrent system, sequential log lines don't show network boundaries or where blocking I/O actually happens. I needed distributed tracing.

---

Setting Up OpenTelemetry and SigNoz

I deployed the OTel Collector as a DaemonSet, routing telemetry to a self-hosted SigNoz instance. I used OTLP over gRPC instead of HTTP to reduce serialization overhead, and configured batch processors to prevent telemetry from saturating the application's network:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 8192
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024

exporters:
  otlp:
    endpoint: "signoz-otel-collector:4317"
    tls:
      insecure: true

The problem was at the application layer. My first deployment broke completely—traces started at the API gateway and terminated immediately. Downstream services generated their own disconnected traces.

I'd missed injecting the OTel trace context into gRPC metadata.

Here's the initialization code that fixed it:

func initProvider(serviceName string) (*trace.TracerProvider, error) {
    ctx := context.Background()

    traceExporter, err := otlptracegrpc.New(ctx, 
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        // TraceIDRatioBased(0.1) samples 10% of traffic. At 10,000 req/sec,
        // AlwaysSample() would overwhelm the observability backend.
        trace.WithSampler(trace.TraceIDRatioBased(0.1)),
        trace.WithBatcher(traceExporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )

    // This propagates trace IDs via W3C TraceContext. Without it, gRPC calls lose the parent span ID.
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp, nil
}

I added business context to spans via custom attributes:

func resolveUserRoles(ctx context.Context, userID string, tier string) ([]Role, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "resolveUserRoles")
    defer span.End()

    span.SetAttributes(
        attribute.String("tenant_id", userID),
        attribute.String("user_tier", tier),
    )

    // business logic...
}

---

Finding the Bottleneck

I ran 1,000 concurrent users against staging. Within three minutes, p99 latency hit 4.2 seconds—matching production exactly.

The SigNoz trace flamegraph showed the problem immediately. Under a single parent span, fifty identical sequential db.Query spans cascaded down, each taking 80ms.

I had an N+1 query problem hidden behind a clean gRPC interface.

When a user logged in, the system fetched their assigned groups. Then, for each group, it made a synchronous blocking database call to resolve RBAC policies:

func (s *AuthzServer) GetUserDetails(ctx context.Context, req *pb.UserReq) (*pb.UserResp, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "GetUserDetails")
    defer span.End()

    groups := fetchUserGroups(ctx, req.UserID)
    var policies []Policy

    for _, group := range groups {
        policy, err := s.db.QueryRowContext(ctx, "SELECT * FROM policies WHERE group_id = $1", group.ID)
        if err != nil {
            return nil, err
        }
        policies = append(policies, policy)
    }

    return buildResponse(policies), nil
}

At low traffic, 5ms database latency was imperceptible. Under high concurrency, connection pool exhaustion and network round-trips compounded into a 4.2-second bottleneck.

---

The Fix

I replaced the sequential loop with a single bulk query using PostgreSQL's ANY operator:

func (s *AuthzServer) GetUserDetails(ctx context.Context, req *pb.UserReq) (*pb.UserResp, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "GetUserDetails")
    defer span.End()

    groups := fetchUserGroups(ctx, req.UserID)
    groupIDs := extractIDs(groups)

    query := "SELECT * FROM policies WHERE group_id = ANY($1)"
    rows, err := s.db.QueryContext(ctx, query, pq.Array(groupIDs))
    if err != nil {
        span.RecordError(err)
        return nil, err
    }
    defer rows.Close()

    var policies []Policy
    // scan rows...

    return buildResponse(policies), nil
}

Staging's p99 dropped from 4.2 seconds back to 120ms immediately after deployment. Connection pool saturation flattened. Error rates returned to baseline.

I configured a SigNoz alert to fire if the resolveUserRoles span exceeded 500ms for more than five minutes, tied directly to span duration instead of generic CPU thresholds.

---

What Changed

Before this, I treated observability as an operational afterthought. After, I started building with distributed tracing in mind from the start. I stopped guessing and started measuring.

The logs-only approach failed because I couldn't reconstruct distributed execution paths across network boundaries. Metrics told me something was broken. Traces showed me exactly where.

​