DEV Community: Tanmay Hathile

How I Cut Authz p99 Latency from 4.2s to 120ms Using OpenTelemetry

Tanmay Hathile — Sun, 19 Jul 2026 05:33:44 +0000

It was 3:14 AM on a Thursday. PagerDuty was firing alerts about API timeouts. My Grafana dashboards showed CPU at 30%, memory stable, no database swapping, no network saturation. But p99 latency on the core authentication flow had jumped from 120ms to 4.2 seconds.

I had no visibility into why.

Grepping Kibana for a session ID returned logs scattered across fifteen pods with no causal links:

[2026-07-18T03:15:12Z] INFO [auth-svc] msg="Incoming login request" user_id="u_9812a"
[2026-07-18T03:15:13Z] DEBUG [user-svc] msg="Fetching user metadata" user_id="u_9812a"
[2026-07-18T03:15:16Z] ERROR [auth-svc] msg="context deadline exceeded" component="grpc_client"

I spent four hours blaming PostgreSQL—locked tables, missing indexes. I added logger.Info() statements throughout the codebase and tried reconstructing execution timelines by subtracting timestamps in a spreadsheet:

startTime := time.Now()
logger.Info("Starting role resolution", zap.String("user_id", req.UserID))

roles, err := resolveUserRoles(ctx, req.UserID)
if err != nil {
    logger.Error("Failed to resolve roles", zap.Error(err))
    return nil, err
}

logger.Info("Finished role resolution", 
    zap.String("user_id", req.UserID), 
    zap.Duration("elapsed", time.Since(startTime)),
)

In a highly concurrent system, sequential log lines don't show network boundaries or where blocking I/O actually happens. I needed distributed tracing.

Setting Up OpenTelemetry and SigNoz

I deployed the OTel Collector as a DaemonSet, routing telemetry to a self-hosted SigNoz instance. I used OTLP over gRPC instead of HTTP to reduce serialization overhead, and configured batch processors to prevent telemetry from saturating the application's network:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 8192
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024

exporters:
  otlp:
    endpoint: "signoz-otel-collector:4317"
    tls:
      insecure: true

The problem was at the application layer. My first deployment broke completely—traces started at the API gateway and terminated immediately. Downstream services generated their own disconnected traces.

I'd missed injecting the OTel trace context into gRPC metadata.

Here's the initialization code that fixed it:

func initProvider(serviceName string) (*trace.TracerProvider, error) {
    ctx := context.Background()

    traceExporter, err := otlptracegrpc.New(ctx, 
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        // TraceIDRatioBased(0.1) samples 10% of traffic. At 10,000 req/sec,
        // AlwaysSample() would overwhelm the observability backend.
        trace.WithSampler(trace.TraceIDRatioBased(0.1)),
        trace.WithBatcher(traceExporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )

    // This propagates trace IDs via W3C TraceContext. Without it, gRPC calls lose the parent span ID.
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp, nil
}

I added business context to spans via custom attributes:

func resolveUserRoles(ctx context.Context, userID string, tier string) ([]Role, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "resolveUserRoles")
    defer span.End()

    span.SetAttributes(
        attribute.String("tenant_id", userID),
        attribute.String("user_tier", tier),
    )

    // business logic...
}

Finding the Bottleneck

I ran 1,000 concurrent users against staging. Within three minutes, p99 latency hit 4.2 seconds—matching production exactly.

The SigNoz trace flamegraph showed the problem immediately. Under a single parent span, fifty identical sequential db.Query spans cascaded down, each taking 80ms.

I had an N+1 query problem hidden behind a clean gRPC interface.

When a user logged in, the system fetched their assigned groups. Then, for each group, it made a synchronous blocking database call to resolve RBAC policies:

func (s *AuthzServer) GetUserDetails(ctx context.Context, req *pb.UserReq) (*pb.UserResp, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "GetUserDetails")
    defer span.End()

    groups := fetchUserGroups(ctx, req.UserID)
    var policies []Policy

    for _, group := range groups {
        policy, err := s.db.QueryRowContext(ctx, "SELECT * FROM policies WHERE group_id = $1", group.ID)
        if err != nil {
            return nil, err
        }
        policies = append(policies, policy)
    }

    return buildResponse(policies), nil
}

At low traffic, 5ms database latency was imperceptible. Under high concurrency, connection pool exhaustion and network round-trips compounded into a 4.2-second bottleneck.

The Fix

I replaced the sequential loop with a single bulk query using PostgreSQL's ANY operator:

func (s *AuthzServer) GetUserDetails(ctx context.Context, req *pb.UserReq) (*pb.UserResp, error) {
    ctx, span := otel.Tracer("authz-svc").Start(ctx, "GetUserDetails")
    defer span.End()

    groups := fetchUserGroups(ctx, req.UserID)
    groupIDs := extractIDs(groups)

    query := "SELECT * FROM policies WHERE group_id = ANY($1)"
    rows, err := s.db.QueryContext(ctx, query, pq.Array(groupIDs))
    if err != nil {
        span.RecordError(err)
        return nil, err
    }
    defer rows.Close()

    var policies []Policy
    // scan rows...

    return buildResponse(policies), nil
}

Staging's p99 dropped from 4.2 seconds back to 120ms immediately after deployment. Connection pool saturation flattened. Error rates returned to baseline.

I configured a SigNoz alert to fire if the resolveUserRoles span exceeded 500ms for more than five minutes, tied directly to span duration instead of generic CPU thresholds.

What Changed

Before this, I treated observability as an operational afterthought. After, I started building with distributed tracing in mind from the start. I stopped guessing and started measuring.

The logs-only approach failed because I couldn't reconstruct distributed execution paths across network boundaries. Metrics told me something was broken. Traces showed me exactly where.

Debugging the Ghost in the Machine: Building a Self-Healing SRE Agent with OpenTelemetry

Tanmay Hathile — Thu, 16 Jul 2026 09:22:37 +0000

AI agents are awesome right up until they silently eat it in production, leaving you with zero logs, empty traces, and a cloud bill that makes you want to cry. Over the last 48 hours, I’ve been building an Autonomous Self-Healing SRE Agent Pipeline. The goal? Instrument it with OpenTelemetry so it doesn’t just fix its own mess, but reports the ROI of that fix in real-time.

Here’s the raw, unvarnished blueprint—complete with the architectural headaches that almost cost me my entire weekend.

The Setup

We wanted something practical: an agent that catches its own crashes (like a choked cache or a dropped database connection) and tries to fix itself.

Our app is a FastAPI service with a /execute-task endpoint. To keep it real—and stressful—we threw in a hardcoded 15% failure rate to simulate resource exhaustion. When it crashes, our middleware catches the exception, spawns an async "self-healing" loop, and logs the outcome to our telemetry backend.

Network Topology

Before we got to the fun stuff, we had to get the plumbing right. If you’ve ever had services fight over the same local port, you know the drill.

Component	Port	Purpose
FastAPI Agent	`8081`	Our main pipeline.
SigNoz UI	`8080`	Visualization dashboard.
OTel Collector	`4317`	gRPC ingest.

Note: Everything defaults to 8080. It took me a solid 30 minutes of checking logs before I realized my app and the SigNoz UI were fighting for the same port. I eventually just moved the agent to 8081 and called it a day.

The Math: $9.94 per Heal

We wanted to track the value of our automation. We based our ROI on a $60/hr SRE cost, a 10-minute manual fix time, and a 3.6-second auto-repair. After factoring in some SLA risk mitigation, every successful heal nets us exactly $9.94.

Seeing that data finally hit the dashboard? It felt like winning the lottery.

The Implementation

import os
import random
import time
from fastapi import FastAPI, HTTPException
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider, Status, StatusCode
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Setup OTel Providers
resource = Resource.create(attributes={"service.name": "sre-agent-pipeline"})

# Tracer setup
trace_provider = TracerProvider(resource=resource)
otlp_trace_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))
trace.set_tracer_provider(trace_provider)
tracer = trace.get_tracer("sre.agent.tracer")

# Metrics setup
otlp_metric_exporter = OTLPMetricExporter(endpoint="http://localhost:4317", insecure=True)
reader = PeriodicExportingMetricReader(otlp_metric_exporter, export_interval_millis=5000)
metric_provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(metric_provider)

meter = metrics.get_meter("sre.agent.metrics")
heals_counter = meter.create_counter(name="pipeline_heals_total", unit="1")
money_saved_counter = meter.create_counter(name="pipeline_money_saved_total", unit="USD")

app = FastAPI()

@app.post("/execute-task")
async def execute_task():
    with tracer.start_as_current_span("ExecuteAgentTask") as span:
        # Simulate a crash
        if random.random() < 0.15:
            with tracer.start_as_current_span("SelfHealingLoop") as heal_span:
                # Trigger recovery...
                heals_counter.add(1, {"type": "cache_purge"})
                money_saved_counter.add(9.94, {"currency": "USD"})
                return {"status": "remediated"}
        return {"status": "success"}

The "I Spent Way Too Long on This" Section

1. The OTel SDK Deprecation Trap

I spent way too long wondering why my imports were failing, only to realize the status submodule was deleted in recent updates. Classic. If you're using opentelemetry-sdk >= 1.27.x, stop trying to import from opentelemetry.sdk.trace.status.

The fix:

# Change this:
from opentelemetry.sdk.trace.status import Status 

# To this:
from opentelemetry.sdk.trace import Status, StatusCode

2. The Port Collision

As I mentioned, don't let your FastAPI app try to claim port 8080. If you're on a workstation, set it explicitly:

$env:APP_PORT="8081"
python main.py

Final Thoughts

Building autonomous systems is wild, but if you can't see what they're doing, you're just adding a "black box" to your production stack. By forcing our agent to log its own recovery, we turned an invisible maintenance cycle into a tracked asset.

Reproducibility: Everything is locked down in casting.yaml and its lock file.
Transparency: Yes, I used AI to brainstorm some of the architectural patterns and clear up my dependency errors—hackathon life is hard enough without fighting syntax errors alone at 3 AM.

If you run into similar OTel issues, check the official OTel Python docs. Happy coding, and may your build never fail on a Sunday night.