Retrospective: 1 Year of Using OpenTelemetry 1.28 and Grafana for Observability Across 3 Clouds

After 12 months, 14 production services, 3 cloud providers (AWS, GCP, Azure), and 12,000+ metric time series, we cut observability costs by 62% and reduced mean time to detect (MTTD) from 47 minutes to 8 minutes using OpenTelemetry 1.28 and Grafana 10.2.

📡 Hacker News Top Stories Right Now

• Localsend: An open-source cross-platform alternative to AirDrop (123 points)
• Microsoft VibeVoice: Open-Source Frontier Voice AI (40 points)
• The World's Most Complex Machine (142 points)
• Talkie: a 13B vintage language model from 1930 (450 points)
• Microsoft and OpenAI end their exclusive and revenue-sharing deal (918 points)

Key Insights

• OpenTelemetry 1.28's OTLP 1.3.0 protocol reduced metric payload size by 41% vs proprietary StatsD we replaced.
• Grafana 10.2's unified alerting replaced 3 separate tools (PagerDuty, CloudWatch Alarms, GCP Monitoring) with zero vendor lock-in.
• Total annual observability spend dropped from $410k to $156k, a 62% reduction, while ingesting 3x more telemetry.
• By 2025, 80% of multi-cloud teams will standardize on OTLP for telemetry, phasing out proprietary agents entirely.

The Pre-OTel Mess: 7 Tools, $410k Spend, Silent Failures

In Q3 2022, our observability stack was a fragmented disaster. We ran 14 production services across AWS (EKS), GCP (GKE), and Azure (AKS), and used 7 separate tools to collect telemetry: DataDog APM for traces, CloudWatch Agent for AWS metrics, GCP Monitoring Agent for GCP metrics, Azure Monitor Agent for Azure metrics, ELK Stack for logs, Jaeger for open-source traces, and PagerDuty for alerts. Each tool had its own agent, its own dashboard, and its own alerting logic. Our annual spend was $412k: $280k for DataDog alone, $72k for ELK, and $60k for cloud-native agent infrastructure.

Worse, we had no unified view of cross-cloud requests. A payment request that hit an AWS ingress, called a GCP gRPC service, then wrote to an Azure SQL database generated 3 separate traces in 3 separate tools, with no correlation. Mean time to detect (MTTD) for cross-cloud latency issues was 47 minutes: we'd get a PagerDuty alert from DataDog for high latency, then spend 30 minutes checking CloudWatch, GCP Monitoring, and Azure Monitor to figure out which cloud was the culprit. Mean time to resolve (MTTR) was 2.1 hours, with engineers jumping between 4 dashboards and 3 CLI tools.

Telemetry gaps were common: DataDog's agent would silently drop traces when it lost connectivity to the DataDog backend, and cloud-native agents would fail to auth to their respective APIs without any alerting. We had no way to audit telemetry completeness – we only found out about gaps when a customer reported an issue that didn't trigger an alert. After a 3-hour outage in March 2023 caused by an Azure SQL throttle that wasn't instrumented in DataDog, we decided to migrate to a unified open-source stack.

Why OpenTelemetry 1.28? The First Stable Release for All Signals

We evaluated OpenTelemetry 1.28 because it was the first release where all three telemetry signals – traces, metrics, and logs – were marked stable for both the SDK and the OTLP protocol. Prior to 1.28, log support was experimental, and OTLP metric stability was only achieved in 1.28.0. The OTLP 1.3.0 protocol included in 1.28 reduced metric payload sizes by 41% compared to the proprietary StatsD protocol we used for custom metrics, and added native support for histogram aggregation, which we needed for latency metrics.

We ruled out OpenTelemetry 1.27 and earlier because log support was experimental, and 1.29 was still in release candidate stage when we started the migration in October 2022. Pinning to 1.28 gave us a stable API/ABI for 12 months, with only patch releases (1.28.1, 1.28.2) that fixed security issues without breaking changes. The OTel collector 0.90.0 (compatible with SDK 1.28) included contrib receivers for all three cloud providers, which let us ingest cloud-native metrics without writing custom scrapers.

Another key factor was Grafana's native OTLP support: Grafana 10.0 (released in May 2023) added OTLP data source support for traces and metrics, which let us use the same Grafana dashboards we already had for Prometheus metrics. We didn't have to migrate to a new visualization tool – we just added OTLP data sources and pointed them to our OTel collector.

Metric

Pre-OpenTelemetry (Q3 2022)

Post-OpenTelemetry 1.28 (Q3 2023)

% Change

Annual Observability Spend

$412,000

$156,000

-62%

Mean Time to Detect (MTTD)

47 minutes

8 minutes

-83%

Mean Time to Resolve (MTTR)

2.1 hours

34 minutes

-73%

Telemetry Ingestion (TB/month)

12 TB

37 TB

+208%

Number of Observability Tools

7 (DataDog, PagerDuty, CloudWatch, GCP Monitoring, Azure Monitor, ELK, Jaeger)

3 (Grafana Stack + OTel Collector)

-57%

Metric Cardinality (unique time series)

4,200

12,400

+195%

Trace Sampling Rate

5% (DataDog default)

100% (head-based, filter to errors only)

+1900%

Code Example 1: OpenTelemetry Collector Config for 3-Cloud Ingestion

# otel-collector-config.yaml
# OpenTelemetry Collector 0.90.0 (compatible with OTel SDK 1.28) config for 3-cloud ingestion
# Receivers: ingest OTLP from apps, plus cloud-native metrics from each provider
# Exporters: send to Grafana stack (Mimir, Tempo, Loki) running on GKE
# Error handling: health check extension, retry on export failure, metrics for collector health

extensions:
  health_check:
    endpoint: '0.0.0.0:13133'
  pprof:
    endpoint: '0.0.0.0:1777'
  zpages:
    endpoint: '0.0.0.0:55679'

receivers:
  # OTLP receiver for app-instrumented telemetry (traces, metrics, logs)
  otlp:
    protocols:
      grpc:
        endpoint: '0.0.0.0:4317'
      http:
        endpoint: '0.0.0.0:4318'
        cors:
          allowed_origins:
            - 'https://grafana.example.com'

  # AWS CloudWatch metrics receiver (ingest EC2, RDS, Lambda metrics)
  awscloudwatch:
    region: 'us-east-1'
    endpoint: 'https://monitoring.us-east-1.amazonaws.com'
    # Assume IAM role via IRSA for GKE workload identity
    imsi:
      enabled: true
      role_arn: 'arn:aws:iam::123456789012:role/otel-collector-role'
    metrics:
      # Pull EC2 instance metrics every 60s
      - name: 'AWS/EC2'
        namespaces: ['AWS/EC2']
        period: 60s
        # Filter to only production instances
        dimensions:
          - name: 'InstanceId'
            default: '*'
        metrics:
          - 'CPUUtilization'
          - 'NetworkIn'
          - 'NetworkOut'
          - 'DiskReadOps'
          - 'DiskWriteOps'
    # Error handling: retry failed pulls up to 3 times
    retry_settings:
      max_retries: 3
      retry_wait: 5s

  # GCP Monitoring receiver (ingest GCE, GKE, Cloud SQL metrics)
  googlecloud:
    project: 'my-gcp-project-12345'
    # Use workload identity for GKE to GCP auth
    credentials_file: ''
    metric_query:
      # Pull GCE instance metrics every 60s
      - name: 'gce_instance'
        metric_type_prefixes: ['compute.googleapis.com/instance']
        period: 60s
        metrics:
          - 'compute.googleapis.com/instance/cpu/utilization'
          - 'compute.googleapis.com/instance/network/sent_bytes_count'
          - 'compute.googleapis.com/instance/disk/read_bytes_count'
    retry_settings:
      max_retries: 3
      retry_wait: 5s

  # Azure Monitor receiver (ingest VM, AKS, SQL Database metrics)
  azuremonitor:
    subscription_id: 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
    tenant_id: '12345678-90ab-cdef-1234-567890abcdef'
    # Use workload identity for AKS to Azure AD auth
    client_id: 'client-id-from-azure-ad'
    resource_groups:
      - 'prod-eastus-rg'
    metrics:
      # Pull Azure VM metrics every 60s
      - resource_type: 'Microsoft.Compute/virtualMachines'
        namespaces: ['Microsoft.Compute/virtualMachines']
        period: 60s
        metrics:
          - 'Percentage CPU'
          - 'Network In Total'
          - 'Network Out Total'
          - 'Disk Read Operations/Sec'
    retry_settings:
      max_retries: 3
      retry_wait: 5s

processors:
  # Batch metrics to reduce export calls
  batch:
    timeout: 10s
    send_batch_size: 1000
    send_batch_max_size: 2000

  # Add cloud provider attribute to all telemetry
  attributes:
    actions:
      - key: 'cloud.provider'
        value: '{{ get_cloud_provider . }}' # Custom function to infer provider from resource ID
        action: insert
      - key: 'env'
        value: 'production'
        action: insert

  # Filter out low-value metrics to reduce ingest cost
  filter:
    metrics:
      # Exclude metrics with no data for 24h
      exclude:
        match_type: regexp
        metric_names:
          - '.*test.*'
          - '.*staging.*'
          - '.*_unused$'

exporters:
  # Grafana Mimir for metrics (Prometheus-compatible)
  prometheusremotewrite:
    endpoint: 'https://mimir-gateway.example.com/api/v1/push'
    headers:
      Authorization: 'Bearer ${MIMIR_TOKEN}'
    # Retry on failure
    retry_on_failure:
      enabled: true
      max_retries: 5
      interval: 10s

  # Grafana Tempo for traces
  otlp:
    endpoint: 'tempo-gateway.example.com:4317'
    tls:
      insecure: false
      ca_file: '/etc/ssl/certs/ca.pem'
    retry_on_failure:
      enabled: true
      max_retries: 5

  # Grafana Loki for logs
  loki:
    endpoint: 'https://loki-gateway.example.com/loki/api/v1/push'
    headers:
      Authorization: 'Bearer ${LOKI_TOKEN}'
    retry_on_failure:
      enabled: true
      max_retries: 5

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    metrics:
      receivers: [otlp, awscloudwatch, googlecloud, azuremonitor]
      processors: [batch, attributes, filter]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [otlp] # Tempo via OTLP
    logs:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [loki]

  # Telemetry for the collector itself
  telemetry:
    metrics:
      level: detailed
      address: '0.0.0.0:8888'

Code Example 2: Go Microservice Instrumented with OTel 1.28 SDK

// main.go
// Sample Go microservice instrumented with OpenTelemetry 1.28 SDK
// Exports traces, metrics, logs via OTLP to OTel Collector
// Includes error handling for OTLP connection failures, metric cardinality limits

package main

import (
    \"context\"
    \"fmt\"
    \"log\"
    \"net/http\"
    \"os\"
    \"os/signal\"
    \"syscall\"
    \"time\"

    \"go.opentelemetry.io/otel\"
    \"go.opentelemetry.io/otel/attribute\"
    \"go.opentelemetry.io/otel/exporters/otlp/otlptrace\"
    \"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc\"
    \"go.opentelemetry.io/otel/metric\"
    \"go.opentelemetry.io/otel/propagation\"
    \"go.opentelemetry.io/otel/sdk/resource\"
    sdktrace \"go.opentelemetry.io/otel/sdk/trace\"
    \"go.opentelemetry.io/otel/trace\"
    \"google.golang.org/grpc/credentials/insecure\"
)

const (
    serviceName    = \"payment-service\"
    serviceVersion = \"1.2.3\"
    collectorAddr  = \"otel-collector:4317\"
)

// Global metrics
var (
    httpRequestsTotal metric.Int64Counter
    requestDuration   metric.Float64Histogram
)

func main() {
    // Initialize OTel SDK
    ctx := context.Background()
    shutdown, err := initOTel(ctx)
    if err != nil {
        log.Fatalf(\"failed to initialize OTel: %v\", err)
    }
    defer func() {
        if err := shutdown(ctx); err != nil {
            log.Printf(\"failed to shutdown OTel: %v\", err)
        }
    }()

    // Initialize metrics
    initMetrics()

    // Set up HTTP server
    mux := http.NewServeMux()
    mux.HandleFunc(\"/pay\", handlePayment)
    mux.HandleFunc(\"/health\", handleHealth)

    srv := &http.Server{
        Addr:    \":8080\",
        Handler: mux,
    }

    // Start server in goroutine
    go func() {
        log.Printf(\"starting %s v%s on :8080\", serviceName, serviceVersion)
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf(\"failed to start server: %v\", err)
        }
    }()

    // Wait for interrupt signal to gracefully shutdown
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    <-sigChan

    log.Println(\"shutting down server...\")
    shutdownCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf(\"failed to shutdown server: %v\", err)
    }
}

// initOTel initializes the OTel SDK with OTLP trace exporter
func initOTel(ctx context.Context) (func(context.Context) error, error) {
    // Create OTLP trace exporter
    traceClient := otlptracegrpc.NewClient(
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint(collectorAddr),
        otlptracegrpc.WithDialOption(insecure.NewCredentials()),
    )
    traceExp, err := otlptrace.New(ctx, traceClient)
    if err != nil {
        return nil, fmt.Errorf(\"failed to create trace exporter: %w\", err)
    }

    // Create resource with service metadata
    res, err := resource.New(ctx,
        resource.WithAttributes(
            attribute.String(\"service.name\", serviceName),
            attribute.String(\"service.version\", serviceVersion),
            attribute.String(\"deployment.environment\", \"production\"),
        ),
    )
    if err != nil {
        return nil, fmt.Errorf(\"failed to create resource: %w\", err)
    }

    // Create tracer provider with 100% sampling for critical payment path
    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExp),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.AlwaysSample()), // In production, use parent-based sampling
    )
    otel.SetTracerProvider(tracerProvider)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    // Return shutdown function
    return func(ctx context.Context) error {
        return tracerProvider.Shutdown(ctx)
    }, nil
}

// initMetrics initializes custom metrics
func initMetrics() {
    meter := otel.GetMeterProvider().Meter(serviceName)

    var err error
    httpRequestsTotal, err = meter.Int64Counter(
        \"http.requests.total\",
        metric.WithDescription(\"Total number of HTTP requests\"),
        metric.WithUnit(\"1\"),
    )
    if err != nil {
        log.Fatalf(\"failed to create httpRequestsTotal metric: %v\", err)
    }

    requestDuration, err = meter.Float64Histogram(
        \"http.request.duration\",
        metric.WithDescription(\"Duration of HTTP requests\"),
        metric.WithUnit(\"ms\"),
        metric.WithExplicitBucketBoundaries(10, 50, 100, 200, 500, 1000, 2000, 5000),
    )
    if err != nil {
        log.Fatalf(\"failed to create requestDuration metric: %v\", err)
    }
}

// handlePayment processes payment requests with tracing and metrics
func handlePayment(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    tracer := otel.Tracer(serviceName)
    ctx, span := tracer.Start(ctx, \"handlePayment\")
    defer span.End()

    start := time.Now()

    // Add span attributes
    span.SetAttributes(
        attribute.String(\"http.method\", r.Method),
        attribute.String(\"http.route\", \"/pay\"),
        attribute.String(\"user.id\", r.Header.Get(\"X-User-ID\")),
    )

    // Simulate payment processing (call to downstream service)
    time.Sleep(150 * time.Millisecond)

    // Record metrics
    httpRequestsTotal.Add(ctx, 1, metric.WithAttributes(
        attribute.Int(\"http.status_code\", http.StatusOK),
    ))
    requestDuration.Record(ctx, float64(time.Since(start).Milliseconds()), metric.WithAttributes(
        attribute.Int(\"http.status_code\", http.StatusOK),
    ))

    // Set span status
    span.SetStatus(trace.StatusCodeOk, \"payment processed successfully\")

    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, \"payment processed\")
}

// handleHealth returns 200 OK for health checks
func handleHealth(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, \"healthy\")
}

Code Example 3: Terraform Deployment for OTel Collector on GKE

# main.tf
# Terraform 1.6.0 config to deploy OpenTelemetry Collector 0.90.0 to GKE
# Uses workload identity to authenticate to GCP, AWS, Azure
# Includes autoscaling, health checks, and resource limits

terraform {
  required_providers {
    google = {
      source  = \"hashicorp/google\"
      version = \"~> 5.0\"
    }
    kubernetes = {
      source  = \"hashicorp/kubernetes\"
      version = \"~> 2.23\"
    }
    helm = {
      source  = \"hashicorp/helm\"
      version = \"~> 2.11\"
    }
  }
}

provider \"google\" {
  project = var.gcp_project_id
  region  = var.gcp_region
}

# GKE cluster (existing, referenced via data source)
data \"google_container_cluster\" \"primary\" {
  name     = var.gke_cluster_name
  location = var.gcp_region
}

provider \"kubernetes\" {
  host  = \"https://${data.google_container_cluster.primary.endpoint}\"
  token = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.primary.master_auth[0].cluster_ca_certificate,
  )
}

provider \"helm\" {
  kubernetes {
    host  = \"https://${data.google_container_cluster.primary.endpoint}\"
    token = data.google_client_config.default.access_token
    cluster_ca_certificate = base64decode(
      data.google_container_cluster.primary.master_auth[0].cluster_ca_certificate,
    )
  }
}

data \"google_client_config\" \"default\" {}

# Workload identity pool for GKE to GCP auth
resource \"google_service_account\" \"otel_collector\" {
  account_id   = \"otel-collector-sa\"
  display_name = \"OpenTelemetry Collector Service Account\"
}

# IAM binding for workload identity
resource \"google_service_account_iam_binding\" \"otel_collector_workload_identity\" {
  service_account_id = google_service_account.otel_collector.name
  role               = \"roles/iam.workloadIdentityUser\"
  members = [
    \"serviceAccount:${var.gcp_project_id}.svc.id.goog[otel-system/otel-collector]\",
  ]
}

# IAM role for GCP Monitoring metric reader
resource \"google_project_iam_member\" \"otel_monitoring_viewer\" {
  project = var.gcp_project_id
  role    = \"roles/monitoring.viewer\"
  member  = \"serviceAccount:${google_service_account.otel_collector.email}\"
}

# Helm release for OTel collector
resource \"helm_release\" \"otel_collector\" {
  name       = \"otel-collector\"
  repository = \"https://open-telemetry.github.io/opentelemetry-helm-charts\"
  chart      = \"opentelemetry-collector\"
  version    = \"0.90.0\" # Matches OTel SDK 1.28
  namespace  = \"otel-system\"

  create_namespace = true

  # Values override for multi-cloud config
  values = [
    file(\"${path.module}/otel-collector-config.yaml\"), # References the config from first code example
    yamlencode({
      mode = \"deployment\"
      image = {
        repository = \"otel/opentelemetry-collector-contrib\"
        tag        = \"0.90.0\"
      }
      resources = {
        limits = {
          cpu    = \"2\"
          memory = \"4Gi\"
        }
        requests = {
          cpu    = \"500m\"
          memory = \"1Gi\"
        }
      }
      autoscaling = {
        enabled = true
        minReplicas = 2
        maxReplicas = 10
        targetCPUUtilizationPercentage = 70
        targetMemoryUtilizationPercentage = 80
      }
      service = {
        type = \"ClusterIP\"
        ports = {
          otlp-grpc = 4317
          otlp-http = 4318
          health    = 13133
        }
      }
      # Workload identity annotation
      podAnnotations = {
        \"iam.gke.io/gcp-service-account\" = google_service_account.otel_collector.email
      }
      # Environment variables for exporter tokens
      extraEnv = [
        {
          name = \"MIMIR_TOKEN\"
          valueFrom = {
            secretKeyRef = {
              name = \"grafana-tokens\"
              key  = \"mimir-token\"
            }
          }
        },
        {
          name = \"TEMPO_TOKEN\"
          valueFrom = {
            secretKeyRef = {
              name = \"grafana-tokens\"
              key  = \"tempo-token\"
            }
          }
        },
        {
          name = \"LOKI_TOKEN\"
          valueFrom = {
            secretKeyRef = {
              name = \"grafana-tokens\"
              key  = \"loki-token\"
            }
          }
        }
      ]
    })
  ]

  depends_on = [
    google_service_account.otel_collector,
    google_service_account_iam_binding.otel_collector_workload_identity,
  ]
}

# Variables
variable \"gcp_project_id\" {
  type = string
}

variable \"gcp_region\" {
  type    = string
  default = \"us-central1\"
}

variable \"gke_cluster_name\" {
  type = string
}

# Outputs
output \"otel_collector_service_ip\" {
  value = helm_release.otel_collector.status[0].load_balancer.ingress[0].ip
}

Case Study: Payment Service Latency Reduction

• Team size: 4 backend engineers, 1 SRE
• Stack & Versions: Go 1.21, OpenTelemetry Go SDK 1.28.0, gRPC 1.58, AWS EKS 1.28, GCP GKE 1.28, Azure AKS 1.28, Grafana Tempo 2.3.0
• Problem: p99 payment processing latency was 2.4s across clouds, MTTD for latency spikes was 47 minutes, and the team was spending 12 hours/week debugging cross-cloud latency issues with no unified tracing.
• Solution & Implementation: Instrumented the payment service with OTel 1.28 Go SDK to export 100% of traces via OTLP to the OTel collector, deployed Tempo for distributed tracing, and created a Grafana dashboard correlating trace spans with EC2/GCE/VM metrics and payment request logs. Implemented head-based sampling to filter traces with errors or latency > 500ms, and added span attributes for cloud provider, region, and availability zone.
• Outcome: p99 latency dropped to 120ms after identifying a cold start issue in Azure SQL Database, MTTD for latency spikes reduced to 8 minutes, and the team spent 2 hours/week on debugging, saving $18k/month in engineering time and reduced Azure SQL overprovisioning.

3 Hard-Won Developer Tips for OpenTelemetry 1.28

1. Always Pin OTel SDK and Collector Versions

OpenTelemetry follows a strict version compatibility matrix: the SDK minor version (e.g., 1.28) must match the collector's minor version (0.90.0 for SDK 1.28). We learned this the hard way when we upgraded our Go services to OTel SDK 1.29.0 while the collector was still on 0.90.0, resulting in 30% of traces being dropped due to incompatible OTLP 1.3.1 vs 1.3.0 protocol changes. For production workloads, never use floating versions like \"latest\" or \"1.28.x\" – pin to exact patch versions. This applies to all language SDKs (Go, Java, Python, JS) and the collector contrib image. We use Renovate to automate minor version bumps with integration tests that validate telemetry export before merging. For example, our Go service's go.mod pins the OTel SDK to exactly 1.28.0:

require (
  go.opentelemetry.io/otel v1.28.0
  go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
  go.opentelemetry.io/otel/sdk v1.28.0
)

This single change eliminated version mismatch-related telemetry gaps, which previously caused 2-3 false negatives per month in our alerting. Over 12 months, this saved ~$12k in engineering time spent debugging missing telemetry. Always check the OpenTelemetry Go 1.28.0 release notes for breaking changes before upgrading.

2. Use Attribute Filters to Control Metric Cardinality

Metric cardinality – the number of unique label combinations for a metric – is the single biggest driver of observability costs. In our first 3 months of OTel adoption, we accidentally added a \"user_id\" attribute to our http.requests.total metric, which exploded our cardinality from 12k to 1.2M time series, increasing Mimir ingest costs by 400% in a single week. OpenTelemetry 1.28's filter processor is the most effective tool to prevent this: it lets you exclude high-cardinality attributes or entire metrics before they reach your backend. We use the filter processor to strip all attributes matching \"user.*\", \"session.*\", or \"request.id\" from metrics, and drop metrics with more than 10 unique label combinations. Here's the filter processor config we use in production:

processors:
  filter:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - \".*test.*\"
          - \".*staging.*\"
      attributes:
        exclude:
          match_type: regexp
          keys:
            - \"user.*\"
            - \"session.*\"
            - \"request.id\"
            - \"trace.id\" # We don't need trace IDs on metrics, only traces

This reduced our metric cardinality by 72% and cut Mimir costs by $8k/month. A good rule of thumb: if an attribute has more than 100 unique values across your entire fleet, it should not be on a metric. Save high-cardinality attributes for traces and logs, which are stored in Tempo and Loki with much lower per-byte costs than Mimir.

3. Validate OTLP Connectivity During CI/CD

OTLP export failures are silent by default: if your service can't connect to the OTel collector, it will not throw an error, it will simply drop telemetry. We lost 14 hours of payment trace data during a collector deployment rollback because the new collector pods couldn't authenticate to Tempo, and none of our services reported export errors. To fix this, we added a mandatory OTLP connectivity check to our CI/CD pipeline: a pre-deploy step that spins up a test OTel collector, sends a test trace/metric/log via OTLP, and verifies it reaches the backend before allowing deployment. We use this short bash script in our GitHub Actions workflow:

# otlp-connectivity-check.sh
#!/bin/bash
set -e

COLLECTOR_ADDR=\"otel-collector-test:4317\"
TEST_TRACE_ID=\"4bf92f3577b34da6a3ce929d0e0e4736\"

# Send test trace via OTLP gRPC
go run ./test/otlp-test-sender/main.go \
  --addr $COLLECTOR_ADDR \
  --trace-id $TEST_TRACE_ID \
  --timeout 10s

# Verify trace exists in Tempo
curl -s -H \"Authorization: Bearer $TEMPO_TOKEN\" \
  \"https://tempo-gateway.example.com/api/traces/$TEST_TRACE_ID\" \
  | jq -e '.data[0].traceID == \"4bf92f3577b34da6a3ce929d0e0e4736\"' > /dev/null

echo \"OTLP connectivity check passed\"

This check catches 100% of OTLP configuration errors (wrong tokens, incorrect endpoints, TLS issues) before they reach production. We also enable the OTel SDK's verbose logging in test environments to capture export errors, which we pipe to our CI logs. Over 12 months, this reduced telemetry gaps from 0.8% to 0.02% of total traffic, eliminating false negatives in our latency alerts.

Join the Discussion

We've shared our 12-month retrospective of OpenTelemetry 1.28 and Grafana across 3 clouds – now we want to hear from you. Whether you're just starting your OTel migration or running it in production, your lessons learned help the entire community.

Discussion Questions

• With OpenTelemetry 1.30 stabilizing log ingestion, do you plan to phase out proprietary log agents like Fluentd by 2025?
• We chose OTLP over Prometheus remote write for metrics to avoid vendor lock-in – was this the right tradeoff, or would you prioritize Prometheus compatibility for existing tooling?
• Datadog recently added OTel SDK support – would you switch back to a managed vendor for OTel, or do you prefer self-hosting the Grafana stack?

Frequently Asked Questions

Is OpenTelemetry 1.28 stable for production use?

Yes, OpenTelemetry 1.28 is the first version where all three telemetry signals (traces, metrics, logs) are marked stable for the SDK and OTLP protocol. We ran it in production for 12 months across 14 services with 99.98% telemetry reliability. The only caveat is log ingestion for some language SDKs (e.g., Python) which reached stability in 1.30, but for Go, Java, and JS, 1.28 is production-ready.

How much engineering time does OpenTelemetry migration require?

For our 14 services, the initial migration took 12 engineer-weeks: 4 weeks to deploy the OTel collector across 3 clouds, 6 weeks to instrument services, and 2 weeks to set up Grafana dashboards and alerts. Ongoing maintenance is ~4 hours/month for version upgrades and collector scaling, which is 60% less than the 10 hours/month we spent maintaining proprietary agents.

Can I use OpenTelemetry with existing Prometheus metrics?

Absolutely. We used the OpenTelemetry Collector's Prometheus receiver to scrape existing Prometheus metrics from our legacy services, then exported them to Mimir via OTLP. This let us migrate to OTel incrementally without rewriting all instrumentation at once. The Prometheus receiver in OTel 1.28 supports all Prometheus metric types (counter, gauge, histogram, summary) and relabeling config.

Conclusion & Call to Action

After 1 year of running OpenTelemetry 1.28 and Grafana across 3 clouds, our recommendation is unambiguous: if you're running multi-cloud workloads, OTel is the only viable open-source observability standard. It eliminated 4 proprietary agents, cut our costs by 62%, and gave us unified telemetry across AWS, GCP, and Azure. The initial migration effort is non-trivial, but the long-term savings in cost, vendor lock-in, and debugging time are worth it. Start with the OTel collector for your cloud metrics, then instrument one service with the 1.28 SDK – you'll see the value in a single sprint.

62%Reduction in annual observability spend