DEV Community

Cover image for Production Observability on a Budget: OpenTelemetry Traces, Structured Logs & Tail Sampling
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

Production Observability on a Budget: OpenTelemetry Traces, Structured Logs & Tail Sampling

#opentelemetry #observability #laravel #node

A customer emails support: "Our invoice shows more AI tokens than our own dashboard does." You open the logs. There are millions of lines from the Laravel API, the Node.js billing worker and the queue, none of them share an ID, and the one request that matters was probably dropped by the 1% head sampling you turned on last quarter to cut the tracing bill. Nothing is broken. You just cannot see anything.

I wrote this with help from AI tools and checked every command, config key and claim myself against the current OpenTelemetry docs. I run Loop Origin, where we handle server setup and maintenance for Laravel and Node.js SaaS platforms, and this is the observability setup I reach for when the budget is small and the incidents are not: OpenTelemetry everywhere, logs that carry trace IDs, and a collector that keeps the traces worth paying for.

📚 Series Roadmap — Production AI & SaaS Systems Engineering

How do you run OpenTelemetry in a Laravel and Node.js SaaS without a huge bill?

Direct answer: Instrument every service with the OpenTelemetry SDK and send 100% of spans to your own OpenTelemetry Collector. Generate request-rate, error and latency metrics there from all spans, then use tail sampling to keep every error, every slow trace and a small random baseline. Put trace IDs in JSON logs so logs and traces join.

Why head sampling fails the incidents you care about

Head sampling decides at the first span whether a trace is recorded, before anyone knows how the request will end. At 1%, you keep 1% of your errors and 1% of your slow checkouts. That is fine for a high-traffic homepage and useless for a billing worker that fails twice a day.

Tail sampling flips the order. The collector buffers all spans of a trace for a few seconds, waits for the trace to finish, then decides. The cost moves from your tracing vendor's storage bill to a little collector memory, which is usually the cheaper side of the trade.

The catch: every span of a trace has to reach the same collector instance, or each instance sees half a trace and makes half a decision. That shapes the whole architecture below.

A steel gold panning pan resting on wet river stones with gravel swirling around a few gold nuggets, representing tail sampling that keeps only the valuable traces, AI-generated
AI-generated image.

The architecture 

 Laravel API ─┐                         ┌─► tier-2 collector A ─┐
 Node worker ─┼─► tier-1 collector(s) ──┤   (spanmetrics +      ├─► traces backend
 Queue jobs  ─┘   loadbalancing by      └─► tier-2 collector B ─┘   metrics backend
                  traceID                    tail_sampling)
Enter fullscreen mode Exit fullscreen mode

On a small platform, tier 1 can be a collector sidecar or a single gateway; tier 2 can start as one instance. Split them when one box can no longer hold the decision buffer.

Tier 1: route by trace ID 

# otel-tier1.yaml (opentelemetry-collector-contrib)
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: { insecure: true }   # private network only; use TLS across hosts
    resolver:
      dns:
        hostname: otel-tier2.internal
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [loadbalancing]
Enter fullscreen mode Exit fullscreen mode

Tier 2: metrics from everything, traces from the interesting part 

# otel-tier2.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }

connectors:
  spanmetrics:
    dimensions:
      - name: http.route
      - name: tenant.plan        # low-cardinality only; never tenant.id
  forward: {}

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/up"'
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 200
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: keep-billing
        type: string_attribute
        string_attribute:
          key: app.flow
          values: [billing, metering]
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch: {}

exporters:
  otlphttp/traces:
    endpoint: https://traces.example.internal
  otlphttp/metrics:
    endpoint: https://metrics.example.internal

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter, filter/health]
      exporters: [spanmetrics, forward]
    traces/sampled:
      receivers: [forward]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/traces]
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [otlphttp/metrics]
Enter fullscreen mode Exit fullscreen mode

Three decisions matter here:

  • spanmetrics runs before sampling. Your request-rate, error and duration metrics are computed from every span, so dashboards and alerts stay accurate even though you store a fraction of the traces. This is what makes aggressive sampling safe.
  • Policies are OR'ed. A trace is kept if any policy says so: every error, every request over 1.5 s, every billing flow, plus 5% of everything else for "what does normal look like?" comparisons.
  • decision_wait must exceed your longest normal trace. Spans that arrive after their trace's decision can be evaluated as a new trace unless the processor's decision cache is configured, and you end up with fragments. Size num_traces for roughly decision_wait × traces per second, with headroom, and watch collector memory.

The health-check filter is a cheap win. Load balancer probes hitting Laravel's /up route every few seconds produce a lot of perfectly boring spans. Check which attribute your instrumentation actually emits (url.path in current semantic conventions, older libraries may use http.target) before you trust the filter.

Instrumenting Laravel

The PHP SDK uses a C extension for automatic hooks plus Composer packages:

pecl install opentelemetry
composer require open-telemetry/sdk open-telemetry/exporter-otlp \
  open-telemetry/opentelemetry-auto-laravel php-http/guzzle7-adapter
Enter fullscreen mode Exit fullscreen mode 
; php.ini / FPM pool env
extension=opentelemetry.so
Enter fullscreen mode Exit fullscreen mode 
# .env or FPM pool
OTEL_PHP_AUTOLOAD_ENABLED=true
OTEL_SERVICE_NAME=api
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-tier1.internal:4318
OTEL_TRACES_SAMPLER=parentbased_always_on
OTEL_PROPAGATORS=tracecontext,baggage
Enter fullscreen mode Exit fullscreen mode

parentbased_always_on is deliberate: the SDK sends everything, the collector decides. Then make every log line carry the trace:

// app/Logging/AddTraceContext.php, registered with 'tap' => [AddTraceContext::class]
namespace App\Logging;

use Illuminate\Log\Logger;
use Monolog\Formatter\JsonFormatter;
use Monolog\LogRecord;
use OpenTelemetry\API\Trace\Span;

class AddTraceContext
{
    public function __invoke(Logger $logger): void
    {
        foreach ($logger->getHandlers() as $handler) {
            $handler->setFormatter(new JsonFormatter());
            $handler->pushProcessor(function (LogRecord $record): LogRecord {
                $ctx = Span::getCurrent()->getContext();
                if ($ctx->isValid()) {
                    $record->extra['trace_id'] = $ctx->getTraceId();
                    $record->extra['span_id']  = $ctx->getSpanId();
                }
                return $record;
            });
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Queued jobs lose context unless you carry it. When the API dispatches work to the Node.js billing worker from Part 4, I inject the W3C traceparent into the payload:

use OpenTelemetry\API\Trace\Propagation\TraceContextPropagator;

$carrier = [];
TraceContextPropagator::getInstance()->inject($carrier);
Redis::rpush('usage:report', json_encode(['tenant_id' => $tenantId, 'otel' => $carrier]));
Enter fullscreen mode Exit fullscreen mode

Instrumenting Node.js 

// instrumentation.ts, loaded with: node --require ./dist/instrumentation.js dist/worker.js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

new NodeSDK({
  serviceName: 'billing-worker',
  traceExporter: new OTLPTraceExporter(), // reads OTEL_EXPORTER_OTLP_ENDPOINT
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fs': { enabled: false }, // noisy, rarely useful
  })],
}).start();
Enter fullscreen mode Exit fullscreen mode 
// worker.ts: continue the trace started in Laravel
import { context, propagation, trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('billing-worker');

export async function handle(job: { tenant_id: number; otel?: Record<string, string> }) {
  const parent = propagation.extract(context.active(), job.otel ?? {});
  return context.with(parent, () =>
    tracer.startActiveSpan('usage.report', async (span) => {
      span.setAttribute('app.flow', 'billing');
      try { await reportUsage(job.tenant_id); }
      catch (e) { span.recordException(e as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw e; }
      finally { span.end(); }
    }));
}
Enter fullscreen mode Exit fullscreen mode

The auto-instrumentation bundle includes Pino and Winston integrations that add trace_id and span_id to log records, so the same join works on both sides. Setting app.flow=billing is what lets the keep-billing policy keep the whole trace, including the Laravel request that started it. That answers the support email: one trace ID, from the API call to the Stripe meter event.

Choosing a sampling strategy

Strategy Where it decides Keeps all errors? Metrics accuracy Cost driver When I use it
No sampling Nowhere Yes Exact Backend storage grows with traffic Low traffic, early stage
Head sampling (e.g. 10%) SDK, at trace start No, ~10% Skewed unless metrics come from elsewhere Lower storage, blind to rare failures Very high traffic, uniform requests
Tail sampling + spanmetrics Collector, after trace ends Yes Exact (computed pre-sampling) Collector RAM, small storage Most B2B SaaS
Logs only, no tracing n/a n/a Depends on log parsing Log ingestion volume Never again, for me

Budget levers beyond sampling

  • Drop debug logs in production at the source, not in the pipeline; you pay to ship what you later delete.
  • Keep metric dimensions low-cardinality. A tenant.id dimension on spanmetrics creates one series per tenant per route. Put tenant IDs on spans, not on metrics.
  • Set retention per signal. Sampled traces for days, aggregated metrics for months, raw logs as short as your compliance allows.
  • Watch the collector itself. It exposes its own telemetry; alert on dropped spans and memory-limiter refusals.

Diagnostic checklist

  1. Can you go from any error log line to its full trace by copying trace_id?
  2. Does a trace started in Laravel continue through the queue into the Node.js worker?
  3. Are all spans of a trace routed to the same tail-sampling collector?
  4. Is decision_wait longer than your slowest normal trace?
  5. Are RED metrics generated before sampling?
  6. Does every policy you need (errors, latency, key flows) exist, plus a small baseline?
  7. Are health checks and other high-volume, low-value spans filtered?
  8. Are metric dimensions free of IDs, emails and URLs with parameters?
  9. Is the SDK sampler parentbased_always_on so the collector sees everything?
  10. Are collector drop and memory metrics alerting someone?

FAQ

Do I need a vendor to start?

No. The collector is open source and exports OTLP, so you can start with self-hosted backends such as Grafana Tempo, Jaeger or an OTLP-capable vendor's free tier, and switch later without touching application code.

How much memory does tail sampling need?

It depends on trace size, traffic and decision_wait. Measure it: run with memory_limiter, load-test at your peak traces per second, and size num_traces from what you observe rather than guessing.

Does sending 100% of spans to the collector slow PHP-FPM down?

The SDK batches exports, but every span still costs a little CPU in the request. Disable instrumentations you do not read, and keep the collector on the same host or network to keep export latency low.

Will tail sampling break my trace-based alerts?

It can if alerts count stored traces. Alert on the spanmetrics output instead, which sees every span.

How do I trace an LLM call inside a request?

Wrap it in a span with attributes for model, token counts and latency. Avoid storing prompts or outputs on spans; they can contain customer data and they inflate storage.

Is OpenTelemetry logging stable enough to replace my log shipper?

It is supported in many SDKs and the collector, but maturity differs by language. My conservative path is JSON logs with trace IDs to stdout, shipped as before, and moving to OTLP logs per service once its SDK is stable.

Next in this series, Part 6 covers LLM gateway and FinOps architecture: semantic caching, model routing and per-tenant token quotas, with the spans from this post as its cost ledger.

Follow & Connect

Top comments (0)