DEV Community

Cover image for Centralized logging in the Otel era
Ashok Nagaraj
Ashok Nagaraj

Posted on

Centralized logging in the Otel era

Centralized logging (paradoxically) is a need in the distributed systems world. How would one possibly go through doing that?
This is a post on some approaches that we tried using Opentelemetry collector and an AppDynamics (for logging data store and visualization)

1. PULL model - Log aggregation through agents

Image description

  • Made popular by the Java way of doing things
  • Suitable for webserver/daemon kind of long running systems
  • Agent needs permissions, could create READ locks, consume resources and could be a network hog
  • Agents can process and filter the logs before sending, there by reducing the load on the collector

2. PUSH model

Image description

  • Logs are shipped out by the applications using configurable SDKs and libraries
  • Decoupled in nature
  • Suitable for transient, run-anywhere, script kind of systems
  • Intermediary will be a resource hog

We proceeded with the PUSH model using Kafka as the intermediary due to it's simplicity and reliability

First cut

Image description
Tried with in-house python-kafka-log-handler, promtail for agent, loki for processor

  • Didn't work as Loki could not EXPORT out in Otel format

Second try

Image description
Directly PUSH logs from Loki to AppDynamics backend

  • Didn't work as Loki and AppD won't talk the same language

One more try - Otel collector to the rescue

Image description
Avoid all intermediaries - Otel collector supports Kafka as a receiver and OtelHttp as an exporter to push to AppD backend (which is fully Otel compliant)

  • This worked like a charm!

Foot notes:

  • opentelemtry has two flavors of collectors - opentelemetry-collector the minimum stable and opentelemetry-collector-contrib which supports a ton of receivers and exporters
  • opentelemetry-collector-contrib has loki support for both receiver and transporter; but we could not figure out a simpler way of pushing logs from applications to loki(processor/aggregator)

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more