DEV Community

Cover image for How Logs Travel From Your EKS Pod to Datadog
Diya
Diya

Posted on

How Logs Travel From Your EKS Pod to Datadog

#aws #kubernetes #monitoring #devops

If you’re running applications on Kubernetes using Amazon EKS and suddenly seeing logs appear in Datadog, you may have wondered:

“How did the logs even get there?”

Your application is running inside a Kubernetes pod.

Datadog is somewhere in the cloud.

Yet somehow every request, every error, and every stack trace magically appears in the Datadog UI.

At first, it feels invisible.

But once you understand the observability pipeline, Kubernetes starts making a lot more sense.

What Is Datadog?

Datadog is an observability platform.

It helps engineers monitor:

  • Infrastructure
  • Kubernetes clusters
  • Applications
  • Logs
  • Metrics
  • Traces
  • Security events

Think of Datadog as a centralized monitoring brain for your systems.

Instead of SSH-ing into servers and manually checking logs, Datadog collects everything into one searchable place.

You can search:

service:map-service status:error
Enter fullscreen mode Exit fullscreen mode

and instantly see logs from hundreds of Kubernetes pods.

You can also:

  • Create dashboards
  • Set alerts
  • Trace requests
  • Monitor pod restarts
  • Watch CPU and memory usage
  • Detect failures in real time

But here’s the important part:

Your application itself usually does NOT directly communicate with Datadog.

That job belongs to the Datadog Agent.

What Is the Datadog Agent?

The Datadog Agent is the collector.

It runs inside your Kubernetes cluster and continuously gathers:

  • Logs
  • Metrics
  • Traces
  • Kubernetes metadata
  • Container information

In Kubernetes, the Datadog Agent is usually deployed as a:

DaemonSet
Enter fullscreen mode Exit fullscreen mode

A DaemonSet means:

“Run one Datadog Agent pod on every Kubernetes node.”

So if your EKS cluster has:

20 worker nodes
Enter fullscreen mode Exit fullscreen mode

Kubernetes automatically creates:

20 Datadog Agent pods
Enter fullscreen mode Exit fullscreen mode

Each agent watches workloads running on its own node.

The Big Question

Here’s what most people wonder:

“My application is inside a pod… so how does Datadog see its logs?”

To understand this, we first need to understand how Kubernetes handles logs internally.

Step 1 Your Application Writes Logs

Inside your container, your application usually writes logs to:

stdout
stderr
Enter fullscreen mode Exit fullscreen mode

For example:

print("hello world")
Enter fullscreen mode Exit fullscreen mode

or:

logger.error("database connection failed")
Enter fullscreen mode Exit fullscreen mode

Or maybe your Gunicorn server prints:

500 Internal Server Error
Enter fullscreen mode Exit fullscreen mode

Your app is simply writing text output.

It doesn’t know anything about Datadog.

It doesn’t know about dashboards.

It doesn’t know about observability.

It is simply talking.

Step 2 Kubernetes Captures Container Logs

Kubernetes containers run through a container runtime like:

  • containerd
  • Docker
  • CRI-O

In EKS today, most clusters use:

containerd
Enter fullscreen mode Exit fullscreen mode

The runtime captures container stdout/stderr and stores it as log files on the Kubernetes node.

Usually under paths like:

/var/log/containers/
/var/log/pods/
Enter fullscreen mode Exit fullscreen mode

At this point:

The logs now physically exist on the EC2 worker node.

This is the key realization:

Your pod logs are not floating magically inside Kubernetes.

They become actual files on the node filesystem.

Step 3 The Datadog Agent Watches Those Logs

Now the Datadog Agent enters the picture.

Because the Agent runs on every node, it can monitor container log files locally.

Conceptually, it does something similar to:

tail -f /var/log/containers/*.log
Enter fullscreen mode Exit fullscreen mode

The Agent continuously watches:

  • new logs
  • new containers
  • restarted pods
  • Kubernetes metadata

Whenever your application writes a log like:

ERROR database timeout
Enter fullscreen mode Exit fullscreen mode

the Datadog Agent immediately sees it.

Step 4 Metadata Enrichment

This is where Datadog becomes powerful.

The Datadog Agent doesn’t just forward raw text.

It enriches logs with Kubernetes metadata.

Example:

{
  "message": "GET /orders 500",
  "pod_name": "map-service-12345abc-west",
  "namespace": "production",
  "service": "map-service",
  "node": "ip-10-0-10-12",
  "cluster": "eks-prod"
}
Enter fullscreen mode Exit fullscreen mode

Now your logs become searchable.

You can filter by:

  • pod
  • namespace
  • service
  • cluster
  • environment
  • container

Without this enrichment, logs would just be random text.

Step 5 Secure Upload to Datadog Cloud

After enrichment, the Agent securely uploads logs to Datadog’s backend using HTTPS.

The flow now looks like this:

That’s the full hidden journey.

What About Metrics?

Datadog does more than logs.

The Agent also collects metrics from Kubernetes.

For example:

  • CPU usage
  • Memory usage
  • Pod restarts
  • RabbitMQ queue depth
  • Redis connections
  • Network traffic

It gathers metrics from:

  • kubelet
  • Kubernetes API
  • cAdvisor
  • integrations
  • DogStatsD

Example:

Kubernetes Node
   ↓
kubelet metrics
   ↓
Datadog Agent
   ↓
Datadog Cloud
Enter fullscreen mode Exit fullscreen mode

This is why you can build dashboards showing:

  • pod CPU %
  • node memory
  • HPA scaling
  • container restarts

in real time.

What About Traces (APM)?

Datadog APM tracks requests flowing across services.

Example:

Frontend
   ↓
API Gateway
   ↓
ROS Service
   ↓
PostgreSQL
Enter fullscreen mode Exit fullscreen mode

Datadog can measure:

  • request latency
  • database queries
  • failed API calls
  • slow endpoints

This works using tracing libraries like:

  • ddtrace (Python)
  • datadog-js
  • Java agent

These libraries send trace data to the Datadog Agent.

Then the Agent forwards it to Datadog.

Why Does the Datadog Agent Need Permissions?

This part surprises many engineers.

The Datadog Agent often mounts host system paths like:

/var/log/containers
/proc
/sys/fs/cgroup
Enter fullscreen mode Exit fullscreen mode

Why?

Because Kubernetes isolation prevents containers from normally seeing host resources.

The Agent needs visibility into:

  • container logs
  • processes
  • cgroups
  • node metrics

Some advanced integrations require even more access.

For example, the Gunicorn integration may require:

hostPID: true
privileged: true
SYS_PTRACE
Enter fullscreen mode Exit fullscreen mode

because the Agent sometimes needs to inspect processes outside its own container namespace.

This is why observability agents can become security discussions between:

  • platform teams
  • DevOps engineers
  • security teams

Monitoring deeply often requires elevated visibility.

The Biggest Realization

Your application is usually just writing logs normally.

The Datadog Agent acts like a collector sitting beside your workloads.

A simple analogy:

Application = person speaking
Datadog Agent = microphone
Datadog Cloud = recording studio
Datadog UI = playback/search system
Enter fullscreen mode Exit fullscreen mode

The application just talks.

The Agent listens.

Once you understand this pipeline, debugging Kubernetes becomes much easier.

When logs disappear, you know exactly where to investigate:

  • Is the app writing logs?
  • Is stdout working?
  • Did the container runtime capture them?
  • Is the Datadog Agent healthy?
  • Are hostPath mounts correct?
  • Is networking blocking uploads?
  • Is metadata enrichment failing?

Observability suddenly becomes understandable instead of magical.

And honestly, that’s one of the coolest parts of Kubernetes infrastructure.

Behind every “simple dashboard” is an entire hidden pipeline quietly moving data across your cluster 24/7.

Top comments (0)