Inside Datadog's Log Pipeline: How "Logging without Limits" Actually Works

YukiOnodera — Mon, 27 Apr 2026 09:54:07 +0000

Introduction

In this post, I want to walk through how Datadog processes logs internally — from raw ingestion all the way to indexed, queryable data.

If you've spent time clicking around Datadog's log management UI, you've probably noticed something satisfying: raw, messy log lines gradually get enriched and structured as they flow through the pipeline. It's a really elegant design, and once you understand the order in which things happen, it becomes clear why Datadog can offer both cost control and deep observability at the same time. Let me break it down.

:::message
This article focuses on the main steps I studied this time around. In practice, there are more detailed processes — sensitive data scanning, Error Tracking, Live Tail, and so on. For the full picture, please refer to the official documentation | Datadog.
:::

The Overall Log Processing Flow

Datadog's log management is built around a design philosophy called Logging without Limits™, which lets you independently control "ingestion," "storage," and "analysis."

The high-level flow looks like this:

Ingest
  ↓
Pipelines (Parse & Enrich)
  ↓
Generate Metrics
  ↓
Exclusion Filters
  ↓
Index

Walking Through Each Step

Ingest

First, logs are collected into Datadog from a wide variety of sources.

Datadog offers over 500 log integrations, covering AWS, GCP, Kubernetes, and all kinds of middleware. I was honestly surprised by just how many there are.

Pipelines (Parse & Enrich)

Once raw logs are ingested, they pass through pipelines that structure and enrich them (adding extra information).

Using processors like the Grok parser, unstructured text logs get broken down into fields, and additional attributes can be attached.

# Before parsing (raw log)
2024-04-27 12:00:00 ERROR [app] Connection timeout: host=db01 duration=5002ms

# After parsing (structured)
{
  "timestamp": "2024-04-27T12:00:00Z",
  "level": "ERROR",
  "service": "app",
  "message": "Connection timeout",
  "host": "db01",
  "duration_ms": 5002
}

Watching unformatted logs get cleaned up and enriched is honestly the most fun part of this whole process to observe.

Generate Metrics

This is the most interesting part of the Logging without Limits design.

Log-based metrics are generated before exclusion filters run.

In other words, even for logs that will later be discarded and never make it to the index, you can still retain statistical information as metrics.

:::message
The benefit of this design is that even if you're aggressively dropping logs to keep costs down, you still get reliable metrics on trends, error rates, and the like.
:::

Exclusion Filters

After metric generation, exclusion filters decide which logs are not saved to the index.

Debug logs, high-volume boilerplate logs, and anything that isn't needed for ongoing search can be dropped here, helping keep indexing costs under control.

Index

Logs that pass through the filters are finally stored in the Index. Once a log is indexed, you can use Datadog's UI for facet search and analysis.

Why This Ordering Matters

The key insight in this processing order is the design principle: "extract metrics before throwing logs away."

Log storage costs balloon quickly, so indexing every single log is rarely realistic. But if you just drop logs, you lose visibility into trends within the discarded data.

Logging without Limits solves this by placing metric generation before exclusion filters. You can lower storage costs while still maximizing observability.

Wrap-up

Datadog's log pipeline has clearly separated stages: ingest, parse, generate metrics, exclude, and index. The design choice to run metric generation before exclusion filters strikes me as especially important — it's what allows you to balance cost and observability rather than trade one off against the other.

Subscribe for more Datadog & Observability deep-dives.

ECR Costs Had Increased Over 10 Times Without Me Noticing

YukiOnodera — Fri, 16 Aug 2024 12:10:15 +0000

Introduction

The other day, while I was looking through Cost Explorer, I discovered that the cost of ECR had ballooned to more than 10 times what it was a few months ago.

Investigation Begins

Realizing this was a serious issue, I immediately began investigating the cause.

Confirming ECR Pricing

I started by reviewing the pricing structure for ECR.

https://aws.amazon.com/ecr/pricing/

The cost of ECR is determined by storage charges based on the amount of image storage and data transfer out.

Reviewing the Invoice

Next, I checked last month’s invoice.

The goal was to identify whether the increased costs were due to storage or data transfer.

At this point, I noticed that in the ECR section of the invoice, only the storage costs were visible. Data transfer costs are listed under a separate data transfer section, so make sure to check that. I initially overlooked this, which left me puzzled about the discrepancy.

In my case, data transfer costs had skyrocketed.

Checking AWS Accounts

Since I was using AWS Organizations, I checked which member account was seeing the increased ECR costs.

Fortunately, only one account had significantly higher costs, so I was able to quickly pinpoint the source.

Reviewing ECR Repositories

However, just identifying the AWS account wasn’t enough to solve the problem.

I decided to open the list of ECR repositories and take a look.

I noticed that several repositories had been created around the time costs started increasing.

The Cause of the Cost Increase

After discussing it with team members and digging deeper, we discovered that this was due to a repository that had been migrated from Docker Hub for use in CI.

CI is executed on GitHub Actions with each commit, and since multiple images of considerable size were being pulled, data transfer costs had surged.

Conclusion

I'm glad I found the cause. In fact, I consider myself lucky to have noticed it through Cost Explorer.

Cloud costs can sometimes spike unexpectedly, so it’s important to stay vigilant.

References

https://docs.docker.com/docker-hub/download-rate-limit/

DEV Community: YukiOnodera

Inside Datadog's Log Pipeline: How "Logging without Limits" Actually Works

Introduction

The Overall Log Processing Flow

Walking Through Each Step

Ingest

Pipelines (Parse & Enrich)

Generate Metrics

Exclusion Filters

Index

Why This Ordering Matters

Wrap-up

ECR Costs Had Increased Over 10 Times Without Me Noticing

Introduction

Investigation Begins

Confirming ECR Pricing

Reviewing the Invoice

Checking AWS Accounts

Reviewing ECR Repositories

The Cause of the Cost Increase

Conclusion

References