DEV Community: Takuya Kajiwara

Tags: The Core Principles of Observability Context

Takuya Kajiwara — Sun, 07 Sep 2025 15:31:10 +0000

When we talk about observability, it’s easy to obsess over dashboards, monitors, or traces. But underneath all of that, one thing quietly determines your ability to understand systems: tags.

Tags (sometimes called labels or attributes) represent the context of telemetry data.

Many people think:

“Entities exist and they emit telemetry.”

But I prefer to flip it:

“Telemetry exists, and it has its context — including the emitting entity.”

Because telemetry becomes independent once it’s released from the emitting entity, each piece must carry its context.
That context lives in tags, which makes tagging the foundation of usable telemetry.

From my experience, effective tagging boils down to two categories of principles: one focused on quantity, and the other on quality.

Principle 1 (Quantity): Add as Much Context as Possible

Tags are the building blocks for asking meaningful questions of your telemetry data. The richer the context, the more powerful your analysis.

Even if you don’t know how you’ll use a tag today, you can’t add it retroactively to past telemetry once you realize it’s important.

With only one tag, such as:

service: checkout

you quickly run into limits. You can’t ask questions like

"Which regions are failing?"
"How many errors are happening in production?"

But with more tags, such as:

service: checkout
region: us-east-1
env: production
team: payments

you can answer those questions.

If you’re not sure what tags to start with, lean on reserved tags from your platform or industry-standard tags such as those defined in OpenTelemetry semantic conventions. In many cases, platforms handle these specially, offering richer analytics and visualizations.
These give you a ready-made foundation of context without guesswork.

Principle 2 (Quality): Keep Tags Canonical and Composable

Adding lots of tags is powerful — but only if they’re consistent and composable.

Canonical: one authoritative way to represent each dimension.
Composable: each tag represents one thing, so you can filter or group them independently.

Scenario A: Only a composed tag

service: auth-prod

Here you lose the ability to filter or group by environment.
If someone asks "How many errors in production?", you can’t answer this.

Scenario B: Mixed duplication

service: auth-prod
environment: prod

Now you’ve got two sources of truth for environment. The env:prod tag already represents the environment, so encoding it again inside service:auth-prod is unnecessary. Hardcoding multiple values to represent the same thing always leads to drift over time.

Being canonical also means unifying the values that point to the same dimension.
For example, you might see both of these in different parts of a system:

env: prod
env: production

Both names are acceptable on their own — but not when they coexist.
If you filter by env:prod, you’ll miss telemetry tagged with env:production, and vice versa.
Pick one naming convention and stick to it.

Defining tags this way keeps them simple, canonical, and composable.

service: auth
env: prod

Wrapping Up

Tags aren’t just metadata. They are the context carriers that make telemetry meaningful to your team — and to everyone else who relies on your data.

By keeping these two principles in mind:

Quantity: Add as much context as possible (start with reserved/standard tags if unsure).
Quality: Keep tags canonical and composable (one dimension per tag, consistent values).

This way, you’ll make your observability data more valuable, reliable, and future-proof.

Datadog, Reframed: A Simple Way to Think About Agents, Pipelines, Indexes, and More

Takuya Kajiwara — Tue, 02 Sep 2025 13:50:36 +0000

This is Part 2 of 3 in the Filters series.

In Part 1, I introduced the Presentation Filter: how Datadog shows data back to users.

Datadog has a lot of products. Agents, tracing libraries, RUM SDKs, AWS integration, log pipelines, retention filters… the list goes on.

It’s not easy to remember how all of these pieces fit together.

That’s where the Filters idea comes in.

In my previous post, I introduced this mental model and zoomed in on the Presentation Filter.

Now, let’s move on to the second one: the Collection Filter.

What’s the Collection Filter?

A Filter is not about a whole product, but about the part of its functionality that plays a certain role.

The Collection Filter is about how data makes its way into Datadog.

It covers the features that gather, refine, and shape signals before they’re stored and shown.

I like to think of data collection in three phases:

Ingestion → bringing signals in.
Enrichment → shaping context so the data makes sense (adding what’s missing, removing what doesn’t belong).
Reduction → trimming or transforming data to keep things practical.

Phase 1: Ingestion

Responsibility: get signals from your systems into Datadog.

Examples: the Datadog Agent on hosts and containers, tracing libraries inside applications, the Lambda Forwarder, RUM SDKs, AWS integration, synthetic bots, private locations, Workflow Automation (when it’s used to gather data), or even custom API calls.

Why it matters: without ingestion, there’s nothing to observe. These components sit close to your systems and act as the first touchpoint for your telemetry.

Phase 2: Enrichment

Responsibility: adjust telemetry so it carries the right context — by adding what’s missing and removing what doesn’t belong.

Examples: log pipelines, event pipelines, and Sensitive Data Scanner, which strips out credentials or personal information so only meaningful context remains.

Why it matters: telemetry without context is just noise. Enrichment makes the data actionable by adding what was missing, and tools like Sensitive Data Scanner make it safe and clean by removing details that aren’t needed. Together, they ensure the data is complete, relevant, and trustworthy.

Phase 3: Reduction

Responsibility: manage volume and cost by filtering, sampling, or transforming data.

Examples: log indexes, generate metrics, APM retention filters, RUM retention filters, tag configurations for metrics.

Why it matters: collecting everything forever is ideal in theory, but rarely possible in practice. Reduction features give you levers to control scale and budget while still keeping the insights you need.

Trade-off: reduction always means some loss of fidelity. The art is to reduce smartly, without cutting away the signals you’ll need later.

Why This Model Helps

Looked at by product name, features like pipelines, indexes, or SDKs can feel scattered.

But through the Collection Filter, they form a simple flow: ingest → enrich → reduce.

This also highlights priorities:

Ingestion and enrichment are essential. They make observability possible.
Reduction is optional. It’s the set of knobs you turn to keep data useful and affordable.

So instead of asking, “Which product does what?”, you can ask:

What data do I need to know my system is healthy, and to find the root cause when it isn’t?

Once that’s clear, you can decide which ingestion, enrichment, and reduction features to use.

Closing Thoughts

This is the second post in the Filters series.

We’ve now covered Presentation and Collection — how data is shown, and how data gets in.

Next up is the Execution Filter, which focuses on how Datadog takes action on your systems.

That one will be shorter, but it completes the picture.

I’d love to hear your thoughts: does the Collection Filter help make Datadog easier to navigate?

Datadog, Reframed: A Simple Way to Think About Dashboards, Monitors, Log Explorer, and Beyond

Takuya Kajiwara — Mon, 01 Sep 2025 13:54:43 +0000

Datadog has a lot of products. Dashboards, Monitors, Logs Explorer, Notebooks, On-call, Service Map, Bits AI SRE Agent… the list goes on.

For many users, it’s hard to keep all of these straight in their heads.

This post introduces a mental model that makes it easier to navigate Datadog.

I call this idea Filter.

What’s a Filter?

A Filter is not about a whole product, but about the part of its functionality that plays a certain role.

I define three Filters that are sufficient to cover all of Datadog’s features:

Presentation Filter → the part that presents data or evaluations back to users.
Collection Filter → the part that gathers data — either from users’ systems (logs, traces, metrics...) or from users directly.
Execution Filter → the part that takes action and has an effect on your systems.

For example:

If we apply the Presentation Filter to Synthetics, we see the alerts it shows to users.
If we apply the Collection Filter to Synthetics, we see how it proactively sends crafted requests to your systems and gathers responses to verify they behave as expected.

Another example is the tracing library:

Through the Collection Filter, it gathers spans from your application and sends them to Datadog.
Through the Execution Filter, features like In-App WAF come into view, because they actively change your application’s behavior by blocking traffic at runtime.

By switching Filters, you don’t need to think of each product as a giant box with dozens of features. Instead, you can decide what to use based on what you want to accomplish. Filters reveal what options you have to achieve your goal, rather than forcing you to start from product names.

The Presentation Filter

Let’s zoom in on one Filter: Presentation.

When this Filter is applied, many Datadog products expose functionality that gives something back to users. These may feel like very different products, but they all share this common role:

Dashboards
Logs Explorer
Monitors
Notebooks
Service Map
On-call
Bits AI SRE Agent
Product-specific pages (like Kubernetes Overview, Watchdog)

This exposed Presentation functionality can be grouped by two key properties.

Property 1: Delivery Mode (Push vs Pull)

Who initiates the transfer of information?

Pull → You open a Dashboard or Logs Explorer to fetch the data yourself.
Push → Datadog notifies you via Monitors when conditions are violated, or sends scheduled reports from Dashboards.

Property 2: Usage Mode (Predefined vs Exploratory)

How do you interact with the data?

Predefined → Dashboards, Monitors, Watchdog, product overview pages. Once defined, they rarely change.
Exploratory → Logs Explorer, Traces Explorer, Product Analytics. Users issue new queries almost every time.

Why This Model Helps

Instead of memorizing product names, you can ask yourself two simple questions:

Do I want Datadog to push results to me, or do I want to pull the data myself?
Do I need a predefined view, or do I want to run exploratory queries?

The answers point you to the right capability:

	Predefined	Exploratory
Pull	Dashboards	Logs Explorer, Traces Explorer
Push	Monitors, Dashboard reports	(rare, but possible in custom setups)

Examples:

If you want to check a predefined set of business metrics whenever you have time → use a Dashboard.
If you want Datadog to alert you in real time when an error pattern appears in logs → use a Log Monitor instead of relying on manual queries in Logs Explorer.
If you want to dig into traces ad-hoc → use an Explorer.

This mental model reduces complexity and helps you reach for the right tool faster.

Closing Thoughts

This is an early attempt to reframe Datadog in a way that’s easier to understand.

I’d love to hear your feedback: does the Presentation Filter make Datadog clearer for you?

In future posts, I’ll explore the other Filters — Collection (how data gets into Datadog) and Execution (how Datadog takes effect on your systems). Together, these three Filters are enough to dissect the entire Datadog product family, and make it easier to understand which tool to use for which purpose.