OpenPipeline routing: create a custom pipeline and route logs to it

All your logs take the same path, and the same processing rule applies to the payment service and to a test job. This tutorial shows OpenPipeline routing in practice: create a custom pipeline for one service, write the route that sends its logs there, and decide what happens to the rest of the stream.

For an SRE or platform team, this is the starting point for controlling log volume, retention and access. OpenPipeline is built on Grail, so this applies to Dynatrace SaaS. It does not apply to Dynatrace Managed, which has neither Grail nor OpenPipeline.

Goal: by the end, a custom log pipeline receives only the logs of one Kubernetes deployment, and nothing else changes. Use cases such as masking, metrics and volume control build on this base and are covered in the linked articles below.

Prerequisites

• A Dynatrace SaaS environment (Grail and AppEngine) with administrator-level access: the documentation states this is needed to create routes and pipelines.
• Logs already ingested with usable attributes, for example k8s.namespace.name and k8s.deployment.name (see the logs and OpenPipeline module).
• Some DQL knowledge: a route condition is a DQL expression.
• Optional: a custom Grail bucket, if this service needs specific retention or permissions. Create it before the pipeline.

How OpenPipeline routing and a pipeline fit together

A record arrives through an ingest source, then it is routed to a pipeline. Routing is dynamic (a DQL condition decides) or static (a custom source points to one fixed pipeline). The documentation stresses one point: the order of the route list sets the order of execution. If no route matches, the record follows the Default route.

Two families of pipelines coexist. Built-in pipelines receive data with no configuration and send it to the built-in Grail bucket. Custom pipelines are yours; the documentation recommends routing as much data as feasible to them, with explicit matching conditions.

Each pipeline runs through stages in a fixed order. Every processor in a stage has a matching condition (matcher) written in DQL, and the output of one processor feeds the next.

Step 1: find what needs routing

Before creating anything, look in a Notebook for the attributes that identify your services. The documentation suggests this query:

fetch logs
| filter k8s.namespace.name == "prod"
| summarize by:{k8s.deployment.name}, count()

The result tells you how many routes you need and what each one matches. Prefer a stable attribute (deployment, namespace) over a message fragment that changes with the code.

Step 2: create the pipeline

1. Open Settings > Process and contextualize > OpenPipeline, pick the Logs configuration scope, then the Pipelines tab.
2. Select Pipeline and enter a name, for example « Checkout service pipeline ».
3. For each stage you need, add a processor: a name, a matching condition, then the processor’s own fields.
4. Select Save.

A pipeline created this way starts empty: it does nothing until you add processors. That is deliberate, so you can create the route first and add processing rules afterwards, for instance those from sensitive data masking.

Step 3: create the dynamic route

Open the dynamic routing tab (Dynamic routing), then Dynamic route. Define three things: a name, a matching condition and the target pipeline.

Name: Checkout service
Matching condition: k8s.deployment.name == "checkoutservice-*"
Target pipeline: Checkout service pipeline

Logs that satisfy the condition go to the target pipeline. The * wildcard comes from the documentation example: it covers suffixed pod or deployment names.

Step 4: decide what happens to the rest

Production logs that match no new condition keep flowing to the Default route, so to the classic pipeline. This lets you migrate service by service without breaking what exists.

The documentation also gives a non-production case: the condition isNotNull(k8s.namespace.name) and k8s.namespace.name != "prod", tied to a pipeline that uses the No storage assignment processor so those logs are not stored. Measure their usefulness first: a route that discards logs is a retention decision, to agree with the teams involved.

How to check that it works

Wait a few minutes after saving: routing only applies to records ingested afterwards, not to logs already stored (expected for ingest-time processing). Generate or wait for traffic, then check the volume in a Notebook:

fetch logs, from:now()-30m
| filter matchesValue(k8s.deployment.name, "checkoutservice-*")
| summarize count()

To see the pipeline itself at work, add a simple processor (for example a field of your choice with the Add fields processor) and look for that field in new logs. If it is missing, check in this order: the route condition, the route’s position in the list, the processor condition.

Keep the limits per configuration scope in mind, as stated in the documentation.

Best practices

• One pipeline per service or team: a pipeline owner can manage its access, which is not the case for routing (the documentation says owner-based access control covers pipelines and ingest sources, not routes).
• Explicit conditions, never « everything »: the documentation recommends this for custom pipelines.
• Watch route order: the position in the list sets the order of execution. The documentation does not describe what happens when several routes match, so avoid overlapping conditions.
• Version pipelines and routes with configuration as code (see the same habit for metrics), and track ingestion with log ingestion monitoring.
• Permissions: for pipelines, the documentation cites settings:objects:read and settings:objects:write (owner) and settings:objects:admin (administrator).

Frequently asked questions

What is the difference between dynamic and static routing?

Dynamic routing relies on a DQL condition. Static routing sends everything to one fixed pipeline, but only for a custom ingest source; for those sources, a match from another route is skipped.

What happens to logs that match no route?

They go through the Default route. In the documentation example, uncovered production logs keep going to the classic pipeline.

What about Dynatrace Managed?

OpenPipeline does not exist in Managed. Sorting can happen upstream, for example in the OpenTelemetry Collector before export (see log ingestion through the API).

Conclusion

Creating an OpenPipeline pipeline comes down to three moves: find the attributes, create the pipeline, write the route with an explicit condition. Masking, metrics and buckets then attach to that pipeline without touching other services. Start with one service, check with a Notebook, then widen the scope.

span.observer is an independent blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Screen labels and limits change: check them in the official OpenPipeline documentation.

---

Originally published on span.observer. French version: https://span.observer/fr/pipeline-openpipeline-creer-router-logs/.