DEV Community

Cover image for OpenPipeline pipeline groups: governing telemetry flows at scale
Span Observer
Span Observer

Posted on Originally published at span.observer

OpenPipeline pipeline groups: governing telemetry flows at scale

On a Tuesday, the payments team ships a new log pipeline and forgets the cost-centre tag. Three weeks later, finance finds a block of volume nobody owns. OpenPipeline pipeline groups exist for exactly this: deciding what must run for everyone, and what each team stays free to tune.

This article starts from an operations decision, not a tour of the UI. The facts come from the official OpenPipeline documentation and the vendor’s announcement, read on 12 November 2026. I did not test the feature on a tenant, so anything unverified is flagged. The topic applies to Dynatrace SaaS; the announcement does not mention Managed and I claim nothing for it.

To create a pipeline and route to it, see our tutorial on the log pipeline and its route. Here we look at what happens when twenty teams do the same.

The problem: a rule copied twenty times drifts

Two common reflexes fail at scale. The first locks everything down: the platform team approves every change, and a simple filter becomes a ticket. The second copies rules: each team pastes the « cost tag plus security context » block, then the copies diverge.

Either way the price is paid in reliability and money: a skipped security step, a surprise bill, a delay before a new service is observable. The real question is which steps the platform should impose, and which it should leave to teams.

OpenPipeline pipeline groups: the principle

A pipeline group collects pipelines and fixes their execution order. The basic rules, per the documentation:

  • a pipeline belongs to one group at a time;
  • groups cannot be nested;
  • a base pipeline stays inactive until it is part of a group.

The group holds a « composition »: an ordered list of base pipelines and a single placeholder for member pipelines. Whatever sits before the placeholder runs first, team pipelines run at the placeholder, and whatever sits after runs last.

Two roles follow. The base pipeline (basePipeline) carries the shared configuration. The member pipeline (memberPipeline) carries one team’s logic. The UI shows a « Member pipeline » badge on members.

Mandate and restrict: two separate levers

The documentation separates two mechanisms. Understand them before designing anything.

  • Mandate: the stages of the composition’s base pipelines always run, in the defined order relative to member stages. The documentation’s example is a permission stage that runs before a department’s logic.
  • Restrict: excluded stages become unavailable to members. A team can add optional stages after the mandated ones, but cannot redefine the restricted ones.

One trap: if a base pipeline and a member pipeline enable the same stage, both run. To block a stage on the member side, exclude it through memberStages.

Steps: designing a group for your logs

Here is a design approach to adapt. I use a fictional case: two business units and one platform team.

  1. List what must never vary. For example: business unit, cost centre, security context, storage bucket by retention. These become your base pipelines.
  2. Place them around the placeholder. What must be set before teams touch the data goes before it; storage assignment can go after.
  3. Set the stages open to members with memberStages: includeAll or include.
  4. Attach the team pipelines in memberPipelines.
  5. Apply the configuration through the Settings API: the UI’s JSON view is read-only.

The schema follows the pattern builtin:openpipeline.<scope>.pipeline-groups, with one scope per data type (logs, spans, metrics, events, bizevents and so on). Each data type therefore has its own groups.

Illustrative pipeline group configuration: displayName, memberStages, a composition with two base pipelines around the member placeholder, and memberPipelines

The field names come from the documentation. The identifiers and values are fictional. I did not verify the exact value expected for stages.type inside the composition: check it against your tenant’s schema before applying.

Routing: what changes for members

Inside a group, a member pipeline receives data from the first matching route. Base pipelines in a group are generally not routable; standalone ready-made extension pipelines are the exception. In practice, your routes point to team pipelines, and the platform does not have to watch them one by one: the baseline applies by itself.

If you already handle masking, a base pipeline is the right home for such a rule, since no team can bypass it. See masking sensitive data with OpenPipeline.

How to check that it works

After applying, check in a Notebook that every record carries the mandated fields. If a field is missing, the base stage did not run.

Example DQL query in a Notebook: fetch logs for one hour, summarize count() by business\_unit and cost\_center, sorted descending

The business_unit and cost_center fields are the ones my fictional group adds; replace them with yours. A row with empty values is the signal to chase: an ingestion path that bypasses the group. To watch for ingestion silences too, see detecting coverage gaps.

Best practices

  • Give group ownership to platform or SRE administrators; leave member pipelines to application teams.
  • Keep base pipelines short: security, cost, storage, nothing else.
  • Test a group change on a low-volume data type before extending it.
  • Document who owns which member pipeline, because the group will not do it for you.

Frequently asked questions

Can a pipeline belong to two groups?

No. The documentation says a pipeline belongs to one group at a time, and that groups cannot be nested.

Can a team bypass a mandated stage?

Not a stage from a base pipeline in the composition: it always runs. A stage excluded through memberStages is unavailable to members. Beware of stages enabled on both sides: both run.

Does the feature exist on Dynatrace Managed?

The vendor announcement states general availability in SaaS version 1.332 and does not mention Managed. I did not verify it for Managed: check the Managed documentation before planning.

Conclusion

OpenPipeline pipeline groups separate « what must always happen » from « what each team decides », and make the platform enforce that line. Start with one data type and three base rules. Main source: the pipeline groups documentation.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Brands mentioned belong to their owners. Features change quickly: check options in the official documentation.


Originally published on span.observer. French version: https://span.observer/fr/groupes-de-pipelines-openpipeline/.

Top comments (0)