DEV Community

Cover image for Journald logs: ship them to Dynatrace with OneAgent
Span Observer
Span Observer

Posted on Originally published at span.observer

Journald logs: ship them to Dynatrace with OneAgent

You open a recent Linux node to look for an error and there is no file to read: everything goes through systemd. The journald logs stay on the machine in a binary format, and they vanish with it if nobody ships them elsewhere. This article starts from that operational decision: who collects the journal, under which rule, and how you prove the lines really reach Dynatrace.

The scope is Dynatrace SaaS with OneAgent. I read the current SaaS documentation for Journald collection and the log module feature flags page, both updated in 2026. I did not build the full chain for this article: the screenshots reproduce documentation excerpts, and whatever I could not verify is said in the text.

For the big picture of log ingestion, read log ingestion: resilience and self-service first. Here we deal with one case only, the systemd journal.

The problem: a local journal does not centralise itself

journald keeps its entries in compressed binary files, usually under /var/log/journal/ when storage is persistent. It manages size, rotation and retention on its own. That is handy locally, with journalctl. Nothing, however, forwards those entries without an extra component.

For on-call, the consequence is direct. A node that crashes or gets replaced takes its system logs with it. An incident across ten machines means ten SSH sessions. Your time to resolve (MTTR) then scales with the fleet size, not with the severity of the problem.

Journald logs: three ways off the node

Before configuring anything, pick one of three approaches:

  • A third-party agent or syslog forwarding. You keep your tools, but you maintain the parsing of severity and units. Dynatrace also documents syslog ingestion, which is a separate project.
  • The OneAgent log module. It reads the journal through the systemd API, with no files to tail by hand. This is the option covered here.
  • The ingestion API. Useful for sources no agent covers. You must write the shipping code yourself.

If OneAgent already runs on your hosts, the second option saves one more component. The vendor’s post (How to observe logs with Journald and Dynatrace, April 2025, updated June 2026) presents it as available for both Dynatrace SaaS and Dynatrace Managed. I could not find an equivalent Managed documentation page: on Managed, check the procedure before applying it.

How OneAgent reads the journal

According to the documentation, the log module opens the libsystemd.so library and behaves as a journald client. It reads entries one by one and remembers its position. No path is hard-coded: the module opens the default location of the journald context.

Each entry is turned into attributes:

  • the MESSAGE field becomes the log content;
  • the UNIT field (or SYSTEMD_UNIT as a fallback) feeds the Journald.unit attribute, and the documentation notes that sometimes neither field is present;
  • the PRIORITY field gives the severity, LogEntryLevel.

That last mapping deserves a pause, because it drives your alerts.

Dynatrace documentation excerpt: mapping of journald priorities from 0 (Emergency) to 7 (Debug) to log levels, and of the MESSAGE and UNIT fields to the Content and Journald.unit attributes

An alert on errors should cover priorities 0 to 3, not only 3. If you want to filter out noise, read how to reduce log alert noise.

Steps: turn on journald log collection

Prerequisites: OneAgent version 1.307 or later on a Linux host with systemd, and permission to change tenant settings. The documentation requires two separate settings, and that is where people trip.

  1. Enable the detector. In the log module feature flags, tick « Enable Journald log detector ». It is on by default for new tenants. Existing tenants must opt in. Without it, the documentation warns that logs do not appear, even with ingest rules in place.
  2. Enable an ingest rule. The built-in « [Built-in] Ingest Journald logs » rule ingests all journald logs. On a new account it is already active.
  3. Or target one unit. Create a rule with two matchers: the « Journald Unit » attribute with a value such as kubelet.service, then the « Log source » attribute with the value « Journald ».

Dynatrace documentation excerpt: the Enable Journald log detector feature flag (OneAgent 1.307 and later), then the built-in Ingest Journald logs rule to enable under Log ingest rules

A navigation detail: the feature flags page puts this setting under « Collect and capture > Log monitoring », while the Journald page still says « Settings > Log Monitoring ». The two pages date from different stages of the documentation. Follow the menu you see in your tenant.

Target one unit instead of ingesting everything

The built-in rule is simple, but it ingests the whole journal. On a Kubernetes node or a busy server, that includes very chatty units. Volume is paid for at ingestion, as the cost of logs: ingestion, retention and query explains.

A sensible path in three moves:

  1. Start with the built-in rule on a single test host, to see what arrives.
  2. List the units that matter for on-call: for example kubelet.service, sshd.service or your application’s unit.
  3. Replace the global rule with one rule per unit, following the model below.

Dynatrace documentation excerpt: an ingest rule named kubelet-journald with the matchers Journald Unit equal to kubelet.service and Log source equal to Journald

The rule name kubelet-journald is my own example, not a required value. I have not measured the volume reduction this targeting brings on a real fleet: measure it on yours before promising it.

How to check that it works

On the host, produce a line whose text you know, for instance with logger -p user.err "test journald dynatrace". It enters the journal with priority 3. Then look for it in Dynatrace, in the Logs app or with a DQL query:

fetch logs, from:now() - 15m
| filter contains(content, "test journald dynatrace")
| limit 20
Enter fullscreen mode Exit fullscreen mode

If the line does not show up, check in this order: the OneAgent version, the Journald detector, the ingest rule, then whether the entry carries a unit or identifier. I did not verify here the exact name of the attribute that carries the source in Grail: do not write a filter on it before reading it in a real record. To go further with DQL, see DQL queries on log errors.

Once the logs are in Grail, OpenPipeline can process them before storage. That is the right place to mask or route: see creating an OpenPipeline pipeline to route logs.

Best practices

  • Enable the detector and the rule together, and record both in your runbook.
  • Prefer one rule per unit once the initial test is conclusive.
  • Alert on severity, not on a fragile keyword.
  • Keep journalctl for local troubleshooting: it still helps when the agent is down.
  • After each OneAgent upgrade, check that logs still arrive.

Frequently asked questions

Do I need to configure anything on systemd?

The documentation I read asks for no journald-side configuration: OneAgent reads the journal through the system library. Just check that the host uses systemd.

Why do my journald logs not arrive?

The most likely cause is an older tenant where the Journald detector was never enabled. Without it, even a correct ingest rule has no effect, according to the documentation.

Does this method apply to Dynatrace Managed?

The vendor’s post says so. I did not find a dedicated Managed page, though: confirm the procedure in the documentation for your version before using it.

Conclusion

Centralising journald logs comes down to two actions and one check: enable the detector, enable a rule, then find a test line. Then target the units that matter to keep volume under control. Main source: the Log Monitoring from Journald page of the Dynatrace documentation.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Trademarks belong to their owners. Products change fast: check settings and versions in the official documentation.


Originally published on span.observer. French version: https://span.observer/fr/logs-journald-dynatrace-oneagent/.

Top comments (0)