DEV Community

Cover image for Log pattern analysis: from raw volume to reusable DQL queries
Span Observer
Span Observer

Posted on Originally published at span.observer

Log pattern analysis: from raw volume to reusable DQL queries

Every on-call engineer knows the moment: an alert fires, you open the logs for the affected service, and the screen fills with tens of thousands of near-identical lines. Only a timestamp, a request ID or an IP address changes. The usual reflex is to stack filters, try full-text searches, then write a regular expression or a parse by hand just to be able to count something. It is slow, and above all it is fragile when you are discovering the source at the same time as the incident. This is where log pattern analysis changes the way you approach the problem.

Log pattern analysis, available in preview in the Dynatrace SaaS Logs app, tackles the problem from the other end. Instead of looking for a needle, you ask the platform to sort the haystack: lines that share the same structure are merged into a single pattern, and the parts that vary are isolated as typed tokens. A few dozen patterns replace hundreds of thousands of records.

This article explains what the feature actually does, when it saves time, and above all how to turn a discovered pattern into something lasting: a DQL query, an OpenPipeline processor or a dashboard tile.

What log pattern analysis does

The principle is similar to what some « log clustering » tools have done for a long time, but here it runs at query time, directly on the data stored in Grail. The analysis examines the content of each message, separates static text from dynamic values and replaces the latter with a token. Lines that share the same skeleton fall into the same group, with an occurrence count and a trend over time.

The interesting part is that each token is typed. Dynatrace distinguishes timestamps, IP addresses, numbers and free text, among others. In practice, a thousand lines saying « connection refused for user X from address Y », scattered across a set of a hundred thousand logs, become a single pattern counted a thousand times, whose X and Y values you can then explore.

Results are sorted by severity, then by frequency. You can narrow the scope in several ways:

  • apply a segment prepared by domain experts (for example « payment platform logs »);
  • add ad hoc filters for a one-off exploration;
  • choose a time window ranging from a few minutes to several days or weeks.

Why it differs from a classic search

A full-text search assumes you know what to look for. A DQL aggregation assumes the fields already exist, meaning someone has written the parsing. Pattern analysis assumes neither: it reveals the shape of the data before you have formed a hypothesis. That is exactly what is missing at the start of an investigation.

It also rebalances frequency and importance. In a raw list, the chattiest message drowns out everything else. Once the lines are condensed, a group of twenty occurrences that appeared last night becomes as visible as a group of a million ordinary lines. Rare, new or changing signals stand out automatically.

Four situations where the tool pays off

An error spike to qualify

A service’s failure rate climbs. Rather than scrolling through messages, run the analysis over the spike window. Within seconds you will know whether a single family of errors explains the rise, whether a new pattern appeared just before, and where to start.

The weak signal buried in noise

An application looks healthy because the vast majority of its logs match the usual messages. A small group of intermittent timeouts to an external API goes unnoticed. Reduced to a handful of patterns, that group jumps out.

High-volume access logs

In web or API access logs, almost every line is a 200 response. The analysis separates these ordinary requests from the atypical cases: unusual status codes, unexpected endpoints, repeated authentication failures. By expanding a pattern, you see the distribution of the IP addresses involved, which lets you spot, for example, a brute-force attempt against a database.

An unknown source to onboard

A new application, or third-party software whose messages nobody really understands: the first question is « what does this source contain? ». Patterns answer it by showing the event types produced, the fields that vary, how often, and which ones deserve parsing or enrichment.

In practice

Step 1 — Scope the analysis

Open the Logs app, apply a segment or a filter on the service or host concerned, and choose a time window that matches your question. For an incident, frame the start of the problem with some margin before it; to onboard a source, take several days so you capture rare messages.

Step 2 — Run the analysis and read the results

Start the pattern analysis. Begin with high-severity patterns, then compare their trends: a pattern that starts exactly when the incident began is a better suspect than a large but stable one.

Step 3 — Explore the tokens

Expand the pattern you have chosen and examine the distribution of its tokens. The question becomes « which host, which ID, which code really matters? ». If a single IP address accounts for most occurrences, the investigation changes in nature.

Step 4 — Get the generated query

For an interesting pattern, the app generates a DQL query containing a parsing statement in Dynatrace Pattern Language (DPL). The result looks like this (simplified example, adapt it to the real pattern):

fetch logs
| filter matchesPhrase(content, "authentication failed")
| parse content, "LD 'user=' WORD:user LD 'from ' IPADDR:client_ip"
| summarize tentatives = count(), by:{client_ip, user}
| sort tentatives desc
| limit 20
Enter fullscreen mode Exit fullscreen mode

You now have named fields (user, client_ip) that you can aggregate, filter or chart as a time series.

Step 5 — Make it permanent

The same DPL pattern can be used wherever DQL can: Notebooks, Dashboards, Workflows and, above all, OpenPipeline processors. In the latter case, parsing happens at ingest and the extracted fields are available to every later query, with no recomputation. For tracking over time, a dashboard tile can rely on a query like this:

fetch logs
| parse content, "LD 'from ' IPADDR:client_ip"
| filter isNotNull(client_ip)
| makeTimeseries echecs = count(), by:{client_ip}
Enter fullscreen mode Exit fullscreen mode

Best practices

  • Start with a narrow scope: a segment or a per-service filter gives more readable patterns than an analysis across all the logs in the environment.
  • Review the generated DPL before promoting it to OpenPipeline: a pattern that is too generic can capture unrelated lines, and one that is too strict will miss some.
  • Parse at ingest what you use often: if a pattern-based query runs in several dashboards, move the parsing into OpenPipeline rather than recomputing it on every view.
  • Share through segments: a segment well defined by the team that owns a domain makes the analysis reproducible for everyone.
  • Watch for new patterns after a deployment: they are often the first sign of a regression.

Availability and limits

At the time of publication, the feature is in preview for Dynatrace SaaS customers with Grail and the DPS licensing model, subject to prior enrolment in the programme. As with any preview, the behaviour and interface may change: check the product documentation for up-to-date filtering options. Also note that the analysis relies on the textual structure of messages: logs that are already heavily structured as JSON will benefit more from classic JSON parsing.

Frequently asked questions

Do you still need to write regular expressions?

Less often. For an interesting pattern, the app itself generates a DQL query with the matching DPL parsing statement. You still need to review and adjust it before reusing it in a dashboard or an OpenPipeline processor.

Who can use pattern analysis today?

At the time of publication, the feature is in preview for Dynatrace SaaS customers with Grail and the DPS licensing model, after enrolling in the programme. Check the product documentation for the current status.

Is the analysis useful on logs already structured as JSON?

It relies on the textual structure of messages: for logs that are already heavily structured as JSON, classic JSON parsing is generally a better fit. The analysis is most valuable on free-text messages.

Conclusion

Pattern analysis does not replace DQL; it is an on-ramp to it. It shortens the most thankless phase of an investigation, when you are discovering the shape of the data, and along the way it produces the parsing you would otherwise have had to write by hand. For a beginner team, it is a gateway to structured log analysis; for an experienced user, it is that many DPL patterns they no longer have to write. Once your sources are under control, the next step is making sure they keep arriving: that is the purpose of monitoring log ingestion.

Further reading: From millions of log lines to actionable patterns in seconds with log pattern analysis (Dynatrace blog).

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.


Originally published on span.observer. French version: https://span.observer/fr/analyse-patterns-logs/.

Top comments (0)