Take Control of your Logs: Top 10 ways using the OpenTelemetry Collector

Let's face it, life as an SRE is tough. Usually you're given a system to work with over which you have no control.

• Developers logging DEBUG in production?
• Logs without a severity?
• Logs with the wrong severity?
• Commercial Off The Shelf (COTS) system that has been unsupported for a decade?
• Personally Identifiable Information (like names, emails, user IDs or credit card numbers) in the logs?
• Drowning in data?

Tough... You're stuck with it...
Well, actually you're not.

By using the OpenTelemetry collector, you can solve all of these problems. You can take back control of your telemetry.

Top 10 LOG processing methods using the OpenTelemetry Collector is a 12 minute video where I cover the top 10 techniques I see used in large enterprises to improve their log quality and control costs. These 10 are:

• Batching
• Removing sensitive information from logs
• Completely dropping low value logs
• Deduplicating logs
• Enriching logs with file and operating system metadata
• Enriching log files with custom business information (like ownership)
• Using CSV files to dynamically enrich log lines
• Dynamically adjusting log severities based on the log line content
• Conditionally adding content to log records based on the log file content
• Ways to standardise log files (extremely useful for COTS products where you can't change anything about the source)
• Methods to standardise log files (useful for cross-organisation standardisation)

All of these tips come with the demo log lines and exact OpenTelemetry collector YAML snippets that you need to recreate this video in your own setup, like this:

Here's the link again: Top 10 LOG processing methods using the OpenTelemetry Collector