DEV Community

James Rivers
James Rivers

Posted on

Production Log Parsing Patterns: How I Fixed Logging in 50+ Kubernetes Clusters

I spent the last 3 years building production observability stacks for Kubernetes clusters (50+ nodes), and I noticed a pattern: every team I worked with spent 2-4 weeks reinventing log parsing.

The problem is concrete. You deploy a service, logs hit your aggregator (Elasticsearch, Loki, Datadog), and suddenly you're writing regex patterns for:

  • CRI-O container timestamps: 2024-01-15T10:23:41.123456789Z (most people miss the nanosecond precision and drop 30-40% of lines)
  • Multiline stacktraces that merge incorrectly (the [FP] flag in CRI-O that nobody documents)
  • IPv6 addresses in access logs ([2001:db8::1] breaks naive regex)
  • Structured JSON with nested exceptions
  • Application-specific formats (Spring Boot, Go, Node.js each log differently)

This pack contains 50+ production-tested regex patterns and ready-to-use configurations for the entire stack:

  • Log collectors: Fluent Bit, Vector, Filebeat, Logstash (complete configs with performance tuning)
  • Aggregators: Elasticsearch, Loki, Datadog (grok patterns, parser pipelines)
  • Specific parsers: CRI-O/containerd, Kubernetes kubelet, Nginx/Apache, PostgreSQL, MySQL
  • Gotchas I learned the hard way: multiline handling, timezone normalisation, cardinality explosion

Example: The CRI-O pattern that works:

^(?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+(?P<stream>stdout|stderr)\s+(?P<flags>[FP])\s+(?P<log>.*)$
Enter fullscreen mode Exit fullscreen mode

Most teams use a simpler pattern, lose the [FP] flag info, and can't reconstruct partial lines correctly.

The pack is £9.50 as a downloadable reference guide (not a subscription). It's practical—copy the configs, adjust for your stack, done.

https://riverbend36.gumroad.com/l/dyrpv

Feedback welcome—I'm iterating on this based on what DevOps teams actually need.

Top comments (0)