DEV Community

Cover image for Why Your rsyslog Config Is Quietly Breaking (And How to Fix It)
Schiff Heimlich
Schiff Heimlich

Posted on

Why Your rsyslog Config Is Quietly Breaking (And How to Fix It)

Why Your rsyslog Config Is Quietly Breaking (And How to Fix It)

RFC 6587 has been sitting there for years. Most of us didn't notice.

Here's what actually happened: you have rsyslog configs from 2012 that work fine on UDP port 514. Then one day someone asks "can you send these logs to our SIEM over TCP?" and you spend three hours wondering why connections keep timing out.

The answer is usually buried in how rsyslog parses messages.

The %TIMESTAMP% Problem

Old rsyslog configs often use TraditionalForward mode, which expects a BSD syslog-style timestamp at a fixed offset in the message. RFC 6587 changed how syslog over TCP works — it uses octet-counting framing (each message is prefixed with its byte count) rather than newline-delimited messages.

If you do this:

action(type="omfwd" protocol="tcp" target="siem.example.com" port="514")
Enter fullscreen mode Exit fullscreen mode

...and your config still has %TIMESTAMP% at a fixed column position, the receiving SIEM (especially one expecting RFC 6587 compliant input) will read garbage.

The fix in rsyslog is straightforward — enable RFC 6587 mode:

action(type="omfwd" protocol="tcp" target="siem.example.com" port="514"
       streaming="on")
Enter fullscreen mode Exit fullscreen mode

That streaming="on" switches to octet-counting. But that's only half the problem.

Structured Logging and the Legacy Template Problem

Compliance auditors are now asking about RFC 5848 (signed log streams) and structured logging (JSON-formatted messages). If your rsyslog templates still use:

$template TraditionalFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"
Enter fullscreen mode Exit fullscreen mode

...you're not sending structured data. You're sending text that a human can read but a SIEM has to parse with regex.

The actual migration path most teams end up on is: keep rsyslog running for the legacy stuff, but bridge it to journald which handles structured logging natively.

The Bridge Setup

On a modern RHEL/Fedora box, syslog-ng sits between rsyslog and journald nicely:

# syslog-ng.conf — receive from legacy rsyslog
source s_net { tcp(port(514) flags(expect-octet-counting())); };

# Forward to journald in structured form
destination d_journal { pipe("/run/systemd/journal/syslog"); };

log { source(s_net); destination(d_journal); };
Enter fullscreen mode Exit fullscreen mode

Then journald handles the structured part — you get JSON via journalctl --output=json, and you can sign streams with journalctl --setup-keys if you need RFC 5848 compliance.

What About Just Replacing rsyslog?

You can. systemd-journald alone is fine for local logging. But most enterprise environments have network devices and appliances that only know how to send traditional BSD syslog. You need something listening on UDP 514 for those.

syslog-ng Premium Edition handles the RFC 6587 stuff properly and has good template support. The community edition works too, just with slightly different syntax.

The Real Lesson

The configs from 2012 worked because the world was simpler — UDP, single SIEM, no compliance requirements. That world is gone. The fix isn't dramatic, but you need to understand what RFC 6587 actually changed about message framing before you can debug why your SIEM is getting half-formed messages at 2am.

Check your rsyslog version (rsyslogd -v) and look at whether your templates assume fixed-width fields. That's usually where things break.

Top comments (0)