DEV Community

Cover image for CEF and Syslog to Sentinel through AMA: connected, but no data
cuong do
cuong do

Posted on Originally published at gitlab.com

CEF and Syslog to Sentinel through AMA: connected, but no data

#azure #security #kql #cybersecurity

The connector page says Connected. The collector VM is running. The Azure Monitor Agent (AMA) is installed. And CommonSecurityLog is as empty as an office on a Friday afternoon. Or worse, every event shows up twice.

Rebuilt in my lab tenant with made-up data, checked against Microsoft Learn (October 2026). My lab: an Ubuntu log collector called LAB-SYSLOG-01 in resource group rg-lab-sentinel, a Contoso workspace, and test messages from a made-up LabFirewall (10.0.0.x addresses). In the screenshots, machine names, accounts, paths and workspace names are masked.

TL;DR

  • A CEF message passes six stops: device, port 514, syslog daemon, AMA on local port 28330, data collection rule (DCR), workspace. "No data" means one hop is broken. Test them one at a time.
  • Send your own test message with logger before you blame the firewall. If yours arrives and the real one does not, read the real CEF header and the daemon config.
  • Duplicates mean two roads into one table: a leftover legacy agent (MMA, retired August 31, 2024) next to AMA, or one facility collected for both Syslog and CEF.

At a glance
Symptom Connector shows Connected, but CommonSecurityLog or Syslog is empty, partial or duplicated
Why it happens One broken hop: DCR, port, daemon config, CEF header, network, an old agent, or two agents at once
Fix Walk the hops in order, one test per hop
Prove it is fixed Your test message lands in the workspace, Heartbeat says Azure Monitor Agent, duplicates near zero
Files queries.kql, enable-ama-autoupgrade.ps1

Flow of a CEF message: device sends to port 514, rsyslog or syslog-ng forwards to AMA on local port 28330, the DCR routes it to the Log Analytics workspace and Sentinel, with the check to run at each hop

Table of contents

1. Symptom

You see one of these:

  • The CEF via AMA or Syslog via AMA connector looks fine, but CommonSecurityLog (CEF) or Syslog has no rows from your device.
  • Some devices arrive, one does not.
  • Data arrives, but the same event is there twice.

None of these tell you where it breaks. The connector status only proves the connector exists. It does not prove that a message from your firewall ever reached the workspace.

2. Why it happens

A message has to survive this trip:

  1. The device sends syslog or CEF to the collector on port 514.
  2. The syslog daemon (rsyslog or syslog-ng) receives it.
  3. The daemon forwards it to AMA on local port 28330 (the mdsd process).
  4. AMA applies the DCR: which facilities and severities to take, and where to send them.
  5. The data lands in the workspace: Syslog for plain syslog, CommonSecurityLog for CEF.

The usual suspects, one per hop:

Hop What breaks
Device → 514 Wrong collector IP, NSG, firewall or load balancer in the way
Daemon AMA forwarding file missing, or an rsyslog input bound to a non-default ruleset (it never forwards to AMA)
Message CEF header with fewer than seven fields, or unescaped `\
AMA Old agent version, service stopped, cannot reach the configuration service to download the DCR
DCR Not linked to the collector, wrong facility or severity
Workspace Duplicates: MMA still installed next to AMA, or one facility in both a Syslog and a CEF DCR

3. Fix, step by step

Step 1. Check the connector and the DCR

In Sentinel, open the connector page. Then open the DCR: is {% raw %}LAB-SYSLOG-01 listed under Resources? Are the facilities and severities the ones your device sends?

DCR data source page for Linux Syslog: one facility, LOG_LOCAL1, is selected with minimum log level LOG_INFO, all others are unselected

The DCR facility and severity decide what AMA picks up. They are not the same thing as the Severity field inside the CEF header, which ends up in the LogSeverity column. Two severities, one name. Classic.

Step 2. Is the collector alive, and which agent is talking?

Run query 1 in queries.kql. No Heartbeat rows means AMA is not talking to the workspace, so fix that first. The Category column tells you the agent: Azure Monitor Agent is AMA, Direct Agent or SCOM usually means the legacy MMA.

Step 3. Update and restart

Update the OS packages (apt update and apt upgrade on Ubuntu, yum update on RHEL). Then restart the agent:

sudo systemctl restart azuremonitoragent
sudo systemctl status azuremonitoragent
Enter fullscreen mode Exit fullscreen mode

Step 4. Are both ports listening? 

sudo ss -lnp | grep -E "28330|514"
Enter fullscreen mode Exit fullscreen mode

You want the syslog daemon (rsyslogd or syslog-ng) on 514 and mdsd on 127.0.0.1:28330.

Step 5. Is the device's traffic reaching the box? 

sudo tcpdump -A -ni any port 514 -vv | grep "CEF:"
Enter fullscreen mode Exit fullscreen mode

Nothing here means the problem is upstream: the device, the route, the firewall, the NSG. Watch port 514 or 28330 instead to see the message leave the daemon for the agent.

Step 6. Send your own test messages

On the collector:

# Plain syslog
logger -p ftp.alert -P 514 -n 127.0.0.1 --rfc5424 --udp "Lab syslog test message"

# CEF through logger (the -t CEF tag already writes "CEF:", so the body starts at 0|)
logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF \
  "0|Contoso|LabFirewall|1.0|100|Lab traffic test|3|src=10.0.0.10 dst=10.0.0.20"

# Raw CEF through netcat, with a PRI prefix
echo -n "<164>CEF:0|Contoso|LabFirewall|1.0|100|Lab traffic test|3|src=10.0.0.10 dst=10.0.0.20" | nc -u -w0 localhost 514
Enter fullscreen mode Exit fullscreen mode

Check with queries 3 and 4 in queries.kql. Logs can take up to 20 minutes, so get a coffee before you panic.

Result Meaning
Test arrives, production does not The problem is the real message or the daemon config for it (steps 7 and 8)
Plain syslog arrives, CEF does not Check the CEF format and the DCR facility
Nothing arrives Back to Heartbeat, DCR link and the agent service

How to read a real message on the wire: a PRI number, a timestamp, the sending host, then CEF: and the header. PRI is facility * 8 + severity, so 142 is local1.info (17 * 8 + 6).

Step 7. Check the CEF header

A CEF header has seven pipe-separated fields before the extensions:

CEF:Version | Device Vendor | Device Product | Device Version | Device Event Class ID | Name | Severity | Extension

If a device sends fewer, the message is malformed and will not parse the way your queries expect. Want to see it break? Drop one field from the LabFirewall test message in step 6 and compare where its values land. Also check escaping: a | inside a header value must be \|, a backslash must be \\, and an = inside an extension value must be \=.

Step 8. Check the syslog daemon config

Look at /etc/rsyslog.conf and /etc/rsyslog.d/ (or /etc/syslog-ng/syslog-ng.conf). The AMA forwarding file (/etc/rsyslog.d/10-azuremonitoragent-omfwd.conf on rsyslog) must exist. And watch for the sneaky one: an input like input(type="imtcp" port="514" ruleset="myruleset") is bound to a non-default ruleset and never forwards to AMA.

Step 9. Run the troubleshooters

  • Sentinel_AMA_troubleshoot.py from the Azure-Sentinel GitHub repository, with --cef for CEF (also --asa and --ftd). Output goes to /tmp/troubleshooter_output_file.log.
  • The agent's own ama_troubleshooter.sh, in the ama_tst folder under /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-<version>/. Run it with -A (interactive) or -L (collect logs).

Output of Sentinel_AMA_troubleshoot.py: DCR, syslog daemon and operating system checks pass, tcpdump finds CEF in the stream, and the output file is written to /tmp/troubleshooter_output_file.log

ama_troubleshooter.sh started from the ama_tst folder of the AzureMonitorLinuxAgent extension, showing its menu: installation failures, connection, unhealthy agent, high CPU or memory, syslog not flowing, custom logs not flowing

Small tip from the first screenshot: "Failed to check" on the syslog daemon only means netstat is not installed. Install net-tools before you chase a ghost.

Step 10. Check DNS and the network

Look at /etc/resolv.conf for the expected nameserver. Test the configuration service:

curl -v https://global.handler.control.monitor.azure.com
Enter fullscreen mode Exit fullscreen mode

If the agent cannot reach it, it never downloads the DCR, and nothing gets collected. Review firewall rules, NSGs and proxy settings.

Step 11. Remove the second road (duplicates)

The legacy Log Analytics agent (MMA, also called OMS agent) was retired on August 31, 2024, and since March 2, 2026 its uploads can stop at any time. A collector that still has it next to AMA can send the same events twice.

Extensions page of a syslog collector VM: AzureMonitorLinuxAgent and OmsAgentForLinux are both installed and provisioned, next to other extensions

Remove the old agent from machines that now have AMA. Then check the DCRs: the same facility in a Syslog DCR and a CEF DCR also duplicates data, so give CEF and plain syslog different facilities on the device side.

Step 12. Keep AMA up to date

An old agent is a quiet cause of odd behavior. Turn on automatic extension upgrade. There is no built-in policy that only does that, so I use enable-ama-autoupgrade.ps1: it loops over the VMs in rg-lab-sentinel, detects Windows or Linux, and sets the AMA extension with -EnableAutomaticUpgrade $true.

The same loop run in Cloud Shell: Set-AzVMExtension with EnableAutomaticUpgrade for each VM, three OK results

Azure Arc machines: turn on automatic upgrade with Update-AzConnectedMachineExtension ... -EnableAutomaticUpgrade. To move to one exact version now, use Update-AzConnectedExtension with a targetVersion. To pin or downgrade, turn automatic upgrade off first so the agent does not jump forward again. Automatic rollout takes weeks, and only agents released in the past year are supported.

4. Prove it is fixed

Don't trust the green "Connected". Trust these:

  1. Your test message arrives. Queries 3 and 4 return the LabFirewall and syslog test rows within 20 minutes.
  2. The right agent is talking. Heartbeat shows LAB-SYSLOG-01 with category Azure Monitor Agent, and no machine reports through both agents (queries 1, 2 and 2b).

Heartbeat query summarized by Computer, OSType and Category: every machine in the result reports through Azure Monitor Agent; machine names are masked

  1. No duplicates. Query 5 groups CommonSecurityLog rows by TimeGenerated, DeviceVendor, DeviceProduct, DeviceEventClassID and Message, and shows the extra copies as a percent of all rows. Near zero is what you want.

Duplicate check on CommonSecurityLog in Defender advanced hunting over the last 30 days: zero duplicate records

If the percentage shows NaN, that is zero divided by zero: the table is empty, and you are back at step 5.

  1. The troubleshooter is green. Sentinel_AMA_troubleshoot.py --cef finds CEF in the stream.

Gotchas

  • "Connected" is not "receiving". The connector status proves nothing about your device.
  • Test data hides the real problem. If your test arrives and production does not, stop retesting. Read the real header.
  • Two severities. The DCR facility and severity are not the CEF LogSeverity.
  • Two agents. AMA plus a forgotten MMA means duplicates. Check Heartbeat after every migration wave.
  • One facility, two streams. The same facility in a Syslog DCR and a CEF DCR means duplicates too.
  • rsyslog rulesets. An input bound to a non-default ruleset never reaches AMA.
  • Patience. Up to 20 minutes before data shows up.

Checklist

  • [ ] DCR linked to the collector, facilities and severities match the device
  • [ ] Heartbeat rows from the collector, category Azure Monitor Agent
  • [ ] Ports 514 and 28330 listening
  • [ ] tcpdump shows CEF: on port 514
  • [ ] logger and nc test messages reach the workspace
  • [ ] Real CEF header has all seven fields, escaping correct
  • [ ] Daemon config reviewed, no non-default ruleset on the input
  • [ ] Troubleshooter run with --cef
  • [ ] Configuration service reachable from the collector
  • [ ] No MMA left, no facility shared by Syslog and CEF, duplicate rate near zero
  • [ ] Automatic upgrade on

What was the sneakiest hop in your own CEF pipeline? I'd like to add it to the gotchas.

Sources

Top comments (0)