Peter Nguyen

Posted on Oct 26

Visualize Tailscale Network Flow Logs with New Relic

#newrelic #tailscale #networkflow #logs

Introduction

Did you know that Tailscale network flow logs can be visualized in New Relic? Take a look at my dashboard:

Technical Details

To get started, you'll need to have Tailscale Enterprise, as this is the only option that lets you stream network flow logs.

While this is not officially supported (yet), network flow logs from Tailscale can be streamed to New Relic. The only thing you need is the log endpoint https://log-api.newrelic.com/log/v1 and your New Relic Ingest license key (ending with NRAL).

At a first glace, most people wouldn't know it can be done, thanks to the Tailscale admin dashboard only showing a few vendors to stream to:

Since we only require the log endpoint and the license key, I tried using these two parametes with each of the vendors to see what actually gets the log data into New Relic.

At the time of writing this, there are 7 supported stream options and results from my testing:

❌AWS S3
✅Axiom
🟡Cribl
❌Datadog
❌Elasticsearch
🟡Panther
❌Splunk

❌ - Streaming Fails or additional parameters required
🟡 - Works, but not optimal for New Relic
✅ - Works, and optimal for New Relic

AWS S3

This option requires AWS credentials such as access key id and secret access key. So, this option won't work to stream data into New Relic. One option, however, is to stream this into AWS S3, then forward the S3 logs into New Relic (inefficient).

Axiom

Axiom was the only option that streamed the network flow logs in near "real-time", with attributes trimmed. Let me explain:

The data for the Network Flow Lows using the Axiom option is streamed almost in real-time, rather than being sent at 1-minute intervals (Cribl and Panther options).

Since network flow logs can get heavy, you'll want to optimize the data ingest for this. If you look at the screenshot, the attributes are not prefixed with event. compared to the other two options that work. As you can imagine, for high log volumes, having the extra event. prefix can add up to excessive data ingestion with little value.

Cribl

This option works. Both configuration and network flow logs are sent to New Relic without issue. The only drawback here is the additional event. prefix in the attributes, which can quickly add up to data ingest. Another thing to note is the Network Flow Logs are sent once a minute.

Datadog

This option doesn't send any logs to New Relic.

Elastic Search

This option has an additional Username option that is required. Using a random username won't send any logs to New Relic.

Panther

Splunk

This option doesn't send any logs to New Relic.

Tailscale Logs in New Relic

Once log streaming for both Configuration and Network Flow Logs are enabled, the logs look something like this:

Here is what this looks like on the Tailscale Admin console:

Here's what I did to create my Tailscale Dashboard.

Network Flow Log Visualization

Utilizing the Sankey Diagram visualization I created with Plotly in my previous post, I used the following NRQL query to generate widget:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT count(*)
FROM Log
WHERE `physicalTraffic` IS NOT NULL
FACET src, dst
SINCE 72 HOURS AGO LIMIT MAX

This query parses the log data and generates a Sankey Diagram showing all of the physical traffic in Tailscale.

Network Flow Traffic

You can also add a line chart to visualize the transmit/received bytes:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT
  average(numeric(txBytes)) AS 'Transmit Bytes',
  average(numeric(rxBytes)) AS 'Received Bytes'
FROM Log WHERE `physicalTraffic` IS NOT NULL
TIMESERIES 90 SECONDS
SINCE '12:45-0400' UNTIL '13:15-0400'
LIMIT MAX

Network Flow Raw Data

Since the data is streaming as-in to New Relic, we can use anchor parsing to make the data look clean in a table like this:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT
src AS 'Source IP',
dst AS 'Destination IP',
  numeric(txPkts) AS 'txPkts',
  numeric(txBytes) AS 'txBytes',
  numeric(rxPkts) AS 'rxPkts',
  numeric(rxBytes) AS 'rxBytes'
FROM Log
WHERE `physicalTraffic` IS NOT NULL
AND rxBytes IS NOT NULL
SINCE 6 HOUR AGO LIMIT MAX

Configuration Log Visualization

Here's another useful visualization to see Configuration Changes at a quick glance:

SELECT count(*) AS 'Event Actions'
FROM Log WHERE `target.type` IS NOT NULL
FACET `action`
SINCE 6 HOURS AGO LIMIT MAX

And for the actual data in a tabular format:

SELECT
  `action`,
  `actor.displayName` AS 'Name',
  `target.name` AS 'Target'
FROM Log
WHERE `target.type` IS NOT NULL
SINCE 6 HOURS AGO LIMIT MAX

Remarks

Overall, streaming both the configuration logs and network flow logs from Tailscale to New Relic was easy. In my scenario, I used the Axiom option. It would be nice if Tailscale had a New Relic option, or at least an "Other" option to add a generic URL and generic header to accommodate other vendors.

DEV Community