TCP packets traffic visualization for kubernetes by k8spacket and Grafana

Darek Barecki — Tue, 09 Aug 2022 11:18:00 +0000

Do you know what your k8s cluster does when you are not watching? Who establishes the TCP communication with him? Who is called by him, e.g., from third-party libraries?

With k8spacket and Grafana, you can visualize TCP traffic in the cluster. See how your workloads communicate together. Inspect how many connections are established, how many bytes are exchanged, and how long these connections are active.

Description

k8spacket is the tool written in Golang and uses the gopacket third-party library to sniff TCP packets on your workloads (incoming and outgoing). It creates TCP listeners on running container network interfaces. When Kubernetes creates a new container, the CNI plugin is responsible for giving the possibility to communicate either with other containers or from cluster to the world. The most common approach uses linux namespace to isolate networks and veth pair to connect the isolated namespace with the bridge. If you want to know more, please refer to a great article written by Safak (How Docker Container Networking Works - Mimic It Using Linux Network Namespaces). Besides type bridge, the CNI plugin can use other types (vlan, ipvlan, macvlan), but all create a network interface for container linux namespace, which is the main handle for k8spacket sniffers.

k8spacket is running with the option

hostNetwork: true

to eliminate the isolation mentioned earlier. It collects TCP streams, processes data, and exposes results through API to be used by Grafana.

Additionally,

k8spacket is a Kubernetes API client and can resolve sniffed workload as a cluster resource name (Pods and Services) visible on visualization.
It is launched as a DaemonSet to listen to network interfaces on all nodes.
Implemented watcher checks every 10s (default) if there is a new (or old) network interface to observe (or forget).

In addition to showing a graph of your cluster resources, it also exposes Prometheus metrics.

Installation

To install k8spacket, you need to have Helm and Grafana installed. Additionally, the tool requires Hamad's Node Grafana API plugin (Node Graph API).
See all installation steps here.

Usage

Types of stats

connection - helps understand how many connections are established between your workloads and from/to outside clients. It tells you which sockets stay open and could cause problems. It is helpful when you are behind some SNAT gateway with port exhaustion (e.g., Azure AKS).
bytes - shows how many bytes are sent or received by workloads.
duration - calculates the lifetime of connections.

Filters

by namespace - select one or more k8s namespaces

by names included - select workload names for visualization
by names excluded - exclude workload names from visualization

Summary

If you want to know what is going on with TCP traffic in your cluster, try k8spacket. Investigation about cutting the traffic (network policy) could be painless when you see a leak of packages outside. Make a review of third-party libraries to check connections with the world.

The source code can be found at github.com/k8spacket