Day 21: High-Performance Networking — Ingress and Egress for AI Clusters

#ai #kubernetes #networking #30dayaiseries

AI traffic does not look like ordinary web traffic. Requests carry multi-megabyte documents, and replies stream token by token over long-lived connections (Server-Sent Events). Network settings built for short page loads break that pattern. This lesson covers three places to tune: traffic coming in, traffic between services, and traffic going out.

In plain terms: Think of the cluster as a building. The front door must stay open long enough for a slow conversation, the corridors between rooms must not jam, and the exits must be locked except for the few doors you approve.

1. Ingress: keep long streams alive

An ingress controller (NGINX, Traefik or Envoy, for example) sits at the front door. Its defaults are tuned for quick requests and break AI streaming in two ways.

Timeouts. Default proxy read and send timeouts are often around 30 seconds, which can cut a connection while a model works through a large context. Raise them to several minutes for your slowest replies.
Buffering. If the proxy buffers the response, it waits for the entire reply before sending anything, which ruins real-time token streaming. Turn buffering off.

Diagram: If an ingress proxy buffers the response it waits for the whole LLM reply before sending anything to the user, which ruins real-time token streaming. Turning buffering off lets each token through as it is generated, over Server-Sent Events. See the animated version.

Diagram: Default proxy timeouts of about 30 seconds cut off long generations over big contexts. Raising the read and send timeouts to several minutes keeps the connection open until the model finishes. See the animated version.

2. East-west traffic: eBPF and Cilium

Standard Kubernetes networking uses kube-proxy, which builds chains of Linux iptables rules. As services multiply, evaluating those chains becomes a bottleneck for chatty traffic such as embeddings moving between gateways, vector stores and inference engines.

An eBPF-based network plugin such as Cilium makes routing decisions with small programs running inside the Linux kernel and direct lookups, and it can replace kube-proxy. In high-throughput datacenters, specialised network cards and kernel-bypass techniques can push latency lower still. How much you gain depends on your cluster, so measure before and after you switch.

Diagram: kube-proxy builds long chains of iptables rules that get slow to evaluate as services multiply. An eBPF-based network plugin such as Cilium makes routing decisions with kernel programs and direct lookups, which helps heavy service-to-service traffic. Gains depend on the cluster, so measure first. See the animated version.

3. Egress: stop data leaving

AI workloads add a new risk: an autonomous agent or a compromised model runtime may try to send internal context to an outside server. Use a NetworkPolicy with default-deny egress, so AI Pods can reach only the internal endpoints you list, and everything else, including the open internet, is blocked. Policies are only enforced if your network plugin supports them, and Cilium and Calico do.

Diagram: A NetworkPolicy with default-deny egress lets an AI Pod reach only approved destinations such as the vector database, plus cluster DNS, and blocks everything else, including unauthorised internet traffic. It needs a network plugin that enforces policies, such as Cilium or Calico. See the animated version.

Here are the two pieces of configuration. They assume the NGINX ingress controller and a namespace called ai, with a vector database Pod labelled app: qdrant.

# Ingress: long timeouts and no buffering, for streaming replies
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: model-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"    # seconds
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
spec:
  ingressClassName: nginx
  rules:
    - host: llm.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: model-svc, port: { number: 80 } }
---
# Egress: AI Pods may talk only to the vector DB and cluster DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ai-default-deny-egress
  namespace: ai
spec:
  podSelector: {}                 # every Pod in the namespace
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector: { matchLabels: { app: qdrant } }
    - to:                          # allow DNS, or nothing will resolve
        - namespaceSelector: {}
          podSelector: { matchLabels: { k8s-app: kube-dns } }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Get the allow-list right and your AI infrastructure handles big payloads with low latency, while keeping strict boundaries around what can leave. That wraps up Week 3. Next week moves to enterprise workflows, starting with autonomous agents.

Coming Up Next

Day 22: Autonomous agents: from passive LLMs to goal-driven execution.

KubernetesNetworking #eBPF #Cilium #NetworkSecurity #DevSecOps

Originally published at https://sureshpallapothu.in/blog/day-21-ai-cluster-networking, where this post includes animated diagrams.

DEV Community