Traefik Rate Limiting That Actually Works on k3s

#kubernetes #traefik #security #homelab

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Prometheus scraped Grafana 30 times in 10 seconds. Grafana's API returned 429s. The dashboard went blank. The root cause: a misconfigured Prometheus service monitor with a 1-second scrape interval — the kind of runaway scrape config that also drives Prometheus's own RAM usage. The fix: Traefik's RateLimit middleware, correctly configured to protect against scrape storms without blocking normal dashboard access. This runs on a BMAX Mini PC* with 20+ services behind Traefik — one misconfigured scrape can cascade into a cluster-wide outage.

View the complete homelab infrastructure source on GitHub 🐙

The Scrape Storm

Grafana runs on k3s behind Traefik. The Prometheus Operator's ServiceMonitor for Grafana had a scrapeInterval: 1s — a copy-paste error from a debug session. Prometheus fired 30 requests per second at Grafana's /api/datasources endpoint. Grafana's Go runtime couldn't handle the concurrent connections, started returning 503s, and the dashboard went blank.

Without rate limiting, any misconfigured scraper — or any user refreshing aggressively — can DoS a service behind Traefik.

The RateLimit Middleware

Traefik's RateLimit middleware uses a token bucket algorithm. It's configured per-entrypoint or per-route:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
  namespace: traefik
spec:
  rateLimit:
    average: 100
    burst: 200
    period: 1s
    sourceCriterion:
      ipStrategy:
        depth: 1

average: 100 — 100 requests per second per source IP
burst: 200 — allow bursts up to 200 requests
period: 1s — the averaging window
sourceCriterion.ipStrategy.depth: 1 — trust the first X-Forwarded-For header (correct behind one proxy)

The depth: 1 is critical. Without it, Traefik uses the raw TCP connection IP, which is always the Traefik pod's IP in a Kubernetes cluster. Every client gets the same rate limit bucket. With depth: 1, Traefik reads the real client IP from X-Forwarded-For.

Applying It Per Service

Not every service needs the same rate limit. Grafana's dashboard needs a higher burst than the API:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: grafana-rate-limit
  namespace: monitoring
spec:
  rateLimit:
    average: 50
    burst: 100
    period: 1s
    sourceCriterion:
      ipStrategy:
        depth: 1
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.lab.woitzik.dev`)
      kind: Rule
      middlewares:
        - name: grafana-rate-limit
          namespace: monitoring
      services:
        - name: grafana
          port: 3000
  tls:
    secretName: lab-wildcard-tls

Whitelisting Internal Traffic

Prometheus, ArgoCD, and other internal services shouldn't be rate-limited. Traefik's RateLimit middleware supports requestHost and requestHeaders as source criteria, but the cleanest approach is a separate middleware for internal services:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: internal-rate-limit
  namespace: traefik
spec:
  rateLimit:
    average: 500
    burst: 1000
    period: 1s
    sourceCriterion:
      requestHost: {}

Internal services use requestHost (the Host header) as the source instead of IP. Since internal services all share the same cluster IP, IP-based rate limiting would bucket them all together.

The 429 Response

When a rate limit is hit, Traefik returns a 429 Too Many Requests with a Retry-After header. The client should respect this. Most HTTP clients don't. The Grafana dashboard showed a blank page because the browser retried immediately. Adding a client-side backoff in the Prometheus scraper config fixed the retry behavior:

scrapeConfig:
  - scrapeInterval: 30s
    scrapeTimeout: 10s
    metricsPath: /metrics

The real fix was the scrape interval. The rate limit was the safety net.

Monitoring Rate Limits

Traefik exposes rate limit metrics in Prometheus format:

traefikmiddleware_ratelimit_requests_total{middleware="monitoring-grafana-rate-limit"} 42
traefikmiddleware_ratelimit_rejected_total{middleware="monitoring-grafana-rate-limit"} 3

Grafana alert: if rejected_total increases by more than 10 per minute, something is hammering the service. This is the early warning for scrape storms, bot attacks, or misconfigured clients.

DEV Community