DEV Community

Achmad Dinofaldi Firmansyah
Achmad Dinofaldi Firmansyah

Posted on

Intermittent high latency (7s+) with Cloudflare Proxy to K3s (Traefik)

#k3s #cloudflare

I am experiencing a frustrating intermittent network issue and would appreciate any guidance on how to debug this further.

I have a web application (Frontend and API) hosted on a VM running K3s. The traffic is routed through Cloudflare to my K3s Traefik Ingress. Sometimes the requests are blazing fast (~50ms), but often they randomly spike to 500ms, take up to 7 seconds, or fail entirely with NS_BINDING_ABORTED in the browser console.

My Architecture & Setup:

  1. DNS/Proxy: Cloudflare (Proxy enabled / Orange Cloud).
  2. SSL Mode: Full (Strict).
  3. Origin Certificate: I am using a Cloudflare Origin CA Certificate (RSA 2048) applied in my K3s cluster.
  4. Host/Origin: A single VM running K3s.
  5. Ingress Controller: The default Traefik that comes with K3s.
  6. Domain: exam.example.com (Frontend) and api-exam.example.com (Backend).

The Symptoms:

  1. Inconsistent Latency: Request times fluctuate wildly without any obvious pattern (sometimes 50ms, sometimes 7+ seconds).
  2. Dropped Connections: The frontend frequently fails to fetch data from the API, resulting in NS_BINDING_ABORTED in the network tab.
  3. App/Pod Performance is Fine: I checked the logs inside my API pods, and the actual execution time is extremely fast (e.g., 24.437µs, 910.061µs). The application and database are not the bottleneck.

The delay is clearly happening at the network layer—either between the browser and Cloudflare, or between Cloudflare and my Traefik ingress. Since I haven't made any recent changes and the issue is intermittent, I suspect it might be related to connection queuing, rate-limiting, or header mismatch.

Here is my Ingress configuration:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: example-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 31536000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "strict-origin-when-cross-origin"
Permissions-Policy: "geolocation=(), microphone=(), camera=()"
Content-Security-Policy: "default-src 'self'; base-uri 'self'; frame-ancestors 'none'; object-src 'none'; script-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com https://cdn.datatables.net https://vjs.zencdn.net https://code.jquery.com 'unsafe-inline'; worker-src 'self' blob:; style-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdn.datatables.net https://vjs.zencdn.net https://fonts.googleapis.com https://code.jquery.com https://cdnjs.cloudflare.com https://s3.example.com 'unsafe-inline'; img-src 'self' data: blob: https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdn.datatables.net https://code.jquery.com https://s3.example.com; font-src 'self' data: https://cdn.jsdelivr.net https://fonts.gstatic.com https://s3.example.com; connect-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://s3.example.com; media-src 'self' https://vjs.zencdn.net https://s3.example.com; upgrade-insecure-requests"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: exam-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 63072000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "strict-origin-when-cross-origin"
Permissions-Policy: "geolocation=(), microphone=(), camera=()"
Content-Security-Policy: "default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval' 'unsafe-eval' https://accounts.google.com https://cdn.jsdelivr.net https://static.cloudflareinsights.com https://unpkg.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com data:; img-src 'self' data: blob: https://example.com https://exam.example.com https://s3.example.com; connect-src 'self' https://example.com https://api-exam.example.com https://static.cloudflareinsights.com https://s3.example.com https://api.iconify.design https://api.simplesvg.com https://api.unisvg.com https://cdn.jsdelivr.net https://unpkg.com https://accounts.google.com; object-src 'none'; base-uri 'self'; frame-ancestors 'none';"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: exam-api-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 31536000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "no-referrer"
X-Permitted-Cross-Domain-Policies: "none"
# This origin only ever returns JSON, so a locked-down CSP here is
# just defence-in-depth (e.g. against an error page ever reflecting
# input) rather than something the API actively relies on. CORS is
# handled by the app itself via CORS_ORIGINS (see main.go), not here.
Content-Security-Policy: "default-src 'none'; frame-ancestors 'none'"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: block-public-metrics
namespace: example
spec:
ipWhiteList:
sourceRange:
- "127.0.0.1/32"
Enter fullscreen mode Exit fullscreen mode

Here is my Middlewares configuration:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: example-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 31536000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "strict-origin-when-cross-origin"
Permissions-Policy: "geolocation=(), microphone=(), camera=()"
Content-Security-Policy: "default-src 'self'; base-uri 'self'; frame-ancestors 'none'; object-src 'none'; script-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com https://cdn.datatables.net https://vjs.zencdn.net https://code.jquery.com 'unsafe-inline'; worker-src 'self' blob:; style-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdn.datatables.net https://vjs.zencdn.net https://fonts.googleapis.com https://code.jquery.com https://cdnjs.cloudflare.com https://s3.example.com 'unsafe-inline'; img-src 'self' data: blob: https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://cdn.datatables.net https://code.jquery.com https://s3.example.com; font-src 'self' data: https://cdn.jsdelivr.net https://fonts.gstatic.com https://s3.example.com; connect-src 'self' https://example.com https://static.cloudflareinsights.com https://cdn.jsdelivr.net https://s3.example.com; media-src 'self' https://vjs.zencdn.net https://s3.example.com; upgrade-insecure-requests"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: exam-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 63072000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "strict-origin-when-cross-origin"
Permissions-Policy: "geolocation=(), microphone=(), camera=()"
Content-Security-Policy: "default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval' 'unsafe-eval' https://accounts.google.com https://cdn.jsdelivr.net https://static.cloudflareinsights.com https://unpkg.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com data:; img-src 'self' data: blob: https://example.com https://exam.example.com https://s3.example.com; connect-src 'self' https://example.com https://api-exam.example.com https://static.cloudflareinsights.com https://s3.example.com https://api.iconify.design https://api.simplesvg.com https://api.unisvg.com https://cdn.jsdelivr.net https://unpkg.com https://accounts.google.com; object-src 'none'; base-uri 'self'; frame-ancestors 'none';"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: exam-api-headers
namespace: example
spec:
headers:
frameDeny: true
contentTypeNosniff: true
stsSeconds: 31536000
stsIncludeSubdomains: true
stsPreload: true
customResponseHeaders:
Referrer-Policy: "no-referrer"
X-Permitted-Cross-Domain-Policies: "none"
# This origin only ever returns JSON, so a locked-down CSP here is
# just defence-in-depth (e.g. against an error page ever reflecting
# input) rather than something the API actively relies on. CORS is
# handled by the app itself via CORS_ORIGINS (see main.go), not here.
Content-Security-Policy: "default-src 'none'; frame-ancestors 'none'"
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: block-public-metrics
namespace: example
spec:
ipWhiteList:
sourceRange:
- "127.0.0.1/32"
Enter fullscreen mode Exit fullscreen mode

Example log from one of my pod:

21:14:49 | 200 | 24.437µs | 10.42.0.1 | GET | /health

21:14:51 | 204 | 21.963µs | 10.42.0.82 | OPTIONS | /api/ujian

21:14:51 | 204 | 9.154µs | 10.42.0.82 | OPTIONS | /api/auth/profile

21:14:52 | 200 | 910.061µs | 10.42.0.82 | GET | /api/ujian

21:14:54 | 200 | 56.765µs | 10.42.0.1 | GET | /health
Enter fullscreen mode Exit fullscreen mode

My Questions:

  1. Since Cloudflare acts as a reverse proxy, all traffic hitting my Traefik comes from Cloudflare IPs. Could Traefik be silently dropping/throttling connections due to built-in rate limits or connection limits (like InFlightReq) thinking it's a DDoS attack from a single IP?
  2. Are there specific forwardedHeaders or proxyProtocol configurations I need to apply to my K3s Traefik HelmChart to properly trust Cloudflare IPs?
  3. How can I properly debug where the request is hanging (Cloudflare edge vs Traefik vs K3s internal networking)?

Thank you in advance for your time and help!

Top comments (0)