Mastering ingress-nginx

Advanced Optimization Techniques for High-Performance Kubernetes Ingress

As a seasoned DevOps engineer with 15 years of experience, I've seen the evolution of ingress controllers in Kubernetes. Today, we'll dive deep into optimizing ingress-nginx, one of the most popular ingress controllers in the Kubernetes ecosystem.

1. Fine-tuning Worker Processes and Connections

One of the first areas to optimize is the NGINX worker configuration. By default, ingress-nginx uses auto-detection, but for high-traffic environments, manual tuning can yield better results.

controller:
  config:
    worker-processes: "8"
    max-worker-connections: "65536"

Adjust these values based on your server's CPU cores and expected concurrent connections.

2. Leveraging HTTP/2 and TLS 1.3

Enable HTTP/2 and TLS 1.3 to improve connection efficiency and security:

controller:
  config:
    use-http2: "true"
    ssl-protocols: "TLSv1.2 TLSv1.3"

3. Optimizing SSL/TLS

Implement OCSP stapling and adjust SSL buffer size:

controller:
  config:
    ssl-buffer-size: "4k"
    ssl-ocsp: "on"

4. Tuning Timeouts and Keepalive

Adjust timeouts and keepalive settings to balance between resource usage and client needs:

controller:
  config:
    keep-alive: "75"
    keep-alive-requests: "100"
    upstream-keepalive-connections: "1000"
    upstream-keepalive-timeout: "60"
    client-header-timeout: "60s"
    client-body-timeout: "60s"

5. Implementing Caching and Compression

Enable and configure caching and compression for improved performance:

controller:
  config:
    use-gzip: "true"
    gzip-level: "6"
    gzip-types: "application/json application/x-javascript text/css text/javascript"
    proxy-cache-path: "/tmp/nginx-cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off"

Then, in your Ingress resource:

annotations:
  nginx.ingress.kubernetes.io/proxy-cache: "my_cache"
  nginx.ingress.kubernetes.io/proxy-cache-valid: "200 60m"

6. Implementing Rate Limiting

Protect your services with intelligent rate limiting:

annotations:
  nginx.ingress.kubernetes.io/limit-rps: "10"
  nginx.ingress.kubernetes.io/limit-rpm: "100"

7. Optimizing Backend Connections

Fine-tune how ingress-nginx connects to your backends:

controller:
  config:
    upstream-keepalive-connections: "1000"
    upstream-keepalive-timeout: "60"
    upstream-keepalive-requests: "1000"

8. Implementing Custom Error Pages

Enhance user experience with custom error pages:

controller:
  config:
    custom-http-errors: "404,503"
  defaultBackendService: "default/custom-error-pages-service"

9. Leveraging Lua for Advanced Functionality

ingress-nginx supports Lua scripting for custom logic. Here's an example of using Lua to add custom headers:

controller:
  config:
    load-balance: "ewma"
    lua-shared-dicts: "configuration_data:5M"
  extraVolumeMounts:
    - name: lua-scripts
      mountPath: /etc/nginx/lua
  extraVolumes:
    - name: lua-scripts
      configMap:
        name: lua-scripts

Then, create a ConfigMap with your Lua script:

apiVersion: v1
kind: ConfigMap
metadata:
  name: lua-scripts
data:
  custom_headers.lua: |
    local headers = ngx.req.get_headers()
    ngx.header["X-Custom-Header"] = "ingress-nginx"

10. Implementing Canary Deployments

Use ingress-nginx's built-in canary deployment feature:

annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-weight: "30"

11. Monitoring and Metrics

Enable Prometheus metrics for comprehensive monitoring:

controller:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

Conclusion

Optimizing ingress-nginx is a continuous process that requires understanding your specific traffic patterns and application requirements. The configurations provided here serve as a starting point for high-performance setups, but always test thoroughly in your environment.

Remember, while these optimizations can significantly improve performance, they may also increase resource usage. Monitor your ingress controller closely after implementing changes and be prepared to fine-tune based on real-world results.

By leveraging these advanced techniques, you can ensure your ingress-nginx setup is performant, secure, and ready to handle enterprise-level traffic with ease.

---

Comments

[Dream]

Great overview on optimizing ingress-nginx! For HTTP/2 and TLS 1.3, do you have any benchmarks comparing performance before and after enabling them?

[Ofir Haim]

Thank you for your comment, @dreama

Yes, we do have benchmarks comparing performance before and after enabling HTTP/2 and TLS 1.3. Here are our findings:

1. The base request to our server with Nginx installed on Docker took 0.190 ms.
2. With Nginx installed on Kubernetes (K8s) without any customizations, the request time was reduced to 0.130 ms.
3. After applying our customizations to Nginx, the request time further improved to 0.080 ms.

We hope this helps illustrate the performance enhancements. Let us know if you have any more questions!