DEV Community: Surendra Kumar

The On-Premise Kubernetes Challenge: A Tale of Two Traffics

Surendra Kumar — Sun, 20 Jul 2025 03:40:27 +0000

Service Mesh: Solving On-Premises Kubernetes Networking

When you're managing your own Kubernetes cluster on-premises, you have unmatched control—but also full responsibility for everything, especially networking. In modern microservices architectures, this responsibility is magnified by the sheer volume and complexity of service-to-service communication.

Two Types of Kubernetes Traffic

Kubernetes networking is commonly divided into:

North-South Traffic: Flows between the outside world and your cluster. Managed by Ingress Controllers.
East-West Traffic: Internal service-to-service communication within the cluster. This is where service meshes excel.

Distribution between these traffic types in a typical microservices setup emphasizes just how critical managing east-west traffic is:

Distribution of Kubernetes Traffic: North-South vs East-West

The On-Premise Struggle Without a Service Mesh

As your application scales, internal communication patterns get intricate. Without a service mesh, developers and operators are left to handle east-west traffic management manually, introducing several challenges:

Complex and Inconsistent Traffic Management: Strategies like canary releases, retries, or circuit breaking must be painstakingly hand-coded for each service.
Security Vulnerabilities: Each internal connection requires manual TLS setup and policy enforcement, leading to “soft target” vulnerabilities.
Opaque Observability: Debugging and monitoring require jumping between siloed logs—tracing a request becomes guesswork.
Developer Overload: Teams waste time implementing infrastructure features rather than business logic.

The impact of these challenges can be visualized as follows:

Challenges of Managing East-West Traffic Without a Service Mesh

Enter the Service Mesh

A service mesh is an infrastructure layer designed to manage east-west traffic through sidecar proxies—tiny, transparent network helpers injected alongside each service. The mesh handles critical concerns centrally and consistently, freeing developers and operators from networking boilerplate.

How a Service Mesh Helps

Feature	Practical Benefit
Advanced Traffic Control	Fine-tuned routing, canary deployments, intelligent load balancing, circuit breaking, and more
Zero-Trust Security	Automatic mutual TLS, identity-driven access policies, consistent enforcement across services
Deep Observability	End-to-end tracing, real-time metrics (latency, errors, traffic), and topology visualization
Developer Empowerment	Reduced boilerplate lets devs focus on features, not infrastructure tools

Visualizing Service Mesh Effectiveness

A service mesh isn’t just a technical luxury—it radically improves cluster resilience, security, and developer happiness for all but the simplest deployments.

Effectiveness of Service Mesh Features in Addressing On-Premises Kubernetes Challenges

Verdict: When Do You Need a Service Mesh?

For small, simple apps, manual traffic management might suffice. But as complexity grows, so does risk. A service mesh offers a robust, production-proven solution, trading a modest increase in stack complexity for vastly improved reliability, visibility, and security.

Bottom Line: In on-premise Kubernetes, a service mesh transforms internal networking from a source of struggle to a foundation for sustainable innovation.

⁂

CentOS 7 (Worker Node)

Surendra Kumar — Sun, 20 Jul 2025 03:28:19 +0000

CentOS 7 Kubernetes Worker Node Setup - Validation & Corrections

1. Repository Configuration

# Backup existing repo files
sudo cp -r /etc/yum.repos.d /etc/yum.repos.d.backup

# Update all CentOS repo files to use vault
sudo sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
sudo sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*

# Clean yum cache
sudo yum clean all
sudo yum makecache

Status: - CentOS 7 is EOL, so vault repositories are necessary.

2. System Package Installation

# Install required system packages
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

Status: - These are required for container runtime.

3. Kubernetes Repository Setup

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Status: - Uses the new Kubernetes repository format.

⚠️ Issues Found & Corrections

1. Container Runtime Configuration

Issue: Script installs containerd.io but doesn't configure it properly for Kubernetes.

Correction:

# Install containerd
sudo yum install -y containerd.io

# Configure containerd for Kubernetes
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

# Enable SystemdCgroup (CRITICAL for Kubernetes)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# Restart and enable containerd
sudo systemctl restart containerd
sudo systemctl enable containerd

2. Kubernetes Package Installation

Issue: Version pinning may cause issues if exact version isn't available.

Correction:

# Install Kubernetes components (use --nogpgcheck if GPG issues occur)
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# Enable kubelet (don't start yet - will fail until joined)
sudo systemctl enable kubelet

3. System Configuration Issues

Issue: Several sysctl parameters are duplicated and kernel modules aren't properly configured.

Correction:

# Load required kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter

# Make modules persistent
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

# Set sysctl parameters
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

4. CNI Configuration Issues

Issue: CNI version might be outdated, and worker nodes don't need manual CNI configuration.

Correction:

# CNI will be configured automatically by the CNI plugin (Flannel/Calico)
# Only ensure CNI directories exist
sudo mkdir -p /etc/cni/net.d
sudo mkdir -p /opt/cni/bin

5. Kubelet Configuration

Issue: Manual kubelet configuration is unnecessary and potentially problematic.

Correction:

# Remove manual kubelet configuration - kubeadm will handle this
# The kubelet will be configured automatically during join

🔧 Recommended Complete Setup Script

#!/bin/bash

# 1. Configure CentOS 7 repositories (EOL)
sudo cp -r /etc/yum.repos.d /etc/yum.repos.d.backup
sudo sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
sudo sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
sudo yum clean all && sudo yum makecache

# 2. Install required packages
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

# 3. Install container runtime
sudo yum install -y containerd.io

# 4. Configure containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

# 5. Add Kubernetes repository
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

# 6. Install Kubernetes components
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable kubelet

# 7. Configure system
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# 8. Load kernel modules
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

# 9. Configure sysctl
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

# 10. Configure firewall
sudo systemctl stop firewalld
sudo systemctl disable firewalld

# 11. Create CNI directories
sudo mkdir -p /etc/cni/net.d
sudo mkdir -p /opt/cni/bin

# 12. Join the cluster (replace with your actual join command)
# sudo kubeadm join <master-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

🚨 Critical Points

Don't start kubelet before joining - It will fail until the node joins the cluster
SystemdCgroup = true is essential for containerd with Kubernetes
Firewall must be disabled or properly configured for Kubernetes ports
Swap must be disabled completely
Use exact join command from your master node
Verify containerd is running before attempting to join

🔍 Verification Steps

After setup, verify before joining:

# Check containerd status
sudo systemctl status containerd

# Check kubelet status (should be inactive until joined)
sudo systemctl status kubelet

# Verify swap is disabled
free -h

# Check if required kernel modules are loaded
lsmod | grep br_netfilter
lsmod | grep overlay

Service Mesh

Surendra Kumar — Sat, 19 Jul 2025 14:04:01 +0000

Imagine you are an engineer at a fast-growing e-commerce company like Amazon. Your team started with a monolithic application—a single, large codebase handling user requests, inventory, and payments. As the company grew, you migrated to a micro-services architecture with separate services for:

User Service (handles user accounts)
Order Service (processes orders)
Inventory Service (tracks stock)
Payment Service (handles payments)

At first, everything ran smoothly. However, as complexity increased, several problems emerged:

🔴 Problem #0: Authentication & Authorization

Each micro-service must handle logins & permissions separately, resulting in duplicated and inconsistent security logic.
🔴 Problem #1: Hard-to-Debug Failures

PopCustomers report orders failing randomly. Logs indicate errors in the Payment Service, but it’s unclear whether these issues are from network glitches, load spikes, or faulty service updates.
🔴 Problem #2: Security Risks

Sensitive data, like payment details, is transmitted between services unencrypted. Without proper encryption, interception becomes a major risk.
🔴 Problem #3: Load Balancing Issues

The Order Service is overwhelmed while the Inventory Service remains underutilized. Efficient traffic distribution is missing.
🔴 Problem #4: Slow Rollouts & Deployments

Introducing a new version of the Payment Service poses a risk. Testing it on only 10% of users is ideal, but gradual rollouts without downtime are challenging.

This growing complexity makes managing your microservices painful. This is where a Service Mesh comes in! 🚀

1. Why Do You Need a Service Mesh?

Before service meshes, organizations manually managed service-to-service communication using:

Custom code in each micro-service
Reverse proxies (e.g., NGINX, HAProxy)
Load balancers
API gateways

While this approach worked, it had serious limitations:

No standardized way to handle retries, timeouts, and failures.
Security risks due to unencrypted communication.
Limited observability making it hard to trace requests across multiple services.

Real-World Example: Netflix

Netflix’s microservices faced issues like:

Slow response times because of ineffective traffic control.
DDoS threats due to the absence of a central security layer.
Authentication issues from each microservice handling its own login logic.

https://netflixtechblog.com/zero-configuration-service-mesh-with-on-demand-cluster-discovery-ac6483b52a51

2. How Service Mesh Helps

A Service Mesh is an infrastructure layer that manages service-to-service communication within a distributed micro-services architecture. It abstracts away networking concerns, thereby enhancing:

Security: Through secure, encrypted communication (mTLS).
Traffic Management: Via intelligent routing, load balancing, and retries.
Observability: With integrated logging, tracing, and monitoring.
Resilience: By using circuit breakers and fault injection.

Key benefits include decoupling networking logic from application code and centralizing security policies.

3. Service Mesh Architecture

A service mesh consists of two primary components:

3.1 Data Plane (Sidecar Proxies)

Each micro-service runs alongside a sidecar proxy that intercepts all incoming and outgoing traffic.

Example: Envoy Proxy

Responsibilities:
- Service Discovery & Load Balancing: Efficiently routes traffic among service instances.
- Retries & Circuit Breaking: Enhances resilience by managing failures.
- Security Enforcement: Uses mTLS for zero-trust security.
- Telemetry Collection: Gathers metrics, logs, and traces.

3.2 Control Plane

Responsibilities:
- Manages and configures the sidecar proxies.
- Applies traffic rules, security policies, and observability settings.

These components work together to enforce consistent communication policies across your micro-services.

How It Works: Without vs. With Service Mesh

Without Service Mesh:

Service A → Service B(Custom retry logic, security, and logging embedded in application code)
Service B → Database(Direct connection without security enforcement)

With Service Mesh:

Service A → Envoy Proxy (handles retries, security, monitoring) → Service B
Service B → Envoy Proxy → Database(All communications are secured and monitored)

4. Popular Service Mesh Implementations

Service Mesh	Proxy Used	Best For
Istio	Envoy	Kubernetes-native applications
Linkerd	Linkerd-proxy	Simplicity and lightweight setups
Consul	Envoy	Multi-cloud and hybrid environments
AWS App Mesh	Envoy	AWS-native applications

For more details, see the Comparison of Service Meshes.

5. Benefits of Service Mesh

5.1 Security: Encrypted & Secure Communication (mTLS)

Problem: Unencrypted traffic is vulnerable to interception.
Example: A banking app transmitting user passwords in plain text.
Solution: Enforce mTLS to ensure all communication is secure and authenticated.

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT
EOF

5.2 Traffic Control

💡 Benefit: Manage how traffic flows between services, allowing canary deployments, fault injection, and load balancing.

🔀 A/B Testing with Traffic Splitting

📌 Route 80% of traffic to v1 and 20% to v2 of the reviews service


kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: default
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2
      weight: 20
EOF

🔹 Now, when you access productpage, most users will see v1, while 20% get v2.

Example: Netflix gradually releases a new video recommendation algorithm.\

5.3 Observability: Debugging Made Easy

Problem: Locating the root cause of failures across multiple services is challenging.
Example: Uber’s thousands of microservices made debugging nearly impossible.
Solution: Integrate with Jaeger and Kiali for distributed tracing and visualization.


istioctl dashboard kiali  # Open the Kiali dashboard for service mesh visualization

5.4 Load Balancing: Efficient Traffic Distribution

Problem: Imbalanced traffic can overwhelm some services while underutilizing others.
Example: Amazon’s checkout service receives millions of requests per minute.
Solution: Dynamically distribute requests using advanced load balancing algorithms.


apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: checkout-service
spec:
  host: checkout-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN  # Directs traffic to the least busy instance

Additional Capabilities

Traffic Management: Intelligent routing, load balancing, retries, and failover.
Security Enhancements: mTLS, role-based access control (RBAC), and centralised authentication/authorization.
Observability & Monitoring: Distributed tracing (Jaeger, Zipkin), logging/metrics collection (Prometheus, Grafana, Kiali), and dynamic service discovery.
Resilience & Fault Tolerance: Circuit breaking, rate limiting, and fault injection for chaos engineering.

7. When to Use a Service Mesh?

Scenario	Use Service Mesh?
Small app with fewer than 5 services	❌ No (Overhead is too high)
10+ microservices	✅ Yes (Provides standardized communication)
Multi-cloud or hybrid deployments	✅ Yes (Simplifies cross-cloud networking)
High-security environments (banking, healthcare)	✅ Yes (Enforces mTLS and centralized authentication)

8. Real-World Benefits & Use Cases

✅ Case Study #1: Airbnb – Scaling Microservices

Problem:

Airbnb had thousands of microservices, making debugging nearly impossible due to scattered logs.
Solution:

They implemented Istio (Service Mesh) along with Jaeger for distributed tracing.
Outcome:

Engineers could trace failures in seconds, reducing incident response time by 70%.

✅ Case Study #2: Stripe – Secure Payment Transactions

Problem:

Stripe processes millions of financial transactions and required end-to-end encryption between microservices.
Solution:

They deployed a Service Mesh with mTLS to encrypt all service-to-service communication automatically.
Outcome:
- Zero unencrypted transactions
- Full compliance with banking security standards

✅ Case Study #3: Netflix – Canary Deployments

Problem:

Netflix needed to test new versions of its recommendation engine on a small subset of users to avoid downtime.
Solution:

They used Istio’s traffic splitting feature to direct 10% of traffic to the new version while maintaining 90% on the stable version.
Outcome:
- Safer rollouts
- No downtime during updates

Conclusion

A Service Mesh—using tools like Istio, Linkerd, or Consul—streamlines inter-service communication in a microservices architecture by abstracting networking, security, and observability away from application code. This approach lets developers focus on core business logic while ensuring robust, secure, and resilient interactions between services.

Feel free to reach out if you need further clarification or want to explore a live demo of these concepts in your environment!

References:
https://www.alibabacloud.com/blog/getting-started-with-service-mesh-origin-development-and-current-status_597241

Istio Config for path based service redirection

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: core-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: core-virtualservice
  namespace: istio-system
spec:
  gateways:
  - core-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /cg
    route:
    - destination:
        host: stg-content-service-api.all-staging.svc.cluster.local
        port:
          number: 8085
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: stg-go-mobile-api.all-staging.svc.cluster.local
        port:
          number: 3333

Istio Config for route based service redirection

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: core-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 30012
      name: content-port
      protocol: HTTP
    hosts:
    - '*'
  - port:
      number: 30013
      name: mobile-port
      protocol: HTTP
    hosts:
    - '*'
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: combined-virtualservice
  namespace: istio-system
spec:
  hosts:
  - '*'
  gateways:
  - core-gateway
  http:
  - match:
    - port: 30012
    route:
    - destination:
        host: stg-content-service-api.all-staging.svc.cluster.local
        port:
          number: 8085
  - match:
    - port: 30013
    route:
    - destination:
        host: stg-go-mobile-api.all-staging.svc.cluster.local
        port:
          number: 3333