DEV Community

Cover image for How we solved cache invalidation in Kubernetes with a headless service
Apoorv Tyagi
Apoorv Tyagi

Posted on • Originally published at apoorvtyagi.tech

How we solved cache invalidation in Kubernetes with a headless service

#computerscience #architecture #kubernetes #distributedsystems

⬅️ Background

At my previous organization, we were building a health platform, and one of the most critical, high-traffic components was our Lab test booking service. One of its core functions is to serve package details like inclusions, prices, and provider information for thousands of user requests daily.

The initial architecture was simple: every single request meant a new query to a database. As traffic scaled, our database struggled. Latency spiked, and CPU usage became a constant concern.

With mostly static data that rarely changes, the clear solution was to cache it. But the execution is where we got creative. This is the story of how we implemented a zero-cost, in-memory cache in our Node.js microservices and used a Kubernetes feature, “The Headless Service” to handle distributed cache invalidation flawlessly.

🛑 The Problem: Fast Caching & Scalable Invalidation

We decided on in-memory caching using a library like @nestjs/cache-manager to slash DB load and boost response times. However, in a Kubernetes cluster, where our service runs across multiple, dynamic pods, this creates a major headache: Cache Inconsistency.

If an admin updates a package price in the MySQL database, only the single pod that handled the write sees the change immediately. The other pods are serving users stale data - consistency, gone*.*

Furthermore, Kubernetes pod IPs are temporary. They change during restarts, deployments, and scaling events. We couldn't rely on hardcoded lists. Our invalidation system needed a reliable way to discover every running pod at the exact moment of a data update.

We needed a broadcasting mechanism.

🛠️ The Solution: The K8s Headless Service

Our strategy was a hybrid approach: local caching for performance, and a targeted internal HTTP broadcast for invalidation.

Step 1: Headless Services for Pod Discovery

This is the key to the whole solution. A standard Kubernetes service acts as a load balancer with a single Virtual IP. But a Headless Service is different. By setting clusterIP: None in the Service manifest (.spec.clusterIP), Kubernetes skips the load balancer and, crucially, returns a list of individual DNS A records for every active pod’s IP address.

Here is the simple(yet powerful) Headless Service manifest we deployed:

apiVersion: v1
kind: Service
metadata:
  name: booking-service-headless
spec:
  clusterIP: None             # THE TRICK: Makes it Headless
  selector:
    app: booking-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
Enter fullscreen mode Exit fullscreen mode

Now, when we resolve booking-service-headless.default.svc.cluster.local, we get an array of all current, healthy pod IPs.

For visualization, here's a diagram of Headless Services in action:

Kubernetes Headless Service Explained

Step 2 & 3: Broadcast Implementation

In the next step, we added kubectl into our Docker image for API queries. This gives the pod the power to query the Kubernetes API directly.

To implement this safely, we followed the Principle of Least Privilege. We created a dedicated ServiceAccount for our booking service and bound it to a specific Role using RBAC (Role-Based Access Control). This Role was strictly limited: it only allowed the list and get operations on the endpoints resource within its own namespace. This ensures that even if a pod is compromised, an attacker cannot delete resources or view sensitive secrets.

Second, we created a simple, internal /invalidate-cache API endpoint on our booking service. When we hit this endpoint, it clears a specific cache key (passed in the request body) using the in-memory cache’s del() method.

When an admin update happens, and the data is written to MySQL, the pod that handles the write executes a bash script using kubectl and curl:

It uses kubectl to get the list of IPs from the Headless Service endpoints, then sends a direct HTTP POST request to the internal /invalidate-cache endpoint on each pod.

# Executed by the Node.js process on update
pod_ips=$(kubectl get endpoints listing-headless-service -o jsonpath='{.subsets[0].addresses[*].ip}')

for ip in $pod_ips; do
   curl -X POST http://$ip:3000/v2/invalidate\
       -H "Content-Type: application/json" \
       -d '{"key": "package-123"}' # Clears the specific key
done
Enter fullscreen mode Exit fullscreen mode

This ensures the system is:

  • Dynamic: The IP list is fresh at invalidation time, handling upscaling, downscaling and restarts seamlessly.

  • Targeted: We clear only the stale package key, maximizing cache retention.

  • Cost-Effective: Zero new service bills.

  • Reliability: Add curl retries and logging for failed broadcasts.

For those looking to keep their Docker images clean, you can avoid installing kubectl and curl entirely by using the official @kubernetes/client-node library. By using the library, your Node.js process can query the Kubernetes API directly from within the code. This makes the logic more testable, allows for better error handling, and removes the overhead of spawning shell processes. It turns your infrastructure logic into standard application code.

📈 Production Results and Takeaways

Post-rollout, our MySQL database load has consistently remained low, and our cache hit rate has remained above 95%+. We have had zero reported incidents of users seeing stale package data. The Headless Service trick proved robust, even during high-traffic deployments.

Graph

This solution proves that sometimes, the most elegant and cheapest solution isn't an expensive managed service, but a creative application of your existing tools*.* It’s about being smart with your infrastructure, not just throwing money at the problem.

Top comments (0)