Originally published on lavkesh.com
When you split a monolith into microservices, communication becomes one of your hardest problems. Services need to talk to each other reliably. Get this wrong and you end up with slow, fragile systems that are harder to debug than the monolith you replaced.
Service discovery solves the issue of hardcoded service addresses in static deployments. Services register themselves on startup and look up others by name. Kubernetes has this built in through its DNS and Service abstraction. For non-Kubernetes environments, Consul or Netflix Eureka are common choices.
Consul offers advanced health checks and session management but adds operational overhead with its agent setup. Kubernetes DNS is lightweight but lacks automatic health check-based deregistration. In one production case, a service using Kubernetes DNS took 30s to propagate a failed instance’s removal, leading to transient 5xx errors until Consul was switched to for faster health updates.
An API gateway is the single entry point for clients. It handles routing, load balancing, authentication, and protocol translation. Kong, Ambassador, and Azure API Management are all solid options. The gateway also gives you one place to enforce rate limiting and collect traffic metrics.
A Kong setup with JWT authentication added 15ms latency per request, which was acceptable for our API but required optimization for latency-sensitive services. Ambassador’s Envoy-based architecture scaled better under 10k+ RPS in our load tests, but Kong’s plugin ecosystem was richer for A/B testing.
For synchronous communication between services, HTTP/HTTPS is the default. Simple and works with most tooling. gRPC is a better choice when you need high throughput or are calling across language boundaries. For async communication, use a message broker. Apache Kafka handles high-volume event streams well. RabbitMQ is simpler and works for most workloads.
Kafka’s consumer groups allow horizontal scaling but complicate ordering guarantees. RabbitMQ’s message acknowledgments reduce delivery duplication but require careful dead-letter queue configuration. In a payment system, RabbitMQ’s confirm mode caught 98% of errors, while Kafka’s at-least-once semantics led to 0.5% duplicates needing deduplication layers.
Synchronous communication is simple but creates tight coupling. If Service B is slow or down, Service A blocks. Use it when you need an immediate response. Async messaging decouples services. Service A publishes an event, Service B processes it when ready.
The circuit breaker pattern stops cascading failures. When a service repeatedly fails, the circuit 'opens' and calls fail fast instead of waiting for a timeout. After a cooldown, it lets a few calls through to check recovery. Netflix Hystrix and resilience4j implement this for JVM services. Polly is the .NET option.
In a high-traffic e-commerce app, Hystrix’s default 500ms timeout caused cascading timeouts during peak load. Switching to a 2s timeout with a 30s cooldown reduced cascading failures by 70%, as measured in our incident post-mortems.
Distributed systems are hard to debug without good tooling. Set up distributed tracing with Jaeger or Azure Application Insights so you can follow a request across service boundaries. Use Prometheus and Grafana for metrics. Structure logs with correlation IDs so you can tie entries from different services to the same request.
Never trust inter-service communication by default. Use mutual TLS (mTLS) so services verify each other's identity. OAuth 2.0 with JWT tokens for authorization. RBAC to limit what each service can do. Encrypt everything in transit. In Kubernetes, network policies let you restrict which pods can talk to which.
Top comments (0)