There is a moment in every distributed system when the application stops being a collection of services and starts becoming a small country.
At first, you have three services.
An API.
An authentication service.
A database.
Everything is simple.
The API calls authentication. Authentication talks to the database. Requests move around. Logs tell you what happened. If something breaks, you open the code and fix it.
Then the system grows.
Ten services.
Twenty.
Fifty.
Suddenly, every service needs to know about retries.
Every service needs timeouts.
Every service needs authentication between services.
Every service needs metrics.
Every service needs distributed tracing.
Every service needs circuit breakers.
Every service needs traffic policies.
And every team implements these things slightly differently.
That is where the idea of a service mesh becomes interesting.
A service mesh is not simply “a proxy in front of every service.”
That description is technically convenient and architecturally incomplete.
A service mesh is a distributed infrastructure layer for controlling service-to-service communication.
The application should focus on business logic.
The mesh should handle much of the communication logic surrounding that business logic.
That distinction is powerful.
Instead of every service learning how to perform retries, enforce mutual TLS, discover other services, emit telemetry, route traffic, and survive network failures, we can move those responsibilities into infrastructure surrounding the services.
The application says:
Process this order.
The mesh says:
How should the request reach the order service?
Should the request be encrypted?
Should it be retried?
How long should we wait?
Which version should receive it?
Should this request be traced?
Is the destination healthy?
Should traffic be shifted?
The application remains concerned with what.
The mesh becomes concerned with how.
And today, we are going to build one.
Not a production replacement for Istio, Linkerd, or another mature service-mesh implementation.
Something more interesting.
A service mesh from first principles.
We are going to discover what actually makes a service mesh a service mesh.
The Problem We Are Actually Solving
Imagine this architecture:
┌──────────────┐
│ API Gateway │
└──────┬───────┘
│
┌─────────┴─────────┐
│ │
┌─────▼─────┐ ┌─────▼─────┐
│ Users │ │ Orders │
└─────┬─────┘ └─────┬─────┘
│ │
└─────────┬─────────┘
│
┌─────▼─────┐
│ Payments │
└───────────┘
It looks simple.
But imagine that Orders calls Payments.
Now ask:
What happens if Payments takes 10 seconds to respond?
What happens if Payments is temporarily unavailable?
What happens if there are three versions of Payments?
What happens if one Payments instance is unhealthy?
What happens if the network connection fails after the payment request reaches the server but before the response reaches Orders?
What happens if Orders needs to prove its identity to Payments?
What happens if we want to send 5% of traffic to Payments v2?
What happens if we need to trace:
API → Orders → Payments → Ledger
across multiple machines?
Without a mesh, application developers often implement these behaviors directly.
That means:
Orders
├── retry logic
├── timeout logic
├── TLS logic
├── discovery logic
├── load balancing
├── metrics
├── tracing
└── circuit breaker
Payments
├── retry logic
├── timeout logic
├── TLS logic
├── discovery logic
├── load balancing
├── metrics
├── tracing
└── circuit breaker
We have duplicated infrastructure logic.
A service mesh changes the architecture.
CONTROL PLANE
┌─────────────────────────┐
│ Service Registry │
│ Configuration │
│ Certificates │
│ Routing Policies │
│ Health Information │
└────────────┬────────────┘
│
┌──────────┴──────────┐
│ │
┌────▼────┐ ┌────▼────┐
│ Proxy │ │ Proxy │
│ Sidecar │ │ Sidecar │
└────┬────┘ └────┬────┘
│ │
┌────▼────┐ ┌────▼────┐
│ Orders │──────────▶│Payments │
└─────────┘ └─────────┘
The application does not need to understand all of this.
The proxy does.
The Two Halves of a Service Mesh
A useful mental model is that a service mesh has two major components:
SERVICE MESH
│
┌─────────┴─────────┐
│ │
CONTROL PLANE DATA PLANE
│ │
Configuration Requests
Discovery Routing
Certificates Retries
Policies Timeouts
Telemetry
The control plane makes decisions.
The data plane executes them.
This separation is one of the most important ideas in modern distributed infrastructure.
The control plane might say:
orders should send 80% of traffic to payments-v1
orders should send 20% to payments-v2
timeout = 2 seconds
retry = 2 attempts
TLS = required
The data plane receives an actual request and applies those rules.
This gives us an important principle:
The control plane should not be in the critical path of every request.
If the control plane disappears for five minutes, existing proxies should ideally continue serving traffic using their last known configuration.
That is the difference between:
control plane controls traffic
and:
control plane carries traffic
A good mesh does the first.
Our Architecture
Let's build a simplified architecture.
┌──────────────────────┐
│ Control Plane │
│ │
│ Service Registry │
│ Config Store │
│ Policy Engine │
│ Certificate Manager │
└──────────┬───────────┘
│
Configuration / Updates
│
┌─────────────────────┴─────────────────────┐
│ │
┌─────▼─────┐ ┌─────▼─────┐
│ Proxy │ │ Proxy │
│ Orders │ │ Payments │
└─────┬─────┘ └─────┬─────┘
│ │
┌─────▼─────┐ ┌─────▼─────┐
│ Orders │ ───────── Service ─────────▶ │ Payments │
└───────────┘ Communication └───────────┘
Our first version needs:
- Service registration
- Service discovery
- Local proxy
- Request forwarding
- Load balancing
- Health checking
- Timeouts
- Retries
- Circuit breaking
- Routing policies
- Mutual TLS
- Telemetry
- Dynamic configuration
That sounds like a lot.
It is.
That is precisely why service meshes are interesting.
They sit at the intersection of networking, distributed systems, security, observability, and operating systems.
Step One: Service Registration
Before a service can communicate with another service, we need to know where it lives.
Suppose Payments has three instances:
payments-1 → 10.0.0.11:8080
payments-2 → 10.0.0.12:8080
payments-3 → 10.0.0.13:8080
We could hardcode these addresses.
That would be terrible.
Instances disappear.
Containers restart.
IP addresses change.
Machines fail.
Instead, services register themselves.
Our registry might expose:
POST /services/register
With:
{
"name": "payments",
"address": "10.0.0.11",
"port": 8080,
"version": "v1"
}
The registry stores:
payments:
- 10.0.0.11:8080
- 10.0.0.12:8080
- 10.0.0.13:8080
But registration alone is not enough.
We need leases.
A service should not remain registered forever if it has died.
So we give every registration a TTL.
For example:
registration TTL = 15 seconds
heartbeat interval = 5 seconds
The service periodically sends:
POST /services/payments/heartbeat
If heartbeats stop, the registry eventually removes the instance.
This gives us:
service lifecycle
│
├── register
│
├── heartbeat
│
├── heartbeat
│
└── disappear
│
▼
expire lease
This is already a distributed-systems problem.
You are not really asking:
“Is this service alive?”
You are asking:
“When should the system stop believing that this service is alive?”
Those are very different questions.
Step Two: Service Discovery
Now Orders wants to call Payments.
Orders does not need to know:
10.0.0.11
10.0.0.12
10.0.0.13
It asks the control plane:
GET /services/payments
The control plane returns:
{
"service": "payments",
"instances": [
{
"address": "10.0.0.11",
"port": 8080,
"version": "v1"
},
{
"address": "10.0.0.12",
"port": 8080,
"version": "v1"
}
]
}
But there is another important optimization.
The proxy should cache discovery information.
We do not want this:
Every request
│
▼
Control Plane
│
▼
Service Instance
That would turn the control plane into a bottleneck.
Instead:
Control Plane
│
configuration
│
▼
Local Proxy
│
┌───────┼───────┐
▼ ▼ ▼
request request request
│ │ │
└───────┴───────┘
The proxy maintains a local view of the service topology.
Step Three: The Sidecar Proxy
Now we reach the heart of the architecture.
Every service gets a proxy.
┌─────────────────────────┐
│ Orders │
│ │
│ application │
│ │
└───────────┬─────────────┘
│
│ localhost
▼
┌─────────────────────────┐
│ Orders Proxy │
│ │
│ discovery │
│ routing │
│ retries │
│ timeout │
│ TLS │
│ metrics │
└───────────┬─────────────┘
│
│ network
▼
Payments Proxy
│
▼
Payments App
The application sends requests to its local proxy.
The proxy forwards them.
This is the sidecar model.
The application can simply call:
http://payments/pay
while the proxy decides how that logical destination maps to an actual instance.
This is where the mesh becomes transparent.
Step Four: Intercepting Traffic
How does the application automatically send traffic through the proxy?
There are multiple approaches.
One is explicit configuration.
The application sends requests to:
localhost:15001
The proxy then handles routing.
Another approach uses network-level traffic interception.
For example:
Application
│
▼
iptables / eBPF / networking layer
│
▼
Proxy
│
▼
Destination
The application thinks it is connecting directly to another service.
The networking layer redirects the connection.
This is one of the reasons service meshes can feel almost magical.
The application has not changed.
The network behavior has.
Step Five: The Forwarding Engine
Our proxy needs a simple pipeline.
Conceptually:
request
│
▼
parse
│
▼
identify destination
│
▼
load configuration
│
▼
select instance
│
▼
apply timeout
│
▼
apply retry policy
│
▼
send request
│
▼
collect telemetry
│
▼
response
A simplified proxy API might look like:
Proxy.forward(request):
destination = resolve(request.host)
policy = policy_for(destination)
instance = load_balancer.choose(destination)
response = send(
instance,
timeout=policy.timeout
)
if should_retry(response, policy):
retry()
return response
But distributed systems immediately make this more complicated.
What counts as a retryable failure?
A TCP connection failure?
HTTP 503?
HTTP 429?
A timeout?
What if the request reached the server?
This is where engineering gets interesting.
Retries Are Dangerous
Retries sound harmless.
They are not.
Imagine:
Orders → Payments
Orders sends:
POST /charge
Payments successfully charges the customer.
But the response is lost.
Orders sees:
timeout
The proxy retries.
Now Payments receives:
POST /charge
again.
The customer may be charged twice.
Therefore:
A retry policy is not simply a networking feature. It is an application semantics problem.
Safe retries usually require idempotency or knowledge that the operation is safe to repeat.
For example:
Idempotency-Key: payment-12345
Payments can recognize duplicate attempts.
Our mesh can enforce conservative retry policies, but it cannot magically determine whether every operation is semantically safe.
That boundary matters.
Load Balancing
Suppose we have:
payments-1
payments-2
payments-3
Our proxy needs to choose one.
The simplest strategy is round robin:
request 1 → payments-1
request 2 → payments-2
request 3 → payments-3
request 4 → payments-1
Another option is weighted routing:
payments-v1 → 90%
payments-v2 → 10%
Or least connections:
payments-1 → 14 active
payments-2 → 4 active
payments-3 → 9 active
Choose:
payments-2
We can define an interface:
LoadBalancer.choose(instances)
and implement:
RoundRobin
Weighted
Random
LeastConnections
ConsistentHash
Now routing becomes policy rather than application code.
Health Checking
A registered service is not necessarily a healthy service.
We therefore need active health checks.
The proxy can periodically call:
GET /health
If the response is:
200 OK
we consider the instance healthy.
If it repeatedly fails:
failure count = 1
failure count = 2
failure count = 3
we temporarily remove the instance from the load-balancing pool.
But health checking introduces another distributed-systems tradeoff.
A service can be healthy from one machine's perspective and unreachable from another.
There is no universal magical concept of “network health.”
Health information is contextual.
Our system should therefore treat health as a signal, not absolute truth.
Circuit Breaking
Now imagine Payments is broken.
Orders keeps sending requests.
Requests fail.
Retries make things worse.
Traffic increases.
The failing service becomes even more overloaded.
We need a circuit breaker.
The classic model is:
failures
│
▼
┌────────────────┐
│ CLOSED │
└───────┬────────┘
│
failure threshold
│
▼
┌────────────────┐
│ OPEN │
└───────┬────────┘
│
cooldown
│
▼
┌────────────────┐
│ HALF-OPEN │
└───────┬────────┘
│
┌──────┴──────┐
▼ ▼
success failure
│ │
▼ ▼
CLOSED OPEN
Suppose:
failure threshold = 5
cooldown = 30 seconds
After five consecutive failures, the proxy stops sending traffic.
Instead, it fails quickly.
This protects the system from cascading failures.
A service mesh is therefore not just routing infrastructure.
It becomes a failure-management layer.
Timeouts
Every network request should have a deadline.
Without a timeout:
Orders
│
└──── waiting forever ────▶ Payments
One blocked dependency can consume all available threads.
Instead:
deadline = 2 seconds
After two seconds:
timeout
But timeout configuration is subtle.
Suppose:
API timeout = 5 seconds
Orders timeout = 4 seconds
Payments timeout = 3 seconds
Database timeout = 2 seconds
The request's total budget must make sense.
A sophisticated mesh can propagate deadlines.
For example:
X-Request-Deadline: 2026-09-18T19:00:05Z
Every downstream service receives the remaining budget.
This creates a powerful concept:
A distributed request should have a distributed deadline.
Otherwise, downstream services may continue working after the caller has already given up.
Mutual TLS
Now we have a security problem.
How does Payments know that a request really came from Orders?
IP addresses are not identities.
We need cryptographic identity.
This is where mutual TLS enters.
Instead of:
Orders ───── TLS ────▶ Payments
we have:
Orders Proxy
│
│ client certificate
▼
Payments Proxy
│
│ server certificate
▼
validated identity
Both sides authenticate each other.
Conceptually:
Orders
│
│ "I am orders"
│
▼
Certificate Authority
│
▼
certificate
The control plane can issue identities.
For example:
spiffe://mesh/services/orders
spiffe://mesh/services/payments
Now authorization can become identity-based.
We can define:
orders → payments = allowed
users → payments = denied
analytics → payments = read-only
This is much stronger than trusting network location.
Certificate Rotation
Certificates expire.
Therefore, the control plane needs certificate rotation.
A proxy might receive:
certificate lifetime = 1 hour
Before expiration, it requests a new certificate.
Proxy
│
├── certificate valid
│
├── renew
│
├── receive new certificate
│
└── continue traffic
The application does not need to know.
This is another example of infrastructure absorbing operational complexity.
Authorization Policies
Now let's build a simple policy model.
{
"source": "orders",
"destination": "payments",
"methods": ["POST"],
"path": "/charge",
"action": "allow"
}
The proxy evaluates the request:
Who are you?
│
▼
orders
Where are you going?
│
▼
payments
What are you doing?
│
▼
POST /charge
Policy?
│
▼
ALLOW
This gives us a policy engine.
The application no longer needs to implement every network authorization rule.
Routing Rules
Suppose Payments v2 is ready.
We don't want to send 100% of traffic to it immediately.
Our control plane can configure:
payments-v1 = 95%
payments-v2 = 5%
The proxy receives:
POST /charge
and chooses:
v1
or:
v2
according to the configured weights.
This enables canary deployments.
We could gradually move:
99 / 1
95 / 5
90 / 10
75 / 25
50 / 50
0 / 100
without changing the application.
That is one of the most compelling features of a service mesh.
Observability
Distributed systems create a nasty debugging problem.
A user reports:
“The checkout is slow.”
Where is the latency?
Gateway
│ 40ms
▼
Orders
│ 200ms
▼
Payments
│ 1500ms
▼
Ledger
Without distributed tracing, this can be painful.
The mesh can automatically generate telemetry.
Every request can receive a trace identifier:
trace-id = 8af7c91
Then:
API
└── trace 8af7c91
└── Orders
└── Payments
└── Ledger
We can measure:
request count
error rate
latency
retry count
timeouts
connection failures
circuit state
The proxy knows about communication because communication is literally its job.
Metrics
Our proxy can expose:
mesh_requests_total
mesh_request_errors_total
mesh_request_duration_seconds
mesh_retries_total
mesh_timeouts_total
mesh_circuit_open_total
Then dashboards can show:
Payments
Requests/sec: 4,200
Error rate: 0.8%
p95 latency: 180ms
p99 latency: 740ms
Retries: 124
Now infrastructure becomes observable without forcing developers to manually instrument every HTTP client.
Distributed Tracing Context
A proxy can propagate tracing metadata.
For example:
traceparent: 00-4bf92f3577b34da6-00f067aa0ba902b7-01
The important idea is not the specific header.
It is propagation.
incoming request
│
▼
proxy extracts context
│
▼
proxy creates child span
│
▼
downstream request
│
▼
destination proxy
│
▼
destination application
This allows a distributed operation to become one observable execution graph.
Dynamic Configuration
One of the most important control-plane capabilities is dynamic configuration.
Suppose we change:
timeout: 2s
to:
timeout: 1s
We should not need to restart every proxy.
The control plane publishes a configuration update.
Control Plane
│
├──────▶ Proxy A
│
├──────▶ Proxy B
│
├──────▶ Proxy C
│
└──────▶ Proxy D
Each proxy updates its local configuration.
This is where a service mesh starts looking less like a collection of proxies and more like a distributed operating system for services.
Configuration Versioning
Dynamic configuration introduces another problem.
What if updates arrive out of order?
Imagine:
version 10
version 11
version 12
Proxy receives:
12
11
It must not roll backward.
Therefore configuration should have versions.
For example:
{
"version": 12,
"routes": [...]
}
The proxy only accepts:
new_version > current_version
This seems trivial.
At scale, details like this become the difference between deterministic infrastructure and mysterious distributed behavior.
The Control Plane Should Be Eventually Consistent
Here is a philosophical shift.
We often want configuration to be instantly consistent everywhere.
But that can be expensive.
Suppose there are:
10,000 proxies
A configuration update must propagate.
There will be some period where:
Proxy A → v12
Proxy B → v12
Proxy C → v11
Proxy D → v11
This is normal.
The important question is:
Is temporary inconsistency safe?
If yes, eventual propagation can be perfectly reasonable.
The control plane can therefore operate using a model where configuration converges.
That is a fundamental distributed-systems idea:
We do not always need everyone to know everything immediately. We need the system to converge toward the correct state.
Failure of the Control Plane
Now let's kill our control plane.
Everything should not immediately collapse.
Existing proxies should have:
cached service endpoints
cached policies
cached certificates
cached routing configuration
Therefore:
Control Plane DOWN
│
▼
Existing Proxy
│
├── cached routing
├── cached endpoints
├── cached policy
└── cached certificates
│
▼
traffic continues
Some capabilities may degrade.
New services may not register.
Configuration changes may not propagate.
Certificates may eventually need renewal.
But existing traffic should remain operational for as long as possible.
This is an extremely important availability principle:
The control plane should fail independently from the data plane.
Backpressure
There is another problem.
Suppose Payments can process:
5,000 requests/sec
but Orders sends:
20,000 requests/sec
Something has to give.
Without backpressure, queues grow until memory disappears and latency explodes.
A proxy can implement connection limits:
max concurrent requests = 1,000
Additional requests may receive:
HTTP 503
or be queued within controlled limits.
The goal is not to make every request succeed.
The goal is to prevent overload from becoming systemic failure.
Connection Pooling
Opening a new TCP connection for every request is wasteful.
Our proxy can maintain connection pools.
Payments connection pool
conn-1
conn-2
conn-3
conn-4
conn-5
Requests reuse existing connections.
For HTTP/2, multiple requests can share a connection.
The mesh therefore becomes responsible for transport optimization.
Again:
The application says:
Call Payments.
The mesh decides:
Which connection?
Which instance?
Which protocol?
Which timeout?
Which policy?
A Simple Proxy State Machine
At this point, our proxy has a lot of state.
We can model it explicitly:
┌───────────────┐
│ INITIALIZING │
└───────┬───────┘
│
▼
┌───────────────┐
│ SYNCING │
│ CONFIGURATION │
└───────┬───────┘
│
▼
┌───────────────┐
│ READY │
└───────┬───────┘
│
├──────────────┐
│ │
▼ ▼
CONFIG UPDATE CONTROL PLANE
DISCONNECTED
│ │
└──────┬───────┘
▼
READY
The proxy should distinguish between:
no configuration
and:
control plane temporarily unavailable
The first might mean:
do not serve traffic
The second might mean:
continue using cached state
These distinctions matter.
Building the Data Plane
A minimal implementation can have these modules:
proxy/
├── listener
├── router
├── discovery
├── load_balancer
├── health
├── retry
├── timeout
├── circuit_breaker
├── tls
├── policy
├── telemetry
└── connection_pool
The request flow becomes:
Listener
│
▼
Router
│
▼
Policy
│
▼
Discovery
│
▼
Load Balancer
│
▼
Circuit Breaker
│
▼
Retry + Timeout
│
▼
TLS Connection
│
▼
Destination
We can implement this in Rust, Go, C++, Java, or another language suited to network services.
Rust is particularly interesting for this kind of system because the proxy is fundamentally a highly concurrent network program.
But the architecture matters more than the language.
Building the Control Plane
Our control plane could expose APIs such as:
POST /services/register
POST /services/heartbeat
GET /services/:name
POST /routes
GET /routes/:service
POST /policies
GET /policies/:service
POST /certificates/issue
Internally:
Control Plane
│
├── Registry
├── Config Store
├── Policy Engine
├── Certificate Authority
└── Event Bus
The event bus allows updates to propagate.
For example:
route changed
│
▼
event: ROUTE_UPDATED
│
├──▶ Proxy A
├──▶ Proxy B
├──▶ Proxy C
└──▶ Proxy D
Watching Instead of Polling
A naive proxy might ask:
GET configuration
every five seconds.
That works.
But it is inefficient.
A better approach is a watch stream:
Proxy ───────────────▶ Control Plane
subscribe
Control Plane
│
│ configuration update
▼
Proxy
The proxy maintains a long-lived connection.
Whenever configuration changes, the control plane pushes an update.
This reduces polling overhead and improves propagation speed.
But now we must handle:
connection dropped
reconnect
resume
missed updates
version synchronization
Again, distributed systems.
The Mesh as a Distributed State Machine
At this point, something deeper becomes visible.
A service mesh is not merely networking software.
It is a distributed state machine.
The control plane maintains a model of:
services
instances
identities
policies
routes
health
configuration
The proxies maintain local replicas of relevant state.
The system continuously tries to converge.
GLOBAL STATE
│
┌───────┴────────┐
▼ ▼
Proxy A Proxy B
local state local state
│ │
└───────┬────────┘
│
traffic
This is why building a service mesh forces you to understand distributed systems.
You are designing:
- failure detection
- replication
- consistency
- identity
- state propagation
- timeouts
- retries
- fault isolation
- concurrency
- observability
The network is only the surface.
The Hardest Problem: Failure
The happy path is easy.
request
↓
proxy
↓
healthy instance
↓
response
The interesting architecture begins when everything goes wrong.
What if:
DNS fails?
What if:
certificate expires?
What if:
control plane disappears?
What if:
proxy crashes?
What if:
service registration becomes stale?
What if:
configuration versions diverge?
What if:
the request reaches the destination but the response is lost?
What if:
all healthy instances become overloaded?
What if:
a retry storm begins?
A service mesh is essentially a machine for making these failures predictable.
Retry Storms
Consider:
API
│
▼
Orders
│
▼
Payments
Payments starts returning errors.
Orders retries twice.
But 100 Orders requests are active.
Suddenly:
100 original requests
+
200 retries
=
300 Payments requests
Now Payments becomes even more overloaded.
Retries have amplified the failure.
This is called a retry storm.
Our mesh should therefore support:
retry budgets
jitter
exponential backoff
maximum attempts
deadline propagation
Instead of:
retry immediately
retry immediately
retry immediately
we use something like:
attempt 1
│
└── 50ms
attempt 2
│
└── 100ms + jitter
attempt 3
Even better, retries should be bounded by an overall request deadline.
Why Jitter Matters
Imagine 10,000 requests fail simultaneously.
If every proxy retries exactly after:
100ms
then all 10,000 requests arrive together.
Again.
Instead:
100ms + random jitter
spreads them out.
request A → 103ms
request B → 117ms
request C → 94ms
request D → 128ms
Randomness becomes a stability mechanism.
That is one of the weirdest things about distributed systems:
Sometimes adding randomness makes a system more predictable.
Security Boundaries
A mesh also changes the security model.
Instead of trusting the internal network, we can define:
identity → policy → action
For example:
service:orders
↓
identity verified
↓
policy evaluated
↓
POST /payments allowed
This moves us toward zero-trust networking.
But the mesh is not automatically secure just because it uses TLS.
We still need to secure:
control plane APIs
certificate authority
configuration distribution
proxy administration
service registration
identity issuance
If an attacker compromises the control plane, the consequences can be enormous.
The control plane is therefore part of the security perimeter.
What Happens When the Proxy Dies?
This is one of the uncomfortable questions.
If the application depends on the sidecar:
Application → Proxy → Network
what happens when:
Proxy = dead
Depending on the deployment architecture, application networking may fail.
Therefore proxies need:
fast startup
resource limits
health checks
automatic restart
minimal memory footprint
And the platform needs to understand that the proxy is infrastructure.
A service mesh introduces another moving part.
That is an important tradeoff.
The Cost of Abstraction
Service meshes solve problems.
They also create problems.
You now have:
application
proxy
control plane
configuration
certificates
telemetry
service registry
instead of:
application
This means:
more CPU
more memory
more latency
more operational complexity
more debugging layers
If you have three services, you may not need a service mesh.
If you have hundreds of services operated by multiple teams, the economics change.
The lesson is not:
Every distributed system needs a service mesh.
The lesson is:
When communication complexity becomes systemic, communication deserves its own infrastructure layer.
A Minimal End-to-End Request
Let's put everything together.
Suppose Orders sends:
POST /charge
The request enters the Orders proxy.
1. Identity
The proxy identifies the source:
orders
2. Destination
The router determines:
payments
3. Authorization
Policy engine checks:
orders → payments
Result:
ALLOW
4. Discovery
Proxy finds:
payments-v1
payments-v2
5. Routing
Policy says:
v1 = 90%
v2 = 10%
Proxy chooses:
payments-v2
6. Load balancing
Three v2 instances exist.
Proxy chooses:
payments-v2-3
7. Circuit breaker
Circuit is:
CLOSED
Proceed.
8. Timeout
Deadline:
2 seconds
9. TLS
Proxy establishes an authenticated encrypted connection.
10. Telemetry
Proxy records:
trace_id
latency
status
destination
retry_count
11. Response
Payments responds:
200 OK
The proxy records the result and returns it to Orders.
The application sees:
200 OK
It does not need to know about most of the machinery.
That is the entire point.
A Better Mental Model
Many people think of a service mesh like this:
service + proxy
I prefer this:
SERVICE MESH
┌──────────────────────┐
│ Communication Policy │
└──────────┬───────────┘
│
┌───────▼───────┐
│ Control Plane │
└───────┬───────┘
│
configuration
│
┌────────────┴────────────┐
▼ ▼
Proxy Proxy
│ │
Service Service
│ │
└──────────network────────┘
The proxy is the executor.
The control plane is the brain.
The network is the environment.
The application is the organism using the environment.
This framing makes the architecture easier to reason about.
What We Have Built
Our service mesh now has:
✓ Service discovery
✓ Service registration
✓ Health checking
✓ Local proxies
✓ Load balancing
✓ Timeouts
✓ Retries
✓ Circuit breakers
✓ Dynamic routing
✓ Mutual TLS
✓ Identity
✓ Authorization
✓ Metrics
✓ Distributed tracing
✓ Configuration propagation
✓ Connection pooling
✓ Backpressure
And yet, this is still only the beginning.
A production-grade mesh introduces even more concerns:
HTTP/2
gRPC
HTTP/3
WebSockets
TCP proxying
DNS handling
xDS-style APIs
certificate rotation
multi-cluster networking
fault injection
traffic mirroring
locality-aware routing
outlier detection
ambient networking
eBPF acceleration
resource isolation
configuration validation
control-plane federation
The rabbit hole goes deep.
Very deep.
The Bigger Idea
There is a broader software architecture lesson hidden inside the service mesh.
Applications keep accumulating responsibilities.
At some point, we recognize that certain responsibilities are not really application responsibilities.
Authentication becomes identity infrastructure.
Logging becomes observability infrastructure.
Databases become persistence infrastructure.
Containers become runtime infrastructure.
And service-to-service communication becomes mesh infrastructure.
This is a recurring pattern in computer science.
A problem becomes common enough that we stop solving it independently inside every application.
We build a platform.
The service mesh is one expression of that idea.
Build the Mesh to Understand the Network
You do not need to build a complete production service mesh to learn something valuable.
Build the smallest possible one.
Start with:
service registry
↓
service discovery
↓
proxy
↓
round-robin load balancing
Then add:
health checks
Then:
timeouts
Then:
retries
Then:
circuit breaking
Then:
TLS
Then:
identity
Then:
routing policies
Then:
telemetry
Then:
dynamic configuration
Each feature teaches you something different.
Service discovery teaches you distributed state.
Retries teach you failure semantics.
Circuit breakers teach you cascading failures.
TLS teaches you machine identity.
Routing teaches you traffic management.
Telemetry teaches you observability.
Dynamic configuration teaches you eventual consistency.
And proxying teaches you something even more fundamental:
the network itself can become programmable infrastructure.
Final Architecture
Our finished conceptual system looks like this:
┌──────────────────────────┐
│ CONTROL PLANE │
│ │
│ Service Registry │
│ Configuration Store │
│ Routing Engine │
│ Policy Engine │
│ Certificate Authority │
│ Event Distribution │
└────────────┬─────────────┘
│
Configuration / Identity
│
┌────────────────────────┴────────────────────────┐
│ │
┌──────▼──────┐ ┌───────▼──────┐
│ Orders Proxy│ │Payments Proxy│
│ │ │ │
│ Discovery │ │ Discovery │
│ Routing │ │ Routing │
│ LB │ │ LB │
│ Retry │ │ Retry │
│ Timeout │ │ Timeout │
│ Circuit │ │ Circuit │
│ TLS │ │ TLS │
│ Telemetry │ │ Telemetry │
└──────┬──────┘ └──────┬───────┘
│ │
┌──────▼──────┐ ┌──────▼───────┐
│ Orders │ ─────────── network ───────────▶│ Payments │
└─────────────┘ └──────────────┘
That diagram looks like networking.
But underneath it is something much deeper.
It is distributed systems theory turned into infrastructure.
A service mesh is a system that takes all the ugly questions surrounding network communication and turns them into explicit mechanisms:
Who are you?
Where are you?
Are you healthy?
Should I trust you?
Should I send traffic to you?
How much traffic?
For how long?
What happens if you fail?
Should I retry?
Should I stop retrying?
Can I encrypt this?
Can I observe this?
Can I change the behavior without redeploying?
Those questions exist whether you build a mesh or not.
The difference is where you answer them.
Without a mesh, they tend to leak into application code.
With a mesh, they can become infrastructure policy.
And that is the real reason to build one from scratch.
Not because the world needs another service mesh.
But because building one forces you to understand what happens between your services.
And that space between services is where distributed systems actually live.
Code is easy when everything is local.
The interesting engineering begins when your code has to cross a network.
That is where latency appears.
That is where packets disappear.
That is where identities matter.
That is where retries become dangerous.
That is where failures become contagious.
And that is where architecture stops being a diagram and starts becoming a machine.
Build the proxy.
Build the control plane.
Break the network.
Kill the services.
Drop the connections.
Expire the certificates.
Send too much traffic.
Watch everything fail.
Then make it recover.
Because if you can build a service mesh from scratch, you are no longer just learning how services communicate.
You are learning how large software systems survive reality.
Top comments (0)