"We deployed a patch to one service. 14 others went down. It was a Tuesday."
The 3 AM Wake-Up Call
It always starts the same way.
Someone merges a small change to the authentication service. CI passes. Staging looks green. The deploy goes out. And then, like dominoes, services start failing.
First it's the user-profile service — reasonable, it depends on auth. Then billing goes down. Then notifications. Then the internal admin dashboard. Then the search indexer. Within 20 minutes, 14 services are throwing 500s and PagerDuty is lighting up every phone in the on-call rotation.
The postmortem reveals the nightmare: the dependency graph had a hidden cycle. Auth depended on user-profile for role resolution. User-profile depended on auth for token validation. Neither team knew about the other's dependency because they'd been introduced six months apart by different engineers.
This isn't hypothetical. This is the single most common architectural failure in microservice systems, and almost nobody tests for it.
Why Circular Dependencies Are Silent Killers
In a monolith, circular dependencies manifest as compile errors or import cycles. Your language's toolchain catches them. In a microservice architecture, there is no compiler. Dependencies are runtime HTTP calls, message queue subscriptions, shared database reads — invisible wires that nobody draws on the architecture diagram.
The damage they cause:
- Cascading failures: Service A calls B calls C calls A. One goes down, they all go down.
- Deployment deadlocks: You can't deploy A without B being up, but B's new version requires A's new version. Neither can go first.
- Infinite retry storms: A calls B, B calls A, both retry on failure, both amplify each other's load until the entire cluster is saturated.
- Impossible rollbacks: Rolling back A requires rolling back B, which requires rolling back C, which requires rolling back A.
The worst part: these cycles are almost never caught during code review. Each individual dependency looks reasonable in isolation. It's only when you look at the graph as a whole that the cycle becomes visible.
Detection: Finding Cycles With DFS
The standard algorithm for cycle detection in directed graphs is Depth-First Search with 3-color marking. Here's the idea:
- White (0): Haven't visited this node yet
- Grey (1): Currently visiting this node (it's in our DFS stack)
- Black (2): Fully explored this node and all its descendants
If during DFS we encounter a grey node — one that's already in our current exploration path — we've found a back-edge, which means a cycle.
def detect_cycles(graph):
"""
Detects circular dependencies in a service dependency graph.
Args:
graph: dict mapping service_name -> {
"depends_on": [list of service names this service calls]
}
Returns:
True if a cycle exists, False if the graph is a valid DAG.
"""
visited = {} # node -> state (0=white, 1=grey, 2=black)
def dfs(node_id):
visited[node_id] = 1 # mark grey (in current path)
for dep in graph[node_id]["depends_on"]:
if visited.get(dep, 0) == 1:
return True # found a back-edge → CYCLE
if visited.get(dep, 0) == 0:
if dfs(dep):
return True
visited[node_id] = 2 # mark black (fully explored)
return False
for node_id in graph:
if visited.get(node_id, 0) == 0:
if dfs(node_id):
return True
return False
Example: a microservice graph with a hidden cycle
services = {
"auth": {"depends_on": ["user-profile"]}, # auth needs roles from profile
"user-profile": {"depends_on": ["auth"]}, # profile needs tokens from auth ← CYCLE
"billing": {"depends_on": ["auth", "payments"]},
"payments": {"depends_on": []},
"notifications": {"depends_on": ["user-profile"]},
"search": {"depends_on": ["user-profile", "billing"]},
}
if detect_cycles(services):
print("🚨 CIRCULAR DEPENDENCY DETECTED — fix this before deploying.")
else:
print("✅ Dependency graph is cycle-free.")
# Output: 🚨 CIRCULAR DEPENDENCY DETECTED — fix this before deploying.
That's it. 20 lines of Python. Run this against your service dependency map and you'll know immediately if you have a cycle.
But detection is only half the battle. The next question is: "If I change service X, what else could break?"
Blast Radius: Computing the Full Impact of a Change
When you deploy a change to a service, the impact isn't limited to its direct dependents. It's transitive. If auth changes, and user-profile depends on auth, and billing depends on user-profile — then billing is impacted too, even though it doesn't directly call auth.
Computing this blast radius is a breadth-first traversal of the reverse dependency graph:
def get_blast_radius(graph, changed_services):
"""
Given a set of changed services, returns ALL downstream services
that could be transitively affected.
This answers the question every team should ask before deploying:
"If I change this service, what else could break?"
"""
# Build the reverse graph: for each service, who depends on it?
dependents = {svc: [] for svc in graph}
for svc, data in graph.items():
for dep in data["depends_on"]:
if dep in dependents:
dependents[dep].append(svc)
# BFS outward from all changed services
affected = set()
queue = list(changed_services)
visited = set(changed_services)
while queue:
current = queue.pop(0)
for downstream in dependents.get(current, []):
if downstream not in visited:
visited.add(downstream)
affected.add(downstream)
queue.append(downstream)
return affected
Using the same service graph (with the cycle fixed):
services_fixed = {
"auth": {"depends_on": []}, # auth is now self-contained
"user-profile": {"depends_on": ["auth"]},
"billing": {"depends_on": ["auth", "payments"]},
"payments": {"depends_on": []},
"notifications": {"depends_on": ["user-profile"]},
"search": {"depends_on": ["user-profile", "billing"]},
}
What happens if we change auth?
blast = get_blast_radius(services_fixed, {"auth"})
print(f"Changing 'auth' impacts: {blast}")
# Output: Changing 'auth' impacts: {'user-profile', 'billing', 'notifications', 'search'}
What about payments?
blast2 = get_blast_radius(services_fixed, {"payments"})
print(f"Changing 'payments' impacts: {blast2}")
# Output: Changing 'payments' impacts: {'billing', 'search'}
Now before every deploy, you know exactly which services are in the blast zone.
Making It Operational: The CI Check
Detection and blast radius computation are useful locally. But the real power comes when you make them CI-enforced — so no one can introduce a circular dependency without the build failing.
Here's a minimal GitHub Actions workflow:
# .github/workflows/dependency-lint.yml
name: Dependency Graph Lint
on: [pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Validate Service Dependencies
run: python scripts/validate_dependencies.py
- name: Check for Undeclared Changes
run: |
git diff --exit-code || \
(echo "ERROR: Dependency graph has undeclared changes." && exit 1)
The script validate_dependencies.py is straightforward — load your service graph (from a JSON file, Terraform outputs, Kubernetes manifests, or whatever your source of truth is), run detect_cycles(), and exit with code 1 if a cycle is found.
Where Does Your Dependency Graph Live?
The graph itself — the data structure mapping services to their dependencies — needs to be a source of truth, not a wiki page that someone updates when they remember. Here are pragmatic options:
Option 1: A JSON manifest in your repo
{
"auth": { "depends_on": [] },
"user-profile": { "depends_on": ["auth"] },
"billing": { "depends_on": ["auth", "payments"] },
"payments": { "depends_on": [] },
"notifications": { "depends_on": ["user-profile"] },
"search": { "depends_on": ["user-profile", "billing"] }
}
Simple. Versionable. Reviewable in PRs. The downside: it can drift from reality.
Option 2: Auto-generated from infrastructure
Parse your Kubernetes manifests, Terraform configs, or API gateway routes to extract actual runtime dependencies. More accurate, but requires tooling investment.
Option 3: Hybrid
Maintain a manually curated graph and run a periodic reconciliation job that compares it against actual network traffic (e.g., from a service mesh like Istio or Linkerd). Alert on discrepancies.
Advanced: Layer Rules
Cycle detection catches the worst violations. But mature architectures also enforce layer rules — structural constraints that prevent dependency spaghetti even when there are no cycles.
Common rules:
- Frontend services may never directly depend on database services
- API gateways must route through business logic services, never call data stores directly
- Shared libraries must have zero runtime dependencies on any service
- Every service must depend on at most N other services (fan-out limit)
These are more nuanced than simple cycle detection, but they follow the same pattern: load the graph, check each node's dependencies against its layer's rules, fail the build if anything violates.
The Dependency Graph You Don't Maintain Will Maintain You
Here's the uncomfortable truth about microservice architectures: the architecture diagram on your Confluence page is almost certainly wrong. It was accurate when someone drew it six months ago. Since then, 47 PRs have introduced new inter-service calls, 3 services have been renamed, and one critical dependency was added "temporarily" and never removed.
The dependency graph you don't validate is the one that wakes you up at 3 AM.
Run cycle detection in CI. Compute blast radius before every deploy. Enforce layer rules. Make your architecture diagram an executable assertion — not a hope.
Because the alternative is discovering your hidden circular dependency in production, at 3 AM, on a Tuesday.
All code in this article uses only Python standard library features. No external packages required. Copy, paste, adapt to your service graph, and run.
Top comments (0)