Your Tests Are Green. Your Distributed System Still Has a Bottleneck.

Amir Marcel — Fri, 31 Jul 2026 17:07:54 +0000

When I started building claims-pipeline, I thought I was building an event-driven backend.

It turned out I was really building a lesson in distributed systems.

The project itself is fairly straightforward:

Generate synthetic healthcare claims
Publish them through SNS
Process them with SQS-backed workers
Compute deterministic provider rankings
Store the results in PostgreSQL
Serve rankings through a FastAPI service
Scale workers automatically with Kubernetes and KEDA

The interesting part wasn't getting the happy path working.

It was discovering what happened when I stopped thinking like a developer and started thinking like production.

The Architecture
Here's the entire pipeline.

Synthetic Claims
│
▼
SNS Topic
│
▼
SQS Queue
│
▼
Validation Worker
│
▼
Scoring Queue
│
▼
Scoring Worker
│
▼
PostgreSQL
│
▼
Ranking API

The goals were simple:

asynchronous processing
independently scalable workers
safe retries
dead-letter queues
deterministic provider scoring
observability under load

Every consumer is idempotent.

Every failure can be replayed.

Every ranking is deterministic.

The language model never computes a score; instead, it only explains one that's already been calculated.

Everything Passed
Unit tests passed.

Integration tests passed.

End-to-end tests passed.

The scoring algorithm behaved exactly as expected.

The retry logic worked.

Dead-letter replay worked.

At that point, it's tempting to believe the system is "done."

It wasn't.

Then I Ran a Load Test
I pushed 10,000 synthetic claim events through the pipeline.

KEDA began scaling workers based on queue depth.

Exactly as designed.

Validation workers climbed from one replica to five and stayed there while the queue drained.

Then something unexpected happened.

The scoring workers kept bouncing.

5 replicas
↓

3 replicas
↓

2 replicas
↓

5 replicas
↓

4 replicas

Throughput became inconsistent.

Nothing had failed.

No alerts fired.

Every test was still green.

Yet the system was clearly fighting itself.

The Bottleneck Was My Own Correctness Guarantee
Earlier in the project I'd added per-provider advisory locks.

The idea was straightforward.

If two workers tried updating the same provider simultaneously, one would wait.

No race conditions.

No corrupted aggregates.

Correctness guaranteed.

Under normal load, it worked perfectly.

Under heavy concurrency, it became the bottleneck.

As KEDA added more workers, more pods competed for the same provider locks.

The system was scaling.

The work wasn't.

The mechanism that guaranteed correctness had become the mechanism limiting throughput.

That wasn't a Kubernetes problem.

It wasn't an autoscaling problem.

It was an architecture problem.

No Test Could Have Shown Me This
Every unit test was green because every unit behaved correctly.

Every integration test passed because the components interacted correctly.

None of those tests simulated dozens of workers competing for the same shared resource.

Load testing exposed a property I hadn't actually tested:

contention.

Correctness and throughput are different qualities.

A system can be perfectly correct while performing terribly under realistic concurrency.

That was probably the biggest lesson from the project.

Why the System Is Still Deterministic
One design decision I intentionally kept was separating deterministic logic from AI.

Provider rankings are computed entirely by code.

The LLM never participates in the decision.

Instead, it receives grounded facts:

provider score
ranking
quality metrics
neighboring providers

Its only job is explaining why the provider ranked where it did.

That means:

rankings are reproducible
explanations can be evaluated independently
business logic remains fully testable

The AI augments the system.

It doesn't control it.

What I'd Change Next
If this were evolving into a production service, the next step wouldn't be tuning KEDA.

I'd revisit the synchronization strategy itself.

Some possibilities include:

partitioning work by provider
reducing lock granularity
batching updates
redesigning aggregation to minimize contention

The important lesson wasn't finding the perfect solution.

It was discovering the real bottleneck before production users did.

Final Thoughts
Building this project reinforced something I think applies well beyond healthcare systems.

Distributed systems aren't difficult because individual components are hard.

They're difficult because every component can be perfectly correct, yet their interactions under real load reveal entirely different behavior.

A green test suite tells you your code works.

A realistic load test tells you whether your architecture does.

The project is open source if you'd like to explore the implementation or follow along as it evolves.

The complete project is open source:

👉 https://github.com/amirmarcel/claims-pipeline

DEV Community: Amir Marcel

Your Tests Are Green. Your Distributed System Still Has a Bottleneck.