The Incident That Forced a Redesign
A while ago, I ran into a backend bottleneck that completely broke our standard API design assumptions.
We had an endpoint that accepted a diagnostic request—like a system health or connectivity check—alongside a payload containing anywhere from 10,000 to 30,000 IDs. For every single ID, the backend needed to run a suite of diagnostic queries, gather the outputs, and send back a combined payload.
Because each diagnostic check took between 1 and 30 seconds, running them sequentially was completely out of the question. Even with an optimistic average runtime, processing 30,000 IDs sequentially would keep the request hanging for hours.
We needed an architecture that could handle massive concurrency without blowing up our servers.
TL;DR
The Problem: Processing up to 30k long-running tasks per request overwhelmed local API thread pools and caused memory spikes.
The Solution: Implemented a Scatter-Gather architecture to split requests into 100-ID batches fanned out over Kafka to stateless workers.
The Key Takeaway: Instead of tracking aggregate state in Kafka, I persisted raw event payloads directly to a database, making job progress tracking fast and enabling asynchronous client updates.
Why "Just Adding Threads" Didn't Work
My initial reaction was to throw thread pools at the problem. Concurrency works well when you're scaling up on a single machine, and it did temporarily boost our throughput.
Request → API Server Thread Pool → Diagnostic Execution
However, offloading this to local threads just shifted the bottleneck rather than solving it. The API server was still forced to hold open long-lived connections, manage thousands of concurrent tasks, handle retries, and aggregate results in memory.
The moment incoming traffic spiked, the node ran out of memory. I realized this wasn't a multithreading problem—it was a workload distribution problem.
Shifting to a Scatter-Gather Pattern
Instead of attempting to process a giant payload in a single lifecycle, we split the incoming IDs into manageable chunks. Breaking 30,000 IDs into 300 distinct batches of 100 allowed me to handle each piece of work independently.
This led to a Scatter-Gather pattern powered by Kafka:
The Scatter phase splits the massive payload and fans out independent tasks across a Kafka topic. Workers pick up these batches asynchronously, and the Gather phase eventually stitches the completed outputs back together for the client.
This gave me two distinct layers of parallelism:
- Horizontal Scaling: Multiple stateless worker instances pull batches from Kafka concurrently.
- Vertical Scaling: Inside each worker node, individual IDs within a 100-item batch are processed concurrently using a local thread pool.
Instead of relying on a single mega-server to orchestrate thousands of tasks, I distributed the execution load across worker nodes that could scale up or down based on queue depth.
When State Stores Become the Bottleneck
This pattern got workers processing tasks quickly, but it introduced a new issue on the aggregation side.
As workers completed batches, they pushed results into a Kafka stream to build the final response. However, keeping and updating this aggregate state directly inside Kafka Streams meant our state stores were inflating rapidly. Transporting massive, growing JSON payloads over Kafka topics added unnecessary network overhead and bloated our storage.
Instead of attempting to maintain an active in-memory aggregate in Kafka, I changed the strategy: I persisted the raw completed event straight to a SQL database.
Every completed batch writes a record containing metadata (requestId, batchId, status, timestamp) along with the complete JSON event payload.
By saving the raw, immutable event instead of a sanitized summary, the database became our source of truth. If a downstream consumer needed a different data format later, or if we had to audit a failed run, the original context was fully preserved in the database without needing to rerun the diagnostics.
Getting Real-Time Status for Free
Switching to event persistence unlocked a huge UX win: real-time progress tracking.
Because every finished batch immediately writes a row to the database, calculating overall job progress becomes a basic query (completed_batches / total_batches).
Instead of holding an HTTP request open for minutes while waiting for all 30,000 checks to finish, the API returns a job ID instantly. Clients can then poll an endpoint or listen over WebSockets to display a live progress bar while work completes in the background.
Architectural Trade-offs
This design solved our scaling issues—workers remained completely stateless, failed batches could be retried without restarting the whole job, and memory consumption dropped significantly—but it came with real trade-offs:
- Eventual Consistency: The system is fully asynchronous, meaning client apps have to be redesigned around polling, WebSockets, or webhooks.
- Operational Overhead: You now have to actively handle idempotency, deduplicate repeated messages, and track correlation IDs across service boundaries.
- Partial Failures: If 298 out of 300 batches succeed, your system needs explicit logic to handle Dead-Letter Queues (DLQs) and mark jobs as partially complete.
Summary
Resolving this bottleneck wasn't just about plugging in Kafka or adding more CPU cores. The breakthrough came from rethinking the lifecycle of a request: breaking monolithic work into bounded chunks, delegating execution to stateless workers, and treating intermediate outputs as immutable events.
Scatter-Gather provided the blueprint, but decoupling processing from state management is what made the pipeline production-ready.
Top comments (0)