Stop Slamming Downstream Services: Singleflight Request Coalescing with Java Virtual Threads

Stop Slamming Downstream Services: Singleflight Request Coalescing with Java Virtual Threads

Virtual threads solved your JVM I/O bottlenecks, but downstream internal services are now on fire because 50,000 concurrent threads are fetching the exact same cache miss simultaneously. You do not need distributed locks or larger database instances; you need the Singleflight request coalescing pattern inside your JVM.

Why Most Developers Get This Wrong

• Reaching for distributed locking: Slapping Redisson or Redis locks over hot cache misses adds unnecessary network hops and operational latency for what is fundamentally an in-process duplicate query problem.
• Blocking with synchronized blocks: Using coarse ReentrantLock or synchronized guards pinned carrier threads in early Loom setups and destroys virtual thread scalability across high-throughput services.
• Relying purely on TTL caches: When a hot Redis or Caffeine key expires under 20k RPS, raw uncoordinated reads trigger immediate thundering herds across downstream gRPC/REST endpoints.

The Right Way

Suppress duplicate concurrent calls locally by routing identical in-flight keys to a single downstream execution using lock-free coordination.

• Use a ConcurrentHashMap to register in-flight CompletableFuture instances keyed by query identity.
• The first virtual thread registers the promise and triggers the downstream RPC, while subsequent threads simply wait on the shared future.
• Leverage CompletableFuture.join()—virtual threads unmount cleanly from OS carrier threads while awaiting the result.
• Automatically evict keys in a finally block or completion callback so future requests trigger fresh executions.

Show Me The Code

Here is an idiomatic, lock-free Singleflight implementation using standard Java concurrency utilities:

public class Singleflight<K, V> {
    private final ConcurrentHashMap<K, CompletableFuture<V>> inFlight = new ConcurrentHashMap<>();

    public V execute(K key, Supplier<V> task) {
        return inFlight.computeIfAbsent(key, k -> {
            var future = new CompletableFuture<V>();
            Thread.ofVirtual().start(() -> {
                try { future.complete(task.get()); }
                catch (Throwable ex) { future.completeExceptionally(ex); }
                finally { inFlight.remove(k); }
            });
            return future;
        }).join();
    }
}

Key Takeaways

• Protect your downstream: Virtual threads can absorb massive ingress spikes; without in-JVM coalescing, that traffic acts as a self-inflicted DDoS against internal dependencies.
• Lock-free coordination wins: Combining ConcurrentHashMap.computeIfAbsent() with CompletableFuture eliminates thread contention while keeping memory overhead negligible.
• Clean up immediately: Always evict keys in finally blocks to avoid memory leaks and prevent transient downstream exceptions from permanently poisoning future callers.

I built javalld.com while prepping for senior roles — complete LLD problems with execution traces, not just theory.