DEV Community: Douglas Carmo

An 8,000 trade settlement simulator shows where virtual threads win big and where CPU-bound work exposes the limit of Loom

Douglas Carmo — Thu, 09 Jul 2026 19:44:33 +0000

Java's Project Loom promised a "free lunch": swap Executors.newFixedThreadPool() for Executors.newVirtualThreadPerTaskExecutor() and watch your throughput fly. It's a great pitch and it's also only half the story. The other half only shows up once you stop simulating I/O-bound work and start hitting the CPU.

I wanted a benchmark that felt closer to something I'd actually build, so instead of a toy "sleep and print" example, I wrote a T+1 stock settlement simulator: 8,000 trades that need to clear, plus a Monte Carlo Value-at-Risk (VaR) calculation running 20,000 simulations per trade. One workload is I/O-bound (waiting on a settlement clearinghouse call), the other is pure CPU. Running both side by side on the same machine made the boundary between "virtual threads help" and "virtual threads don't help" impossible to miss.

Part 1 — I/O-Bound Work: Settling 8,000 Trades

The setup: each trade settlement blocks on a simulated I/O call (think: a REST call to a custodian or a clearinghouse). I ran it three ways.

Strategy	Tasks	Time (ms)	Throughput (ops/s)	Peak OS Threads	Memory
Platform threads (pool=200)	8,000	9,788	817.3	207	46 MB
Virtual threads	8,000	651	12,288.8	24	30 MB
Virtual threads (with pinning)	8,000	539	14,842.3	24	105 MB

Virtual threads came out ~15x faster than a platform thread pool capped at 200 threads, while using roughly a tenth of the OS threads. This is the result everyone quotes when they talk about Loom, and it holds up: when a task spends most of its life waiting, a virtual thread can unmount from its carrier thread during that wait and let another virtual thread run in its place. You get massive concurrency without paying for an OS thread per task.

The "with pinning" row is the interesting one, though. It ran faster here, but faster isn't always better — pinning is a trap, not a feature. Pinning happens when a virtual thread can't unmount from its carrier during a blocking operation, most commonly inside a synchronized block or, historically, during certain native calls. When that happens, the carrier thread is stuck too, and you silently lose the whole point of using virtual threads: your effective concurrency collapses back toward the size of the carrier pool. It didn't hurt in this run because the pinned regions were short and the carrier pool (by default, one per CPU core) could still absorb the load — but at higher contention, this is exactly the kind of thing that turns into a production incident that's brutal to diagnose, because it doesn't throw, it just quietly gets slower.

If you suspect pinning, don't guess — run with:

-Djdk.tracePinnedThreads=full

This prints the exact stack trace of where and why a virtual thread got pinned, which is the fastest way to find synchronized blocks worth converting to ReentrantLock.

The code behind it

The settlement pipeline models something close to a real T+1 workflow: fetch the trade, validate the counterparty against a clearinghouse, check buying power, confirm share availability for sells, then write the final status back — five blocking round-trips per trade, each backed by JDBC or a simulated network call:

public void settle(long tradeId) {
    try {
        Trade trade = fetchTrade(tradeId);          // JDBC - blocking
        if (trade == null) return;

        Object localLock = new Object();
        boolean ok;

        synchronized (localLock) {                    // <-- pinning trigger
            ok = runValidationPipeline(trade);
        }

        updateStatus(tradeId, ok ? "SETTLED" : "FAILED");
    } catch (SQLException e) {
        updateStatusSilently(tradeId, "FAILED");
    }
}

private boolean runValidationPipeline(Trade trade) {
    boolean counterpartyOk = clearinghouse.validateCounterparty(trade);
    boolean buyingPowerOk = clearinghouse.checkBuyingPower(trade);
    boolean sharesOk = "SELL".equals(trade.side())
            ? clearinghouse.confirmShareAvailability(trade)
            : true;
    return counterpartyOk && buyingPowerOk && sharesOk;
}

That synchronized (localLock) block is doing exactly one job in this benchmark: it's the pinning trigger for the third scenario. It doesn't need to guard shared state, a synchronized block wrapping any blocking call is enough to pin the virtual thread to its carrier for the duration, on JDKs prior to JEP 491 (JDK 24), which finally removes this restriction. If you're on an older JDK and you see virtual threads not scaling the way the marketing promised, this is one of the first places to look, anything from a legacy synchronized DAO method to a third-party library you don't control.

The clearinghouse itself is a stand-in for the network calls a real settlement service would make to DTCC/NSCC, a broker's margin engine, and a custodian, each one a Thread.sleep standing in for HTTP/gRPC latency:

public boolean validateCounterparty(Trade trade) {
    sleepRandom(60, 160);
    return true;
}

public boolean checkBuyingPower(Trade trade) {
    sleepRandom(40, 120);
    return true;
}

public boolean confirmShareAvailability(Trade trade) {
    sleepRandom(50, 140);
    return true;
}

Three calls, 40–160ms each, per trade, times 8,000 trades — that's the "waiting" that virtual threads turn from a resource cost into a rounding error.

Part 2 — CPU-Bound Work: Monte Carlo VaR on 8,000 Trades

This is where the "just use virtual threads everywhere" advice falls apart. Same 8,000 trades, but now each one runs 20,000 Monte Carlo simulations to estimate Value-at-Risk — no I/O, no waiting, just math. Machine had 16 available cores.

Strategy	Tasks	Time (ms)	Throughput (ops/s)	Peak OS Threads	Memory
Sequential (1 thread)	8,000	13,141	608.8	1	94 MB
Virtual threads (misused for CPU work)	8,000	15,339	521.5	24	96 MB
Fixed pool = 16 cores	8,000	16,750	477.6	24	54 MB
Parallel Stream (ForkJoinPool, 15 workers)	8,000	16,271	491.7	15	110 MB

Notice what didn't happen: virtual threads didn't win. They didn't even reliably beat a plain sequential loop — they came in barely behind it, and every "smarter" concurrency strategy landed in roughly the same narrow band, within a few percentage points of each other. That's the tell.

The reason is structural, not incidental. Virtual threads solve scheduling problems: when you have thousands of tasks and most of them are idle most of the time, Loom lets the JVM interleave them cheaply on a small number of carrier threads. But when a task never yields — because it's pure CPU computation with nothing to wait on — there's no idle time to reclaim. A virtual thread doing CPU work behaves exactly like a platform thread doing CPU work, except now you've added the bookkeeping overhead of the virtual thread scheduler on top, for zero benefit. Once every strategy is CPU-bound, the ceiling is the same for all of them: Runtime.availableProcessors(). You cannot schedule your way past the number of physical cores you have.

The code behind it

The VaR engine simulates a day of price movement via geometric Brownian motion, 20,000 times per trade, then reads the loss off the 5th percentile of the resulting P&L distribution — no network, no disk, no sleeps, just floating-point math and a sort:

public double calculateVaR(Trade trade, int simulations) {
    double[] pnl = new double[simulations];
    double price = trade.price().doubleValue();
    double notional = trade.notional().doubleValue();
    ThreadLocalRandom rnd = ThreadLocalRandom.current();

    for (int i = 0; i < simulations; i++) {
        double z = rnd.nextGaussian();
        double simulatedPrice = price * Math.exp(
                (DRIFT - 0.5 * DAILY_VOLATILITY * DAILY_VOLATILITY) + DAILY_VOLATILITY * z
        );
        double priceChangePct = (simulatedPrice - price) / price;
        pnl[i] = notional * priceChangePct;
    }

    java.util.Arrays.sort(pnl);
    int varIndex = (int) ((1 - CONFIDENCE) * simulations);
    return -pnl[varIndex]; // loss = negative of the 5th percentile
}

Compare this to ClearinghouseClient from Part 1: same shape of "task," radically different runtime behavior. One spends its time parked, waiting on something external; the other spends its time saturating a core. Running the second one inside a virtual thread doesn't change what the CPU has to do — it just adds a scheduler in the middle that has nothing useful to schedule around.

The Rule of Thumb

I/O-bound (waiting on a DB, a REST call, a queue, a file): virtual threads. Executors.newVirtualThreadPerTaskExecutor() and let thousands of tasks block "for free."
CPU-bound (hashing, simulation, serialization, encryption): a fixed-size pool sized to your core count, or a parallelStream() / ForkJoinPool. Virtual threads add nothing here except the risk that someone downstream assumes I/O-scale concurrency is safe and floods you with work the CPU can't actually absorb any faster.
Watch out for pinning any time virtual threads touch legacy synchronized code — it's the one way I/O-bound workloads can quietly regress toward platform-thread performance.

The mental model that held up for me: virtual threads don't make your CPU faster, they make waiting cheaper. If your bottleneck is a clock cycle, not a network socket, Loom has nothing to offer you — and the benchmark numbers make that a lot more concrete than the general advice usually does.

Setup: JDK 21+ (pinning behavior discussed here is fixed in JDK 24 via JEP 491), benchmark simulates T+1 equity settlement (8,000 trades) for the I/O-bound scenario and Monte Carlo VaR (20,000 simulations/trade) for the CPU-bound scenario. Full harness available on GitHub — link in comments / profile.

Stop Guessing Batch Sizes: High-Performance Bulk Inserts in Spring Boot with PostgreSQL COPY

Douglas Carmo — Wed, 01 Jul 2026 17:22:23 +0000

When you can't afford to guess the right batch size and don't want to pay for that lesson in production

When I designed the persistence layer of a real-time interbank settlement pipeline, I had a problem that many financial systems share: I didn't have a precise estimate of daily order volume. Some days would be normal. Others would be outliers and in a settlement system, outlier days are not a hypothesis. They are a certainty.

The question wasn't whether the system could handle average load. It was whether it could handle the days it wasn't designed for.

That question led me away from JDBC batch inserts and toward PostgreSQL's COPY protocol, exposed in Spring Boot via the CopyManager API.

Why Not JDBC Batch?

The standard Spring Boot approach for bulk inserts is JDBC batch with JdbcTemplate:

jdbcTemplate.batchUpdate(
    "INSERT INTO settlement_order (id, amount, status, ...) VALUES (?, ?, ?, ...)",
    orders,
    500, // batch size
    (ps, order) -> {
        ps.setObject(1, order.getId());
        ps.setBigDecimal(2, order.getAmount());
        ps.setString(3, order.getStatus().name());
        // ...
    }
);

This works. For predictable volumes, it works well. The problem is the number 500.

That batch size is a tuning decision. Too small, and you make too many roundtrips to the database. Too large, and you risk memory pressure or statement timeouts under high load. Getting it right requires knowing your volume and knowing your volume requires data you may not have yet.

In a financial system with variable daily throughput, you are essentially betting that your batch size stays optimal across all load scenarios. Some days you win that bet. Others you don't, and you find out in production.

I didn't want to pay for that lesson.

The `COPY` Protocol

PostgreSQL's COPY command was designed for bulk data transfer. It bypasses the standard row-by-row insert path entirely, streaming data directly into the table in a single operation. No per-row parsing overhead, no per-row plan execution, no per-row roundtrip.

The Java PostgreSQL driver exposes this through CopyManager, which accepts an InputStream and streams it into the database using the COPY FROM STDIN syntax.

The key difference from JDBC batch: you don't choose a batch size. You stream the entire dataset in one pass, and PostgreSQL handles the ingestion. The throughput ceiling is determined by the database and the network, not by a tuning parameter in your application code.

Implementation in Spring Boot

The main challenge in a Spring Boot application is that CopyManager requires a native PostgreSQL connection, not the wrapped connection that HikariCP hands to your code by default. You need to unwrap it:

@Component
public class SettlementOrderCopyBulkAdapter {

    private final JdbcTemplate jdbcTemplate;

    public SettlementOrderCopyBulkAdapter(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    @Transactional
    public void bulkInsert(List<SettlementOrder> orders, UUID batchId) {
        if (orders.isEmpty()) return;

        jdbcTemplate.execute((Connection connection) -> {
            // Unwrap HikariCP's proxy to reach the real PostgreSQL driver connection
            BaseConnection baseConnection = connection.unwrap(BaseConnection.class);
            CopyManager copyManager = new CopyManager(baseConnection);

            String copySql = """
                COPY settlement_order (
                    id, originator_id, destination_id, batch_id, order_type,
                    amount, currency, settlement_date, status, created_at
                ) FROM STDIN WITH (FORMAT csv, DELIMITER ',', QUOTE '"', ESCAPE '"')
                """;

            StringBuilder csv = new StringBuilder();
            for (SettlementOrder order : orders) {
                csv.append(order.getId()).append(",")
                   .append(order.getOriginator().getId()).append(",")
                   .append(order.getDestination().getId()).append(",")
                   .append(batchId).append(",")
                   .append(order.getType().getCode()).append(",")
                   .append(order.getAmount()).append(",")
                   .append("BRL").append(",")
                   .append(order.getSettlementDate()).append(",")
                   .append(order.getStatus().name()).append(",")
                   .append(order.getCreatedAt()).append("\n");
            }

            byte[] bytes = csv.toString().getBytes(StandardCharsets.UTF_8);
            try (InputStream inputStream = new ByteArrayInputStream(bytes)) {
                copyManager.copyIn(copySql, inputStream, 65536);
            }

            return null;
        });
    }
}

A Note on Memory Usage: In the implementation above, I used a StringBuilder to serialize the data into CSV format in memory. This is concise and works perfectly for moderate volumes. However, for extremely large datasets, building a massive string in memory can lead to OutOfMemoryError or heavy Garbage Collection pressure. For high-throughput systems, consider streaming the data directly to the InputStream using a custom implementation or a library like Apache Commons CSV or Jackson CSV to write to a Writer or OutputStream directly. This keeps the memory footprint constant regardless of the batch size.

Three Details Worth Explaining

connection.unwrap(BaseConnection.class)

HikariCP wraps the real JDBC connection in a proxy for connection pool management. CopyManager is a PostgreSQL-specific class that needs the actual driver connection, not the wrapper. unwrap() pierces through the proxy and returns the underlying org.postgresql.core.BaseConnection. Without this, you get a ClassCastException at runtime.

FROM STDIN WITH (FORMAT csv, DELIMITER ',', QUOTE '"', ESCAPE '"')

This is the COPY format declaration. FORMAT csv tells PostgreSQL to parse the stream as CSV. The QUOTE and ESCAPE settings ensure that values containing commas or quotes are handled correctly. For a financial system where amounts and identifiers have predictable formats, CSV is the right choice; it's compact and fast to serialize.

65536 as the buffer size

The third argument to copyIn() is the read buffer size in bytes — 64KB here. This controls how much data is read from the InputStream per internal read cycle during the stream transfer. It is not a batch size in the JDBC sense; the entire dataset is still transferred in a single COPY operation. The buffer size affects internal I/O efficiency, not correctness. 64KB is a reasonable default for most workloads.

Transactional Participation

One concern with native PostgreSQL features is whether they play nicely with Spring's transaction management. CopyManager does because the connection it uses is the same connection that Spring's transaction manager already enrolled in the active transaction.

The @Transactional annotation on the adapter method ensures that the entire COPY operation participates in the surrounding transaction. If anything fails after the bulk insert - a downstream status update, a Kafka publish anything - the COPY rolls back along with everything else. No partial state reaches the database.

@Transactional          ← Spring opens transaction
  bulkInsert()          ← COPY streams into PostgreSQL
  updateStatus()        ← JPQL update on same connection
  publishToKafka()      ← fires only after commit (afterCommit)
                        ← commit or rollback applies to all

The Risk Argument

The choice between JDBC batch and CopyManager is not purely technical. It is also a risk decision.

JDBC batch with a fixed batch size is a bet that your volume estimate stays valid. In a system where daily throughput varies significantly, and where outlier days are expected that bet has a real probability of failing in production. When it fails, you find out under load, at the worst possible time.

CopyManager removes that bet. The application serializes the dataset and streams it. PostgreSQL ingests it. The throughput ceiling is the database, not a tuning parameter. On a normal day the difference is negligible. On an outlier day it is the difference between a pipeline that holds and one that doesn't.

For a settlement system processing interbank transactions under regulatory time constraints, that margin matters.

When Not to Use CopyManager

CopyManager is the right tool when you need to insert a large, complete dataset in one operation. It is not always the right tool:

Small, frequent inserts — for single-row or low-volume writes, regular JPA or JdbcTemplate is simpler and sufficient
Upsert semantics — COPY is insert-only; if you need ON CONFLICT DO UPDATE, you need a different approach (a staging table + merge, or INSERT ... ON CONFLICT)
Complex validation per row — if each row needs application-level validation before insert, building that into the CSV serialization loop adds complexity that JDBC batch handles more naturally

Takeaway

If your system has unpredictable or variable bulk insert volume, tuning a JDBC batch size is an ongoing maintenance burden with production risk attached. PostgreSQL's COPY protocol delegates that responsibility to the database, where it belongs.

The implementation in Spring Boot requires one non-obvious step: unwrapping HikariCP's connection proxy but once that is in place, the rest is straightforward: serialize your dataset to CSV, stream it in, let PostgreSQL do what it was designed to do.

Don't tune your way around the database. Use what the database gives you.

This is part of a series on the STR-XML-Pipeline, a high-throughput interbank settlement system built with Spring Boot 3.5, Java 21, Apache Kafka, PostgreSQL 16, Redis 7, and AWS Fargate. The previous article covered how distributed locks were moved off the API hot path and onto the scheduler.

Don t Lock Your API - Lock Your Scheduler Instead

Douglas Carmo — Tue, 23 Jun 2026 18:09:23 +0000

How inverting responsibility eliminated distributed lock contention from a high-throughput settlement pipeline

When building the ingestion layer of a real-time interbank settlement pipeline, I faced a classic distributed systems problem: how do you guarantee that all orders belonging to the same settlement window are grouped together before being published to Kafka, without killing your API throughput?

The naive answer involves a distributed lock on the hot path. I went a different direction.

The Problem

The system receives settlement orders via a REST API built with Spring Boot 3.5 and Java 21. Orders must be grouped by settlement window (a time-based boundary, e.g., 17:00) before being published as a batch to Kafka. The consumer downstream expects a complete, immutable batch, not a partial one.

The constraint: orders can arrive at any rate. The window must close atomically. No order should be published before the window closes, and no order from a closed window should sneak into the next batch.

The Naive Approach And Why It Fails

The first instinct is to coordinate at ingestion time:

POST /orders → acquire distributed lock → check window → persist → release lock

This works at low volume. At scale, it becomes your bottleneck:

Every incoming request competes for the same lock
Lock contention grows with throughput
Latency spikes under load, exactly when you need stability most
Redis or ZooKeeper become single points of failure on the critical path

You've introduced coordination overhead into the one place that should be as fast as possible: the API entry point.

The Inversion

Instead of coordinating at ingestion, I moved all coordination out of the hot path entirely.

The architecture splits responsibility across two distinct phases:

Phase 1 — API: Persist and Forget

@PostMapping("/orders")
public ResponseEntity<Void> receiveOrder(@RequestBody SettlementOrderRequest request) {
    settlementOrderService.persist(request, OrderStatus.PENDING);
    return ResponseEntity.accepted().build();
}

The API does exactly one thing: persist the order with status PENDING. No window check. No lock. No coordination. Just a fast write via HikariCP and a 202 back to the client.

Phase 2 — Scheduler: Close, Batch, Publish

The scheduler entry point iterates over all participants. Before the loop, the JPA first-level cache is cleared to avoid stale reads:

@Scheduled(cron = "0 0 17 * * MON-FRI")
@ShedLock(name = "settlement-window-close", lockAtMostFor = "PT10M")
public void closeWindowAndPublish(String windowKey) {
    entityManager.clear(); // flush JPA first-level cache before processing

    List<Participant> participants = participantPort.findAll()
        .stream()
        .filter(p -> p.getType() != ParticipantType.BACEN)
        .toList();

    for (Participant participant : participants) {
        try {
            processor.process(window, today, participant);
        } catch (Exception e) {
            log.error("Critical error processing participant [{}]. Skipping.", participant.getIspb(), e);
        }
    }
}

Each participant is processed in complete isolation via REQUIRES_NEW, so a failure in one does not roll back the others:

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void process(SettlementWindow window, LocalDate today, Participant participant) {

    if (!window.isOpen(clock)) {
        rejectPendingOrders(window, today, participant); // explicit cutoff rejection
        return;
    }

    if (batchPort.existsActiveBatch(window, today, participant.getId())) {
        log.warn("Batch already exists for window [{}] participant [{}]. Skipping.",
            window.getPartitioningKey(), participant.getIspb());
        return;
    }

    List<SettlementOrder> pending = orderPort.findPendingForWindow(window, today, participant.getIspb());
    if (pending.isEmpty()) return;

    List<SettlementOrder> batched = pending.stream().map(SettlementOrder::batch).toList();
    FileBatch savedBatch = batchPort.save(new FileBatch(window, today, batched, participant));
    orderPort.updateStatusBatch(batched.stream().map(o -> o.associateWithBatch(savedBatch.getId())).toList());

    // Kafka is fired only after the transaction commits successfully
    TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
        @Override
        public void afterCommit() {
            publisherPort.publish(savedBatch);
        }
    });
}

The scheduler owns the window boundary. It runs once per window, outside the request lifecycle, under zero user-facing load pressure. ShedLock with Redis (SET NX PX) ensures only one instance executes across the entire ECS/EKS cluster, but this lock fires once per window, not once per request.

Four Details That Matter

1. entityManager.clear() before the loop

Some operations in the batching flow rely on JPQL bulk updates. Since JPQL bulk updates bypass Hibernate's managed entities and go directly to the database, the persistence context may contain stale state after those updates, Hibernate still believes it holds the correct version of those entities in memory. Clearing the persistence context before processing ensures that subsequent reads are served from the database rather than from outdated managed entities.

2. REQUIRES_NEW per participant

Each participant gets its own isolated transaction. If processing fails for participant A, participants B through N are unaffected and their transactions commit independently. Without this, a single failure rolls back the entire batch cycle.

3. afterCommit() before publishing to Kafka

This is a correctness guarantee, not a performance optimization. Publishing inside the transaction would risk sending a Kafka message for a batch that never actually committed to the databas a phantom message that consumers would try to process against data that doesn't exist. afterCommit() ensures the database write is durable before Kafka ever sees the event.

4. Explicit rejectCutoff() for late orders

Orders that arrive after the window closes are not silently ignored, they are explicitly marked as REJECTED_CUTOFF. This makes the system auditable: at any point you can query exactly which orders missed which window and why.

Why This Works: The Key Insight

The distributed lock didn't disappear, it moved.

Concern	Naive Approach	Inverted Approach
Lock location	API hot path	Scheduler (cold path)
Lock frequency	Every request	Once per window
Lock contention	High under load	Zero
API latency	Unpredictable	Consistent
Throughput impact	Degrades with volume	None

ShedLock on a scheduler that fires once per settlement window has negligible throughput impact. ShedLock on an API endpoint receiving thousands of requests per minute does not.

Kafka Partitioning by Window

Once the batch reaches Kafka, ordering guarantees are maintained by partitioning on the window key:

kafkaTemplate.send("str.batch.emission", batch.windowKey(), batch);
// windowKey: "STR-D0-17h00", "STR-D1-17h00", etc.

Each settlement window maps to a dedicated partition. This guarantees temporal ordering without any additional locking mechanism. Kafka's own partition semantics do the work.

Status Flow

The order lifecycle makes the phase separation explicit:

PENDING  →  BATCHED  →  EMITTED
  (API)    (Scheduler)  (Consumer)
             ↓
       REJECTED_CUTOFF
      (window closed)

Each transition is unambiguous. At any point you can query exactly where in the pipeline an order is and why it got there.

What About Late Orders?

Orders that arrive after the scheduler closes the window are not silently dropped or deferred, they are explicitly rejected with status REJECTED_CUTOFF. This is intentional on two levels: the STR protocol requires complete, bounded batches, and a silent failure is always worse than an explicit one. The system knows exactly which orders missed which window, and so does the operator.

Takeaway

If you find yourself reaching for a distributed lock on an API hot path to coordinate time-based grouping, ask whether the coordination can be deferred to a scheduled process instead.

Moving the lock from the hot path to the cold path cost nothing in correctness and eliminated the primary throughput bottleneck. The API became a pure write endpoint. The scheduler became the sole owner of window semantics.

Lock the thing that controls the boundary. Not the thing that feeds it.

This article is part of a series on the STR-XML-Pipeline, a high-throughput interbank settlement system built with Spring Boot 3.5, Java 21, Apache Kafka, PostgreSQL 16, Redis 7, and AWS S3 & Fargate.

When NASDAQ Freezes: Chaos Engineering a Stock Quotes API with Java and ToxiProxy

Douglas Carmo — Sun, 22 Feb 2026 20:16:56 +0000

I wanted to understand what really happens to a distributed system when things go wrong. So I broke it on purpose.

This article walks through a chaos engineering experiment I built around a real-time stock quotes API for NASDAQ-listed companies. The stack involves Java 21, Spring Boot 4, Redis, PostgreSQL, Resilience4j and ToxiProxy — and the results were more interesting than I expected.

What is Chaos Engineering?

Chaos Engineering is the practice of intentionally injecting failures into a system to observe how it behaves under stress. The idea, popularized by Netflix with their Chaos Monkey tool, is simple: if failures are going to happen in production anyway, it's better to discover how your system reacts in a controlled experiment than during a real outage.

In a financial context like stock quotes, the stakes are clear. Imagine a system that feeds real-time prices to traders during market open at 9:30 AM EST — and suddenly Redis starts responding in 2 seconds instead of 6 milliseconds. What happens? Does the system degrade gracefully, or does it collapse?

That's what I set out to find out.

Architecture

The project simulates a production-like environment where every external dependency passes through ToxiProxy — a programmable network proxy that can inject latency, bandwidth limits, connection resets and more.

Client
  └─► Nginx (port 8000 / 20000)
        └─► Backend (Spring Boot :8080)
              ├─► ToxiProxy :20001 ──► PostgreSQL :5432
              ├─► ToxiProxy :20002 ──► Redis :6379
              └─► ToxiProxy :20003 ──► finnhub.io:443

The key detail: the backend never connects directly to PostgreSQL, Redis or Finnhub. Every call passes through ToxiProxy, so we can degrade any dependency at any time without touching application code.

Tech stack:

Java 21 + Spring Boot 4
PostgreSQL 17 (persistence)
Redis 7 (cache via Spring Cache + Lettuce)
Resilience4j (Circuit Breaker + Retry)
ToxiProxy (chaos injection)
Nginx (reverse proxy + SSL termination for Finnhub)
Finnhub API (real market data)
Docker + Docker Compose

How the Cache Works

The application fetches stock quotes from Finnhub, saves them to PostgreSQL and caches them in Redis using @Cacheable:

 @Cacheable(value = "quotes", key = "#symbol")
    public StockQuote getQuote(String symbol) {
        log.info("Cache MISS for symbol: {}", symbol);

        FinnhubQuoteResponse response = finnhubClient.fetchQuote(symbol);

        if (Objects.equals(response.currentPrice(), BigDecimal.ZERO)) {
            throw new AssetNotFoundException("Asset not found for symbol: " + symbol);
        }

        StockQuote quote = stockQuoteMapper.toEntity(symbol, response);

        return stockQuoteRepository.save(quote);
 }

The flow is:

1st call (cache miss): Finnhub → PostgreSQL → Redis → response
2nd call (cache hit): Redis → response (PostgreSQL and Finnhub are never touched)

A background scheduler also refreshes all quotes every 60 seconds, so the cache stays warm automatically.

The Circuit Breaker

The Finnhub client is wrapped with Resilience4j annotations:

@CircuitBreaker(name = "financialApi", fallbackMethod = "fetchQuoteFallback")
@Retry(name = "financialApi")
public FinnhubQuoteResponse fetchQuote(String symbol) {
    return finnhubWebClient.get()
            .uri(uri -> uri.path("/quote")
                    .queryParam("symbol", symbol)
                    .queryParam("token", properties.token())
                    .build())
            .retrieve()
            .bodyToMono(FinnhubQuoteResponse.class)
            .timeout(Duration.ofSeconds(3))
            .block();
}

private FinnhubQuoteResponse fetchQuoteFallback(String symbol, Exception e) {
    log.error("Fallback triggered for symbol: {} - reason: {}", symbol, e.getMessage());
    return FinnhubQuoteResponse.empty();
}

The Circuit Breaker is configured to open when 50% of calls are slow (above 3 seconds) across a sliding window of 10 calls:

resilience4j:
  circuitbreaker:
    configs:
      default:
        registerHealthIndicator: true
        slidingWindowSize: 5
        minimumNumberOfCalls: 3
        permittedNumberOfCallsInHalfOpenState: 3
        automaticTransitionFromOpenToHalfOpenEnabled: true
        waitDurationInOpenState: 30s
        slowCallRateThreshold: 50
        failureRateThreshold: 50
        eventConsumerBufferSize: 10
    instances:
      financialApi:
        baseConfig: default
        waitDurationInOpenState: 60s
        recordExceptions:
          - org.springframework.web.client.HttpServerErrorException
          - java.util.concurrent.TimeoutException
          - java.io.IOException

Chaos Scripts

Three scripts control the experiment:

setup-toxiproxy.sh — creates the proxies, no chaos yet:

#!/bin/bash
set -e
echo "Configuring Toxiproxy proxies..."

curl -X POST http://localhost:8474/proxies \
  -H "Content-Type: application/json" \
  -d '{"name":"postgres_proxy","listen":"0.0.0.0:20001","upstream":"stock-quotes-postgres:5432","enabled":true}'

curl -X POST http://localhost:8474/proxies \
  -H "Content-Type: application/json" \
  -d '{"name":"redis_proxy","listen":"0.0.0.0:20002","upstream":"stock-quotes-redis:6379","enabled":true}'


curl -X POST http://localhost:8474/proxies \
  -H "Content-Type: application/json" \
  -d '{
    "name":"finnhub_proxy",
    "listen":"0.0.0.0:20003",
    "upstream":"finnhub.io:443",
    "enabled":true
  }'

echo "Toxiproxy configuration complete."

inject-chaos.sh — injects 2000ms ±500ms latency on all three proxies:

#!/bin/bash
set -e

TOXIPROXY_URL="http://localhost:8474"

echo "Injecting chaos: 2000ms latency (±500ms jitter) to postgres_proxy, redis_proxy and finnhub_proxy..."

curl --fail -X POST "$TOXIPROXY_URL/proxies/postgres_proxy/toxics" \
  -H "Content-Type: application/json" \
  -d '{"name":"latency_downstream","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":500}}'

curl --fail -X POST "$TOXIPROXY_URL/proxies/redis_proxy/toxics" \
  -H "Content-Type: application/json" \
  -d '{"name":"latency_downstream","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":500}}'

curl --fail -X POST "$TOXIPROXY_URL/proxies/finnhub_proxy/toxics" \
  -H "Content-Type: application/json" \
  -d '{"name":"latency_downstream","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":500}}'

echo "Chaos injected. Monitor your Resilience4j Circuit Breaker status."

remove-chaos.sh — removes all toxics, system recovers:

#!/bin/bash
set -e

TOXIPROXY_URL="http://localhost:8474"

echo "Removing chaos from postgres_proxy, redis_proxy and finnhub_proxy..."

curl --fail -X DELETE "$TOXIPROXY_URL/proxies/postgres_proxy/toxics/latency_downstream"

curl --fail -X DELETE "$TOXIPROXY_URL/proxies/redis_proxy/toxics/latency_downstream"

curl --fail -X DELETE "$TOXIPROXY_URL/proxies/finnhub_proxy/toxics/latency_downstream"

echo "Chaos removed. System should be recovering — watch the Circuit Breaker close."

Experiment Results

Baseline — No Chaos

# First call (cold cache)
curl -w "\nTotal time: %{time_total}s\n" http://localhost:8080/api/v1/stock-quotes/get-by-symbol/AAPL
# Total time: 0.029s

# Second call (Redis cache hit)
curl -w "\nTotal time: %{time_total}s\n" http://localhost:8080/api/v1/stock-quotes/get-by-symbol/AAPL
# Total time: 0.006s

Redis responding in 6ms is the happy path. The PostgreSQL metrics confirm this — the database itself responds in under 2ms, the overhead comes from the ToxiProxy layer even without chaos injected.

With Chaos — 2000ms ±500ms on All Services

Here's where things get interesting.

./scripts/inject-chaos.sh

# First call
curl -w "\nTotal time: %{time_total}s\n" http://localhost:8080/api/v1/stock-quotes/get-by-symbol/AAPL
# Total time: 9.4s

# Second call (expected Redis hit...)
curl -w "\nTotal time: %{time_total}s\n" http://localhost:8080/api/v1/stock-quotes/get-by-symbol/AAPL
# Total time: 10.8s

The second call was slower than the first. That was unexpected.

The reason: with 2000ms of latency on the Redis proxy, the write to Redis after the first call was delayed so much that by the time the second request arrived, the cache entry hadn't been written yet. Both calls went all the way to Finnhub and PostgreSQL — each adding their own 2000ms latency.

This is the invisible failure mode: Redis latency doesn't just slow down cache reads, it breaks the cache population itself. The system continued to function, but silently lost all caching benefits. Every request was hitting the full stack.

Real Latency Numbers per Asset

Figure 1: Comparison between healthy Redis hits (green) and system performance under 2000ms ±500ms injected latency (red). Note that even a cold cache miss under normal conditions takes only 29ms, proving that chaos doesn't just slow down the system—it fundamentally breaks the efficiency of the architecture.

Asset	No Chaos (Redis hit)	With Chaos
AAPL	5.9ms	1823.9ms
GOOGL	6.7ms	1870.6ms
NVDA	7.4ms	2279.5ms
AMZN	7.2ms	2295.1ms
META	6.4ms	1522.2ms
MSFT	6.0ms	2445.1ms
AMD	5.6ms	2180.9ms

Redis is approximately 300x faster than the chaos scenario.

Circuit Breaker Opens

After enough slow calls, the Circuit Breaker opened automatically:

ERROR: Fallback triggered for symbol: INTC - reason: CircuitBreaker 'financialApi' is OPEN
WARN:  Financial API Circuit Breaker state changed: OPEN -> HALF_OPEN

curl http://localhost:8080/actuator/circuitbreakers | jq

{
  "circuitBreakers": {
    "financialApi": {
      "bufferedCalls": 3,
      "failedCalls": 0,
      "failureRate": "0.0%",
      "failureRateThreshold": "50.0%",
      "notPermittedCalls": 0,
      "slowCallRate": "100.0%",
      "slowCallRateThreshold": "50.0%",
      "slowCalls": 3,
      "slowFailedCalls": 0,
      "state": "OPEN"
    }
  }
}

Recovery

After removing chaos:

./scripts/remove-chaos.sh

The Circuit Breaker transitioned automatically: OPEN → HALF_OPEN → CLOSED. The system recovered without any manual intervention.

Key Takeaways

1. Redis latency silently breaks your cache. This is the most important finding. The system didn't throw errors — it just stopped caching, and every request paid the full cost. Without observability, this would be invisible in production.

2. Circuit Breakers are essential, but need correct wiring. Getting the @CircuitBreaker annotation to work required: creating an interface for the client class (so Spring AOP could create a proxy), aligning the instance name across the annotation, YAML config and registry, and using the right exception types. The annotation is simple — the configuration is not.

3. ToxiProxy gives you a realistic baseline before chaos. Even with zero toxics configured, routing through ToxiProxy adds ~250ms of overhead from TCP proxy traversal and deserialization. This is your real baseline, not localhost-to-localhost speed.

4. The scheduler is also affected. The background job that refreshes quotes every 60 seconds suffered the same latency — meaning cached data would eventually become stale even if the circuit breaker protected the API endpoint.

Running the Project

git clone https://github.com/Doug16Yanc/stock-quotes.git
cd stock-quotes
cp .env-example .env
# fill in your Finnhub API key and credentials

# start infrastructure
docker compose up -d postgres redis toxiproxy

# configure proxies
./scripts/setup-toxiproxy.sh

# start application
docker compose up -d backend nginx

# test
curl http://localhost:8080/api/v1/stock-quotes/get-by-symbol/AAPL

What's Next? (Hardening the Resilience)

Now that we’ve proven how the system behaves under stress and confirmed that the PostgreSQL fallback keeps the lights on, the next steps to achieve true high availability are:

Stale-While-Revalidate Pattern: Implement a strategy where Redis serves "stale" (slightly outdated) data instantly while a background thread fetches the update, eliminating user-facing latency during cache refreshes.
Bulkhead Isolation: Isolate resource pools (threads and connections) to ensure that a catastrophic slowdown in the NASDAQ API doesn't exhaust the backend's thread pool, which could otherwise crash unrelated services.
Deep Observability: Integrate Micrometer, Prometheus, and Grafana to visualize Circuit Breaker state transitions in real-time and create alerts based on "Anomalous Cache Miss Rates."
Stress & Load Testing: Use k6 to simulate a massive volume of concurrent requests during chaos injection to observe the "Thundering Herd" effect—where multiple requests try to rebuild the cache simultaneously.

If you found this useful or have questions, drop a comment. The full source code is on GitHub: Doug16Yanc/stock-quotes

Breaking things on purpose is the best way to learn how they work.

DEV Community: Douglas Carmo

An 8,000 trade settlement simulator shows where virtual threads win big and where CPU-bound work exposes the limit of Loom

Part 1 — I/O-Bound Work: Settling 8,000 Trades

The code behind it

Part 2 — CPU-Bound Work: Monte Carlo VaR on 8,000 Trades

The code behind it

The Rule of Thumb

Stop Guessing Batch Sizes: High-Performance Bulk Inserts in Spring Boot with PostgreSQL COPY

Why Not JDBC Batch?

The COPY Protocol

Implementation in Spring Boot

Three Details Worth Explaining

Transactional Participation

The Risk Argument

When Not to Use CopyManager

Takeaway

Don t Lock Your API - Lock Your Scheduler Instead

The Problem

The Naive Approach And Why It Fails

The Inversion

Phase 1 — API: Persist and Forget

Phase 2 — Scheduler: Close, Batch, Publish

Four Details That Matter

Why This Works: The Key Insight

Kafka Partitioning by Window

Status Flow

What About Late Orders?

Takeaway

When NASDAQ Freezes: Chaos Engineering a Stock Quotes API with Java and ToxiProxy

What is Chaos Engineering?

Architecture

How the Cache Works

The Circuit Breaker

Chaos Scripts

Experiment Results

Baseline — No Chaos

With Chaos — 2000ms ±500ms on All Services

Real Latency Numbers per Asset

Circuit Breaker Opens

Recovery

Key Takeaways

Running the Project

What's Next? (Hardening the Resilience)

The `COPY` Protocol