DEV Community: William Nogueira

Distributed Transactions in Java: Why I Skipped the Outbox Pattern

William Nogueira — Mon, 06 Jul 2026 20:39:13 +0000

Picture a payment flowing through a system: authorize the funds, capture the payment, post it to the ledger. Three steps, three services, and no way to wrap them in a single @Transactional, each service has its own database, and the network sits between them.

Now the interesting part: the authorization succeeds, and the capture fails. You've reserved money on a customer's card for a payment that will never happen. Nothing crashed, no exception went unhandled, every service did its job. The transaction is what broke.

This is the problem the Saga pattern exists for, and I built an orchestration-based saga coordinator in Spring Boot to explore it properly, compensation, retries, crash recovery, the works. In this article I'll focus on the most controversial decision I made along the way: I didn't use the Transactional Outbox pattern, the thing every microservices reference architecture tells you to use. I'll show you why, and when you still should.

The Saga Pattern in 60 Seconds

A saga replaces one impossible distributed transaction with a sequence of local ones. Each step has a compensation: an action that semantically undoes it. If step 3 fails, you run the compensations of steps 2 and 1, in reverse, and the system ends up consistent again.

I chose orchestration over choreography: a central engine owns the state machine, so "where is my payment?" has one answer and one place to attach timeouts, retries, and manual intervention. The orchestrator talks to participants exclusively over RabbitMQ command and reply queues: never in-process, because lost replies and duplicate deliveries are the whole point of the exercise.

The Textbook Problem: The Dual Write

Here's the trap every saga implementation has to deal with. When the orchestrator advances a step, it must do two things:

Write the new state to its database ("step 2 is now RUNNING")
Publish the command to the broker ("capture the payment")

These are two different systems, so you can't commit them atomically. Publish first and the DB write can fail: you've commanded a capture you have no record of. Commit first and the process can crash before publishing: your database says a command is in flight that never left the building.

The textbook answer is the Transactional Outbox: write the outgoing message into an outbox table in the same DB transaction as the state change, and let a relay process publish from that table afterwards. It works. It's also a second table, a polling relay, ordering concerns, cleanup jobs, and one more moving part that can fail in its own creative ways.

Before adding all that, I stopped and asked what the outbox actually buys here.

The Realization: My Saga Table Was Already an Outbox

Look at what the orchestrator persists when it dispatches a step:

public void markDispatched(Instant when) {
    transitionTo(StepStatus.RUNNING);
    this.attempts++;
    this.dispatchedAt = when;
}

A step in RUNNING with a dispatched_at timestamp and no reply is the record of a command that must be in flight. That's exactly the information an outbox row carries and I was about to persist it twice.

So instead, commands leave after the state is durable, using a post-commit hook:

private void dispatchAfterCommit(StepCommand command, int attempt) {
    TransactionUtils.executeAfterCommit(
            () -> commandDispatcher.dispatch(command),
            () -> log.error("dispatch of step '{}' for saga {} failed; "
                    + "the armed timeout will fail the step",
                    command.stepName(), command.sagaId()));

    // armed even if the dispatch above fails: a command that never left is
    // indistinguishable from a reply that never came back, and both end here
    TransactionUtils.executeAfterCommit(
            () -> stepTimeoutScheduler.scheduleTimeoutCheck(
                    command.sagaId(), command.stepId(), attempt));
}

That comment is the entire architecture in one sentence. If the process crashes between commit and dispatch, the exact window the outbox exists for, the command is simply gone. And from the orchestrator's perspective, a command that never left looks identical to a participant that never answered. I already needed timeout handling for dead participants. Why build a second recovery mechanism for a failure that's indistinguishable from the first?

Timeouts Are the Relay

Every dispatched command arms a timeout. If no reply lands in time, the step is retried with linear backoff, and only silence retries. An explicit business failure ("card declined") goes straight to compensation, because retrying a decision is pointless:

if (step.getStatus() == StepStatus.RUNNING) {
    if (step.getAttempts() < sagaProperties.stepMaxAttempts()) {
        retryDispatch(saga, step, step.getName());
        return;
    }

    step.transitionTo(StepStatus.FAILED);
    beginCompensation(saga);
}

Notice the retries are attempt-guarded: each timeout check carries the attempt number it was armed for, so a stale timer from attempt 1 can't kill a step that's already on attempt 2.

Crash Recovery: One Sweep, Same Path

The timers live in memory, so a restart loses them. That's fine, they were never the source of truth. A recovery sweeper runs at startup and on a schedule, re-deriving every deadline from persisted state and routing stragglers through the same timeout handler:

@Scheduled(fixedDelayString = "${saga.sweep-interval}")
public void sweep() {
    var now = clock.instant();
    var inFlight = sagaStepRepository.findByStatusIn(
            List.of(StepStatus.RUNNING, StepStatus.COMPENSATING));

    for (var step : inFlight) {
        var deadline = step.getDispatchedAt()
            .plus(sagaProperties.stepTimeout().multipliedBy(step.getAttempts()));
        if (now.isAfter(deadline)) {
            sagaOrchestrator.onStepTimeout(
                    step.getSaga().getId(), step.getId(), step.getAttempts());
        }
    }
}

One recovery story, two triggers. And because the handler is state-guarded and attempt-guarded, and the saga row carries an optimistic lock, the sweep is safe to run concurrently, five pods running it against one database would just be four no-ops and a winner.

Idempotency Without a Dedupe Table

The same thinking killed the message-deduplication table. RabbitMQ redelivers, participants retry, timers race replies, but a reply can only act on a step that's in exactly the right state:

switch (step.getStatus()) {
    case RUNNING -> onStepReply(saga, step, reply);
    case COMPENSATING -> onCompensationReply(saga, step, reply);
    default -> log.warn("ignoring duplicate or late reply for saga {} step {}",
            reply.sagaId(), reply.stepId());
}

A duplicate success reply finds the step already COMPLETED and falls into the default arm. A reply that arrives after the timeout already failed the step? Same. Two threads processing the same reply concurrently? One commits, the other gets an OptimisticLockException and rolls back. The state machine is the dedupe.

The Trade-off: When You Should Still Use an Outbox

What I gave up: recovery latency. An outbox relay republishes a lost message within seconds. My design waits out the step timeout first: bounded and tunable, but slower. For a payment saga where the participant itself might take seconds to respond, that's acceptable. For a system where a lost message must be recovered near-instantly, it isn't.

Where the outbox genuinely wins: when you're publishing events rather than commands: fire-and-forget notifications where no reply will ever come back to tell you something was lost. No reply channel means no timeout, and no timeout means my whole recovery story evaporates. If I ever split the participants into real services that publish their own domain events, they will get outboxes.

The pattern isn't wrong. I just didn't have the problem it solves: my state machine already solved it as a side effect of existing.

Conclusion

The saga engine ended up with exactly one recovery mechanism: persist intent first, dispatch after commit, and let timeouts plus a sweeper re-drive anything the real world loses. Fewer moving parts, one code path to test: and the test suite hammers it with duplicate replies, dead participants, and simulated crashes against real Postgres and RabbitMQ containers.

Patterns are answers to specific problems. Before you add one, check whether your system already answers the question: sometimes the table you have is the table the pattern would have given you.

The full implementation, compensation, manual intervention endpoints, metrics, the whole state machine is on GitHub.

Scaling Java with Write-Behind Caching

William Nogueira — Wed, 24 Dec 2025 03:02:17 +0000

If you ask a Junior Developer to build a URL Shortener (like Bit.ly) that tracks click analytics, the logic usually looks like this:

User clicks the short link (/abc).
App fetches the original URL from the database.
App increments the click counter in the database (UPDATE urls SET clicks = clicks + 1 ...).
App redirects the user.

This works great for 100 users. It works okay for 1,000 users.
But at 10,000 requests per second (RPS)? Your database locks up, latency spikes, and your application crashes.

Why? Because Database I/O is expensive. Blocking a thread to write a "+1" to disk for every single click is a massive waste of resources.

In this article, I’ll show you how I solved this using the Write-Behind Caching pattern, Redis, and Java 25 Virtual Threads.

The Bottleneck: Synchronous Writes

In the naive approach, the "Read" (finding the URL) and the "Write" (tracking the click) are coupled.

If your database takes 50ms to perform the update, your maximum throughput per thread is capped. If traffic spikes, your connection pool exhausts, and requests start timing out.

The Solution: Write-Behind Caching

To achieve high scale, we need to decouple the analytics from the redirect. The user wants to go to Google.com; they don't care if we counted their click right now or 5 minutes from now.

Here is the architecture I implemented in my project:

Fast Path: Increment the counter in Redis (Memory is fast!).
Dirty Tracking: Add the Short Code to a "Dirty Set" in Redis so we know which records changed.
Slow Path (Async): A background scheduler wakes up, reads the Dirty Set, and flushes the counts to the Database (DynamoDB/Postgres) in batches.

Let's look at the code.

Step 1: The "Fire and Forget" Increment

I used Spring Boot 4 and RedisTemplate. When a request comes in, we hit Redis. This operation takes sub-milliseconds.

@Async("clickExecutor")
public void incrementClickCount(String code) {
    // 1. Atomic Increment in Redis (Super fast)
    redisTemplate.opsForValue().increment("clicks:" + code, 1);

    // 2. Mark this code as "Dirty" so the scheduler knows it changed
    redisTemplate.opsForSet().add("dirty_urls", code);
}

Notice the @Async. The controller returns the redirect immediately. This logic ensures the user experiences zero latency penalty for our analytics tracking:

@Configuration
@EnableAsync
public class AsyncConfig {

    @Bean(name = "clickExecutor")
    public Executor clickExecutor() {
        return new TaskExecutorAdapter(Executors.newVirtualThreadPerTaskExecutor());
    }
}

Step 2: The Virtual Thread Scheduler

This is where Java 25 shines.

Historically, background tasks that talk to databases were heavy because they blocked OS threads. With Project Loom (Virtual Threads), we can spawn threads that are lightweight. If the database is slow, the Virtual Thread "parks" itself, freeing up the carrier thread to do other work.

Here is the scheduler that runs every 5 minutes:

@Scheduled(cron = "0 */5 * * * *") // Runs every 5 mins
public void persistClicks() {
    log.info("Starting sync job...");

    while (true) {
        // 1. Pop a batch of codes (e.g., 1000 at a time) from the Set
        var codes = redisTemplate.opsForSet().pop("dirty_urls", 1000);

        if (Objects.isNull(codes) || codes.isEmpty()) {
            break;
        }

        // 2. Process in parallel using Virtual Threads
        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
            codes.forEach(obj -> executor.submit(() -> syncToDb(obj.toString())));
        }
    }
}

Step 3: The Atomic Sync

The syncToDb method is responsible for moving the data from Cache to Storage.

private void syncToDb(String code) {
    // 1. Get the count from Redis and RESET it to 0 atomically
    var clickCountObj = redisTemplate.opsForValue().getAndSet("clicks:" + code, "0");

    if (clickCountObj == null) return;
    long clicks = Long.parseLong(clickCountObj.toString());

    if (clicks > 0) {
        // 2. Update the Database (in this case, DynamoDB)
        repository.updateClickCount(code, clicks);
    }
}

public void updateClickCount(String code, long totalClicks) {
    dynamoDbClient.updateItem(req -> req
            .tableName("urls")
            .key(Map.of("code", AttributeValue.builder().s(code).build()))
            .updateExpression("ADD clicks :inc")
            .expressionAttributeValues(Map.of(
                    ":inc",
                    AttributeValue.builder().n(String.valueOf(totalClicks)).build()
            ))
    );
}

The Trade-off: What about Data Loss?

In distributed systems, everything is a trade-off. By using Write-Behind, we gain massive performance, but we introduce a risk: Durability.

If the Redis instance crashes completely before the 5-minute sync triggers, we lose the clicks that happened in that window.

Why is this acceptable?

Business Value: Losing 5 minutes of click analytics is usually acceptable. Losing the actual URL mappings (Create operation) is not. (Creation is synchronous; Analytics is asynchronous).
Mitigation: We can enable Redis AOF (Append Only File) persistence to minimize data loss on crash.

The Results

By implementing this pattern:

Latency: The GET /{code} endpoint response time dropped from ~60ms (DB write) to ~5ms (Redis write).
Throughput: The database is no longer hammered by every single click. It only receives bulk updates every few minutes.
Cost: We reduced Database Write Units (WCUs), which is the most expensive part of cloud databases like DynamoDB.

Conclusion

You don't always need microservices to scale. Sometimes, you just need to stop treating your database like a notepad. Patterns like Write-Behind Caching, combined with modern tools like Virtual Threads, allow monoliths to handle massive loads efficiently.

You can check out the full implementation (including the Docker Compose setup and Testcontainers integration) in the repository below:

GitHub Repo: Distributed URL Shortener

Comparing Database Management Systems: MySQL vs PostgreSQL

William Nogueira — Sat, 02 Sep 2023 19:08:04 +0000

If you're a beginner looking to dive into the world of relational database management systems (RDBMS), you may find yourself faced with the question of which one to learn first. Two popular options in the RDBMS landscape are MySQL and PostgreSQL. In this article, we will explore the key differences between these two systems, their areas of application, and help you decide which one might be the best fit for your needs.

Overview of MySQL and PostgreSQL

MySQL and PostgreSQL are both open-source RDBMS that have been widely adopted by developers and organizations around the world. While they share some similarities, they also have distinct characteristics that set them apart. Let's take a closer look at each of these database systems.

MySQL

MySQL has a long history and is known for its simplicity and ease of use. It was initially developed by Michael Widenius and David Axmark in 1994 and has since become one of the most popular RDBMS options available. MySQL is particularly favored in web development, thanks to its seamless integration with popular web technologies like PHP. It offers a wide range of features, including support for multiple storage engines, replication, and high availability.

PostgreSQL

PostgreSQL, often referred to as Postgres, is a powerful and feature-rich RDBMS that has gained a strong following in the developer community. Initially developed at the University of California, Berkeley, in 1985, PostgreSQL has evolved into a robust and highly extensible database system. It boasts advanced features such as support for complex data types, JSON, and full-text search. PostgreSQL is known for its adherence to SQL standards and its ability to handle complex queries and large amounts of data.

Key Differences between MySQL and PostgreSQL

While MySQL and PostgreSQL are both excellent choices for managing relational databases, they have some fundamental differences that may influence your decision. Let's examine these differences in more detail.

Data Types and Flexibility

One significant distinction between MySQL and PostgreSQL lies in their approach to data types. MySQL is a purely relational database, meaning it primarily deals with structured data. On the other hand, PostgreSQL is an object-relational database, which allows for the storage and manipulation of more complex data structures. PostgreSQL supports features such as arrays, hstore, and even user-defined data types, making it a more versatile choice for developers working with diverse data formats.

ACID Compliance

ACID compliance is crucial for ensuring data integrity and consistency in a database. MySQL offers ACID compliance when used with certain storage engines like InnoDB and NDB Cluster. In contrast, PostgreSQL is fully ACID compliant in all configurations. This means that PostgreSQL guarantees transactional integrity, even in complex scenarios, making it a reliable choice for applications that require strict data consistency.

Concurrency Control

Concurrency control is essential for handling multiple simultaneous database operations. MySQL uses a locking mechanism for concurrency control, which can sometimes result in performance bottlenecks when dealing with high levels of concurrent transactions. PostgreSQL, on the other hand, employs a more advanced approach called Multiversion Concurrency Control (MVCC). MVCC allows for better concurrency by creating duplicate copies of records, enabling multiple users to read and update the same data simultaneously without conflicts.

Indexing Options

Indexes play a crucial role in optimizing database performance by enabling faster data retrieval. MySQL supports B-tree and R-tree indexes, which are suitable for hierarchical and spatial data, respectively. PostgreSQL offers a wider range of indexing options, including B-tree, expression indexes, partial indexes, and hash indexes. This flexibility allows developers to fine-tune their database performance based on specific requirements.

Views and Materialized Views

Views are virtual tables derived from the result of a query. They provide a convenient way to organize and access data from multiple tables. While both MySQL and PostgreSQL support views, PostgreSQL takes it a step further with materialized views. Materialized views are precomputed views that store the results of a query as a physical table. This can significantly improve query performance, especially when dealing with complex calculations or aggregations.

Stored Procedures and Triggers

Stored procedures and triggers are essential tools for implementing complex business logic within a database. Both MySQL and PostgreSQL support stored procedures, which are reusable blocks of code that can be called from within the database. However, PostgreSQL has an advantage in that it allows stored procedures to be written in languages other than SQL, providing greater flexibility and enabling developers to leverage their preferred programming languages.

Triggers, on the other hand, are database actions that automatically execute in response to specific events. MySQL supports triggers for SQL INSERT, UPDATE, and DELETE statements. PostgreSQL goes a step further by offering INSTEAD OF triggers, which allow for more complex SQL statements using functions.

Choosing the Right RDBMS for Your Needs

Now that we've explored the key differences between MySQL and PostgreSQL, you may be wondering which one is the best choice for you as a beginner. The answer ultimately depends on your specific requirements and the nature of the projects you'll be working on.

If you're looking for a lightweight and easy-to-use database system with seamless integration into web development frameworks, MySQL may be the better option. Its simplicity and wide adoption make it an excellent choice for beginners and small to medium-sized projects.

On the other hand, if you're working on more complex applications that require advanced features, support for diverse data types, and robust transactional capabilities, PostgreSQL may be the preferred choice. Its adherence to SQL standards and extensive feature set make it well-suited for enterprise-level applications and developers seeking maximum flexibility and control.

The Problem of Bloat in Web Development: Do We Really Need Front-End Frameworks?

William Nogueira — Fri, 23 Jun 2023 07:26:54 +0000

In recent years, the web development industry has witnessed a surge in the use of front-end frameworks for building web applications. These frameworks, such as Angular, React, and Vue.js, offer developers a plethora of tools and functionalities to streamline the development process. However, with the increasing popularity of these frameworks, concerns have been raised about the issue of bloat in web development. This article aims to explore the problem of bloat and question the necessity of front-end frameworks. Is it possible to build efficient and user-friendly web applications without relying on these frameworks? Let's delve deeper and find out.

Understanding Bloat in Web Development

The Growing Size of Web Pages

Over the years, web pages have become increasingly bloated in terms of size. This bloat can be attributed to various factors, including the excessive use of JavaScript libraries, heavy media content, and overcomplicated design elements. To put things into perspective, let's take a look at the average size of web pages over the past decade:

March 2011: Average size 735kB
March 2012: Average size 1008kB
March 2013: Average size 1335kB
March 2014: Average size 1728kB

As we can see, there has been a significant increase in the average size of web pages, indicating a growing problem of bloat. This bloat not only affects the loading speed of websites but also leads to a poor user experience.

The Role of Front-End Frameworks in Bloat

Front-end frameworks, while offering numerous advantages in terms of development speed and code organization, can contribute to the problem of bloat in web development. These frameworks often come bundled with a plethora of features and functionalities, many of which may not be necessary for every project. This results in unnecessary code bloat, leading to larger file sizes and slower loading times.
Additionally, front-end frameworks often rely heavily on JavaScript, which can further contribute to the bloat. JavaScript files can be large and resource-intensive, especially when multiple libraries and dependencies are included. This not only affects the performance of the web application but also increases the load on the user's device, particularly on mobile devices with limited resources.

The Case for Static Pages

The Simplicity and Efficiency of Static Pages

Static pages, built using basic HTML and CSS, offer a simpler and more efficient approach to web development. By eliminating the need for complex JavaScript frameworks, static pages can significantly reduce the bloat and improve the overall performance of a website. With static pages, the focus is on delivering content quickly and efficiently, without unnecessary bells and whistles.

The Advantages of Using Basic HTML and CSS

Building web pages with basic HTML and CSS has several advantages:

Lightweight and Fast: Static pages are lightweight and load quickly, providing a seamless user experience. Without the overhead of complex JavaScript frameworks, the page size is significantly reduced, resulting in faster load times.
Better SEO: Static pages are highly optimized for search engine indexing. The simplicity of static pages allows search engines to crawl and index the content more efficiently, leading to better search engine rankings.
Improved Accessibility: Basic HTML and CSS ensure compatibility across different devices and browsers, making the website accessible to a wider audience. This is particularly important for users with slower internet connections or older devices.
Easier Maintenance: With static pages, there is less code complexity, making maintenance and updates easier. Changes can be made quickly without the need to navigate through a complex framework structure.

The Role of Front-End Frameworks in Modern Web Development

The Benefits of Front-End Frameworks

Despite the concerns surrounding bloat, front-end frameworks offer several benefits that make them a popular choice among web developers:

Rapid Development: Front-end frameworks provide pre-built components, libraries, and tools that streamline the development process. This allows developers to build complex web applications more efficiently and with fewer lines of code.
Code Organization: Frameworks promote modular code structures, making it easier to manage and scale large projects. Components can be reused across different parts of the application, improving code maintainability.
Developer Community: Front-end frameworks have vibrant and active developer communities, providing access to extensive documentation, tutorials, and support. This can be invaluable for developers seeking assistance or looking to learn new techniques.
Rich User Interfaces: Front-end frameworks offer a range of UI components and styling options, enabling developers to create visually appealing and interactive user interfaces. This can enhance the overall user experience and engagement.

Assessing the Need for Front-End Frameworks

While front-end frameworks offer numerous advantages, it is essential to assess the specific needs of a project before deciding to incorporate them. Consider the following factors:

Project Complexity: If the project involves complex user interactions, real-time updates, or extensive data manipulation, a front-end framework may be beneficial. Frameworks provide the necessary tools and abstractions to handle these complexities efficiently.
Development Speed: Front-end frameworks excel in rapid development scenarios, where time-to-market is crucial. If the project requires quick iteration and deployment, a framework can expedite the development process.
Resource Constraints: Consider the target audience and their devices. If the majority of users have limited resources, such as slow internet connections or older devices, opting for a lightweight approach without a framework might be more appropriate.
Long-Term Maintenance: Evaluate the long-term maintenance and scalability requirements of the project. Front-end frameworks can facilitate code organization and scalability, making it easier to maintain and extend the application over time.

Conclusion

In conclusion, the problem of bloat in web development is a growing concern, as web pages continue to increase in size and complexity. While front-end frameworks offer numerous benefits in terms of development speed and code organization, they can contribute to the bloat problem. However, static pages built with basic HTML and CSS provide a lightweight and efficient alternative, particularly for simpler projects or those with resource constraints.

The decision to use a front-end framework should be based on a careful assessment of the project's needs, considering factors such as complexity, development speed, resource constraints, and long-term maintenance. By weighing the advantages and drawbacks of front-end frameworks, developers can make informed decisions and strike a balance between functionality and performance in their web applications.