DEV Community: Rohit Babu Gaddeti

The work and its checkpoint can never be atomic

Rohit Babu Gaddeti — Tue, 27 Oct 2026 15:15:49 +0000

Description

When one service does the work and another records it, some work runs twice. Idempotency is a correctness requirement.

This post covers two writes that look like one step: doing some work, and recording that it was done.

In this design the work runs in one service. Another service writes the record. No transaction spans both. So some work will run twice. The shape of the design guarantees it, and high load is not required. That makes idempotency a correctness requirement. If a stage cannot be made idempotent, the architecture has to change. Tighter timeouts will not fix it.

The architecture

The platform is an enterprise quoting system. It runs as Spring Boot services on EKS, and around 30 of them share one self-managed Oracle 12c instance.

Each in-flight workflow has one row in a PENDING_WORK table. The row is keyed by GROUP_ID and QUOTE_ID and carries a STAGE and a STAGE_STATE. A scheduler runs on every pod, and ShedLock makes it a singleton. It orchestrates and holds no business logic. Each stage is an HTTP endpoint on a core service. The orchestrator calls the endpoint, waits for success, then writes the next stage. Both services can update the row, but only the core service writes the business tables. Retry is bounded, after which the row is marked non-resumable. One user submission fans out into roughly 120 external API calls and database writes.

Two commits, one gap

The core service commits the work. Later, the orchestrator commits the checkpoint. These are two commits in two processes, and every ordering leaves a gap. Work then record risks duplication. Record then work risks silent loss, because the row says done and nothing will retry it. There is no third ordering. This design does the work first, which is the better failure.

Here is the window:

The orchestrator reads a row at PRICING / PENDING.
It sends POST /stages/pricing to the core service.
The core service does the work and commits.
The core service returns 200 OK.
The orchestrator pod dies before it updates the row. A node drain or an OOM kill is enough.
The ShedLock lease expires. A scheduler on another pod acquires it.
That scheduler reads the row. It still says PRICING / PENDING.
It sends the same request. The work runs twice.

Step 5 can also be a read timeout after step 3. The orchestrator sees a failure and retries. The team built that retry as the recovery path. It is also what produces the duplicate. The team also spent real effort sizing executors and queues against the connection pool. That work controls load. It has no effect on whether a stage runs twice. Shorter timeouts shrink the window. Nothing closes it.

Why not two-phase commit

XA works. Oracle supports it, and Spring can drive it through a JTA transaction manager such as Narayana. The price is extra round trips and forced log writes on every transaction, plus a coordinator that is a failure domain of its own. If the coordinator dies after prepare, participants hold in-doubt locks until someone resolves them. On a database shared by 30 services, those locks block other teams. Most teams decline XA for these reasons, and they are right to.

This system has a more basic problem. A stage is an HTTP call, and HTTP has no prepare phase. The core service cannot join a transaction owned by the orchestrator. The third-party APIs behind it cannot either.

Exactly-once is not on offer

At-most-once means possible loss. At-least-once means possible duplication. Exactly-once across a network boundary does not exist, because a sender cannot tell a lost request from a lost reply. Kafka's exactly-once guarantee holds inside Kafka and ends at the first external side effect. What a system can build is effectively-once: at-least-once delivery plus a receiver that makes repeats harmless.

Semantics	Guarantee	Cost	Choose it when
At-most-once	No duplicates. Loss is possible.	Almost none. No retry, no durable state.	Occasional loss is acceptable, as with metrics.
At-least-once	No loss. Duplicates are possible.	Retries and a durable record of pending work.	The receiver is already idempotent.
Effectively-once	Each effect lands once.	At-least-once plus a dedup store or idempotent writes, with retention and cleanup.	A duplicate costs money or breaks correctness.

The workflow above is at-least-once by construction. Every stage has to be effectively-once at the receiver.

Making a stage idempotent

The cheapest option is a natural key already in the data, and it is the one the team uses. Every stage endpoint is idempotent on GROUP_ID and QUOTE_ID, the identifiers that key the work row. That is the only reason the trace above ends safely.

With a unique constraint, a second insert fails with ORA-00001, which Spring raises as DuplicateKeyException. The endpoint has to catch it and return the original result as a success. An error sends the orchestrator into retries until the row goes non-resumable.

Endpoint idempotency covers the database writes keyed on those identifiers. External calls need more. If a stage composes several calls with CompletableFuture and one fails, a retry repeats the calls that succeeded, unless the stage checks for a stored result before each one.

Without a natural key, the caller can send an explicit idempotency key. It must be stable across retries, so it is derived from the row identifiers and the stage. A fresh UUID per attempt would defeat it. The receiver stores each key with its response, which costs an extra write per call, an index and a cleanup job.

Some operations are idempotent by nature. A MERGE to a known value is safe to repeat, and so is SET STATUS = 'PRICED'. An increment such as SET TOTAL = TOTAL + ? is not, and neither is an unkeyed insert into a history table. Rewriting increments as absolute sets often fixes a stage with no new infrastructure.

A dedup table generalises the key store, and its hard question is retention. Keys must outlive every path by which a request can be replayed, including a manual resume of a non-resumable row weeks later. Keys kept for 24 hours protect nothing against that.

The hard case is a third-party API that is not idempotent and takes no key. The caller writes an intent row before the call and an outcome after it. An intent with no outcome is unknown, and a blind retry may repeat the effect. Unknowns go to reconciliation through a status lookup or the provider's reports. A confirmed duplicate needs a compensating action, such as a cancel. At an illustrative rate of 1 ambiguous call in 10,000, 120 calls per submission leaves about 1.2% of submissions with at least one unknown.

The outbox, when both sides are owned

The outbox pattern closes the gap when the team owns both sides. The service writes its business change and an event row in one local transaction. A publisher sends unpublished rows afterwards.

CREATE TABLE OUTBOX_EVENT (
  ID            NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  AGGREGATE_KEY VARCHAR2(100) NOT NULL,
  EVENT_TYPE    VARCHAR2(50)  NOT NULL,
  PAYLOAD       CLOB          NOT NULL,
  CREATED_AT    TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL,
  PUBLISHED_AT  TIMESTAMP
);
CREATE INDEX OUTBOX_PENDING_IX ON OUTBOX_EVENT (PUBLISHED_AT, ID);

The publisher still sends first and marks second. That is the same dual-write, moved to a place where a retry is safe.

@Scheduled(fixedDelay = 1000)
@SchedulerLock(name = "outboxPublisher", lockAtMostFor = "PT30S")
public void publishPending() {
    List<OutboxEvent> batch = jdbc.query(
        "SELECT * FROM OUTBOX_EVENT WHERE PUBLISHED_AT IS NULL " +
        "ORDER BY ID FETCH FIRST 100 ROWS ONLY", outboxMapper);
    for (OutboxEvent e : batch) {
        sender.send(e.aggregateKey(), e.id(), e.payload()); // a crash here causes a resend
        jdbc.update("UPDATE OUTBOX_EVENT SET PUBLISHED_AT = SYSTIMESTAMP WHERE ID = ?",
            e.id());
    }
}

The outbox buys atomicity between a state change and its message. It costs a table, a publisher and constant polling against a database that 30 services already share. Delivery is still at-least-once, so consumers dedupe on ID.

A narrower version fits the database-only stages. The core service already writes PENDING_WORK. If it advanced STAGE in the same transaction as its business writes, those stages would have no gap. That needs one datasource inside one @Transactional method. Two datasources are two transactions, even on one Oracle instance. The cost is that stage transitions would live in two services, and the orchestrator would have to read an already-advanced row as success. External API stages gain nothing.

Fencing for concurrent work

Idempotency handles repeated work. Fencing handles concurrent work, and this system needs both. ShedLock grants a lease with an expiry. If the holder stalls in a long GC pause, or a run outlives lockAtMostFor, another pod takes the lock while the first is still running. The row also has two writers by design, the orchestrator and the core service.

The team already has part of a fence. An ATTEMPT_COUNT column is updated on every execution of a stage. The orchestrator stops triggering a stage once the count reaches a configurable threshold. That bounds how often a stage runs. It does not stop two workers running it at once. If the count is read, checked and then incremented, two workers that both read 2 can both start attempt 3.

The same column becomes a fencing token once every write is conditional on it.

// Claim the attempt. Only one worker can move the count from n to n + 1.
int claimed = jdbc.update(
    "UPDATE PENDING_WORK SET ATTEMPT_COUNT = ATTEMPT_COUNT + 1 " +
    "WHERE GROUP_ID = ? AND QUOTE_ID = ? AND STAGE = ? AND ATTEMPT_COUNT = ?",
    groupId, quoteId, stage, readCount);
if (claimed == 0) return; // another worker owns this attempt

// ...call the stage endpoint...

int rows = jdbc.update(
    "UPDATE PENDING_WORK SET STAGE = ?, STAGE_STATE = ? " +
    "WHERE GROUP_ID = ? AND QUOTE_ID = ? AND STAGE = ? AND ATTEMPT_COUNT = ?",
    nextStage, nextState, groupId, quoteId, stage, readCount + 1);
if (rows == 0) throw new StaleCheckpointException(groupId, quoteId, stage);

A stale worker's checkpoint now matches zero rows. Without the condition, it could move a row back to a stage already passed. The stale call to the endpoint still happens, and idempotency makes it harmless. The conditional claim also makes the threshold exact. The core service's writes to the row need the same condition.

A decision guide

For each stage, the questions go in this order.

Does running it twice leave the same state? Call it twice with the same input and compare the rows. If it does, the stage is already idempotent.
Is there a natural key, or can the orchestrator send a stable key that the receiver stores? Then idempotency is cheap. The cost is storage and a retention rule that covers manual resumes.
Is the effect owned by a third party that accepts no key? Then it needs an intent record, reconciliation and compensation. It also belongs in its own stage, so retries of other work do not repeat it.
If none of these hold and a duplicate is unacceptable, the architecture has to change. Timeouts and retry counts are the wrong tools.

Isn't this what Temporal and Step Functions are for?

Yes. Both solve durable execution properly. Either one would replace the work table, the ShedLock scheduler and the hand-written retry logic.

Neither removes the need for idempotency. A Temporal activity or a Step Functions task is still a call that can succeed and lose its reply. Temporal's own documentation recommends idempotent activities. The dual-write moves to the activity boundary and stays there.

The counterweights are specific. Step Functions Standard workflows bill per state transition, and a 120-way fan-out multiplies the transitions in every submission. Workflow state lives in the engine's store, outside Oracle, so it is not transactional with business data. That is two systems without a shared transaction, again. A new platform also brings new on-call knowledge to a team that already runs its own database. With either tool in place, the team would still be writing idempotent stages.

Threads Are Not Connections: A Spring Boot Concurrency Incident

Rohit Babu Gaddeti — Fri, 16 Oct 2026 10:17:40 +0000

Description:

A Spring Boot service slowed down under load because API calls and database calls shared one thread pool. How the team found it, fixed it, and why virtual threads would not have helped.

Load went up, and the service slowed down with it. p99 latency climbed from 400 ms to 18 s. CPU sat around 20%. Connection timeout errors started showing up in the logs.

A slow service usually has something working too hard. This one did not. Requests were piling up, yet very little real work was getting done. Most of the time was going to waiting, and the engineers had to find out what for.

What the service was doing

Each unit of work was a short chain of blocking calls. Some went to external APIs. Some went to the database. A single unit could touch both, more than once, one after another.

The service ran many units at the same time with CompletableFuture. Every stage of every unit ran on one shared thread pool of 100 threads. Database access went through a Hikari connection pool with 20 connections.

Both kinds of calls block the thread that makes them. A thread that sends an HTTP request sits idle until the response comes back. A thread that needs the database first asks Hikari for a connection. If none is free, it waits. Then it waits again for the query. While a thread waits, it does no useful work. It is also unavailable for anything else.

So the 100 threads were one shared budget for two different kinds of waiting. When the APIs were fast and connections were free, the budget was enough. When either side slowed down, it spent the budget for both.

That design was the team's own choice, and it was easy to set up. The problem was that 100 did not describe any real limit in the system. It was just a number.

Finding the cause

The team took thread dumps during the slowdown and lined them up with Hikari's metrics. Nearly every pool thread was blocked. Some were waiting on HTTP responses. Many more were parked inside getConnection. Hikari showed all 20 connections in use and up to 60 threads waiting for one.

A thread could get stuck in two ways, and each made the other worse.

The first was slow API calls. Under load, external calls took longer and held their threads longer. That left fewer threads for every other stage, including database work, which sat in the executor's queue.

The second was the database. With 100 threads and only 20 connections, most of the pool could end up waiting inside Hikari. A thread waiting for a connection is just as unavailable as one waiting on HTTP. API stages then queued behind them.

Because both kinds of work drew from the same threads, slowness on one side became delay on the other. Every unit of work waited in the same queue, whatever resource it needed next.

The root cause was a mismatch between thread demand and available connections. The thread pool and the connection pool had been sized as two separate settings. They should have been one decision. This also explained the low CPU. Blocked threads use almost none.

Transaction scope made it worse. One code path opened a transaction, called an external API, and wrote the result before committing. The connection stayed checked out for the whole API call. In that path, every slow response took a connection out of the pool while no query was running.

The fix: one executor per resource

The team split the shared pool into two executors. One runs database work. The other runs external API calls. Each stage now runs on the executor for the resource it blocks on.

@Configuration
class ExecutorConfig {
    // 18 = Hikari maximum-pool-size (20) minus 2 for request-path queries
    @Bean ThreadPoolTaskExecutor dbExecutor()  { return pool("db-", 18); }
    // 80 = ~200 calls/s x 0.3 s p95 latency = 60, plus headroom
    @Bean ThreadPoolTaskExecutor apiExecutor() { return pool("api-", 80); }

    private static ThreadPoolTaskExecutor pool(String prefix, int size) {
        var ex = new ThreadPoolTaskExecutor();
        ex.setCorePoolSize(size);
        ex.setMaxPoolSize(size);
        ex.setThreadNamePrefix(prefix);
        return ex;
    }
}

The DB executor was resized to align with the Hikari pool. A database thread without a connection can only wait, so threads beyond the connection count add waiting and nothing else. With the sizes aligned, extra database work waits in the executor queue, where it holds no thread and no connection. The executor got 18 threads rather than 20, because a few request-path queries use the same Hikari pool outside the executor.

The API executor needed a different method, and Little's Law gives it. In plain words, the calls in flight equal the calls started per second multiplied by the seconds each call takes. At peak, the service handled about 100 units of work per second, each making two API calls. That is about 200 calls per second. The API's p95 latency under load was around 300 ms, so about 60 calls are in flight at any moment. The team added headroom and set the executor to 80 threads.

The total thread count barely changed, from 100 to 98. What changed was which work could use which threads.

Latency is a live input here. If the API gets twice as slow, the same throughput needs twice as many threads. A call with no timeout breaks the math completely, because it can hold a thread for as long as the remote side likes.

The stages then hop between executors explicitly. A simplified version:

CompletableFuture<Void> process(Job job) {
    return CompletableFuture
        .supplyAsync(() -> externalApi.fetch(job), apiExecutor)
        .thenApplyAsync(data -> repository.save(job, data), dbExecutor)
        .thenApplyAsync(saved -> externalApi.confirm(saved), apiExecutor)
        .thenAcceptAsync(reply -> repository.markDone(job, reply), dbExecutor);
}

The Async variants matter. Plain thenApply can run the next stage on whatever thread completed the previous one. In this chain, that would quietly put database work back on an API thread.

The split also assumes each stage touches one resource. The transactional path broke that rule, so the team moved its API call out of the transaction. The transaction now covers only the writes. They also added dashboards and alerts on pending connections and on each executor's queue depth.

After the fix, p99 at peak load settled around 450 ms. Connection timeout errors stopped. Peak throughput went from about 70 units per second during the incident back to the expected 100.

It worked because the two kinds of waiting stopped sharing a budget. A slow API can fill its own executor, but it cannot take threads from database work. A busy database cannot stall API stages. A slow API still lowers overall throughput, since every unit needs both. It just no longer drags the database side down with it.

Would virtual threads have prevented this?

Java 21 made virtual threads a standard feature. They are cheap, and a service can run a very large number of them. When a virtual thread blocks on I/O, it usually releases the platform thread underneath it. Spring Boot 3.2 added a spring.threads.virtual.enabled property that moves several of its own components onto virtual threads on Java 21 or later.

The incident was a story about blocked threads, so the question is fair. The answer is no. Virtual threads make threads cheap. They do nothing about connections, and there would still be only 20 of those.

Picture the same service with one virtual thread per task, using Executors.newVirtualThreadPerTaskExecutor(). The 100-thread limit is gone, and every unit starts

Notify via email when an event occurs in a website

Rohit Babu Gaddeti — Wed, 20 May 2020 11:22:29 +0000

Ever wanted to know as soon as

a product comes into stock?
tickets are available for any event?

I think most of us have had those questions. After reading this, you'll know how to implement answers for the above questions/similar questions.

This implementation had paved ways to find answers to the above questions.

The project mentioned above was built in last summer(2019 April) during the IPL season. What is IPL? check here.

The main objective of this project is to notify the user when someone hits a SIX in the IPL match. With few modifications, you can build a system to notify yourself when an event occurs in a website.

Diving into how I did it, the first and core concept I should discuss about is Web Scraping. Web Scraping allows you to get data from websites even if the website owner is not providing any API kind of thing. I used Python language in this project and will be talking about python packages throughout this post. Python has web scraping libraries like BeautifulSoup, Scrapy etc. I used BeautifulSoup here because it is simple and fits our need.

BeautifulSoup consists of powerful parsers that can parse response objects into traversable objects. It also has several inbuilt methods to filter out the data extracted from the website. We can achieve a lot of things using them.

So, you first need to make a request to the website you wanted to scrape using its URL. This can be done with packages like urllib, urllib3, requests etc. You'll get a HttpResponse object from the server of the website you requested. Now you have to pass that response object to BeautifulSoup class specifying which parser to use. It will give a parsed bs4 object accordingly.

Now, you have to check for the required data from the parsed object using builtin methods or regular expressions.

You'll want to notify after checking for the data. To notify the user via email, you need two packages: smtplib, email.

email is used to frame email object which we will be using in sendmail method of smtplib.

Using threads to send notifications can prevent the pausing of main activity.