DEV Community: Ed Legaspi

The Inbox Transaction Boundary: Getting Event Processing Right in Spring Boot

Ed Legaspi — Wed, 16 Sep 2026 00:52:36 +0000

The Inbox Transaction Boundary: Getting Event Processing Right in Spring Boot

One of the hardest parts of building an event-driven system isn't getting messages from one service to another.

It's making sure the system behaves correctly when something fails halfway through processing.

While building NERV Event, I ran into an important reliability problem:

Where should the transaction boundary be when processing an event?

At first, an Inbox can look deceptively simple.

Receive a message.

Check whether it was already processed.

Execute the business operation.

Mark the message as processed.

Acknowledge the message.

But these operations don't automatically share the same transactional boundary.

And that creates some uncomfortable failure scenarios.

The Inbox Pattern

The Inbox pattern is commonly used to make event consumers idempotent.

A simplified flow looks like this:

                    ┌─────────────────┐
                    │     Broker      │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │     Consumer    │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │      Inbox      │
                    │                 │
                    │ event_id        │
                    │ status          │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │ Business Logic  │
                    └─────────────────┘

The Inbox records that an event has been received and tracks its processing state.

For example:

RECEIVED
   │
   ▼
PROCESSING
   │
   ▼
PROCESSED

If the broker delivers the same event again, the consumer can inspect the Inbox and determine whether the event has already been processed.

This gives us a foundation for idempotent event processing.

But there's an important question:

How does the Inbox transaction relate to the business transaction?

The Transaction Boundary Problem

Consider a consumer processing an OrderCreated event.

The consumer needs to:

Receive the event.
Create an Inbox record.
Execute the business operation.
Mark the Inbox record as processed.
Acknowledge the broker message.

It is tempting to think of this as one atomic operation.

It isn't necessarily one transaction.

The database transaction and the broker acknowledgment belong to different systems.

Even within the database, the Inbox update and business operation can accidentally end up in different transactional boundaries.

That matters when something fails.

When the Business Operation Succeeds but the Inbox Doesn't

Consider this sequence:

Business operation
       │
       ▼
    SUCCESS
       │
       ▼
Inbox update
       │
       ▼
    FAILURE

If the business operation and Inbox update participate in the same database transaction, the failure can cause the entire transaction to roll back.

The event can then be retried.

But consider a design where they use separate transactions:

Transaction A
    │
    └── Business operation COMMITTED

Transaction B
    │
    └── Inbox update FAILED

Now the business operation has already committed.

The Inbox doesn't indicate that processing completed.

The broker may redeliver the event.

The consumer sees an event that appears unprocessed and executes the business operation again.

Now idempotency becomes critical.

When the Inbox Says Processed but the Business Operation Rolls Back

The reverse situation can be even more problematic.

Suppose the Inbox is marked as processed before the business operation is committed:

Inbox
  │
  └── PROCESSED

Then the business transaction fails:

Business transaction
  │
  └── ROLLBACK

If those changes aren't part of the same appropriate transactional boundary, the system can end up with contradictory state.

The Inbox says:

PROCESSED

But the business data says:

NOT PROCESSED

A retry may then be suppressed because the Inbox says the event has already been processed.

The system has effectively lost the opportunity to recover.

What Should Share the Transaction?

For a database-backed Inbox, the important relationship is between the Inbox state and the business state.

Conceptually:

┌──────────────────────────────────────────┐
│          Database Transaction            │
│                                          │
│  Inbox state                             │
│       +                                  │
│  Business state                          │
│                                          │
│       └── Atomic commit / rollback       │
└──────────────────────────────────────────┘

The goal is that the Inbox processing state and the business changes have a consistent relationship.

A simplified Spring Boot example might look like:

@Transactional
public void process(Event event) {
    Inbox inbox = inboxRepository.findByEventId(event.id());

    if (inbox.isProcessed()) {
        return;
    }

    businessService.apply(event);

    inbox.markProcessed();
}

The important part isn't the @Transactional annotation itself.

The important part is understanding which operations actually participate in that transaction.

If businessService.apply(event) and the Inbox update use the same database transaction, the database can commit or roll back those changes together.

But What About the Broker?

This is where things become more interesting.

A database transaction normally cannot make a Kafka acknowledgment atomic with the database commit.

You effectively have two systems:

        Database                    Broker
           │                          │
           │                          │
     DB transaction              Message ack
           │                          │
           └──────────┬───────────────┘
                      │
                Different systems

Even with a correctly designed Inbox transaction, there is still a failure window around the broker acknowledgment.

For example:

Database transaction
        │
        ├── Business change
        └── Inbox = PROCESSED
        │
        ▼
     COMMIT
        │
        ▼
Broker acknowledgment
        │
        └── FAILURE

The broker may deliver the message again.

But this is exactly where the Inbox pattern provides its value.

On redelivery:

Event arrives
     │
     ▼
Check Inbox
     │
     ▼
Already PROCESSED
     │
     ▼
Skip business operation

The consumer can safely tolerate duplicate delivery.

This is the fundamental relationship between the Inbox pattern and at-least-once delivery.

The Goal Isn't Exactly-Once Processing

A common misconception is that the Inbox pattern makes event processing exactly-once.

It doesn't.

The broker may still deliver the same message more than once.

Instead, the goal is closer to:

At-least-once delivery
          +
Idempotent processing
          =
Reliable consumer behavior

The Inbox gives the consumer persistent knowledge about an event's processing state.

That state can then be used to make retries deterministic.

Processing State Matters

A simple boolean such as:

processed = true

often isn't enough for a production event-processing system.

A useful Inbox model can distinguish states such as:

RECEIVED
PROCESSING
PROCESSED
FAILED

This makes failures inspectable.

For example:

RECEIVED
   │
   ▼
PROCESSING
   │
   ├──────────────► PROCESSED
   │
   └──────────────► FAILED

Now the system has an explicit record of what happened.

That becomes particularly valuable when retries are involved.

What Happens During a Retry?

Suppose processing fails:

Event
  │
  ▼
PROCESSING
  │
  ▼
Business operation fails
  │
  ▼
FAILED

A retry can then use the Inbox state to determine what should happen next.

Depending on the architecture, the consumer can:

retry immediately
retry after a delay
increment an attempt counter
record the failure
eventually move the event to a failure or DLQ mechanism

The important thing is that the failure becomes persistent state rather than disappearing with the consumer process.

The Inbox Is More Than Deduplication

This is the architectural distinction I find most important.

An Inbox is often introduced as:

A table that prevents duplicate messages.

That's only part of the story.

A production Inbox can provide:

idempotency
processing state
retry state
failure information
inspection
operational visibility
deterministic recovery

That makes the Inbox part of the consumer's reliability boundary.

The question therefore isn't simply:

"Have we seen this event before?"

It becomes:

"What do we know about this event's processing lifecycle?"

Designing the Boundary Explicitly

When designing an event consumer, it helps to make the boundaries explicit.

Think about three separate concerns:

             Broker
                │
                ▼
        Message Delivery
                │
                ▼
        ┌───────────────┐
        │    Inbox      │
        │               │
        │ Receipt/state │
        └───────┬───────┘
                │
                ▼
        ┌───────────────┐
        │   Database    │
        │  Transaction  │
        │               │
        │ Inbox +       │
        │ Business Data │
        └───────────────┘
                │
                ▼
          Broker Ack

The database transaction provides atomicity between the Inbox state and the business changes.

The Inbox provides persistent processing state.

The broker acknowledgment controls message delivery semantics.

These are related, but they are not the same thing.

Spring Boot Makes the Boundary Easy to Hide

Spring Boot makes transaction management straightforward.

That is useful, but it can also make the boundary easy to overlook.

For example:

@Transactional
public void handle(OrderCreatedEvent event) {
    Inbox inbox = inboxService.startProcessing(event);

    if (inbox.isProcessed()) {
        return;
    }

    orderService.createOrder(event);

    inboxService.markProcessed(inbox);
}

The code looks simple.

But several architectural questions are hiding behind it:

Which database transaction is active?
Which operations participate in it?
Does orderService start another transaction?
Can the Inbox update commit independently?
When is the Inbox state persisted?
When is the broker message acknowledged?
What happens if the application crashes after the database commit?
What happens if the broker acknowledgment fails?

The annotation doesn't answer those questions.

The architecture does.

Failure Windows Are Part of the Design

One of the most useful lessons from event-driven architecture is that failure isn't an exceptional edge case.

It is part of the normal operating model.

A consumer can crash after committing its database transaction:

Business operation
       │
       ▼
Inbox + business data
       │
       ▼
    COMMIT
       │
       X
  Application
    crashes
       │
       ▼
Broker redelivers

Or:

Inbox + business transaction
       │
       ▼
    COMMIT
       │
       X
Broker acknowledgment fails
       │
       ▼
Redelivery

A reliable system doesn't try to pretend these scenarios cannot happen.

It defines what should happen when they do.

That's the real purpose of the Inbox.

Building NERV Event

These are the kinds of engineering decisions that shaped NERV Event.

The goal isn't simply to provide an Inbox implementation.

It's to provide explicit persistence and processing semantics that make event-driven behavior easier to understand, operate, and debug.

The Inbox is treated as part of the consumer reliability model rather than just a mechanism for detecting duplicate event IDs.

That distinction becomes increasingly important as systems move from simple message consumption toward production workloads involving:

multiple application instances
retries
failures
duplicate delivery
long-running processing
operational recovery

Final Thoughts

The Inbox transaction boundary is easy to overlook.

Most event-driven examples focus on the happy path:

Receive → Process → Acknowledge

Production systems need to think differently:

Receive
   │
   ▼
Persist state
   │
   ▼
Process
   │
   ├── success ──► Commit
   │
   └── failure ──► Retry / Failure state
                         │
                         ▼
                    Redelivery

The important question isn't whether failures will happen.

They will.

The important question is whether the system's state still makes sense after they do.

That's why the transaction boundary around the Inbox and business operation deserves deliberate architectural attention.

NERV Event

NERV Event is an open-source Spring Boot library for building reliable event-driven systems with patterns such as:

Transactional Outbox
Inbox
Idempotent consumers
Retries
Failure handling
Broker integrations

If you're building event-driven systems with Spring Boot, the transaction boundary is one of those details worth getting right before production forces you to think about it.

Transactional Outbox Pattern with Spring Boot: Reliable Event

Ed Legaspi — Thu, 10 Sep 2026 10:58:02 +0000

A service saves data to the database.

Then it publishes an event to Kafka.

Simple enough.

Until the database transaction succeeds and the Kafka publish fails.

Now your application says something happened, but the rest of the system never hears about it.

This is the dual-write problem, and it's one of the fundamental reliability problems in event-driven systems.

The Transactional Outbox Pattern gives us a practical way to solve it without trying to coordinate a distributed transaction between the database and the message broker.

In this article, we'll look at how it works, what guarantees it actually provides, and some of the production concerns that appear once you start running it across multiple application instances.

The Dual-Write Problem

Consider a typical Spring Boot service:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);

    kafkaTemplate.send(
        "payments",
        new PaymentCompletedEvent(payment.getId())
    );
}

At first glance, this looks reasonable.

But two independent systems participate in this operation:

The application database
Kafka

Your database transaction does not automatically include Kafka.

That creates failure scenarios.

Database succeeds, Kafka fails

Database
Payment = COMPLETED  ✓

Kafka
PaymentCompleted     ✗

The payment exists, but downstream services never receive the event.

Kafka succeeds, database fails

The opposite can also happen depending on when publishing occurs:

Kafka
PaymentCompleted     ✓

Database
Payment = COMPLETED  ✗

Consumers may now process an event representing state that was never committed.

What we really want is this:

If the business transaction commits, the intention to publish its event must commit with it.

That's the problem the Transactional Outbox Pattern addresses.

Introducing the Transactional Outbox

Instead of publishing directly to Kafka inside the business transaction, we persist the event in the same database transaction as the business data.

          DATABASE TRANSACTION
     ┌─────────────────────────────┐
     │                             │
     │  Update Payment             │
     │         +                   │
     │  Insert Outbox Event        │
     │                             │
     └──────────────┬──────────────┘
                    │
                  COMMIT
                    │
                    ▼
             Outbox Dispatcher
                    │
                    ▼
                  Kafka

Now the database provides the atomic boundary.

Either both records commit:

Payment       ✓
Outbox Event  ✓

or neither does:

Payment       ✗
Outbox Event  ✗

Kafka no longer has to participate in the business transaction.

That's the key idea.

We're not making the database and Kafka transactional together.

We're making the business state and the intention to publish transactional together.

What Goes Into the Outbox?

An outbox record contains enough information to publish the event later.

Conceptually:

event_id       = 01J...
event_type     = PaymentCompleted
source         = payment-service
payload        = {...}
content_type   = application/json
status         = PENDING
created_at     = ...

The important part is that both the event payload and its delivery state are persisted.

I prefer keeping the payload readable rather than hiding it behind opaque serialization.

Why?

Because eventually something will fail in production.

When that happens, being able to inspect exactly what your service intended to publish makes debugging considerably easier.

The outbox isn't only a delivery mechanism.

It also becomes useful operational history.

Dispatching the Event

Once the business transaction has committed, a separate dispatcher processes the outbox.

A simplified lifecycle might look like this:

PENDING
   │
   ▼
PROCESSING
   │
   ├───────────────► PUBLISHED
   │
   └── failure
          │
          ▼
       retry later

If an event repeatedly fails and exhausts its retry policy, it can eventually transition to:

FAILED

This separation is important.

The business transaction is responsible for recording what happened.

The dispatcher is responsible for delivering that information.

Those responsibilities should be able to fail independently.

Why Not Just Retry Kafka Inside the Transaction?

A common alternative is to retry the Kafka operation:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);

    retryTemplate.execute(context ->
        kafkaTemplate.send("payments", event)
    );
}

Retries help with temporary broker failures.

But they don't solve the underlying coupling.

Imagine Kafka is unavailable for several minutes.

Should your payment transaction stay open while your application waits for Kafka to recover?

Usually, no.

Long-running transactions can:

hold database resources longer
increase lock duration
increase contention
couple business availability to broker availability

With an outbox, the responsibilities are separated:

Business Transaction
        │
        ├── commits quickly
        │
        ▼
      Outbox
        │
        │ Kafka unavailable?
        │
        └── retry asynchronously

Your business operation doesn't need to wait for the broker.

The Important Catch: At-Least-Once Delivery

The outbox solves message loss, but it introduces another architectural consideration.

Imagine this:

1. Dispatcher publishes event to Kafka
2. Kafka accepts the event
3. Application crashes
4. Outbox wasn't marked PUBLISHED
5. Application restarts
6. Dispatcher publishes the event again

The same event may be delivered twice.

This isn't necessarily a bug.

It's a consequence of favoring reliable delivery over silently losing events.

So a reliable event architecture usually becomes:

Transactional Outbox
        +
At-Least-Once Delivery
        +
Idempotent Consumer

The producer cannot simply assume that every event will be processed exactly once.

Consumers must be designed to tolerate duplicates.

That naturally leads us to the Inbox Pattern, which I'll cover in the next article in this series.

What Happens With Multiple Application Instances?

Production applications rarely run as a single instance.

Suppose we have three pods:

                 OUTBOX
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
        Pod A    Pod B    Pod C

If every instance polls the same outbox table without coordination, multiple workers could select the same events.

Database-level locking can provide one solution.

For databases that support it, a common approach is conceptually similar to:

SELECT ...
FROM outbox
WHERE status = 'PENDING'
FOR UPDATE SKIP LOCKED;

Suppose Pod A locks:

1  2  3

Pod B doesn't wait for those rows.

It skips them and can claim:

4  5  6

This makes it possible for multiple workers to process the outbox concurrently without all competing for the same records.

But locking strategy matters.

A naive SELECT ... FOR UPDATE implementation can create unnecessary contention as throughput and worker count increase.

The important goal is:

Multiple workers should be able to safely claim work without serializing the entire outbox.

Polling vs Change Data Capture

Polling isn't the only way to implement the outbox pattern.

Another common architecture uses Change Data Capture (CDC):

Application
     │
     ▼
Database Outbox
     │
     ▼
    CDC
     │
     ▼
   Kafka

Tools such as Debezium can stream database changes into Kafka instead of having application workers poll the outbox.

CDC can be a great choice, particularly when the required infrastructure already exists.

But it comes with different operational trade-offs.

Application-Level Polling

Advantages include:

simpler infrastructure
straightforward local development
application-controlled retries
easier application-level debugging
potential broker independence

Change Data Capture

Advantages can include:

reduced application polling
high-throughput change streaming
natural Kafka integration
separation of event extraction from application workers

Neither approach is universally better.

More importantly, polling vs CDC isn't the core of the pattern.

The architectural guarantee comes from this:

Business State + Outbox Event
              │
       SAME TRANSACTION

How you transport that committed event afterward is a separate design decision.

Why I Built NERV Event

Once you've implemented this architecture several times, you start seeing the same infrastructure repeatedly:

Outbox persistence
       │
       ├── Serialization
       ├── Dispatching
       ├── Retries
       ├── Concurrency
       ├── Failure states
       ├── Metrics
       ├── Kafka integration
       ├── SQS integration
       └── Operational tooling

That's infrastructure every business application shouldn't have to rebuild.

It's one of the reasons I created NERV Event.

NERV Event is an open-source Spring Boot library focused on reliable event processing using transactional Outbox and Inbox patterns.

The architecture is intentionally straightforward:

Business Service
       │
       ▼
   NERV Event
       │
       ▼
Transactional Outbox
       │
       ▼
   Dispatcher
       │
     ┌─┴─┐
     ▼   ▼
  Kafka  SQS

The business application owns the event.

The infrastructure owns its reliable delivery.

And importantly, the state remains inspectable when something goes wrong.

Reliability Doesn't End at the Producer

The Transactional Outbox Pattern answers an important question:

How do I prevent an event from being lost after my business transaction commits?

But successfully publishing the event isn't the end of the reliability story.

Now we have another service receiving it.

What happens if:

the same event arrives twice?
processing succeeds but acknowledgement fails?
the consumer crashes halfway through processing?
processing needs to be retried?
we need to determine whether an event has already been handled?

These are consumer-side reliability problems.

And that's where the Inbox Pattern comes in.

In Article #2, we'll look at:

Inbox Pattern and Idempotent Consumers

and build the other half of reliable event processing.

NERV Event

NERV Event is open source and available on GitHub:

👉 github.com/czetsuyatech/nerv-event

If you're building event-driven Spring Boot systems and have dealt with outbox/inbox implementations in production, I'm interested in hearing how you're handling these problems — particularly polling vs CDC and multi-instance event claiming.

This article is part of my series on building reliable event-driven systems with Spring Boot.

Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Ed Legaspi — Tue, 01 Sep 2026 14:56:23 +0000

Event-driven architecture looks simple at first.

Your application performs a business operation, publishes an event to Kafka or SQS, and another service consumes it.

@Transactional
public void createOrder(CreateOrderCommand command) {
  Order order = orderRepository.save(...);

  kafkaTemplate.send(
      "orders",
      new OrderCreatedEvent(order.getId())
  );
}

Looks reasonable.

The order is saved, an OrderCreatedEvent is published, and other services can react to it.

But there is a problem hiding in those few lines:

What happens if the database transaction succeeds, but publishing the event fails?

And on the consumer side:

What happens if the same event is delivered twice?

These questions lead to several patterns that become essential once event-driven systems move beyond simple demos:

Transactional Outbox
Inbox Pattern
Idempotent Consumers
Durable Retries
Multi-instance-safe processing
Observable failure handling

Let's build the architecture step by step.

The Dual-Write Problem

Imagine an order service that needs to:

Save an order to PostgreSQL.
Publish OrderCreated to Kafka.

Conceptually:

Database ──────> COMMIT ✓
                   |
Kafka ─────────> PUBLISH ✗

The database and Kafka are independent systems.

A successful database commit does not guarantee a successful Kafka publish.

Consider:

1. INSERT order
2. COMMIT
3. Publish OrderCreated
4. Application crashes

If the application crashes between steps 2 and 3, the order exists but the event doesn't.

Other services may never know that the order was created.

Reversing the operations doesn't fix it:

1. Publish OrderCreated
2. INSERT order
3. Database transaction fails

Now consumers may receive an event for an order that doesn't exist.

This is the classic dual-write problem.

The Transactional Outbox Pattern

Instead of trying to atomically update the database and message broker, persist the event as part of the same database transaction as the business operation.

             Database Transaction
        ┌─────────────────────────────┐
        │                             │
Request ──> Business Data             │
        │       +                     │
        │   Outbox Event              │
        │                             │
        └────────── COMMIT ───────────┘
                       |
                       v
                Outbox Dispatcher
                       |
                       v
                  Kafka / SQS

When an order is created, we persist:

ORDER
  +
OUTBOX EVENT

inside the same transaction.

Either both commit or neither commits.

A separate dispatcher finds pending outbox records and publishes them to the broker.

A simplified lifecycle could look like:

PENDING
   |
   v
PROCESSING
   |
   +──── success ────> PUBLISHED
   |
   └──── failure ────> RETRY / FAILED

This removes the dangerous database-and-broker dual write from the business transaction.

It also gives us something extremely useful:

persistent delivery state.

Make the Outbox Debuggable

Reliability isn't only about retrying failed operations.

Eventually, someone will need to answer:

What happened to this event?

A useful outbox should contain enough information to answer that question:

eventId
eventType
source
correlationId
payload
status
attempts
createdAt
availableAt
publishedAt
lastError

Instead of reconstructing everything from distributed logs, an engineer can inspect the actual state:

SELECT *
FROM event_outbox
WHERE status = 'FAILED';

This leads to an important principle:

Reliability mechanisms should also improve debuggability.

If an event cannot be delivered, that failure should be visible and inspectable.

Reliable Publishing Is Only Half the Problem

Suppose our outbox works perfectly.

Every event eventually reaches Kafka.

We're still not finished.

Most event-driven systems operate with at-least-once delivery.

That means the same event may arrive more than once.

Producer
   |
   v
Kafka
   |
   +──── OrderCreated #123 ────> Consumer
   |
   +──── OrderCreated #123 ────> Consumer

A consumer could successfully process an event and then crash before acknowledging it.

The broker delivers the event again.

If the handler sends an email, perhaps the customer receives two emails.

If it performs a payment operation, the consequences can be considerably worse.

Consumers therefore need to assume:

Every event can arrive more than once.

The Inbox Pattern

The Inbox Pattern provides a durable record of received events.

Before processing an event, the consumer registers its unique event ID.

Broker
   |
   v
Inbox Registration
   |
   +── event already exists ──> DUPLICATE
   |
   └── new event
          |
          v
       RECEIVED
          |
          v
      PROCESSING
        /     \
       v       v
 PROCESSED   FAILED

If the same eventId arrives again, the consumer knows that it has already seen it.

The event ID becomes an idempotency boundary.

Instead of depending on exactly-once delivery, we make duplicate delivery safe.

The Inbox Is More Than a Deduplication Table

A minimal inbox could contain nothing more than processed event IDs.

In a production system, however, it can become a durable history of event processing.

Consider storing:

eventId
eventType
source
correlationId
payload
status
attempts
receivedAt
processedAt
availableAt
lastError

Now imagine an incident:

Order 123 was created, but the downstream action never happened.

You can inspect the inbox.

Was the event received?

Did processing start?

Did the handler fail?

How many attempts were made?

When is the next retry?

What was the last error?

Those questions become much easier to answer when processing state is explicit.

Retries Should Survive Application Restarts

Spring provides excellent retry mechanisms.

For example:

@Retryable
public void handle(OrderCreatedEvent event) {
  ...
}

This can be perfectly appropriate for short-lived transient failures.

But durable event processing introduces another question:

What happens if the JVM dies?

Event processing fails
        |
        v
Retry scheduled in memory
        |
        v
Application restarts

If retry state exists only in memory, it disappears with the process.

For critical event processing, retry state can instead be persisted:

FAILED
   |
   | availableAt <= now
   v
PROCESSING
   |
   +──── success ────> PROCESSED
   |
   └──── failure ────> FAILED
                         |
                         + attempts++
                         + availableAt = next retry

A scheduler periodically finds events whose retry time has arrived.

Because the state lives in the database, restarting the application doesn't destroy the retry information.

Back Off Instead of Hammering a Failing Dependency

Retrying continuously can make an outage worse.

If a downstream service is unavailable, thousands of failed events retrying as quickly as possible only add pressure.

A better strategy is exponential backoff:

Attempt 1 → immediate
Attempt 2 → +1 second
Attempt 3 → +2 seconds
Attempt 4 → +4 seconds
Attempt 5 → +8 seconds

Eventually, the configured retry limit is exhausted.

At that point, the event can remain explicitly failed:

status = FAILED
availableAt = null

Automatic retries stop.

But importantly, the failure doesn't disappear.

The event remains available for investigation and operational recovery.

What About Dead-Letter Queues?

Dead-letter queues are useful, particularly for broker-level failures.

But the broker's DLQ doesn't necessarily need to become the application's primary record of processing failure.

There is a useful distinction:

Broker concern               Application concern

Delivery failure             Processing failure
Malformed message            Business handler failure
Transport problem            Retry exhaustion
        |                            |
        v                            v
       DLQ                         INBOX

The two mechanisms can coexist.

A database-backed inbox gives the application direct visibility into its own processing state, while a DLQ remains available for appropriate broker- and transport-level failures.

Then You Deploy Multiple Pods

Everything becomes more interesting once the application runs more than one instance.

                  OUTBOX
                     |
               pending event
                     |
          ┌──────────┴──────────┐
          v                     v
        Pod A                 Pod B
     Dispatcher             Dispatcher

Both instances may discover the same pending event.

Without concurrency control, both may attempt to process it.

Production implementations therefore need a concept of claiming or locking.

For example:

status
lockOwner
lockedAt

An instance claims records transactionally before processing them.

Other instances can then determine that those records are already being handled.

But this creates another question:

What happens if a pod claims an event and then dies?

The system needs a deterministic mechanism for recovering stale claims after an appropriate timeout.

At this point, the outbox is no longer just a database table plus a scheduled query.

It has become infrastructure.

Even the Scheduler Can Fail

There is another failure mode that is surprisingly easy to overlook.

Suppose the dispatcher completes successfully and schedules its next execution:

Dispatcher completes
        |
        v
schedule(nextRun)
        |
        X
TaskScheduler rejects the task

If the scheduler continues reporting itself as running, the application has entered a dangerous state.

Everything appears healthy.

But no future dispatch will happen.

Events can quietly accumulate in the outbox.

A reliable scheduler therefore benefits from an explicit lifecycle:

STOPPED
   |
   v
STARTING
   |
   v
RUNNING
   |
   +──── scheduling failure ────> FAILED

A useful invariant is:

A scheduler must not report itself as running if no task is scheduled and no work is currently executing.

Scheduling infrastructure itself needs observable failure semantics.

Observability Is Part of Reliability

Imagine receiving a production incident at 2 AM:

We created the order, but the downstream system didn't process it.

Ideally, you should be able to follow the event:

Order
  |
  v
Outbox Event
  |
  +── created
  +── claimed
  +── publish attempts
  +── published
  |
  v
Broker
  |
  v
Inbox Event
  |
  +── received
  +── processing attempts
  +── failure reason
  +── retry schedule
  +── processed

Correlation metadata should connect the pieces.

Payloads should be readable.

State transitions should be explicit.

Failures should remain inspectable.

Logs should explain what the infrastructure is doing without becoming the only source of truth.

A reliable system isn't only one that recovers from failures. It is one that helps engineers understand those failures.

Putting It All Together

Once these pieces are combined, the architecture starts looking like this:

Business Transaction
        |
        v
Transactional Outbox
        |
        v
Outbox Dispatcher
        |
        v
   Kafka / SQS
        |
        v
      Inbox
        |
        v
Idempotency Check
        |
        v
 Event Handler
        |
        +── Success
        |
        └── Durable Retry

Underneath it all is persistent state:

Persistence
├── Outbox delivery state
├── Inbox processing state
├── Attempts
├── Retry scheduling
├── Lock ownership
└── Failure information

That's quite a bit of infrastructure around what originally looked like:

kafkaTemplate.send(...);

From Architecture to Implementation: NERV Event

These are the problems I wanted to solve consistently across Spring Boot applications.

None of the individual patterns are new.

Transactional outbox is well understood.

Idempotent consumers are well understood.

Retries, locking, and message brokers are well understood.

The difficult part is making them work together consistently in a production application.

That's what led me to build NERV Event.

NERV Event is an open-source event infrastructure library for Spring Boot that implements the architecture described in this article.

It brings together:

Transactional outbox persistence
Durable inbox processing
Idempotent event consumption
Persistent retries
Exponential retry policies
Multi-instance-safe processing
Kafka integration
AWS SQS integration
Scheduler lifecycle and failure visibility
Event retention
Operational inspection
Correlation metadata
Human-readable persisted payloads

The goal isn't to hide event-driven architecture behind magic.

The goal is to make its behavior predictable, observable, and easy to debug.

NERV Event

The project is open source:

👉 NERV Event on GitHub

The repository contains the source code, documentation, configuration, and examples.

Even if you don't use the library directly, I hope the architecture and implementation can be useful as a reference when designing reliable event-driven Spring Boot applications.

I'll cover individual parts of NERV Event in future articles, including transactional publishing, inbox processing, retries, Kafka and SQS integration, multi-pod deployments, and operational tooling.

Final Thoughts

Adding Kafka or SQS to a Spring Boot application doesn't automatically make the application reliably event-driven.

The difficult parts exist around the broker:

Business Transaction
        |
        v
Transactional Outbox
        |
        v
Reliable Delivery
        |
        v
At-Least-Once Messaging
        |
        v
Inbox + Idempotency
        |
        v
Durable Processing
        |
        v
Retries + Recovery
        |
        v
Observability

Each layer addresses a different failure mode.

And in distributed systems, those failures aren't theoretical.

Processes restart. Networks fail. Messages are redelivered. Dependencies become unavailable. Schedulers fail. Multiple instances compete for the same work.

The objective isn't to pretend these failures won't happen.

It's to design the system so that when they do:

State is preserved. Recovery is predictable. And engineers can understand exactly what happened.

That's the philosophy behind NERV Event.

Why I Stopped Writing Exception Handlers in Every Spring Boot Service

Ed Legaspi — Sat, 01 Aug 2026 08:50:46 +0000

Why I Stopped Writing Exception Handlers in Every Spring Boot Service

One of my favorite things about Spring Boot is how easy it is to get started.

Until exception handling enters the picture.

At first, it's simple.

You create a @RestControllerAdvice.

Handle a couple of business exceptions.

Return a consistent error response.

Done.

Then the application grows.

Suddenly you're handling validation exceptions.

Authentication failures.

Access denied.

JPA exceptions.

Media type errors.

JSON parsing errors.

Method argument errors.

Missing parameters.

Type mismatches.

And before long, your once-simple exception handler has become one of the biggest classes in the application.

I've been there more than once.

After building enough services, I noticed I wasn't solving business problems anymore—I was rebuilding the exact same exception handling infrastructure in every project.

That became the motivation behind nerv-exception.

The real problem isn't writing exception handlers

Creating an exception handler isn't difficult.

The difficult part is keeping every service consistent.

One service returns:

{
  "message": "User not found"
}

Another returns:

{
  "error": "USER_NOT_FOUND",
  "message": "..."
}

A third includes timestamps.

Another includes trace IDs.

HTTP status codes aren't always consistent.

Some services expose raw exception messages.

Others don't.

Eventually every microservice speaks a different error language.

For consumers, that's frustrating.

For developers, it's even worse.

What I wanted instead

I wanted exception handling to be something applications configured, not something they rebuilt.

The goal wasn't to replace Spring's exception handling.

It was to standardize it.

Applications should be able to share:

a consistent error response
centralized error codes
framework integrations
sensible defaults
the ability to customize when necessary

Without copying hundreds of lines of infrastructure between repositories.

A modular approach

One design decision became clear very early.

Not every Spring application uses the same stack.

Some applications use Spring Security.

Others don't.

Some use Spring Data JPA.

Others use MongoDB.

Some don't even have a database.

So instead of creating one large dependency that tries to handle everything, I chose a modular architecture.

The core library remains lightweight.

Framework integrations live in dedicated modules.

That means applications only add the functionality they actually need.

The latest addition

The newest release expands framework support with built-in integrations for two of the most common Spring technologies.

Spring Security

Authentication and authorization exceptions can now be translated into the same standardized error response used throughout the rest of the application.

No additional exception handlers required.

Spring Data JPA

Persistence-related exceptions are now supported through a dedicated module:

nerv-exception-spring-data-jpa

Keeping JPA support separate means applications that don't use Spring Data JPA don't inherit unnecessary dependencies.

It's a small architectural decision, but one that keeps the library flexible as the ecosystem grows.

The philosophy hasn't changed

Whenever I add support for another framework, I ask the same question.

Should every application have to implement this itself?

If the answer is "probably not," it belongs in the library.

The goal isn't to hide Spring.

The goal is to remove repetitive infrastructure while still feeling like Spring.

What's next?

I'm continuing to expand the Nerv ecosystem one problem at a time.

Not by building another framework...

...but by extracting the infrastructure I've implemented repeatedly across real-world projects into reusable, production-ready libraries.

If you've ever copied an exception handler from one project into another, I'd love to hear how you've approached it.

I'm always interested in seeing the different patterns teams use.

The latest version of nerv-exception is available on Maven Central.

Source code:
https://github.com/czetsuyatech/nerv-exception

Feedback, ideas, and contributions are always welcome.

Stop Rewriting the Same Spring Data JPA Code in Every Project

Ed Legaspi — Sat, 01 Aug 2026 08:40:47 +0000

I Got Tired of Rewriting the Same Spring Data JPA Code

After building enough Spring Boot applications, I noticed something strange.

No matter what the application was—a REST API, an internal tool, or a microservice—the persistence layer always started the same way.

I would create a BaseEntity.

Then an AuditableEntity.

Then a repository base class.

Then pagination models.

Then DTOs.

Then a generic Specification builder.

Then projection utilities.

Every.

Single.

Project.

The business logic changed.

The persistence infrastructure didn't.

Eventually I asked myself:

Why am I rebuilding the same foundation every time I start a new application?

That question became nerv-persistence.

Not another ORM.

Not a replacement for Spring Data JPA.

Just a reusable persistence foundation for applications that have already outgrown copy-pasting infrastructure between repositories.

The moment I realized something was wrong

It happened while creating a new service.

I copied my old BaseEntity.

Then my audit fields.

Then my repository interfaces.

Then my Specification utilities.

Again.

Nothing I was writing was specific to the application.

I was spending the first few hours rebuilding plumbing instead of solving business problems.

I'm guessing many Spring developers have done exactly the same thing.

The hidden cost of "just copy it"

Copying code feels free.

Until six months later.

One project fixes a bug.

Another project improves the Specification builder.

A third project adds projection support.

Now every application has a slightly different version of the same infrastructure.

Maintaining consistency becomes harder than writing the code in the first place.

I wanted a single place where those common building blocks could evolve together.

What ended up inside the library?

Instead of trying to become another framework, I focused on extracting the pieces I kept rebuilding.

The library includes reusable entity foundations like:

BaseEntity
BusinessEntity
AuditableEntity
EnableEntity

It also provides common API models for things like slice pagination and reference data.

On the repository side, I extracted reusable repository implementations, projection support, and slice-based repositories that I found myself implementing over and over.

But the part I'm probably happiest with is the dynamic query support.

If you've worked with Spring Data JPA Specifications, you know how powerful they are...

...and how quickly the surrounding infrastructure grows.

Search criteria.

Builders.

Operators.

Generic specifications.

Projection utilities.

Most applications eventually reinvent this wheel.

So I stopped copying it between projects and turned it into reusable components instead.

One philosophy guided the entire project

Whenever I had to make a design decision, I asked myself one question:

Would I want to maintain this in ten different repositories?

If the answer was "no," it belonged in the library.

That kept the project surprisingly focused.

Rather than replacing Spring Data JPA, it simply removes the repetitive code that tends to accumulate around it.

What's next?

This is only the initial release.

There are still plenty of ideas I'd like to explore as the project grows.

More importantly, I want the library to evolve from real-world usage rather than assumptions.

If there's a persistence problem you've solved repeatedly, I'd genuinely like to hear about it.

Those are usually the best candidates for reusable abstractions.

The project is available on Maven Central, and the source code is on GitHub:

🔗 https://github.com/czetsuyatech/nerv-persistence

If you have feedback, ideas, or feature requests, I'd love to hear them.

After all, this library exists because I got tired of solving the same problem over and over again. I'm sure I'm not the only one.

Stop Rebuilding Exception Handling in Every Spring Boot Project

Ed Legaspi — Mon, 06 Jul 2026 13:20:38 +0000

After working on multiple Spring Boot projects over the years, I noticed an interesting pattern.

Every application had different business requirements.

But almost every application ended up with the same exception handling infrastructure.

I found myself recreating the same classes over and over again:

@RestControllerAdvice
Custom exceptions
Error response models
Error codes
Validation handlers
Feign error decoders
Utility classes
Configuration

The business logic changed.

The exception handling didn't.

Eventually I asked myself:

Why am I rebuilding this for every project?

That question became nerv-exception.

Instead of copying hundreds of lines of exception handling infrastructure from project to project, I wanted a reusable, production-ready library that could become the foundation for every Spring Boot application.

The Problem

A typical Spring Boot application usually starts with something like this.

@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(UserNotFoundException.class)
    public ResponseEntity<ApiError> handle(UserNotFoundException ex) {
        return ResponseEntity.status(HttpStatus.NOT_FOUND)
                .body(new ApiError(
                        "USER_NOT_FOUND",
                        ex.getMessage()));
    }

    @ExceptionHandler(MethodArgumentNotValidException.class)
    public ResponseEntity<ApiError> handle(MethodArgumentNotValidException ex) {
        ...
    }

}

Looks simple.

Until the application grows.

Now you have:

business exceptions
validation exceptions
security exceptions
Feign client exceptions
Kafka consumer exceptions
asynchronous processing
scheduled jobs

Every new integration introduces another place where exceptions need to be translated.

Soon enough, different parts of your application return completely different error payloads.

Your REST API becomes inconsistent.

What I Wanted

I wasn't trying to replace Spring Boot.

Spring Boot already provides excellent exception handling capabilities.

Instead, I wanted a reusable foundation that provides:

consistent API responses
centralized error codes
minimal boilerplate
RFC 7807 Problem Details
Spring Boot auto-configuration
modular architecture

Most importantly...

I wanted something I would actually use in production.

Introducing nerv-exception

The library is intentionally split into multiple modules.

nerv-exception-api
nerv-exception-core
nerv-exception-spring-web
nerv-exception-spring-boot-starter
nerv-exception-spring-feign
nerv-exception-event
nerv-exception-spring-kafka

Each module has a single responsibility.

For example:

api contains only the public contracts.
core contains the domain exception model.
spring-web integrates with Spring MVC.
starter provides auto-configuration.
feign handles remote service errors.
event standardizes exception events.
spring-kafka integrates with Kafka consumers.

Applications only depend on what they actually need.

Adding the Library

Getting started is intentionally simple.

<dependency>
    <groupId>com.czetsuyatech</groupId>
    <artifactId>nerv-exception-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

That's it.

Spring Boot auto-configuration registers everything automatically.

Standardized Error Responses

One of the biggest goals of the project was consistency.

Instead of every controller returning its own custom JSON, APIs can return standardized RFC 7807 Problem Details.

Example:

{
  "type": "https://example.com/problems/user-not-found",
  "title": "User not found",
  "status": 404,
  "detail": "User 123 was not found.",
  "instance": "/users/123"
}

This makes APIs significantly easier to consume.

Frontend developers know exactly what to expect.

API documentation becomes simpler.

Clients become easier to implement.

Designed for Extension

One design decision I made very early was:

Avoid forcing applications into one way of doing things.

The library provides sensible defaults while remaining extensible.

For example:

custom error codes
custom exception mapping
trace context propagation
Feign integration
Kafka integration
event publishing

Applications can override behavior without modifying the library itself.

Runnable Examples

Documentation is helpful.

Runnable code is even better.

That's why I also created a companion repository containing complete Spring Boot applications demonstrating how to use the library in real projects.

Rather than reading isolated snippets, developers can clone the repository, run the application, and explore the implementation themselves.

Repository:

https://github.com/czetsuyatech/nerv-examples

Why Open Source?

Throughout my career I've benefited enormously from open source.

Many libraries have saved me countless hours.

Building nerv-exception is my opportunity to give something back.

If another developer no longer has to rebuild the same exception handling infrastructure for their next project, then this library has already accomplished its goal.

Resources

GitHub

https://github.com/czetsuyatech/nerv-exception

Maven Central

<dependency>
    <groupId>com.czetsuyatech</groupId>
    <artifactId>nerv-exception-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

Example Projects

https://github.com/czetsuyatech/nerv-examples

I'm continuously improving the library based on real-world experience.

If you have ideas, suggestions, or find it useful, I'd love to hear your feedback.

Happy coding! 🚀

Stop Rewriting CI/CD: Reusable GitHub Actions for Maven Projects

Ed Legaspi — Tue, 28 Apr 2026 00:36:48 +0000

If you’ve worked on multiple Java projects, you’ve probably run into this:

Every repository has its own version of a CI/CD pipeline.

Slightly different YAML.

Slightly different setup steps.

Slightly different ways of doing the same thing.

And somehow… none of them are reusable.

😤 The Real Problem

It’s not that CI/CD is hard.

It’s that it’s repetitive.

You end up doing the same things over and over:

Set up Java
Cache Maven dependencies
Run tests
Build artifacts
Handle versioning
Publish or release

Then copy that workflow into another repo…

…and tweak it just enough to break consistency.

This is exactly the kind of problem automation should solve — yet we keep duplicating it.

💡 A Different Approach

Instead of treating each pipeline as a one-off YAML file, I started thinking:

What if CI/CD steps were modular and reusable — just like code?

That’s where NERV-Actions came from.

👉 https://github.com/czetsuyatech/nerv-actions

⚙️ What NERV-Actions Is

NERV-Actions is a collection of reusable GitHub Actions designed for Maven-based projects.

The idea is simple:

Break CI/CD into small, composable actions
Reuse them across repositories
Keep pipelines consistent without rewriting everything

Instead of this:

# repeated in every repo
- uses: actions/setup-java@v4
- run: mvn clean install

You plug in reusable building blocks that already handle those concerns.

🧱 Why This Matters

Reusable workflows aren’t just about convenience — they solve real scaling problems.

When you define things once and reuse them:

You reduce duplication
You avoid inconsistencies across repos
You make updates easier (change once, apply everywhere)
You spend less time debugging YAML

🧠 Design Philosophy

I didn’t try to build a “one-size-fits-all” pipeline.

Instead, NERV-Actions focuses on:

Composable units → mix only what you need
Sensible defaults → minimal setup required
Consistency → same patterns across projects
Extensibility → override when necessary

It’s closer to building blocks than a framework.

🚀 Example Use Case

For a typical Maven project, your pipeline usually boils down to:

Run tests
Build artifacts
Publish packages
Create releases

With reusable actions, you don’t redefine these steps every time—you just assemble them.

🤔 Why Not Just Copy-Paste?

Because copy-paste doesn’t scale.

It works for 2–3 repos.

It breaks down at 10+.

Suddenly:

Fixes don’t propagate
Pipelines drift
Debugging becomes inconsistent

Reusable actions solve that by centralizing the logic.

🧪 Still Evolving

This isn’t meant to be a “perfect CI/CD solution.”

It’s a practical attempt to reduce the friction I kept running into across projects.

If you’re dealing with the same repetition in GitHub Actions, this might be useful:

👉 https://github.com/your-repo-link

💬 Curious About Your Setup

Are you:

copying workflows between repos?
using reusable workflows heavily?
or building your own internal action libraries?

Would be interesting to hear how others are handling this.

I Got Tired of Rewriting Audit Logs in Spring Boot — So I Built nerv-audit

Ed Legaspi — Fri, 17 Apr 2026 07:18:23 +0000

Every backend system eventually hits this moment:

“Who changed this record?”
“What was the previous value?”
“When did it happen?”

Simple questions… until you actually need answers in production.

⚠️ Note: The core module is commercial, but you can explore the public modules and examples here:
👉 https://github.com/czetsuyatech/nerv-audit
👉 https://github.com/czetsuyatech/nerv-examples

The Problem

In most of my Spring Boot projects, I relied on Hibernate Envers.

It works—but in real systems, it starts to hurt:

You repeat the same setup across services
Audit queries are hard to read and maintain
Business-level audit logic gets scattered
Small mistakes become painful in production

After a few projects, I realized:

I wasn’t building features anymore—I was rebuilding audit infrastructure.

What I Actually Wanted

Not a replacement for Envers.

Just something that:

Standardizes audit handling
Reduces boilerplate
Makes queries readable
Works consistently across projects

So I Built nerv-audit

nerv-audit is a lightweight layer on top of Envers that focuses on developer experience.

Instead of wiring everything manually, you get a cleaner way to work with audit data.

Example

Without nerv-audit

You end up dealing with low-level Envers APIs and custom query logic.

With nerv-audit

service.getVerticalAudits(entity, criteria);

That’s it.

Want to see how it's implemented in a real project?
👉 https://github.com/czetsuyatech/nerv-audit
👉 https://github.com/czetsuyatech/nerv-examples

What It Improves

1. Consistent Audit Handling

Define audit behavior once, reuse everywhere.

@AuditedEntity
public class Order {
    private String status;
}

2. Cleaner Queries

No more complex Envers query construction.

You focus on:

What changed
When it changed

Not how to retrieve it.

3. Less Repetition

Across projects, the pattern is always the same.

nerv-audit abstracts that pattern so you don’t rewrite it every time.

Why I Built This

This came from real production work:

Multiple systems
Repeated audit requirements
Real incidents where audit data mattered

At some point, it made more sense to abstract the solution than keep rebuilding it.

About the Model (Transparency)

nerv-audit is not fully open-source.

There is a free tier you can use
Some advanced capabilities are part of a paid core

I chose this approach to:

Keep the project sustainable
Continue improving it over time
Focus on real-world use cases

Who This Is For

This might be useful if you:

Build Spring Boot applications
Use Hibernate Envers (or plan to)
Want a cleaner way to handle audit logs

I’d Like Your Input

How are you handling audit logs today?

Pure Envers?
Custom implementation?
Something else?

I’m especially interested in:

Pain points
Query challenges
Scaling issues

If you're working with Envers, try it out and let me know what breaks—or what works better.

🔗 Resources

Hands-On Coding: Exploring Hyperparameters for Programmers

Ed Legaspi — Thu, 07 Mar 2024 04:54:31 +0000

Introduction

In this article, we will explore different techniques for finding the optimal hyperparameter values from a given set of parameters in a grid. Particularly we will look at RandomizedSearchCV, GridSearchCV, and BayesSearchCV.

In this blog you will learn:

How to initialize the parameter grid.
How to find the optimal hyperparameters based on a given technique.
How to build a model (XGBClassifier) to use the hyperparameters.
How to score the performance of the model.

RandomizedSearchCV

param_grid = {
    "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
    "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
    "max_depth": [1, 2, 3, 4, 5, 6, 8, 12],
    "n_estimators": [25, 50, 65, 80, 100, 115, 200]
}

grid_search = RandomizedSearchCV(estimator=classifier_0, param_distributions=param_grid, scoring=scoring)

GridSearchCV

param_grid = {
    "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
    "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
    "max_depth": [2, 3, 4, 5, 6, 8, 12],
    "n_estimators": [25, 50, 65, 80, 100, 115, 200]
}

grid_search = GridSearchCV(estimator=classifier_0, param_grid=param_grid, scoring=scoring)

BayesSearchCV

param_bayes = {
    'gamma': Categorical(param_grid['gamma']),
    'learning_rate': Categorical(param_grid['learning_rate']),
    'max_depth': Categorical(param_grid['max_depth']),
    'n_estimators': Categorical(param_grid['n_estimators'])
}

grid_search = BayesSearchCV(estimator=classifier_0, search_spaces=param_bayes, scoring=scoring, n_jobs=-1, cv=10)

Finding the Best HyperParameters

best_model = grid_search.fit(X_train, y_train)
hyperparams = best_model.best_params_

Building and Scoring the Classifier using the HyperParameters

# Fitting the Model
ne = hyperparams['n_estimators']
lr = hyperparams['learning_rate']
md = hyperparams['max_depth']
gm = hyperparams['gamma']
print("Recommended Params >>", f"ne: {ne},", f"lr: {lr}", f"md: {md}", f"gm: {gm}")

# Build Classification Model
classifier_1 = XGBClassifier(
    base_score=0.5,
    colsample_bylevel=1,
    colsample_bynode=1,
    objective=objective,
    booster="gbtree",
    eval_metric=eval_metric_list,
    n_estimators=ne,
    learning_rate=lr,
    max_depth=md,
    gamma=gm,
    subsample=0.8,
    colsample_bytree=1,
    random_state=1
)

# Fit Model
eval_set = [(X_train, y_train)]
classifier_1.fit(
    X_train,
    y_train,
    eval_set=eval_set,
    verbose=False
)

# Get predictions for training data
train_yhat = classifier_1.predict(X_train)
print("Training Preds: \n", train_yhat[:5])

# Set K-Fold Cross Validation Levels
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)

# Training Results
train_results = cross_val_score(classifier_1, X_train, y_train, scoring=scoring, cv=cv, n_jobs=1)

# Brief Review of Training Results
print("Average Accuracy K-Fold: ", round(train_results.mean(), 2))
print("Std Deviation K-Fold: ", round(train_results.std(), 2))
print("Precision Score 0: ", round(precision_score(y_train, train_yhat, average=None)[0], 3))
print("Precision Score 1: ", round(precision_score(y_train, train_yhat, average=None)[1], 3))

Performance

Machine: Laptop
Processor: AMD Ryzen 7
OS: Windows
DataFrame Shape: (7282, 17)

Understanding How Scope Affects Values in Your Spring REST Controller

Ed Legaspi — Sat, 02 Mar 2024 03:33:59 +0000

Below we explore how a scope annotation affects an instance value in a Spring REST controller.

Each controller is annotated with scope.

@RestController
@Scope([SCOPE_VALUE])
public class XXXScopeController {}

How to Convert Vertically Stored Asset Data into Columnar Format for Cointegration Analysis

Ed Legaspi — Sat, 02 Mar 2024 01:12:16 +0000

Introduction

This piece of code fetches asset information from a table stored vertically.

Dependencies

Install the following package.

conda install pandas
conda install numpy as np
conda install mysql-connector-python
conda install sqlalchemy
conda install pymysql

Hands-on Coding

Connect to the database

def query_df(query):
    try: 
        engine_uri = f"mysql+pymysql://db_user:db_pass_123@localhost:3306/tradewise_pse"
        db_conn = create_engine(engine_uri)        
        df_result = pd.read_sql(query, db_conn)    
        return df_result

    except Exception as e:    
        print(str(e))

Fetching the Dataset

if not load_existing:
    sql_distinct_tickers = "select ticker from candlestick where event_time='2023-12-29' and ticker not like '^%%'"
    df_tickers = query_df(sql_distinct_tickers)

    df = pd.DataFrame(index=['event_time'])

    ### Get the candlesticks
    for ticker in df_tickers['ticker']:
        sql_ticker_col = "select event_time, close from candlestick where ticker='{0}'"
        df_temp = query_df(sql_ticker_col.format(ticker))
        df_temp.set_index('event_time', inplace=True)
        df_temp.rename(columns={'close': ticker}, inplace=True)        
        df = df.add(df_temp, fill_value=0)

    df.to_csv(file_name)

Load the dataset from file

df = pd.read_csv(file_name, index_col=0)
df.drop(index=df.index[-1],axis=0, inplace=True)

Drop NA

df.dropna(axis=1, inplace=True)

Print the Dataset

print(f"Shape: {df.shape}")
print(f"Null values: {df.isnull().values.any()}")
df

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ypd0yfpe8ihrfvx58cl.png)

Learn to Incorporate Rolling Hurst Values into Your DataFrame

Ed Legaspi — Sat, 02 Mar 2024 01:08:30 +0000

The hurst function.

def hurst(ts, min_lag=1, max_lag=7):
    lags = range(min_lag, max_lag)
    tau = [np.sqrt(np.std(np.subtract(ts[lag:], ts[:-lag]))) for lag in lags]
    poly = np.polyfit(np.log(lags), np.log(tau), 1)
    return poly[0]*2.0

Adding to our DataFrame

The hurst value is computed with the last 14 close values.

df['Hurst'] = df['close'].rolling(14).apply(hurst, raw=True)
df[10:20]