DEV Community: Odumosu Matthew

A practical guide to race conditions and five ways engineers make sure it's never both

Odumosu Matthew — Sat, 14 Nov 2026 14:26:33 +0000

Two Travellers, One Hotel Room, the Same Second: Who Gets It?

Sofia is in Madrid. James is in Toronto.

Both are planning the same weekend in Lisbon. Both find the same small hotel by the river. Both see the same message: "Only 1 room left!"

At exactly the same second, both tap "Book now."

Who gets the room?

In a badly built system, both do. Two confirmation emails. Two happy travellers. One room. And one very awkward conversation at the front desk.

This is a race condition, and it's one of the most common and most dangerous bugs in software. It hides in the few milliseconds nobody tests, and it shows up in production when real people do things at the same time.

In payments, I deal with the same problem, just with money instead of rooms: two withdrawals hitting the same balance at the same moment. Get it wrong, and someone spends money that isn't there.

Let's break down why it happens and how to prevent it.

What is a race condition?

A race condition happens when two operations run at the same time, and each makes a decision based on information that is about to become wrong.

Here's the hotel booking, step by step:

Time   Sofia's request                James's request
----   ---------------                ---------------
t1     READ: rooms_left = 1
t2                                    READ: rooms_left = 1
t3     1 > 0, so book it
t4                                    1 > 0, so book it
t5     WRITE: rooms_left = 0
t6                                    WRITE: rooms_left = 0
       Result: two bookings, one room

Each check was correct when it was made. The problem is the gap between checking and acting. Engineers call this the check-then-act bug, and the gap is the race window.

The naive code looks perfectly reasonable:

var room = await _db.Rooms.FindAsync(roomId);

if (room.RoomsLeft > 0)
{
    room.RoomsLeft -= 1;
    _db.Bookings.Add(new Booking(roomId, guestId, night));
    await _db.SaveChangesAsync();
}

It passes every test written by one developer clicking one button. It fails the first busy Friday night.

Fix 1: Make check-and-act a single atomic step

The simplest and often best fix is to let the database do the check and the update in one statement:

UPDATE rooms
SET rooms_left = rooms_left - 1
WHERE id = 42
  AND rooms_left > 0;

If two requests run this at the same time, the database serialises them. The first changes one row. The second changes zero rows, because rooms_left is no longer greater than zero.

Your code checks the affected row count:

var updated = await _db.Database.ExecuteSqlInterpolatedAsync(
    $"UPDATE Rooms SET RoomsLeft = RoomsLeft - 1 WHERE Id = {roomId} AND RoomsLeft > 0");

if (updated == 0)
    return BookingResult.SoldOut();

This is exactly how a safe balance debit works in a payment system:

UPDATE accounts
SET balance = balance - @amount
WHERE id = @accountId
  AND balance >= @amount;
-- 0 rows updated = insufficient funds, even under concurrency

Fix 2: Pessimistic locking

Pessimistic locking assumes conflicts are likely, so it locks the record before working on it. The second request waits until the first is finished, then sees the updated value.

In PostgreSQL or MySQL:

BEGIN;
SELECT rooms_left FROM rooms WHERE id = 42 FOR UPDATE;
-- row is now locked until COMMIT
UPDATE rooms SET rooms_left = rooms_left - 1 WHERE id = 42;
INSERT INTO bookings (...) VALUES (...);
COMMIT;

In SQL Server, the equivalent hint is WITH (UPDLOCK, ROWLOCK).

In Spring Data JPA:

@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("select r from Room r where r.id = :id")
Optional<Room> findByIdForUpdate(@Param("id") Long id);

Trade-off: it's simple and safe, but locks make other requests wait. Hold them only for the shortest possible time, never across a slow external call like a payment provider.

Fix 3: Optimistic concurrency

Optimistic concurrency assumes conflicts are rare. Instead of locking, every record carries a version number. An update only succeeds if the version hasn't changed since you read it.

UPDATE rooms
SET rooms_left = 0, version = 4
WHERE id = 42 AND version = 3;
-- 0 rows updated = someone else changed it first

In EF Core, mark a concurrency token:

public class Room
{
    public int Id { get; set; }
    public int RoomsLeft { get; set; }

    [Timestamp]
    public byte[] RowVersion { get; set; } = default!;
}

try
{
    await _db.SaveChangesAsync();
}
catch (DbUpdateConcurrencyException)
{
    // Someone else booked first: re-read, then retry or tell the user.
}

In JPA:

@Entity
public class Room {
    @Id private Long id;
    private int roomsLeft;

    @Version
    private long version;
}
// A conflicting update throws OptimisticLockException.

Trade-off: no waiting and great performance when conflicts are rare, but under heavy contention many requests fail and retry.

Fix 4: Let the database be the final referee

Even with careful application code, bugs happen. A unique constraint is your last line of defence:

ALTER TABLE bookings
ADD CONSTRAINT uq_room_night UNIQUE (room_id, night);

If every other layer fails, the database simply refuses the second booking for the same room and night. In payments, the equivalent is a unique constraint on a transaction reference or idempotency key, so the same payment can never be recorded twice.

Constraints don't replace good logic. They catch what your logic misses.

Fix 5: Hold, then confirm

Many booking sites say: "We're holding this room for you for 10 minutes."

That's a reservation with a time-to-live (TTL). The room is set aside the moment you start checking out, so you never enter card details for a room that's already gone. If you don't complete payment in time, the hold expires and the room is released.

INSERT INTO holds (room_id, night, guest_id, expires_at)
VALUES (42, '2026-11-14', 7781, NOW() + INTERVAL '10 minutes');
-- with a unique constraint on (room_id, night) for active holds

A background job, or the next booking attempt, clears expired holds. The same pattern is used in payments for authorisation holds: funds are reserved first, then captured or released.

Which fix should you use?

Atomic conditional update: your default for counters, balances and stock levels.
Pessimistic locking: when you must read, decide and write several things together, and conflicts are common.
Optimistic concurrency: when conflicts are rare and you want to avoid waiting.
Unique constraints: always, as a safety net.
Reservation holds: when there's a gap between choosing and paying.

Most production systems combine several. A booking might use a hold during checkout, an atomic update to confirm, and a unique constraint as the final guard.

How to catch race conditions before production

Race conditions rarely appear when one developer tests one request. You have to create the race on purpose:

Write integration tests that fire many identical requests in parallel and assert that only one succeeds.
Load-test busy endpoints like checkout, transfers and withdrawals.
In code reviews, look for any read, then decide, then write sequence on shared data and ask: "What if two of these run at once?"

[Fact]
public async Task Only_one_booking_succeeds_for_the_last_room()
{
    var tasks = Enumerable.Range(0, 20)
        .Select(i => _client.PostAsJsonAsync("/bookings", new { roomId = 42, guestId = i }));

    var responses = await Task.WhenAll(tasks);

    Assert.Equal(1, responses.Count(r => r.StatusCode == HttpStatusCode.Created));
}

What James sees

So, if Sofia wins, what does James see?

Not a crash. Not a charge for a room he didn't get. Just an honest message:

"Sorry, this room was just booked. Here are similar rooms nearby."

That's what good engineering feels like from the outside: calm, clear and fair, even when two people reach for the same thing at once.

Key takeaways

A race condition happens in the gap between checking and acting.
Close the gap with atomic conditional updates wherever possible.
Use pessimistic locks when conflicts are frequent, and keep them short.
Use optimistic concurrency when conflicts are rare.
Add unique constraints as the final referee.
Use reservation holds when there's a delay between choosing and paying.
Test concurrency on purpose, because it won't show up by accident.

Race conditions hide in the milliseconds nobody tests. Good engineering decides, calmly and in advance, who wins.

Matthew Odumosu is a Senior Software Engineer with 8+ years of experience building payment, banking and insurance systems. He writes about engineering principles in plain language, for engineers at every level.

Connect on LinkedIn: linkedin.com/in/matthew-odumosu

Tags: Software Engineering, Concurrency, Databases, System Design, Backend Development

Money In Freely, Money Out Carefully: The 24-Hour Rule That Stops Fraudsters

Odumosu Matthew — Sat, 14 Nov 2026 14:24:24 +0000

How payment systems protect people from fraud without turning customers away

Tolu opened a new bank account on Monday morning.

By Monday afternoon, a client paid her for a big job. The alert came in. Joy.

Then she tried to pay her supplier.

"Limit exceeded."

The money was right there in her account. Nothing was wrong with it. So why couldn't she move it?

To Tolu, it felt like the bank was getting in her way. But what she couldn't see was that the same rule stopping her that afternoon was protecting someone else's life savings.

This article explains that rule, why it exists, and the engineering decisions behind enforcing it well.

The fraudster's playbook

Some of the most valuable tools in a fraudster's kit are brand new bank accounts.

The pattern looks like this:

A victim is tricked into sending money, or their account is compromised.
The money is sent to a freshly opened account, often one opened only hours earlier.
Within minutes, the money is moved on to a second account, then a third.
By the time the victim reports it, the money has passed through several accounts and is gone.

These pass-through accounts are often called mule accounts, and moving money quickly through several of them is called layering. Speed is the fraudster's greatest advantage. The faster money can leave a new account, the harder it is to recover.

So a new account is, in effect, the getaway car.

The rule: let money in, slow money out

The protection is simple in principle: for a short period after an account is opened, and especially before the customer has completed fuller identity verification, limit how much money can leave the account.

Notice what this rule does not do. It does not reject money coming in.

That distinction matters more than it first appears.

If the system rejected inflows:

Tolu's client gets an error and assumes something is wrong with her account.
Tolu loses a payment, and possibly a client.
The bank or fintech loses a customer on her first day.

If the system limits outflows:

Tolu gets paid. Her money is safe in her account.
She can still spend within a reasonable limit straight away.
Larger amounts can move once the window passes or she upgrades her account.
A fraudster who receives stolen money can't run with it.

The customer still gets paid. The business keeps the customer. And stolen money has nowhere to go.

This connects to tiered KYC (Know Your Customer), used in many markets including Nigeria: the more a customer verifies their identity, the more freedom their account has. A brand new, lightly verified account starts with tighter limits, and those limits widen as trust is established.

The engineering behind it

Enforcing a rule like this sounds easy: "if the account is new and the amount is above the limit, block it." In production, there's more to get right. I've worked on enforcing this kind of rule in payment infrastructure, and these are the principles that matter.

1. Enforce it on the right path

The check belongs on the outflow path: transfers out, withdrawals, card payments. Inflows should still be credited normally.

Putting the check in the wrong place either blocks legitimate customers or leaves a gap fraudsters will find. Every way money can leave an account (app transfers, USSD, cards, POS, agent withdrawals) needs to go through the same check, ideally in one shared component rather than copied into each channel.

2. Measure a rolling window, not a single transaction

A limit on a single transfer is easy to get around: just send several smaller ones. The rule needs to look at the total outflow over the window.

public async Task<LimitCheckResult> CheckOutflowAsync(
    Account account, decimal amount, CancellationToken ct)
{
    var rules = _limits.CurrentValue;              // from configuration
    var accountAge = _clock.UtcNow - account.OpenedAtUtc;

    if (accountAge >= rules.NewAccountWindow || account.KycTier >= rules.ExemptFromTier)
        return LimitCheckResult.Allowed();

    var windowStart = _clock.UtcNow - rules.NewAccountWindow;

    var alreadySent = await _db.Transactions
        .Where(t => t.AccountId == account.Id
                 && t.Direction == Direction.Debit
                 && t.Status != TransactionStatus.Failed
                 && t.CreatedAtUtc >= windowStart)
        .SumAsync(t => t.Amount, ct);

    var remaining = rules.NewAccountOutflowLimit - alreadySent;

    return amount <= remaining
        ? LimitCheckResult.Allowed()
        : LimitCheckResult.Blocked(remaining, account.OpenedAtUtc + rules.NewAccountWindow);
}

3. Keep the rule outside the code

Limits change. Regulators adjust them. Risk teams tune them after studying fraud patterns. If the limit is hardcoded, every change means a code release.

So the values live in configuration, ideally a dynamic configuration service that can be updated without redeploying or restarting anything.

public class NewAccountLimitOptions
{
    public TimeSpan NewAccountWindow { get; set; } = TimeSpan.FromHours(24);
    public decimal NewAccountOutflowLimit { get; set; }
    public int ExemptFromTier { get; set; }
}

// Program.cs
builder.Services.Configure<NewAccountLimitOptions>(
    builder.Configuration.GetSection("Limits:NewAccount"));

// Inject IOptionsMonitor<NewAccountLimitOptions> so updated values
// are picked up without a restart.

During a fraud wave, being able to tighten a limit in minutes rather than days can make a real difference.

4. Beware of two withdrawals at the same moment

Here's the subtle bug. Two outgoing transfers arrive at almost exactly the same time. Both check the limit. Both see enough headroom. Both go through. Together, they exceed the limit.

That's a race condition, and it's exactly the gap a scripted fraud attempt will try to exploit. The fix is to make the check and the update a single atomic step: lock the account row while checking and debiting, or track the window's total in a counter that's updated with a conditional update. (I cover race conditions in depth in a separate article.)

5. Say "not yet" kindly and clearly

A rule that protects people can still frustrate them if it's communicated badly.

"Transaction failed" tells Tolu nothing. She'll assume the bank is broken and call customer care.

Compare that with:

"For your security, new accounts can send up to [limit] in the first 24 hours. You'll be able to send more from [time], or straight away if you upgrade your account."

Same rule. Completely different experience. The second message turns a frustration into a reassurance.

6. Leave an audit trail

Every block should be logged with the account, the amount, the remaining limit and the reason. Risk teams use this to tune the rules, support teams use it to answer customers, and compliance teams use it to show regulators the rule is being enforced.

The balance every payment system has to strike

Every extra security step protects customers. Every extra security step also risks losing them.

Good engineering in fintech lives in that tension. Block too much, and legitimate customers leave. Block too little, and fraudsters move in.

The "money in freely, money out carefully" approach is a good example of finding the balance: maximum protection with minimum disruption to honest customers.

Key takeaways

New accounts are a common tool for moving stolen money quickly.
Limiting outflows, not inflows, protects victims without losing customers.
Measure the rolling total, not individual transactions.
Keep limits in configuration so they can change without a release.
Guard against race conditions on the limit check.
Explain blocks clearly, with when and how the limit lifts.
Log every decision for audit.

To Tolu, that limit felt like an obstacle. To someone else, the same rule was the reason their stolen savings were still there when they called the bank.

Good engineering doesn't always say no. Sometimes it just says "not yet."

Connect on LinkedIn: linkedin.com/in/matthew-odumosu

Tags: Fintech, Fraud Prevention, Software Engineering, Payments, System Design

Message queues and asynchronous processing, explained through a trip to the dry cleaner

Odumosu Matthew — Mon, 09 Nov 2026 10:41:49 +0000

Why Your Bank App Says "Successful" Before the SMS Arrives

When you drop your clothes at the dry cleaner, nobody asks you to wait at the counter while they wash, dry and iron everything.

They hand you a ticket and say, "We'll call you when it's ready."

You go about your day. The work still happens. It just doesn't happen while you stand there waiting.

That simple idea is behind one of the most important patterns in modern software. It's the reason your banking app can say "Transfer successful" in about a second, while the SMS alert, the receipt and the rewards points arrive a few moments later.

Engineers call it asynchronous processing, and the tool that makes it work is the message queue.

In this article, I'll explain what a message queue is, why payment systems depend on it, and the rules experienced engineers follow so that nothing gets lost in between.

What must happen now, and what can happen next

When you tap "Send" on a transfer, a lot happens. But not all of it needs to happen before you see a result.

Some things must happen immediately:

Check that you have enough money
Debit your account
Record the transaction in the ledger
Confirm the result to you

Other things can safely happen a moment later:

Send the SMS or push notification
Generate the receipt
Update rewards or loyalty points
Feed fraud analytics and reporting

If a payment service did all of this in one long chain, every transfer would be as slow as its slowest step. And if the SMS provider had a bad day, your transfer would fail even though the money part worked perfectly.

So instead, the payment service does the critical work, then drops a message into a queue that says, in effect, "A payment just completed. Whoever needs to know, here are the details." Then it moves on.

Other services pick up that message and do their part in the background.

How a message queue works

There are three main roles:

Producer: the service that publishes a message. Here, the payment service.
Broker: the system that stores messages safely until they're processed. Apache Kafka and RabbitMQ are the most common.
Consumer: a service that reads messages and acts on them. The SMS service, the receipt service, the rewards service.

[Payment Service] --publishes--> [ payment-completed queue ] --> [SMS Service]
   (must happen NOW)                                         --> [Receipt Service]
                                                             --> [Rewards Service]
                                                             --> [Fraud Analytics]
                                                                (can happen NEXT)

The beauty of this design is that the producer doesn't need to know who is listening. Tomorrow, the team can add a new consumer, say a monthly spending insights service, without touching the payment service at all.

That's called loose coupling, and it's one of the reasons large systems can keep growing without collapsing under their own weight.

The rules experienced engineers follow

Queues make systems faster and more resilient. They also introduce new ways for things to go wrong. Here are the rules I hold teams to when building this way.

1. Make every consumer idempotent

Most message brokers guarantee that a message will be delivered at least once. Not exactly once. At least once.

That means the same message can arrive twice: after a network hiccup, a consumer restart, or a retry. If your consumer isn't prepared for that, one payment can turn into two SMS alerts. Annoying. Or one credit can turn into two credits. Expensive.

An operation is idempotent if doing it once or ten times gives the same result. The common way to achieve this is to record every message ID you've processed, protected by a unique constraint:

@KafkaListener(topics = "payment-completed", groupId = "sms-service")
public void onPaymentCompleted(PaymentCompletedEvent event) {
    // Insert the event ID into a table with a UNIQUE constraint.
    // Returns false if this event was already processed.
    boolean firstTime = processedEvents.markProcessed(event.eventId());

    if (!firstTime) {
        log.info("Duplicate event {} ignored", event.eventId());
        return;
    }

    smsService.sendPaymentAlert(event.customerPhone(), event.amount());
}

For anything that moves money, this isn't optional. One credit must never become two.

2. Retry with backoff, then park it in a dead letter queue

Sometimes a message fails because of a temporary problem: the SMS provider is slow, the database is busy. Retrying makes sense.

But retrying instantly and forever just hammers a system that's already struggling. So we retry with exponential backoff: wait one second, then two, then four, then eight.

If it still fails after a few attempts, we don't let it block every message behind it. We move it to a dead letter queue (DLQ), where it can be inspected and replayed later.

In Spring Kafka, that looks like this:

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);

    var backOff = new ExponentialBackOffWithMaxRetries(4);
    backOff.setInitialInterval(1_000L);   // 1 second
    backOff.setMultiplier(2.0);           // 1s, 2s, 4s, 8s
    backOff.setMaxInterval(10_000L);

    return new DefaultErrorHandler(recoverer, backOff);
}

Failed messages land on a topic like payment-completed.DLT, and nothing behind them gets stuck.

3. Choose your keys carefully when order matters

Kafka splits a topic into partitions so it can process messages in parallel. Order is only guaranteed within a partition.

If events for the same account must be processed in order (a debit before a reversal, for example), give them the same key. Kafka sends every message with the same key to the same partition.

kafkaTemplate.send("account-events", event.accountNumber(), event);
//                                    ^ key: same account, same partition, same order

Pick the wrong key, or no key, and two events for the same account can be processed out of order.

4. Watch the backlog

The danger with background work is that it can fall behind quietly. Customers won't complain about a slow queue. They'll complain that their alert never came.

So monitor consumer lag: the gap between the latest message published and the latest message processed. If lag keeps growing, consumers can't keep up, and you want to know before your customers do.

5. Don't lose the message between the database and the queue

Here's a subtle trap. The payment service writes to the ledger, then publishes a message. What if the database write succeeds and the publish fails? The money moved, but no SMS, no receipt, no fraud check.

The transactional outbox pattern solves this. Instead of publishing directly, the service writes the ledger entry and an "outbox" row in the same database transaction. A separate relay reads the outbox and publishes to the queue.

BEGIN;
  INSERT INTO ledger_entries (...) VALUES (...);
  INSERT INTO outbox (event_id, topic, payload) VALUES (...);
COMMIT;
-- A relay process publishes outbox rows to Kafka, then marks them sent.

Either both are saved, or neither is. Nothing gets lost in between.

When not to use a queue

Asynchronous processing isn't free. It adds moving parts, makes debugging harder, and means data is briefly out of sync.

Don't make something asynchronous if the user genuinely needs the result before continuing. Checking the balance and debiting the account must happen synchronously. A customer can wait a few seconds for an SMS. They can't wait a few seconds to find out whether their money moved.

The skill is in drawing that line well.

What the customer feels

Customers never see a queue. What they feel is:

"Transfer successful" in about a second
The alert and receipt arriving shortly after
An app that keeps working even when one part of the system is having a bad day

And when it's done well, nothing gets lost along the way.

Key takeaways

Split work into what must happen now and what can happen next.
Message queues let services communicate without waiting on each other.
Delivery is at least once, so consumers must be idempotent.
Retry with exponential backoff, then move failures to a dead letter queue.
Use message keys to keep related events in order.
Monitor consumer lag.
Use the outbox pattern so a database write and a published event can't drift apart.

Good engineering isn't doing everything at once. It's knowing what must happen now, what can happen next, and making sure nothing gets lost in between.

Just like a good dry cleaner never loses the ticket.

Connect on LinkedIn: linkedin.com/in/matthew-odumosu

Tags: Software Engineering, System Design, Kafka, Microservices, Fintech

API Versioning: How to Evolve Your API Without Breaking Existing Clients

Odumosu Matthew — Thu, 29 Oct 2026 11:29:13 +0000

𝐃𝐚𝐧𝐢𝐞𝐥 𝐥𝐚𝐧𝐝𝐬 𝐢𝐧 𝐑𝐨𝐦𝐞 𝐟𝐨𝐫 𝐚 𝐜𝐨𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞. 𝐏𝐡𝐨𝐧𝐞 𝐚𝐭 4%.

He reaches the hotel, pulls out his charger, and stops.

The plug doesn’t fit the socket.

Nothing is broken. The phone works. The socket works. They just no longer speak the same shape

Software has the same problem every day.

A team improves their API. They rename a field, change a format, remove something “nobody uses.” It works perfectly on their side.

Then thousands of people who haven’t updated their app open it on Monday morning, and it stops working. Payments fail. Screens go blank. Support lines fill up.

The new socket was better. It just didn’t fit the plugs already out in the world.

Good engineers know that once an API is public, it becomes a promise. You can improve it, but you can’t quietly break the people already relying on it.

So instead of forcing everyone to change overnight, we give them an adapter: the old version keeps working while the new one is introduced, and nobody is left standing in a hotel room with a dying phone.

What’s the most painful breaking change you’ve had to recover from?

Backpressure in System Design: How to Handle Traffic Spikes Without Crashing

Odumosu Matthew — Thu, 29 Oct 2026 11:25:41 +0000

𝐈𝐭’𝐬 𝐭𝐡𝐞 𝐟𝐢𝐫𝐬𝐭 𝐌𝐨𝐧𝐝𝐚𝐲 𝐨𝐟 𝐭𝐡𝐞 𝐡𝐨𝐥𝐢𝐝𝐚𝐲𝐬. 𝐄𝐯𝐞𝐫𝐲 𝐟𝐥𝐢𝐠𝐡𝐭 𝐥𝐞𝐚𝐯𝐞𝐬 𝐛𝐞𝐟𝐨𝐫𝐞 9𝐚𝐦.

Thousands of passengers reach airport security at once.

Imagine the airport let everyone through the entrance together. People crowd the scanners. Bags pile up. Staff can’t move. Nobody gets through, and everyone misses their flight.

Real airports don’t do that. They control the flow at the door: a rope queue, a gate that opens in batches, a fast-track lane for anyone about to miss a flight. The scanners keep a steady pace, and the queue absorbs the rush.

Software faces the same rush on payday, during a flash sale, or when something goes viral. A system that accepts everything eventually serves no one.

Engineers call the fix 𝐛𝐚𝐜𝐤𝐩𝐫𝐞𝐬𝐬𝐮𝐫𝐞. I sketched how it works, from the API gateway down to the database.

What’s the biggest traffic spike your system has had to survive?

Race Conditions in Production: When Two Requests Compete for One Resource

Odumosu Matthew — Thu, 29 Oct 2026 11:10:46 +0000

𝐒𝐨𝐟𝐢𝐚 𝐢𝐬 𝐢𝐧 𝐌𝐚𝐝𝐫𝐢𝐝. 𝐉𝐚𝐦𝐞𝐬 𝐢𝐬 𝐢𝐧 𝐓𝐨𝐫𝐨𝐧𝐭𝐨.

Both find the same small hotel in Lisbon for the same weekend. Both see “Only 1 room left!”

At exactly the same second, both tap “Book now.”

Who gets the room?

In a badly built system, both do. Two confirmations. One room. One very awkward morning at the front desk.

Engineers call this a race condition: two operations each make a decision on information that’s about to become wrong.

Exactly the same scenario plays out in payments, just with money instead of rooms. Two withdrawals, one balance, same moment.

Which of the five fixes have you used in production?

THE 24 - HOUR ACCOUNT RULE IN TECH

Odumosu Matthew — Thu, 29 Oct 2026 10:52:19 +0000

𝐓𝐨𝐥𝐮 𝐨𝐩𝐞𝐧𝐞𝐝 𝐚 𝐧𝐞𝐰 𝐚𝐜𝐜𝐨𝐮𝐧𝐭 𝐨𝐧 𝐌𝐨𝐧𝐝𝐚𝐲 𝐦𝐨𝐫𝐧𝐢𝐧𝐠.

By afternoon, a client paid her for a big job. The alert came in. Joy.

Then she tried to pay her supplier. “Limit exceeded.”

The money was right there. So why couldn’t she move it?

Because brand new accounts are a fraudster’s favourite getaway car: receive stolen money within hours, move it on before the victim even reports it.

So for the first 24 hours, the rule is simple. Money can come in. Large amounts can’t leave yet.

The customer still gets paid. The business doesn’t lose a customer. And stolen money has nowhere to run.

Good engineering doesn’t always say no. Sometimes it just says “not yet.”

Why Records Are the Better Choice for Clean, Modern, and Principle-Driven .NET Development

Odumosu Matthew — Sat, 18 Jul 2026 10:48:20 +0000

If you've been writing C# for more than a few years, you've written thousands of DTOs. Request models, response models, event payloads, integration contracts — the connective tissue of every distributed system. And if you're like most of us, you've written nearly all of them as classes with { get; set; } properties, because that's what we've done since .NET Framework 2.0.

Here's the uncomfortable truth: a mutable class is the wrong data structure for a DTO, and it has been since C# 9 gave us a better one. Not "stylistically wrong." Wrong in ways that show up as production incidents — duplicate payments, corrupted audit trails, and race conditions in async pipelines.

This article makes the case from first principles, with examples drawn from banking and fintech systems — payment processing, reconciliation, maker-checker workflows — because that's where the cost of getting this wrong is measured in money, not just bugs. Whether you're a junior engineer writing your first API contract or a solutions architect reviewing a payment platform design, the goal is the same: understand why records exist, not just how to type them.

🚚 What Are DTOs in Practice?

Imagine you're building a ride-sharing app like Uber or Bolt. When the frontend requests a list of nearby drivers, you return a DriverDto:

{
  "id": 142,
  "name": "Linda",
  "rating": 4.9
}

In C#, you might write:

public class DriverDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public double Rating { get; set; }
}

This works — but it's bloated, mutable, and doesn't reflect the true intent of a DTO: a simple, structured data carrier.

Now zoom out from ride-sharing to where DTOs really earn their keep: system boundaries in enterprise software. In a payments platform, DTOs are everywhere:

The inbound webhook payload from a card scheme or a partner bank notifying you of a settlement.
The API response your mobile app receives when a customer checks their balance.
The Kafka event published when a transfer moves from PENDING to SETTLED.
The reconciliation record you compare against the processor's end-of-day file.

Every one of these is a contract. It represents a fact — "this transaction, with this reference, for this amount, at this time" — communicated across a boundary. And here is the key insight the whole article rests on:

A DTO is a snapshot of data at a point in time. Snapshots don't change. So why are we modeling them with a type whose defining feature is that it changes?

A junior engineer can think of it this way: a DTO is like a printed bank statement. Once it's printed, nobody should be able to grab a pen and change the numbers. If the numbers need to be different, you print a new statement. Classes with set; accessors hand everyone a pen. Records take the pen away.

🔍 The Real Problems with Class-Based DTOs

Let's be precise about what's wrong, because "bloated" alone wouldn't justify changing a decade of habit. There are four concrete failures.

1. Mutability is a liability at boundaries

Consider a simplified inbound payment notification in a fintech platform:

public class PaymentNotificationDto
{
    public string TransactionRef { get; set; }
    public decimal Amount { get; set; }
    public string Currency { get; set; }
    public string SenderAccount { get; set; }
    public DateTime ReceivedAt { get; set; }
}

This object will pass through validation, enrichment, fraud screening, ledger posting, and notification services — often across async hops and sometimes across threads. Any one of those stages can silently mutate it. A well-meaning teammate writes:

// "Normalizing" inside the fraud check — seems harmless
notification.Amount = Math.Round(notification.Amount, 2);

Now the amount that reaches your ledger is not the amount you received from the partner bank. When reconciliation runs at end of day and finds a ₦0.005 discrepancy across two million transactions, you'll spend a very unpleasant week finding this line. The compiler could have prevented it. With a mutable class, it can't.

In banking, this isn't hypothetical. The single most valuable property of financial data in flight is that it cannot be tampered with between the boundary and the ledger. Mutability is an open invitation to tampering — usually accidental, occasionally worse.

2. Reference equality is the wrong equality

Two DTOs carrying identical data are, for every practical purpose, the same fact. But classes compare by reference:

var a = new PaymentNotificationDto { TransactionRef = "TXN-8842", Amount = 50_000m, Currency = "NGN" };
var b = new PaymentNotificationDto { TransactionRef = "TXN-8842", Amount = 50_000m, Currency = "NGN" };

Console.WriteLine(a == b);        // false
Console.WriteLine(a.Equals(b));   // false

Why does this matter? Idempotency — the backbone of every payment system. Partner banks retry webhooks. Kafka delivers at-least-once. Mobile apps resubmit on timeout. Your system must recognize "I've seen this exact fact before" cheaply and correctly. With classes, you either hand-write Equals/GetHashCode (and forget to update them when a property is added — a classic source of subtle dedup bugs), pull in a library, or compare serialized JSON like a barbarian.

3. The boilerplate hides the contract

A class-based DTO with proper equality, immutability via constructor injection, and a useful ToString for structured logging is 40–60 lines. The contract — the thing an engineer actually needs to read — is 5 lines. The other 90% is ceremony that must be maintained, reviewed, and kept in sync. In an enterprise codebase with 400 DTOs, that's tens of thousands of lines of pure noise, every line a place for a bug to hide.

4. Nothing communicates intent

public class PaymentNotificationDto tells a reader nothing about how the type is meant to be used. Is it safe to cache? Safe to share across threads? Is mutation expected? The type system is silent, so the answer lives in tribal knowledge and code review vigilance — the two most expensive and least reliable enforcement mechanisms in software.

✨ Enter Records: The Type That Says What It Means

The same DTO as a record:

public record PaymentNotificationDto(
    string TransactionRef,
    decimal Amount,
    string Currency,
    string SenderAccount,
    DateTime ReceivedAt);

Five lines. And the compiler generates, for free:

Init-only properties — the object is frozen after construction.
Value-based Equals and GetHashCode — two records with the same data are equal.
A readable ToString — PaymentNotificationDto { TransactionRef = TXN-8842, Amount = 50000, ... }, which is a quiet superpower for structured logging in Serilog.
Deconstruction — var (txnRef, amount, currency, _, _) = dto;
with expressions — non-destructive "mutation" that produces a new copy.

Our idempotency check now just works:

var a = new PaymentNotificationDto("TXN-8842", 50_000m, "NGN", "0123456789", timestamp);
var b = new PaymentNotificationDto("TXN-8842", 50_000m, "NGN", "0123456789", timestamp);

Console.WriteLine(a == b); // true — same fact, recognized as such

And the tampering bug from earlier? It's now a compile-time error:

notification.Amount = Math.Round(notification.Amount, 2);
// CS8852: Init-only property can only be assigned in an object initializer

The pen has been taken away. If a stage in your pipeline genuinely needs a normalized amount, it must be explicit about creating a new fact:

var normalized = notification with { Amount = Math.Round(notification.Amount, 2) };

Notice what happened to the semantics: mutation-in-place (invisible, untraceable) became transformation (visible, reviewable, loggable). In a regulated environment where auditors ask "show me every place this value could have changed," that difference is not cosmetic.

🏦 Records in the Wild: Enterprise Fintech Scenarios

Abstract arguments are cheap. Let's walk through four scenarios that recur in nearly every banking or fintech platform and see how records change the engineering.

Scenario 1: Idempotent webhook ingestion

Your platform receives settlement notifications from multiple partner banks. Networks are unreliable; partners retry aggressively. A duplicate posting means a customer gets credited twice — a direct financial loss.

public record SettlementNotification(
    string ProviderRef,
    string SessionId,
    decimal Amount,
    string Currency,
    string BeneficiaryAccount,
    DateTimeOffset SettledAt);

Because records give you correct GetHashCode for free, deduplication within a processing window becomes trivially correct:

// In-memory dedup for a processing window — HashSet just works
private readonly HashSet<SettlementNotification> _seenInWindow = new();

public async Task HandleAsync(SettlementNotification notification)
{
    if (!_seenInWindow.Add(notification))
    {
        _logger.LogWarning("Duplicate settlement ignored: {Notification}", notification);
        return; // structured log includes full payload via generated ToString
    }
    await _ledger.PostAsync(notification);
}

With a class, HashSet.Add would happily accept ten copies of the same notification, because each is a distinct reference. You'd need hand-rolled equality — and every hand-rolled GetHashCode is a future bug when someone adds NarrationText to the DTO and forgets to update the hash.

(In production you'd also have a durable idempotency store keyed on ProviderRef + SessionId — value equality complements that; it doesn't replace it. More on this in the limitations section.)

Scenario 2: Event-driven architecture on Kafka

In an event-driven core banking system, events are facts about the past: TransferInitiated, FraudCheckPassed, TransferSettled. The first rule of event sourcing is that facts are immutable — you never rewrite history; you append new events.

public record TransferInitiated(
    Guid TransferId,
    string SourceAccount,
    string DestinationAccount,
    decimal Amount,
    string Currency,
    DateTimeOffset InitiatedAt) : IDomainEvent;

When a consumer replays this event during recovery or a new projection is built from the event log, the guarantee that no handler anywhere could have mutated the event is what makes replay deterministic. With mutable class events, one handler that "fixes up" a field poisons every downstream consumer in the same process — and you'll only discover it when a rebuilt projection disagrees with the ledger.

Records make the architectural rule ("events are immutable facts") a compiler-enforced invariant instead of a wiki page nobody reads.

Scenario 3: Maker-checker workflows

Regulated back-office operations — bulk account onboarding, limit changes, manual reversals — follow maker-checker: one officer initiates, another approves. The state transition must be explicit and auditable.

public record ReversalRequest(
    Guid RequestId,
    string TransactionRef,
    decimal Amount,
    string MakerId,
    string? CheckerId,
    ApprovalStatus Status,
    DateTimeOffset CreatedAt,
    DateTimeOffset? DecidedAt);

The with expression models the transition beautifully — the original request is preserved untouched, and approval produces a new state:

public ReversalRequest Approve(ReversalRequest request, string checkerId) =>
    request with
    {
        CheckerId = checkerId,
        Status = ApprovalStatus.Approved,
        DecidedAt = _clock.UtcNow
    };

For audit purposes you can log both objects — before and after — and the generated ToString gives the auditor a complete, readable diff. Try producing that with a mutated class: the "before" state is gone; you overwrote it.

This is the event-sourcing mindset in miniature: state changes are new values, not edits. Records make that the path of least resistance.

Scenario 4: End-of-day reconciliation

Reconciliation is fundamentally a set-comparison problem: our view of the day's transactions versus their file. Value equality turns this from a bespoke matching engine into LINQ:

public record ReconRow(string Ref, decimal Amount, string Currency, DateOnly ValueDate);

var internalRows  = await _ledger.GetRowsAsync(date);
var providerRows  = ParseProviderFile(file);

var missingOnOurSide   = providerRows.Except(internalRows);  // they have it, we don't
var missingOnTheirSide = internalRows.Except(providerRows);  // we have it, they don't

Except, Intersect, Distinct, GroupBy — the entire LINQ set-algebra toolkit works correctly by default on records because they carry proper value semantics. With classes you'd be writing IEqualityComparer<T> implementations, one per comparison, each a maintenance liability.

🧭 How Records Fulfill Software Engineering Principles

This is where the article stops being about syntax and starts being about design. Records aren't just terser — they align data contracts with the principles we already claim to follow.

Single Responsibility Principle (SRP)

A DTO has one job: carry data across a boundary. A mutable class quietly takes on a second job — being a workspace that pipeline stages scribble on. That second job is precisely what causes the tampering bugs above. Records structurally reject the second job. The type does one thing, and the compiler enforces it.

Junior-friendly analogy: a DTO should be an envelope, not a whiteboard. Records seal the envelope.

Open/Closed Principle (OCP)

"Open for extension, closed for modification" is usually discussed for behavior, but it applies to data too. The with expression is OCP for values: you extend an existing fact into a new one without modifying the original.

var enriched = notification with { Narration = ResolveNarration(notification) };

Every stage of an enrichment pipeline receives an input it cannot break for other consumers and emits a new output. Pipelines composed this way are trivially parallelizable and testable, because no stage has side effects on shared state.

Liskov Substitution Principle (LSP)

Records support inheritance (public record ChildDto : ParentDto) and — unlike hand-written class equality — the generated equality is hierarchy-aware. A record's Equals checks the runtime type via a synthesized EqualityContract, so a ParentDto and a ChildDto with identical property values are correctly not equal. Hand-rolled class equality gets this wrong constantly (the classic symmetric-equality bug in Equals overrides). Records give you LSP-safe equality out of the box.

That said — and this is a genuine senior-level guideline — prefer flat records and composition for DTOs. Deep DTO inheritance hierarchies are usually a design smell regardless of the type mechanism. Records just fail more safely when you do it.

Interface Segregation Principle (ISP)

Positional records make fat contracts visible. A 22-property class DTO looks like any other file. A 22-parameter positional record looks alarming in the constructor — and that discomfort is a feature. It nudges you toward segregating the contract:

public record BeneficiaryDetails(string Account, string BankCode, string Name);
public record TransferAmount(decimal Value, string Currency);
public record TransferRequest(Guid Id, TransferAmount Amount, BeneficiaryDetails Beneficiary);

Nested records compose value equality automatically: two TransferRequests are equal if their nested records are equal, all the way down. You get well-factored, comparable aggregate contracts with zero extra code.

Dependency Inversion Principle (DIP)

DIP is about depending on stable abstractions. In distributed systems, the most stable abstraction between two services is an immutable contract. A record shared in a contracts package is a promise: "this shape, these semantics, no hidden mutation." Consumers can cache it, memoize on it, and key dictionaries by it, all safely — because the contract's stability is enforced, not merely documented.

YAGNI (You Aren't Gonna Need It)

Every set; accessor on a DTO is speculative capability: "someone might need to mutate this someday." YAGNI says: don't build it until they do. Records are YAGNI applied to data — you ship exactly the capability a data carrier needs (construction, reading, comparison, copying) and nothing more. When a genuine mutation need appears, with handles it explicitly, and the code review sees it.

DRY (Don't Repeat Yourself)

The 40 lines of Equals/GetHashCode/ToString/copy-constructor boilerplate per DTO is repetition at industrial scale — the same pattern re-implemented hundreds of times, each an opportunity to diverge. Records move that pattern into the compiler, where it's written once and correct forever. This is DRY in its strongest form: not just avoiding copy-paste, but eliminating the category of code entirely.

Principle of Least Astonishment

a == b returning false for two identical payment notifications is astonishing to every engineer who hasn't been burned by it yet. Records make equality behave the way domain intuition expects for data: same values, same fact, equal. Fewer surprises, fewer 2 a.m. debugging sessions.

Fail Fast

Positional records force construction through a single point — the primary constructor — which is exactly where validation belongs:

public record TransferAmount
{
    public decimal Value { get; init; }
    public string Currency { get; init; }

    public TransferAmount(decimal value, string currency)
    {
        if (value <= 0)
            throw new ArgumentOutOfRangeException(nameof(value), "Amount must be positive.");
        if (currency is not { Length: 3 })
            throw new ArgumentException("ISO 4217 code required", nameof(currency));

        Value = value;
        Currency = currency;
    }
}

Because the object can't be mutated afterward, valid at construction means valid forever. Compare that with a mutable class, where validity is a moving target you must re-verify at every use site.

DDD: Records are Value Objects

If you practice Domain-Driven Design, you've been hand-writing value objects — Money, AccountNumber, Bvn, IdempotencyKey — for years, with all the equality ceremony that entails. Records are the value-object pattern promoted to a language feature. Eric Evans's definition — "an object that describes some characteristic... with no conceptual identity, compared by value" — is a description of a C# record. This is the deepest reason records feel right for DTOs: DTOs were always value objects; we just lacked the language support to say so.

CQS/CQRS alignment

In a CQRS system (MediatR users, this is you), queries return read models and commands carry intent. Both are one-way messages — facts flowing in one direction. Immutable records match that semantics exactly: a GetAccountBalanceQuery or its AccountBalanceResult has no business being mutated after creation. Making commands, queries, and results records turns your MediatR pipeline into a chain of pure-ish transformations, which is dramatically easier to reason about, test, and parallelize.

⚡ Performance: What Records Actually Buy You (and What They Don't)

Senior engineers should be suspicious of performance claims, so let's be honest and specific.

Where records genuinely help

1. Correct, fast equality without reflection. The compiler-generated Equals and GetHashCode are direct field comparisons — no reflection, no boxing of the comparands, no serialization tricks. Teams that fall back to comparing DTOs via JsonSerializer.Serialize(a) == JsonSerializer.Serialize(b) (it happens more than anyone admits) are paying two full serializations plus a string comparison per check. Record equality is orders of magnitude cheaper and allocation-free for the comparison itself.

2. Dictionary and set performance by construction. Because GetHashCode is generated correctly, records are first-class citizens as keys in Dictionary<TKey,TValue>, HashSet<T>, ConcurrentDictionary, and memoization caches. In a fraud-screening service that caches risk scores per (Account, Channel, AmountBand) tuple, a record key gives you O(1) lookups with hash distribution you didn't have to think about.

3. Lock-free sharing under concurrency. This is the big one for throughput. An immutable object can be read by any number of threads with zero synchronization. In a high-TPS inbound payments pipeline — parallel Kafka consumers, Parallel.ForEachAsync enrichment, fan-out to multiple downstream handlers — mutable shared DTOs force you into locks, defensive copies, or prayer. Immutable records eliminate the entire category: no locks, no torn reads, no defensive Clone() calls scattered through the code. The performance win isn't the object itself; it's all the synchronization machinery you get to delete.

4. Defensive copying disappears. Mature codebases with mutable DTOs accrue defensive copies — "clone it before passing it to that service, just in case." Each copy is allocation and CPU spent to compensate for mutability. With records, passing a reference is safe sharing. The copies you no longer make are a real, measurable saving in hot paths.

5. record struct for the truly hot path. Since C# 10, readonly record struct gives you value semantics with stack allocation — zero GC pressure for small, short-lived carriers:

public readonly record struct FxRate(string Pair, decimal Rate, DateTimeOffset AsOf);

For a pricing engine evaluating thousands of FX quotes per second, this avoids heap allocation entirely for the quote objects, which shows up directly in Gen0 collection frequency. The guidance mirrors regular structs: keep them small (≈16 bytes as a rule of thumb, pragmatically a few fields), and prefer readonly record struct to prevent the defensive-copy costs of non-readonly structs.

Where you must be honest about costs

with expressions allocate. Every with produces a full new instance. In a pipeline that transforms a record five times, that's five allocations where a mutable class would have zero. For typical API/webhook DTOs (short-lived, Gen0-collected) this is noise — Gen0 collection of small objects is extremely cheap, and correctness wins. But if profiling shows a genuinely hot loop churning large records through repeated with copies, that's a legitimate case for a builder or a mutable working type inside the tight scope, converting to a record at the boundary.

Record equality is O(n) in fields. Comparing two 30-property records field-by-field costs more than a reference check. If you're doing millions of comparisons and references would suffice (i.e., you know it's the same instance), ReferenceEquals still exists. This is rarely a real problem, but it's worth knowing the shape of the cost.

The performance argument, summarized for an architect: records are not "faster objects." They are objects whose guarantees let you delete slower code — locks, defensive copies, hand-rolled comparers, serialization-based equality — and whose struct variant removes allocation where it matters. The system gets faster because the design gets simpler.

⚠️ Limitations and Sharp Edges (Read This Before You Migrate)

Records are the right default for DTOs. They are not the right choice for everything, and they have gotchas that will bite teams who treat them as "classes but shorter."

1. Do not use records for EF Core entities

This is the most common misapplication. ORM entities are the opposite of DTOs: they have identity (primary key), a lifecycle, and mutation is the mechanism — EF Core's change tracker works by detecting property modifications on tracked instances. Records fight this at every turn: value equality confuses identity-based logic and DbSet semantics, with copies break change tracking (the copy is a different, untracked object), and lazy-loading proxies don't play well with record equality. Keep entities as classes. Map them to record DTOs at the query boundary — which is exactly the layering clean architecture prescribes anyway.

2. Collection properties break value equality

The gotcha that surprises everyone:

public record BatchDto(string BatchId, List<string> TransactionRefs);

var a = new BatchDto("B-01", new List<string> { "T1", "T2" });
var b = new BatchDto("B-01", new List<string> { "T1", "T2" });

Console.WriteLine(a == b); // FALSE — List<T> compares by reference

Generated equality compares each member with its own Equals — and List<T> equality is reference equality. So a record containing a collection silently loses the value-equality guarantee you adopted records for. In a reconciliation or dedup context, this is a correctness bug, not a style issue. Mitigations: use ImmutableArray<T>/ImmutableList<T> with a custom Equals override for that member, model the data to avoid collections in equality-sensitive records, or explicitly override Equals/GetHashCode with sequence comparison. Whatever you choose, make it a documented team convention.

3. `with` is a shallow copy

with copies field values. If a field is a reference to a mutable object (that List<string> again), the original and the copy share it:

var copy = original with { BatchId = "B-02" };
copy.TransactionRefs.Add("T3");   // also visible through original — same list!

The record's own scalar fields are safe; nested mutable state is not. The clean fix is to make records immutable all the way down — nested records, immutable collections — so shallow copy and deep copy become indistinguishable. Half-immutable records are arguably more dangerous than honest mutable classes, because they look safe.

4. Serializer and framework friction

Modern System.Text.Json and current Newtonsoft.Json handle records and constructor binding well, but the wider ecosystem varies: some mappers, legacy binders, and reflection-based tools still assume a parameterless constructor and settable properties. AutoMapper needs ForCtorParam configuration for positional records. If a tool forces you to add set; accessors back onto a record to make it work, you've kept the syntax and lost the point — reconsider the tool or use init with object-initializer-style records.

5. Inheritance equality can surprise you

Record inheritance is safer than hand-rolled class equality, but the EqualityContract behavior — a base-typed variable holding a derived instance never equals a base instance with identical values — surprises engineers who expected structural comparison across the hierarchy. Combined with the general smell of deep DTO hierarchies, the pragmatic rule stands: flat records, composition over inheritance.

6. Wide records get unwieldy

A positional record with 20+ parameters is miserable to construct and brittle to reorder (positional args have no names at most call sites). For wide contracts, use nominal records with init properties and object initializers, or — better — apply ISP and decompose the contract, as shown earlier.

7. Migration is not free in a brownfield

Introducing records into a 7-year-old codebase means auditing every place that mutates DTOs today. Some of those mutations are bugs you'll be glad to find; others are load-bearing hacks. Migrate boundary-first: new contracts, new events, and new endpoints as records; refactor legacy DTOs opportunistically when you're already touching their pipelines. A big-bang conversion of 400 DTOs is a two-week PR nobody can review.

🧰 When a Class Is Still the Right Answer

To be a trustworthy advocate for records, you have to know their boundary. Reach for a class when the type:

Has identity and a lifecycle — ORM entities, aggregate roots, anything where "same object over time despite changing state" is the point.
Is fundamentally behavioral — services, handlers, strategies. Records model data; classes model actors.
Requires managed mutable state — builders, accumulators, caches, connection pools.
Must integrate with mutation-assuming frameworks you can't replace — some legacy binders, certain UI frameworks, older SOAP/WCF tooling.
Is a large object copied in a proven hot loop — where with-copy allocation genuinely dominates a profile (measure first; it almost never does).

The mental model to give a junior engineer:

Records are for facts. Classes are for actors. A payment notification is a fact. The service that processes it is an actor. Facts don't change; actors do things.

📋 A Practical Decision Guide

Type of thing	Use
API request/response contracts	`record`
Kafka / message-bus events	`record`
MediatR commands, queries, results	`record`
DDD value objects (`Money`, `AccountNumber`)	`record` (with constructor validation)
Small hot-path carriers (FX ticks, coordinates)	`readonly record struct`
EF Core entities / aggregate roots	`class`
Services, handlers, middleware	`class`
Builders, accumulators, mutable caches	`class`
Configuration bound at startup (`IOptions`)	`record` with `init` properties

🏁 Conclusion: This Was Never About Syntax

The record keyword saves you 40 lines per DTO, and that alone would justify using it. But the real argument is architectural:

Mutable class DTOs make correctness a matter of discipline. Records make it a matter of compilation.

In consumer software, the cost of a mutated DTO is a weird bug. In banking and fintech, it's a duplicated settlement, a reconciliation break, a corrupted audit trail — failures with monetary and regulatory consequences. Every principle we invoked — SRP, OCP, YAGNI, DRY, fail-fast, least astonishment, DDD value semantics — converges on the same conclusion: data crossing a boundary should be an immutable, value-comparable fact, and C# now has a first-class type for exactly that.

So the next time you type public class SomethingDto { get; set; }, stop. Ask yourself: is this a fact or an actor? If it's a fact — and a DTO always is — reach for the record, take the pen away, and let the compiler enforce what your code reviews have been trying to enforce for years.

Your reconciliation team will thank you. Eventually. Silently. By never having to call you at 2 a.m. about a ₦0.005 discrepancy.

If you found this useful, drop a comment with the worst mutable-DTO bug you've debugged in production — the war stories are half the education.

LinkedIn Account : LinkedIn
Twitter Account: Twitter

🛑 Why Records Are the Better Choice for Clean, Modern, and Principle-Driven .NET Development

Odumosu Matthew — Sat, 18 Jul 2026 10:28:57 +0000

If you've been writing C# for more than a few years, you've written thousands of DTOs. Request models, response models, event payloads, integration contracts, the connective tissue of every distributed system. And if you're like most of us, you've written nearly all of them as classes with { get; set; } properties, because that's what we've done since .NET Framework 2.0.

Here's the uncomfortable truth: a mutable class is the wrong data structure for a DTO, and it has been since C# 9 gave us a better one. Not "stylistically wrong." Wrong in ways that show up as production incidents, duplicate payments, corrupted audit trails, and race conditions in async pipelines.

This article makes the case from first principles, with examples drawn from banking and fintech systems, payment processing, reconciliation, maker-checker workflows, because that's where the cost of getting this wrong is measured in money, not just bugs. Whether you're a junior engineer writing your first API contract or a solutions architect reviewing a payment platform design, the goal is the same: understand why records exist, not just how to type them.

🚚 What Are DTOs in Practice?

Imagine you're building a ride-sharing app like Uber or Bolt. When the frontend requests a list of nearby drivers, you return a DriverDto:

{
  "id": 142,
  "name": "Linda",
  "rating": 4.9
}

In C#, you might write:

public class DriverDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public double Rating { get; set; }
}

This works, but it's bloated, mutable, and doesn't reflect the true intent of a DTO: a simple, structured data carrier.

Now zoom out from ride-sharing to where DTOs really earn their keep: system boundaries in enterprise software. In a payments platform, DTOs are everywhere:

The inbound webhook payload from a card scheme or a partner bank notifying you of a settlement.
The API response your mobile app receives when a customer checks their balance.
The Kafka event published when a transfer moves from PENDING to SETTLED.
The reconciliation record you compare against the processor's end-of-day file.

A DTO is a snapshot of data at a point in time. Snapshots don't change. So why are we modeling them with a type whose defining feature is that it changes?

🔍 The Real Problems with Class-Based DTOs

Let's be precise about what's wrong, because "bloated" alone wouldn't justify changing a decade of habit. There are four concrete failures.

1. Mutability is a liability at boundaries

Consider a simplified inbound payment notification in a fintech platform:

public class PaymentNotificationDto
{
    public string TransactionRef { get; set; }
    public decimal Amount { get; set; }
    public string Currency { get; set; }
    public string SenderAccount { get; set; }
    public DateTime ReceivedAt { get; set; }
}

This object will pass through validation, enrichment, fraud screening, ledger posting, and notification services, often across async hops and sometimes across threads. Any one of those stages can silently mutate it. A well-meaning teammate writes:

// "Normalizing" inside the fraud check — seems harmless
notification.Amount = Math.Round(notification.Amount, 2);

2. Reference equality is the wrong equality

Two DTOs carrying identical data are, for every practical purpose, the same fact. But classes compare by reference:

var a = new PaymentNotificationDto { TransactionRef = "TXN-8842", Amount = 50_000m, Currency = "NGN" };
var b = new PaymentNotificationDto { TransactionRef = "TXN-8842", Amount = 50_000m, Currency = "NGN" };

Console.WriteLine(a == b);        // false
Console.WriteLine(a.Equals(b));   // false

Why does this matter? Idempotency the backbone of every payment system. Partner banks retry webhooks. Kafka delivers at-least-once. Mobile apps resubmit on timeout. Your system must recognize "I've seen this exact fact before" cheaply and correctly. With classes, you either hand-write Equals/GetHashCode (and forget to update them when a property is added — a classic source of subtle dedup bugs), pull in a library, or compare serialized JSON like a barbarian.

3. The boilerplate hides the contract

A class-based DTO with proper equality, immutability via constructor injection, and a useful ToString for structured logging is 40–60 lines. The contract — the thing an engineer actually needs to read, is 5 lines. The other 90% is ceremony that must be maintained, reviewed, and kept in sync. In an enterprise codebase with 400 DTOs, that's tens of thousands of lines of pure noise, every line a place for a bug to hide.

4. Nothing communicates intent

✨ Enter Records: The Type That Says What It Means

The same DTO as a record:

public record PaymentNotificationDto(
    string TransactionRef,
    decimal Amount,
    string Currency,
    string SenderAccount,
    DateTime ReceivedAt);

Five lines. And the compiler generates, for free:

Init-only properties - the object is frozen after construction.
Value-based Equals and GetHashCode - two records with the same data are equal.
A readable ToString - PaymentNotificationDto { TransactionRef = TXN-8842, Amount = 50000, ... }, which is a quiet superpower for structured logging in Serilog.
Deconstruction - var (txnRef, amount, currency, _, _) = dto;
with expressions - non-destructive "mutation" that produces a new copy.

Our idempotency check now just works:

var a = new PaymentNotificationDto("TXN-8842", 50_000m, "NGN", "0123456789", timestamp);
var b = new PaymentNotificationDto("TXN-8842", 50_000m, "NGN", "0123456789", timestamp);

Console.WriteLine(a == b); // true — same fact, recognized as such

And the tampering bug from earlier? It's now a compile-time error:

notification.Amount = Math.Round(notification.Amount, 2);
// CS8852: Init-only property can only be assigned in an object initializer

The pen has been taken away. If a stage in your pipeline genuinely needs a normalized amount, it must be explicit about creating a new fact:

var normalized = notification with { Amount = Math.Round(notification.Amount, 2) };

🏦 Records in the Wild: Enterprise Fintech Scenarios

Abstract arguments are cheap. Let's walk through four scenarios that recur in nearly every banking or fintech platform and see how records change the engineering.

Scenario 1: Idempotent webhook ingestion

public record SettlementNotification(
    string ProviderRef,
    string SessionId,
    decimal Amount,
    string Currency,
    string BeneficiaryAccount,
    DateTimeOffset SettledAt);

Because records give you correct GetHashCode for free, deduplication within a processing window becomes trivially correct:

// In-memory dedup for a processing window — HashSet just works
private readonly HashSet<SettlementNotification> _seenInWindow = new();

public async Task HandleAsync(SettlementNotification notification)
{
    if (!_seenInWindow.Add(notification))
    {
        _logger.LogWarning("Duplicate settlement ignored: {Notification}", notification);
        return; // structured log includes full payload via generated ToString
    }
    await _ledger.PostAsync(notification);
}

(In production you'd also have a durable idempotency store keyed on ProviderRef + SessionId — value equality complements that; it doesn't replace it. More on this in the limitations section.)

Scenario 2: Event-driven architecture on Kafka

public record TransferInitiated(
    Guid TransferId,
    string SourceAccount,
    string DestinationAccount,
    decimal Amount,
    string Currency,
    DateTimeOffset InitiatedAt) : IDomainEvent;

Records make the architectural rule ("events are immutable facts") a compiler-enforced invariant instead of a wiki page nobody reads.

Scenario 3: Maker-checker workflows

Regulated back-office operations — bulk account onboarding, limit changes, manual reversals , follow maker-checker: one officer initiates, another approves. The state transition must be explicit and auditable.

public record ReversalRequest(
    Guid RequestId,
    string TransactionRef,
    decimal Amount,
    string MakerId,
    string? CheckerId,
    ApprovalStatus Status,
    DateTimeOffset CreatedAt,
    DateTimeOffset? DecidedAt);

The with expression models the transition beautifully — the original request is preserved untouched, and approval produces a new state:

public ReversalRequest Approve(ReversalRequest request, string checkerId) =>
    request with
    {
        CheckerId = checkerId,
        Status = ApprovalStatus.Approved,
        DecidedAt = _clock.UtcNow
    };

This is the event-sourcing mindset in miniature: state changes are new values, not edits. Records make that the path of least resistance.

Scenario 4: End-of-day reconciliation

Reconciliation is fundamentally a set-comparison problem: our view of the day's transactions versus their file. Value equality turns this from a bespoke matching engine into LINQ:

public record ReconRow(string Ref, decimal Amount, string Currency, DateOnly ValueDate);

var internalRows  = await _ledger.GetRowsAsync(date);
var providerRows  = ParseProviderFile(file);

var missingOnOurSide   = providerRows.Except(internalRows);  // they have it, we don't
var missingOnTheirSide = internalRows.Except(providerRows);  // we have it, they don't

🧭 How Records Fulfill Software Engineering Principles

This is where the article stops being about syntax and starts being about design. Records aren't just terser — they align data contracts with the principles we already claim to follow.

Single Responsibility Principle (SRP)

Junior-friendly analogy: a DTO should be an envelope, not a whiteboard. Records seal the envelope.

Open/Closed Principle (OCP)

var enriched = notification with { Narration = ResolveNarration(notification) };

Liskov Substitution Principle (LSP)

Records support inheritance (public record ChildDto : ParentDto) and — unlike hand-written class equality - the generated equality is hierarchy-aware. A record's Equals checks the runtime type via a synthesized EqualityContract, so a ParentDto and a ChildDto with identical property values are correctly not equal. Hand-rolled class equality gets this wrong constantly (the classic symmetric-equality bug in Equals overrides). Records give you LSP-safe equality out of the box.

Interface Segregation Principle (ISP)

public record BeneficiaryDetails(string Account, string BankCode, string Name);
public record TransferAmount(decimal Value, string Currency);
public record TransferRequest(Guid Id, TransferAmount Amount, BeneficiaryDetails Beneficiary);

Dependency Inversion Principle (DIP)

YAGNI (You Aren't Gonna Need It)

DRY (Don't Repeat Yourself)

Principle of Least Astonishment

Fail Fast

Positional records force construction through a single point — the primary constructor — which is exactly where validation belongs:

public record TransferAmount
{
    public decimal Value { get; init; }
    public string Currency { get; init; }

    public TransferAmount(decimal value, string currency)
    {
        if (value <= 0)
            throw new ArgumentOutOfRangeException(nameof(value), "Amount must be positive.");
        if (currency is not { Length: 3 })
            throw new ArgumentException("ISO 4217 code required", nameof(currency));

        Value = value;
        Currency = currency;
    }
}

DDD: Records are Value Objects

CQS/CQRS alignment

⚡ Performance: What Records Actually Buy You (and What They Don't)

Senior engineers should be suspicious of performance claims, so let's be honest and specific.

Where records genuinely help

1. Correct, fast equality without reflection. The compiler-generated Equals and GetHashCode are direct field comparisons - no reflection, no boxing of the comparands, no serialization tricks. Teams that fall back to comparing DTOs via JsonSerializer.Serialize(a) == JsonSerializer.Serialize(b) (it happens more than anyone admits) are paying two full serializations plus a string comparison per check. Record equality is orders of magnitude cheaper and allocation-free for the comparison itself.

5. record struct for the truly hot path. Since C# 10, readonly record struct gives you value semantics with stack allocation — zero GC pressure for small, short-lived carriers:

public readonly record struct FxRate(string Pair, decimal Rate, DateTimeOffset AsOf);

Where you must be honest about costs

⚠️ Limitations and Sharp Edges (Read This Before You Migrate)

Records are the right default for DTOs. They are not the right choice for everything, and they have gotchas that will bite teams who treat them as "classes but shorter."

1. Do not use records for EF Core entities

2. Collection properties break value equality

The gotcha that surprises everyone:

public record BatchDto(string BatchId, List<string> TransactionRefs);

var a = new BatchDto("B-01", new List<string> { "T1", "T2" });
var b = new BatchDto("B-01", new List<string> { "T1", "T2" });

Console.WriteLine(a == b); // FALSE — List<T> compares by reference

3. `with` is a shallow copy

with copies field values. If a field is a reference to a mutable object (that List<string> again), the original and the copy share it:

var copy = original with { BatchId = "B-02" };
copy.TransactionRefs.Add("T3");   // also visible through original — same list!

4. Serializer and framework friction

5. Inheritance equality can surprise you

6. Wide records get unwieldy

7. Migration is not free in a brownfield

🧰 When a Class Is Still the Right Answer

To be a trustworthy advocate for records, you have to know their boundary. Reach for a class when the type:

Has identity and a lifecycle — ORM entities, aggregate roots, anything where "same object over time despite changing state" is the point.
Is fundamentally behavioral — services, handlers, strategies. Records model data; classes model actors.
Requires managed mutable state — builders, accumulators, caches, connection pools.
Must integrate with mutation-assuming frameworks you can't replace — some legacy binders, certain UI frameworks, older SOAP/WCF tooling.
Is a large object copied in a proven hot loop — where with-copy allocation genuinely dominates a profile (measure first; it almost never does).

The mental model to give a junior engineer:

Records are for facts. Classes are for actors. A payment notification is a fact. The service that processes it is an actor. Facts don't change; actors do things.

📋 A Practical Decision Guide

Type of thing	Use
API request/response contracts	`record`
Kafka / message-bus events	`record`
MediatR commands, queries, results	`record`
DDD value objects (`Money`, `AccountNumber`)	`record` (with constructor validation)
Small hot-path carriers (FX ticks, coordinates)	`readonly record struct`
EF Core entities / aggregate roots	`class`
Services, handlers, middleware	`class`
Builders, accumulators, mutable caches	`class`
Configuration bound at startup (`IOptions`)	`record` with `init` properties

🏁 Conclusion: This Was Never About Syntax

The record keyword saves you 40 lines per DTO, and that alone would justify using it. But the real argument is architectural:

Mutable class DTOs make correctness a matter of discipline. Records make it a matter of compilation.

Your reconciliation team will thank you. Eventually. Silently. By never having to call you at 2 a.m. about a ₦0.005 discrepancy.

If you found this useful, drop a comment with the worst mutable-DTO bug you've debugged in production - the war stories are half the education.

LinkedIn Account : LinkedIn
Twitter Account: Twitter
Credit: Graphics sourced from Dev.To

The Missing Row: Auto-Provisioning Derived Records Without the Race Condition

Odumosu Matthew — Sat, 18 Jul 2026 10:00:37 +0000

Why some records should be created by your system, not your users, and how to do it safely in .NET.

A support ticket lands on your desk: "The Teams page is empty. I added a member, but no team shows up."

You check the API. It's behaving exactly as written:

  { "items": [], "totalCount": 0 }

Nothing is broken. And that's the problem. The system is faithfully returning nothing, because the row that the page reads from was never created. Somewhere in your design,

you assumed a human would create it first.

This article is about a small, recurring design decision that quietly causes empty dashboards, confused users, and "is this a bug?" tickets: who is responsible for creating
derived records the user, or the system? and how to let the system do it without introducing duplicate rows or race conditions.

The problem

Let's use a fictional product: a collaboration tool called Loop.

In Loop, the important entities are:

An Organization (a paying customer).
A Member (a person invited into an organization under a plan).
A Team a grouping that members belong to, keyed by (OrganizationId, PlanCode).

The admin dashboard lists Teams. Each team card shows a member count.

Here's the catch in the original design: creating a Member wrote a member row. Creating a Team was a separate, manual step an admin was expected to do first. If an

admin invited members without first creating the matching team, the dashboard showed nothing even though the members clearly existed.

From the user's point of view, they did everything right. From the system's point of view, a required row simply didn't exist.

Why it matters

The Team record isn't independent information. It is fully derivable from the first member invited under a plan. When one entity's existence is implied by another, forcing a
human to create it manually is a design smell. It leads to:

Empty states that look like outages. Users can't tell "no data" from "misconfigured."
Support load. Every skipped step becomes a ticket.
Silent data drift. Members exist with no grouping to roll them up.

The fix is to let the system provision the derived record at the moment it first becomes necessary the first write, instead of relying on a prerequisite step.

The naive solution (and its two traps)

The obvious move: when a member is created, create the team if it isn't there yet.

  public async Task AddMemberAsync(NewMember input, CancellationToken ct)
  {
      var member = Member.Create(input);
      await _members.AddAsync(member, ct);

      // Create the team if this is the first member on this plan.
      var exists = await _teams.AnyAsync(
          t => t.OrganizationId == input.OrganizationId
            && t.PlanCode == input.PlanCode, ct);

      if (!exists)
      {
          var team = new Team(input.OrganizationId, input.PlanCode);
          await _teams.AddAsync(team, ct);
      }

      await _unitOfWork.SaveChangesAsync(ct);
  }

This works in a demo. In production it has two traps.

Trap 1: the check-then-insert race

AnyAsync(...) and AddAsync(...) are not atomic. If two members are invited to the same new team at the same moment, both requests can observe exists == false, and both

insert a team. Now the dashboard shows two identical team cards, and every downstream count is split across them.

An application-level AnyAsync check can never close this window on its own. The only reliable guard is a unique constraint, so the database, the one component that sees
all writes, rejects the second insert.

  // EF Core model configuration
  modelBuilder.Entity<Team>()
      .HasIndex(t => new { t.OrganizationId, t.PlanCode })
      .IsUnique();

With the constraint in place, turn the "check" into a "try, and treat a uniqueness violation as success", because a violation means someone else already created exactly the

row you wanted:

  private async Task EnsureTeamExistsAsync(
      Guid organizationId, string planCode, CancellationToken ct)
  {
      var exists = await _teams.AnyAsync(
          t => t.OrganizationId == organizationId
            && t.PlanCode == planCode, ct);
      if (exists) return;

      _teams.Add(new Team(organizationId, planCode));

      try
      {
          await _unitOfWork.SaveChangesAsync(ct);
      }
      catch (DbUpdateException ex) when (IsUniqueViolation(ex))
      {
          // A concurrent request created the same team first.
          // That is the desired end state, so this is a no-op, not an error.
      }
  }

The application-level AnyAsync still earns its keep: it avoids a failed insert on the common path. The constraint is what makes the code correct under concurrency. You

want both, the check for efficiency, the constraint for truth.

###Trap 2: the normalization mismatch

The team and the member are joined on PlanCode. If one side stores "pro" and the other stores "pro " (trailing space) or "PRO", the join silently returns nothing.

You'll create a team and members, and the dashboard still shows a count of zero a bug that looks identical to the original one.

Normalize the key in exactly one place, and make both sides use it:

  public Team(Guid organizationId, string planCode)
  {
      OrganizationId = organizationId;
      // Trim once, here, so the stored key matches how members store it.
      PlanCode = planCode.Trim();
      Id = Guid.NewGuid();
  }

Whatever rule you choose, trim, lowercase, collapse, apply it on every path that reads or writes the key. A join is only as reliable as the normalization behind it.

The retroactive gap

Here's the part teams forget. This fix runs at write time. It fixes every member invited after you ship it.

It does nothing for data that already exists. Organizations that invited members before the change still have no team row. They will look broken until either:

someone invites one more member (the new code then provisions the missing team), or
you run a backfill for the existing data.

Don't skip the backfill and hope the write path heals everything. Write a one-time job that creates the missing rows from what's already there:

  -- Create the missing team for every distinct (org, plan) that has members
  -- but no team yet. Idempotent: safe to run more than once.
  INSERT INTO teams (id, organization_id, plan_code, created_at_utc)
  SELECT gen_random_uuid(), m.organization_id, TRIM(m.plan_code), now()
  FROM   members m
  LEFT JOIN teams t
    ON t.organization_id = m.organization_id
   AND t.plan_code = TRIM(m.plan_code)
  WHERE  t.id IS NULL
  GROUP BY m.organization_id, TRIM(m.plan_code);

A forward fix plus a backfill is one complete change, not two optional ones. Ship them together.

Best practices

Let the system create what it can derive. If record B's existence is implied by record A, don't make a human create B first.
Guard uniqueness at the database, not just in code. Application checks are an optimization; constraints are the guarantee.
Make "already exists" a success, not an exception. Idempotent provisioning should converge on the same state no matter how many callers race.
Normalize join keys once and everywhere. Most "the data is there but the count is zero" bugs are a normalization mismatch.
Pair every write-time fix with a backfill. The forward path won't repair history.

Common mistakes

Relying on AnyAsync / SELECT ... IF NOT EXISTS alone under concurrency.
Adding the unique index but not handling the violation — turning a race into a 500 for the second caller.
Backfilling but forgetting to normalize in the backfill query, so it re-inserts near-duplicates.
Provisioning the derived row in a separate transaction from the triggering write, so a partial failure leaves one without the other.

Lessons learned

An empty response is not always "no data." Sometimes it's "the row you're reading from was never anyone's job to create." The most durable fix is to move that responsibility

from the user to the system, and then make the system's version safe under concurrency and honest about history.

The interesting engineering isn't the get-or-create. It's everything around it: the constraint that makes it correct, the normalization that makes the join real, and the

backfill that makes it true for data that already exists.

LinkedIn Account : LinkedIn
Twitter Account: Twitter
Credit: Graphics sourced from Medium

Mastering the Senior Java/Spring Boot Engineer Interview

Odumosu Matthew — Thu, 19 Mar 2026 12:14:25 +0000

I've been on both sides of the interview table for over a decade. I've bombed interviews at companies I really wanted to join. I've also hired engineers who looked mediocre on paper but turned out to be the best technical decisions I ever made.

Here's what I've learned: Senior interviews aren't knowledge tests. They're thinking tests.

Anyone can memorize that HashMap has O(1) lookup. What interviewers actually want to know is whether you'll make good decisions when the documentation runs out and Stack Overflow doesn't have your exact problem.

This guide is different. I'm not going to list 50 questions with textbook answers. Instead, I'll show you how senior engineers think through problems, because that's what actually gets you hired.

The Uncomfortable Truth About Senior Interviews

Let me be blunt: if you have 10+ years of experience and you're still getting rejected, it's probably not your technical knowledge. It's one of these:

You're answering questions like a junior - giving definitions instead of demonstrating judgment
You're not showing trade-off thinking - every solution has costs, and you're not acknowledging them
You're not telling stories - your experience is your differentiator, but you're not using it
You're solving the wrong problem - you're answering what was asked, not what was meant

Let's fix all of these.

Part 1: Core Java - But Not The Way You Think

Every Java interview covers the basics. The difference is depth. Here's what separates a senior answer from a mid-level one.

The `equals()` and `hashCode()` Question

What they ask: "What happens if you override equals() but not hashCode()?"

Mid-level answer: "It breaks hash-based collections because equal objects might have different hash codes."

Senior answer:

"It violates the hashCode contract, which creates subtle bugs that are incredibly hard to track down in production.

Here's why this is dangerous: the bug won't show up in your unit tests. It'll show up three months later when someone adds your object to a HashSet in a completely different part of the codebase.

public class TradingAccount {
    private String accountId;
    private String owner;

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        TradingAccount that = (TradingAccount) o;
        return Objects.equals(accountId, that.accountId);
    }

    // Intentionally missing hashCode — this is a bug
}

Now imagine this in a trading system:

Set<TradingAccount> processedAccounts = new HashSet<>();
TradingAccount account1 = loadFromDatabase(12345);
processedAccounts.add(account1);

// Later, different code path loads the same account
TradingAccount account2 = loadFromDatabase(12345);
if (!processedAccounts.contains(account2)) {
    // This executes! We process the same account twice.
    // In a trading system, this could mean duplicate orders.
    processAccount(account2);
}

The contains() check fails because HashSet first checks the bucket using hashCode(). Since we didn't override it, account1 and account2 have different hash codes (based on memory address), so they land in different buckets. The equals() method is never even called.

I've seen this exact bug cause duplicate transactions in a payment system. Took two weeks to find because it only happened under specific load conditions where the same entity was loaded through different code paths.

The fix isn't just adding hashCode() - it's establishing team conventions. We now require that any class overriding equals() must also have a test that verifies the hashCode contract."

See the difference? The senior answer:

Explains the mechanism (hash buckets)
Describes real consequences (duplicate transactions)
Shares a war story (the two-week debugging session)
Proposes prevention (team conventions and tests)

This is what interviewers want. They're not testing if you know Java. They're testing if you've been burned by Java and learned from it.

Memory Model and Concurrency

What they ask: "Explain the Java Memory Model and when you'd use volatile."

This question is a trap. Most candidates launch into a textbook explanation of happens-before relationships and CPU caches. That's fine, but it's not impressive.

Senior approach:

"The Java Memory Model defines how threads interact through memory, but honestly, the practical implications matter more than the theory.

Here's when I've actually needed volatile in production:

1. Status flags for graceful shutdown

public class OrderProcessor implements Runnable {
    private volatile boolean running = true;

    public void shutdown() {
        running = false;  // Visible to the processing thread immediately
    }

    @Override
    public void run() {
        while (running) {
            Order order = queue.poll(100, TimeUnit.MILLISECONDS);
            if (order != null) {
                process(order);
            }
        }
        // Cleanup resources
    }
}

Without volatile, the processing thread might cache running = true and never see the shutdown signal. I've seen services that couldn't shut down cleanly because of this - they'd hang during deployments.

2. Double-checked locking (though I avoid it now)

public class ExpensiveResourceHolder {
    private volatile ExpensiveResource instance;

    public ExpensiveResource getInstance() {
        if (instance == null) {
            synchronized (this) {
                if (instance == null) {
                    instance = new ExpensiveResource();
                }
            }
        }
        return instance;
    }
}

This pattern is technically correct with volatile, but I've stopped using it. It's too easy to get wrong, and the performance benefit over simple synchronization is negligible in most applications. I now use either:

Lazy holders for singletons
ConcurrentHashMap.computeIfAbsent() for cached values
Or just accept the synchronization cost

3. Publishing immutable objects

public class ConfigurationHolder {
    private volatile ImmutableConfiguration config;

    public void updateConfiguration(ImmutableConfiguration newConfig) {
        // Single atomic write, immediately visible to all readers
        this.config = newConfig;
    }

    public ImmutableConfiguration getConfiguration() {
        return config;  // No synchronization needed for reads
    }
}

This is probably my most common use case — when you have a reference that's written rarely but read constantly.

What I'd avoid:

Using volatile for compound operations:

private volatile int counter = 0;

public void increment() {
    counter++;  // NOT THREAD-SAFE! Read-modify-write is not atomic
}

This is a classic mistake. volatile guarantees visibility, not atomicity. You need AtomicInteger or synchronization here.

The broader point is: most concurrency problems are better solved by avoiding shared mutable state entirely. Immutable objects, message passing, and higher-level constructs like CompletableFuture are almost always better choices than low-level primitives like volatile."

Part 2: Spring Boot — Beyond the Annotations

Every Java developer knows @Autowired. Senior developers know when not to use it.

Dependency Injection Philosophy

What they ask: "Explain dependency injection in Spring."

What they mean: "Do you understand why we use DI, or do you just follow patterns blindly?"

Senior answer:

"Dependency injection is fundamentally about inverting control of object creation. Instead of a class creating its dependencies, they're provided from outside. This seems simple, but the implications are profound.

Here's what DI actually buys us:

1. Testability without magic

// Without DI — hard to test
public class PaymentProcessor {
    private final PaymentGateway gateway = new StripeGateway();

    public PaymentResult process(Payment payment) {
        return gateway.charge(payment);  // How do you test this without hitting Stripe?
    }
}

// With DI — trivially testable
public class PaymentProcessor {
    private final PaymentGateway gateway;

    public PaymentProcessor(PaymentGateway gateway) {
        this.gateway = gateway;
    }

    public PaymentResult process(Payment payment) {
        return gateway.charge(payment);
    }
}

// Test
@Test
void shouldHandleDeclinedCard() {
    PaymentGateway mockGateway = mock(PaymentGateway.class);
    when(mockGateway.charge(any())).thenReturn(PaymentResult.declined("Insufficient funds"));

    PaymentProcessor processor = new PaymentProcessor(mockGateway);
    PaymentResult result = processor.process(new Payment(100));

    assertThat(result.isDeclined()).isTrue();
}

2. Configuration flexibility

@Configuration
public class PaymentConfig {

    @Bean
    @Profile("production")
    public PaymentGateway productionGateway() {
        return new StripeGateway(stripeApiKey);
    }

    @Bean
    @Profile("sandbox")
    public PaymentGateway sandboxGateway() {
        return new StripeSandboxGateway();
    }

    @Bean
    @Profile("test")
    public PaymentGateway testGateway() {
        return new InMemoryPaymentGateway();  // No external calls
    }
}

Same code, different behavior based on environment. No if-statements scattered through your business logic.

3. Lifecycle management

Spring manages object lifecycles, which matters more than people realize:

@Component
public class DatabaseConnectionPool {
    private final HikariDataSource dataSource;

    @PostConstruct
    public void initialize() {
        // Warm up connections
        dataSource.getConnection().close();
        log.info("Connection pool warmed up");
    }

    @PreDestroy
    public void shutdown() {
        // Graceful shutdown — finish in-flight queries
        dataSource.close();
        log.info("Connection pool shut down gracefully");
    }
}

What I've learned to avoid:

Field injection:

// Don't do this
@Service
public class OrderService {
    @Autowired
    private PaymentService paymentService;  // Hidden dependency
    @Autowired
    private InventoryService inventoryService;  // Another hidden dependency
}

Problems:

Dependencies are invisible in the constructor
Can't create instances without Spring (harder to test)
No compile-time safety if dependency is missing
Encourages too many dependencies (code smell for SRP violation)

Constructor injection is always better:

@Service
public class OrderService {
    private final PaymentService paymentService;
    private final InventoryService inventoryService;

    // If this constructor has 8 parameters, you know something's wrong
    public OrderService(PaymentService paymentService, InventoryService inventoryService) {
        this.paymentService = paymentService;
        this.inventoryService = inventoryService;
    }
}

With constructor injection, a class with too many dependencies becomes obvious — the constructor is unreadable. That's a feature, not a bug. It's a signal that the class has too many responsibilities."

The N+1 Problem — A Production Story

What they ask: "How do you solve N+1 queries in JPA?"

What they mean: "Have you actually debugged performance issues in production?"

Senior answer:

"The N+1 problem has cost me more production incidents than any other JPA issue. Let me tell you about one.

We had an order history page that worked fine in development — maybe 50ms response time. In production, with real data, it took 12 seconds. Users were furious.

The code looked innocent:

@GetMapping("/orders")
public List<OrderDto> getOrders(@RequestParam Long customerId) {
    List<Order> orders = orderRepository.findByCustomerId(customerId);
    return orders.stream()
        .map(this::toDto)
        .collect(toList());
}

private OrderDto toDto(Order order) {
    return new OrderDto(
        order.getId(),
        order.getTotal(),
        order.getItems().size(),  // BOOM — lazy load
        order.getCustomer().getName()  // BOOM — another lazy load
    );
}

One customer had 500 orders. That's:

1 query to get orders
500 queries to get items (one per order)
500 queries to get customer (yes, the same customer 500 times — JPA doesn't know)

1,001 queries for one page load.

Solution 1: JOIN FETCH

@Query("SELECT o FROM Order o " +
       "JOIN FETCH o.items " +
       "JOIN FETCH o.customer " +
       "WHERE o.customer.id = :customerId")
List<Order> findByCustomerIdWithDetails(@Param("customerId") Long customerId);

This generates a single query with JOINs. But there's a catch — if you have multiple collections, you get a cartesian product:

// DON'T DO THIS
@Query("SELECT o FROM Order o " +
       "JOIN FETCH o.items " +
       "JOIN FETCH o.payments " +  // Two collections = cartesian product
       "WHERE o.customer.id = :customerId")

If an order has 10 items and 3 payments, you get 30 rows per order. Hibernate deduplicates, but you've transferred way more data than needed.

Solution 2: @EntityGraph

@EntityGraph(attributePaths = {"items", "customer"})
List<Order> findByCustomerId(Long customerId);

Cleaner syntax, same result. I prefer this for simple cases.

Solution 3: Batch fetching

@Entity
public class Order {
    @OneToMany(mappedBy = "order")
    @BatchSize(size = 25)  // Load 25 orders' items in one query
    private List<OrderItem> items;
}

This changes N+1 into N/25+1. For 500 orders, that's 21 queries instead of 501. Not perfect, but much better, and it doesn't require changing your repository methods.

Solution 4: DTO projections (my preferred approach for read-heavy endpoints)

public interface OrderSummary {
    Long getId();
    BigDecimal getTotal();
    Integer getItemCount();
    String getCustomerName();
}

@Query("SELECT o.id as id, o.total as total, SIZE(o.items) as itemCount, c.name as customerName " +
       "FROM Order o JOIN o.customer c " +
       "WHERE c.id = :customerId")
List<OrderSummary> findOrderSummaries(@Param("customerId") Long customerId);

This is the nuclear option. One query, exactly the data you need, no entity mapping overhead. For read-heavy endpoints where you're just displaying data, this is usually the best choice.

How I prevent N+1 now:

Enable SQL logging in development:

spring:
  jpa:
    show-sql: true
    properties:
      hibernate:
        format_sql: true

Use a query counter in tests:

@Test
void shouldLoadOrdersInSingleQuery() {
    QueryCounter counter = new QueryCounter();

    orderRepository.findByCustomerIdWithDetails(customerId);

    assertThat(counter.getQueryCount()).isLessThanOrEqualTo(1);
}

Monitor query count in production — we alert if any endpoint exceeds 10 queries."

Part 3: System Design Thinking

Senior interviews always include system design. Here's how to approach them.

The Payment Service Question

What they ask: "Design a payment service for an e-commerce platform."

How seniors approach it:

"Before I start designing, I need to understand the constraints. Let me ask a few questions:

What's the expected transaction volume? Tens per second or thousands?
Do we need to support multiple payment providers or just one?
What's the consistency requirement? Can we ever lose a transaction?
What's the latency budget for a payment request?

[Interviewer provides context: 100 TPS, multiple providers, zero data loss, 500ms budget]

Okay, let me think through this systematically.

The core challenge with payments is reliability. Users will retry if something looks stuck. Payment providers might timeout but still process the charge. Network partitions happen. We need to handle all of this without charging someone twice or losing their order.

Architecture:

┌─────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Client    │────▶│  Payment API    │────▶│  Payment Worker │
└─────────────┘     └─────────────────┘     └─────────────────┘
                            │                        │
                            ▼                        ▼
                    ┌─────────────┐          ┌─────────────┐
                    │  Database   │          │   Stripe    │
                    │  (source    │          │   Adyen     │
                    │  of truth)  │          │   etc.      │
                    └─────────────┘          └─────────────┘

Key design decisions:

1. Idempotency is non-negotiable

Every payment request must include a client-generated idempotency key:

@PostMapping("/payments")
public PaymentResponse createPayment(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody PaymentRequest request) {

    // Check if we've seen this request before
    Optional<Payment> existing = paymentRepository.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return toResponse(existing.get());  // Return cached result
    }

    // Process new payment
    Payment payment = processPayment(request, idempotencyKey);
    return toResponse(payment);
}

This prevents duplicate charges when clients retry.

2. Two-phase state machine

Payments go through explicit states:

public enum PaymentState {
    PENDING,      // Created, not yet sent to provider
    PROCESSING,   // Sent to provider, awaiting response
    COMPLETED,    // Successfully charged
    FAILED,       // Provider declined
    REFUNDED      // Money returned
}

The critical insight: we persist state before calling the payment provider, and update it after we get a response. This way, if we crash mid-processing, we can recover:

@Transactional
public Payment initiatePayment(PaymentRequest request, String idempotencyKey) {
    // Step 1: Create payment record in PENDING state
    Payment payment = Payment.builder()
        .idempotencyKey(idempotencyKey)
        .amount(request.getAmount())
        .state(PaymentState.PENDING)
        .build();
    paymentRepository.save(payment);

    // Step 2: Transition to PROCESSING before calling provider
    payment.setState(PaymentState.PROCESSING);
    paymentRepository.save(payment);

    return payment;
}

// Separate method, possibly async
public void executePayment(Payment payment) {
    try {
        ProviderResponse response = paymentProvider.charge(payment);
        payment.setState(PaymentState.COMPLETED);
        payment.setProviderTransactionId(response.getTransactionId());
    } catch (PaymentDeclinedException e) {
        payment.setState(PaymentState.FAILED);
        payment.setFailureReason(e.getMessage());
    }
    paymentRepository.save(payment);
}

3. Recovery process for stuck payments

A background job finds payments stuck in PROCESSING:

@Scheduled(fixedDelay = 60000)
public void recoverStuckPayments() {
    Instant cutoff = Instant.now().minus(5, ChronoUnit.MINUTES);
    List<Payment> stuck = paymentRepository.findByStateAndCreatedBefore(
        PaymentState.PROCESSING, cutoff);

    for (Payment payment : stuck) {
        // Check with provider if payment actually went through
        ProviderStatus status = paymentProvider.checkStatus(payment.getIdempotencyKey());

        switch (status) {
            case COMPLETED -> payment.setState(PaymentState.COMPLETED);
            case FAILED -> payment.setState(PaymentState.FAILED);
            case NOT_FOUND -> {
                // Provider never received it — safe to retry or fail
                payment.setState(PaymentState.FAILED);
                payment.setFailureReason("Provider timeout — please retry");
            }
        }
        paymentRepository.save(payment);
    }
}

4. Provider abstraction for multi-provider support

public interface PaymentProvider {
    ProviderResponse charge(PaymentDetails details);
    ProviderStatus checkStatus(String idempotencyKey);
    RefundResponse refund(String transactionId, BigDecimal amount);
}

@Service
@Primary
public class RoutingPaymentProvider implements PaymentProvider {
    private final Map<String, PaymentProvider> providers;

    @Override
    public ProviderResponse charge(PaymentDetails details) {
        PaymentProvider provider = selectProvider(details);
        return provider.charge(details);
    }

    private PaymentProvider selectProvider(PaymentDetails details) {
        // Route based on card type, amount, merchant category, etc.
        if (details.getAmount().compareTo(new BigDecimal("10000")) > 0) {
            return providers.get("stripe");  // Lower fees for large transactions
        }
        return providers.get("adyen");  // Better for small transactions
    }
}

What I'd monitor:

Payment success rate (alert if drops below 95%)
Provider latency (p99 should be under 2 seconds)
Stuck payments count (alert if any)
Idempotency key collision rate (should be near zero)

Trade-offs I'm making:

Complexity for reliability — this is more complex than a simple REST call to Stripe
Eventual consistency — the client might get 'processing' and need to poll
Storage overhead — we're keeping full payment history"

Part 4: Behavioral Questions That Actually Matter

Senior roles require leadership and judgment. Here's how to demonstrate both.

"Tell me about a technical decision you regret"

Bad answer: "I can't think of any" (implies you don't reflect on your work)

Good answer:

"Two years ago, I pushed hard for us to adopt microservices. We were a team of six, and our monolith was getting messy. I'd read all the blog posts about how microservices would solve our problems.

We spent four months splitting the monolith into eight services. And honestly? It made everything worse.

Deployments went from 10 minutes to 2 hours because we had to coordinate across services. Debugging a request meant checking logs in five different places. We didn't have the observability infrastructure to support it. And the 'messy monolith' problems? They were still there, just distributed across services now.

What I learned: microservices are an organizational scaling solution, not a technical one. They make sense when you have multiple teams that need to deploy independently. For a team of six working on one product, a well-structured modular monolith is almost always better.

We eventually merged three of those services back. It was painful to admit the mistake, but the team's velocity doubled after we simplified."

"How do you handle disagreements with other engineers?"

Senior approach:

"I had a significant disagreement last year about our caching strategy. A colleague wanted to cache aggressively at the application layer using Redis. I thought we should rely more on database query optimization and HTTP caching.

Here's how I approached it:

First, I made sure I understood their position. Not just what they wanted, but why. Turned out they'd been burned by a slow database at a previous company, so caching was their instinctive solution.

Then I focused the discussion on data, not opinions. I proposed we measure our actual bottlenecks. We profiled a few endpoints and found that 70% of our latency was network calls to external services, not database queries.

We found a compromise based on evidence. We implemented Redis caching for external API responses (where it genuinely helped) but invested in query optimization for our own database. Both of us got part of what we wanted, and the solution was better than either original proposal.

The relationship actually improved. Because I'd taken their concerns seriously and proposed measuring instead of arguing, they trusted my judgment more afterward. We've collaborated well since then.

What I've learned is that technical disagreements are almost never purely technical. There's usually context — past experiences, assumptions, priorities — that shapes someone's position. Understanding that context is often more productive than debating the technical merits."

Final Advice

After conducting hundreds of senior interviews, here's what I look for:

Depth over breadth — I'd rather you know one thing deeply than ten things superficially
Opinions with reasons — "I prefer X because Y" is always better than "I use whatever works"
Humility about mistakes — The best engineers have strong opinions loosely held
Clear communication — If you can't explain it simply, you don't understand it well enough
Curiosity — Do you ask good questions, or just answer what's asked?

The best interview advice I ever got: treat the interview as a conversation between potential colleagues, not an exam. Ask questions. Challenge assumptions. Share your real opinions.

Good luck.

I've also written guides for C# interviews if you work across both ecosystems.

What's the hardest interview question you've faced? Drop it in the comments — I'll add my take.

Why I Built a Centralised Platform Admin in a Microservices Architecture -And What I Learned

Odumosu Matthew — Thu, 19 Mar 2026 07:31:34 +0000

It Started With a Slack Message

Our CTO posted in #engineering at 9:47 AM on a Tuesday:

"Can someone tell me how many active users we have across all products right now?"

Silence. For forty-five minutes.

Not because nobody cared - because nobody knew. The auth service had login counts. The user service had registration numbers. The billing service knew who was paying. But no single place could answer: "How many active users do we have?"

That Slack message cost us a morning of three engineers querying different databases and reconciling numbers in a Google Sheet. That's when we knew we had a problem.

The Microservices Hangover

Here's the thing nobody warns you about in the "Why Microservices" blog posts: decentralisation is fantastic for business logic and terrible for everything else.

We had twelve services. Each one managed its own roles. Its own permissions. Its own metrics. Its own version of the truth. And when anyone outside engineering asked a straightforward question - "Who has admin access?" "What's our uptime?" "When did this permission change?" - the answer was always the same: "Give us a few hours."

We had three specific nightmares:

The Permission Nightmare. Service A thought a user was an admin. Service B disagreed. Both were technically correct according to their own databases. A customer escalation later, we discovered the role had been revoked in one service but not the other. Fun times.

The Dashboard Nightmare. Leadership wanted a single executive dashboard. We had twelve Grafana dashboards, each telling a different story. Our "platform uptime" was a vibes-based estimate that someone calculated manually every Monday morning.

The Audit Nightmare. A compliance review asked: "Show us every admin role change in the last 90 days, who approved it, and why." We spent a week grepping logs across services and reconstructing a timeline in a spreadsheet. The auditor was not impressed.

So we did what any reasonable team would do: we built another microservice. But this one had a different job.

The Platform Admin: One Service to Rule the Cross-Cutting Concerns

The mandate was simple: own the things that get worse when distributed.

Authorization - One source of truth for roles and permissions. Every service asks us: "Can this user do this thing?" We answer in under 10ms.
Observability - One place where metrics from every service converge into a single dashboard.
Audit - One immutable, append-only log of every permission change, role assignment, and admin action across the entire platform.

Not a God service. Not a monolith in disguise. A focused service with a clear boundary: cross-cutting platform governance.

Here's what we learned building it.

Bet #1: Let Services Push Metrics (Don't Pull)

Our first instinct was to have the Platform Admin poll every service. Hit their health endpoints, scrape their stats, aggregate.

Then someone on the team asked the obvious question: "What happens when the auth service is down and we're trying to pull its uptime metric?"

Right. The monitoring system can't depend on the availability of the things it's monitoring. That's not architecture — that's a comedy sketch.

So we flipped it. Every service pushes its own metrics to us via POST endpoints. Auth service reports its call counts. Payment service reports its API volumes. User service reports active users. We receive, validate, store, and serve.

Auth Service ──→ POST /ingest/subsystem-metrics ──→ Platform Admin DB
Payment Service ──→ POST /ingest/api-calls ──→ Platform Admin DB
User Service ──→ POST /ingest/active-users ──→ Platform Admin DB

Frontend ←── GET /dashboard/executive-summary ←── Platform Admin DB

The beauty of this: we don't need to know about new services in advance. A team spins up a new microservice, starts posting to our ingestion endpoint, and their metrics appear on the dashboard. Zero configuration on our side.

And when a service goes silent? Its metrics expire via TTL. The absence of data is the signal. No heartbeat checks needed.

Bet #2: One Table for All Metrics (Yes, Really)

We needed to serve seven different metric types on the dashboard. The obvious move was seven database tables with seven migrations and seven sets of CRUD operations. Then someone on the team muttered: "And next quarter, when product wants three more metric types...?"

Instead, we went with a single table. One row per metric. A composite string key, a JSONB value column, and a TTL timestamp. That's it.

Want to add a new metric type? Create a DTO, pick a key prefix, and start writing. No migration. No PR to update the schema. No waiting for the DBA.

"But you can't query individual fields inside the JSON!"

Correct. And we don't need to. Dashboard data is always read as a complete object and handed to the frontend. We're not running analytical queries against it. We're serialising, storing, and deserialising. The table is a smart cache with persistence, not a relational data model.

Months later, we've added four more metric types. Zero migrations. The table doesn't care.

Bet #3: Validate Tenants at the Gate, Not in Business Logic

Multi-tenant platforms have a sneaky bug pattern: someone passes a tenant ID that doesn't exist, and the system happily returns empty results instead of saying "that's not a real tenant."

We caught this one during testing when someone called our endpoint with /dashboard/metrics/banana/30 and got a 200 OK with zeros. Technically correct — there's no data for the "banana" tenant. Practically useless - it should have been a 400 error.

So we built tenant validation into the request pipeline. Every request with a tenant identifier gets checked against the database before it reaches any business logic. The special keyword all (meaning "aggregate across all tenants") short-circuits and never touches the database.

Since tenant codes change roughly never, we cache the valid set in a HashSet for 10 minutes. The hot path is an O(1) lookup. The cold path hits the database once every 10 minutes. Thirteen endpoints, one validation rule, enforced consistently through FluentValidation in the MediatR pipeline. No endpoint can skip it, even by accident.

Bet #4: CQRS, Because Reads and Writes Live Different Lives

Here's the access pattern for a platform admin:

Writes: Rare, high-stakes. An admin changes a role, ingests metrics, creates an audit entry. Maybe a few hundred writes per hour.
Reads: Constant, latency-sensitive. Permission checks from other services, dashboard queries from the frontend, audit log lookups. Thousands per minute.

Treating these the same way is like designing a library and a nightclub with the same floor plan. They have fundamentally different requirements.

CQRS let us optimise each side independently. The write side does thorough validation, persists carefully, and emits domain events. The read side is aggressively cached and optimised for speed. A permission check from another service hits the cache and returns in single-digit milliseconds. The write path doesn't even notice.

Bet #5: Side Effects Belong in Event Handlers, Not Command Handlers

When an admin assigns a role, here's everything that needs to happen:

Validate the request
Persist the role assignment
Invalidate the permission cache for that user
Send an email notification to the affected user
Notify all admins for visibility
Write an immutable audit log entry
Refresh platform statistics cache

Our first version had all seven steps in the command handler. It was 200+ lines, touched four different infrastructure services, and was a nightmare to test. Mocking the email service to test the cache invalidation? Really?

We refactored to domain events. The command handler does steps 1 and 2 - validate and persist. Then it publishes an event: "A role was assigned." Separate, independent handlers pick up the event and handle their own concern. Cache handler invalidates keys. Notification handler fires emails (fire-and-forget, never blocking the response). Audit handler writes the log.

Now each handler is 20 lines, testable in isolation, and completely unaware of the others. Adding a new side effect - say, posting to a Slack channel, means adding a new event handler. No existing code changes.

Bet #6: Don't Make Reads Pay for Expired Data

Metrics have a TTL. The lazy approach: add WHERE ExpiresAt > NOW() to every query. It works... until you have 50,000 expired rows cluttering the table and your index scans are doing unnecessary work.

We built a background service that wakes up every 30 minutes, deletes expired metrics in batches, and goes back to sleep. Batch size is capped so a single cleanup cycle can't lock the table. If it hits the limit, it logs a warning so ops can tune the configuration.

The metrics table stays lean. Queries stay fast. And instead of every read request paying a tiny tax to filter garbage, a background job handles it once in bulk. It's not clever — it's just good housekeeping.

Bet #7: Design the Audit Log for the Auditor, Not the Developer

Our first audit log was a message text column with entries like: "User admin@company.com assigned role Admin to user john@company.com". Human-readable. Unsearchable. Useless for compliance.

The redesigned audit log captures:

Who initiated the action (with IP address)
Who was the target
What changed (previous state → new state, as structured JSON)
When it happened
Why (a mandatory reason field)
Whether it succeeded or failed

It's append-only. No updates. No deletes. Not even soft deletes. Once a record exists, it exists forever. That's not a technical constraint — it's a compliance requirement. When an auditor asks "has anyone tampered with these logs?" the answer needs to be architecturally impossible, not "we promise we didn't."

We also added CSV export, because auditors don't want to learn your API. They want to open a file in Excel, apply some filters, and go home. Meeting your users where they are applies to internal users too.

The Caching Philosophy That Changed Everything

Here's what I wish someone had told me earlier: caching isn't a performance optimisation. It's an architectural decision.

When fifteen microservices are calling your permission endpoint on every request, the difference between 200ms and 5ms is the difference between a platform that feels snappy and one that feels sluggish. And the bottleneck isn't your code — it's the database round trip.

Our caching strategy is tiered by how often data actually changes:

Data	Cache TTL	Invalidation
User permissions	5 min	Immediate on role change (domain event)
Reference data (role list)	30 min	On any role modification
Dashboard metrics	5 min	On ingestion writes
Tenant codes	10 min	Almost never changes

The critical insight: TTLs are safety nets, not invalidation strategies. When we know data changed (because we just processed the command), we invalidate immediately via domain events. The TTL only exists for the edge case where an event gets lost or a cache entry somehow becomes stale.

This gives us sub-10ms permission checks in production. Other services call us on every request and barely notice the network hop. That's not optimisation - that's making centralised auth viable.

Honest Mistakes We Made

We bolted on audit logging after building RBAC. Retrofitting event publishing into existing command handlers was a week of tedious work. If we'd designed the audit schema first, every handler would have been built with events from day one. Lesson: compliance requirements should shape your architecture, not be stapled on later.

We let each team define their own ingestion payload. One team used count, another used total, a third used value. Same concept, three different field names. A shared contract or schema registry would have saved us a normalisation sprint.

We underinvested in cache invalidation testing. Our unit tests mocked the cache and passed beautifully. The bugs lived in timing — a role change followed by an immediate permission check, where the cache hadn't been invalidated yet. These only surfaced under concurrent load testing. Test your cache invalidation under race conditions, not just in isolation.

What It Looks Like Today

Thirteen endpoints. Six ingestion (microservices pushing data in), seven dashboard (frontend pulling data out). Every single one has:

Tenant validation against the database
Input validation in the request pipeline
JWT authentication
In-memory caching with event-driven invalidation
Structured logging
Automated TTL cleanup

Our CTO's question - "How many active users do we have?" — now takes about 3 seconds to answer. Open the dashboard. It's right there.

TL;DR for the Architects

Centralise cross-cutting concerns, decentralise business logic. Auth, audit, and observability get worse when spread across services. Give them a home.
Push over pull for metrics. The monitoring system shouldn't depend on the availability of what it's monitoring.
Schema flexibility beats relational purity for metrics. A key-value store with JSONB means zero migrations when product wants new dashboard widgets.
Event-driven cache invalidation, not just TTLs. You know when data changes. Act on it immediately.
Design for the auditor. Immutable logs, CSV exports, structured change records. It feels like overhead until the compliance review. Then it's the best thing you built.
Caching is architecture. When your service is in the critical path of every other service, response time isn't a nice-to-have — it's existential.

Building platform infrastructure isn't the kind of work that gets you on stage at a conference. Nobody's tweeting about your audit log schema. But the next time an auditor asks "who had admin access on March 3rd?" and you answer in three seconds flat — or when the CEO opens one dashboard instead of pinging five engineering leads — you'll know it was worth it.

Built with .NET 9, Clean Architecture, CQRS, Entity Framework Core, and PostgreSQL.

DEV Community: Odumosu Matthew

A practical guide to race conditions and five ways engineers make sure it's never both

Two Travellers, One Hotel Room, the Same Second: Who Gets It?

What is a race condition?

Fix 1: Make check-and-act a single atomic step

Fix 2: Pessimistic locking

Fix 3: Optimistic concurrency

Fix 4: Let the database be the final referee

Fix 5: Hold, then confirm

Which fix should you use?

How to catch race conditions before production

What James sees

Key takeaways

Money In Freely, Money Out Carefully: The 24-Hour Rule That Stops Fraudsters

How payment systems protect people from fraud without turning customers away

The fraudster's playbook

The rule: let money in, slow money out

The engineering behind it

1. Enforce it on the right path

2. Measure a rolling window, not a single transaction

3. Keep the rule outside the code

4. Beware of two withdrawals at the same moment

5. Say "not yet" kindly and clearly

6. Leave an audit trail

The balance every payment system has to strike

Key takeaways

Message queues and asynchronous processing, explained through a trip to the dry cleaner

Why Your Bank App Says "Successful" Before the SMS Arrives

What must happen now, and what can happen next

How a message queue works

The rules experienced engineers follow

1. Make every consumer idempotent

2. Retry with backoff, then park it in a dead letter queue

3. Choose your keys carefully when order matters

4. Watch the backlog

5. Don't lose the message between the database and the queue

When not to use a queue

What the customer feels

Key takeaways

API Versioning: How to Evolve Your API Without Breaking Existing Clients

Backpressure in System Design: How to Handle Traffic Spikes Without Crashing

Race Conditions in Production: When Two Requests Compete for One Resource

THE 24 - HOUR ACCOUNT RULE IN TECH

Why Records Are the Better Choice for Clean, Modern, and Principle-Driven .NET Development

🚚 What Are DTOs in Practice?

🔍 The Real Problems with Class-Based DTOs

1. Mutability is a liability at boundaries

2. Reference equality is the wrong equality

3. The boilerplate hides the contract

4. Nothing communicates intent

✨ Enter Records: The Type That Says What It Means

🏦 Records in the Wild: Enterprise Fintech Scenarios

Scenario 1: Idempotent webhook ingestion

Scenario 2: Event-driven architecture on Kafka

Scenario 3: Maker-checker workflows

Scenario 4: End-of-day reconciliation

🧭 How Records Fulfill Software Engineering Principles

Single Responsibility Principle (SRP)

Open/Closed Principle (OCP)

Liskov Substitution Principle (LSP)

Interface Segregation Principle (ISP)

Dependency Inversion Principle (DIP)

YAGNI (You Aren't Gonna Need It)

DRY (Don't Repeat Yourself)

Principle of Least Astonishment

Fail Fast

DDD: Records are Value Objects

CQS/CQRS alignment

⚡ Performance: What Records Actually Buy You (and What They Don't)

Where records genuinely help

Where you must be honest about costs

⚠️ Limitations and Sharp Edges (Read This Before You Migrate)

1. Do not use records for EF Core entities

2. Collection properties break value equality

3. with is a shallow copy

4. Serializer and framework friction

5. Inheritance equality can surprise you

6. Wide records get unwieldy

7. Migration is not free in a brownfield

🧰 When a Class Is Still the Right Answer

3. `with` is a shallow copy

3. `with` is a shallow copy

The `equals()` and `hashCode()` Question