DEV Community: Williams Ashibuogwu

The Happy Path Is Lying to You: What Production Systems Taught Me About Idempotency, Reversals, and Financial Truth

Williams Ashibuogwu — Wed, 07 Oct 2026 21:08:24 +0000

There is a point where a software project stops being "an app."

The UI might still look the same.

There are forms, tables, buttons, APIs, and database rows.

But suddenly a bug does not mean:

The page displayed the wrong thing.

It means:

The system charged twice.

Or:

The accounting entry was duplicated.

Or:

Someone reversed a transaction they were not authorized to touch.

Or:

The balance shown to the user was stale, but the system treated it as authoritative.

That changes the way you have to build software.

Over the last few weeks, I have been working on two production systems from very different domains.

One handles business operations such as sales, inventory, accounting, reporting, staff permissions, and branch activity.

The other handles accounts, balances, transfers, pricing, reconciliation, and asynchronous financial events.

Different products.

Same engineering problem.

Once software starts carrying real consequences, the main job is no longer creating features.

The job is preserving truth when the system is under pressure.

This post is about some of the patterns that become important at that point.

CRUD works until your rows start meaning something

A typical application starts simply:

Create
Read
Update
Delete

You build a few resources.

Then you add authentication.

Then permissions.

Then reports.

Then background jobs.

Then integrations.

Eventually, one of those database rows starts representing something that matters outside the database.

A stock movement affects whether an item can be sold.

A payment affects receivables.

A journal entry changes financial statements.

A transfer affects someone's available money.

At that point, this becomes dangerous:

UPDATE transactions
SET amount = 125000
WHERE id = 42;

Why?

Because you may have just changed history.

That transaction might already have affected:

an account balance
a journal
a report
a notification
an external transfer
a reconciliation process

The system now needs something stronger than CRUD.

It needs invariants.

Invariants are the real feature

For a manual accounting entry, the screen can be simple.

Account	Debit	Credit
Cash	50,000	0
Sales Revenue	0	50,000

The hard part is not rendering those inputs.

The hard part is guaranteeing that the data can never become financially impossible.

For example:

function validateJournal(lines: JournalLine[]) {
  if (lines.length < 2) {
    throw new Error("A journal requires at least two lines");
  }

  let debit = 0n;
  let credit = 0n;

  for (const line of lines) {
    const hasDebit = line.debit > 0n;
    const hasCredit = line.credit > 0n;

    if (hasDebit === hasCredit) {
      throw new Error(
        "Each line must contain either a debit or a credit"
      );
    }

    debit += line.debit;
    credit += line.credit;
  }

  if (debit === 0n || debit !== credit) {
    throw new Error("Journal is not balanced");
  }
}

That validation is useful.

It is not enough.

The system still needs to know:

Is the account active?
Can this account receive manual postings?
Is the accounting period open?
Does the user belong to this branch?
Has this exact operation already been posted?
Did the account state change after the form loaded?

That last question is where things get interesting.

Validate again at commit time

Imagine a finance user opens a posting screen at 2:00 PM.

The accounting period is open.

They prepare a long journal.

At 2:05 PM, an administrator closes the accounting period.

At 2:07 PM, the user clicks Post.

If the application only validated the period when the form loaded, you now have a posting inside a closed period.

The rule needs to be enforced inside the transaction that performs the write.

Conceptually:

await db.transaction(async tx => {
  const period = await tx.accountingPeriods
    .where({ id: input.periodId })
    .forUpdate()
    .first();

  if (!period || period.status !== "OPEN") {
    throw new Error("Accounting period is closed");
  }

  const accountIds = [
    ...new Set(input.lines.map(line => line.accountId))
  ].sort();

  const accounts = await tx.accounts
    .whereIn("id", accountIds)
    .orderBy("id")
    .forUpdate();

  for (const account of accounts) {
    if (!account.active || !account.manualPostingAllowed) {
      throw new Error("Account cannot accept this posting");
    }
  }

  await createJournal(tx, input);
});

The important idea is not the syntax.

It is this:

A rule that matters financially should be checked as close as possible to the write that makes the operation permanent.

The check-then-insert race

Here is one of my favorite production bugs because the code often looks completely reasonable.

const existing = await db.journals.findFirst({
  where: {
    sourceType,
    sourceId,
    sourceEvent
  }
});

if (existing) {
  return existing;
}

return db.journals.create({
  data: {
    sourceType,
    sourceId,
    sourceEvent
  }
});

Looks fine.

Until two requests arrive together.

Request A

Checks.

Nothing exists.

Request B

Checks.

Nothing exists.

Request A

Inserts.

Request B

Also tries to insert.

If the database has no uniqueness constraint, congratulations:

You now have two financial effects.

If the database does have a uniqueness constraint, that is much better.

But Request B may still receive an ugly database exception even though the operation actually succeeded.

The stronger implementation treats the constraint as part of the control flow.

try {
  return await db.journals.create({
    data: {
      sourceType,
      sourceId,
      sourceEvent
    }
  });
} catch (error) {
  if (!isUniqueViolation(error)) {
    throw error;
  }

  const winner = await db.journals.findFirst({
    where: {
      sourceType,
      sourceId,
      sourceEvent
    }
  });

  if (!winner) {
    throw error;
  }

  return winner;
}

And at the database layer:

CREATE UNIQUE INDEX journal_source_identity_unique
ON journals (
  source_type,
  source_id,
  source_event
);

Now the database decides who wins.

The application knows how to handle the loser.

Important: Application checks can reduce unnecessary conflicts. Database constraints enforce reality.

Idempotency is not a frontend feature

A common mistake is thinking this solves duplicate operations:

button.disabled = true;

It helps UX.

It does not solve duplication.

A user can still:

open two browser tabs
resend an HTTP request
refresh after a timeout
trigger a mobile retry
cause a queue redelivery
receive the same webhook twice

A consequential operation should have a stable identity.

For example:

{
  "operation_id": "01K7H4N9S9P1D9JYC0H7...",
  "amount_minor": 250000,
  "currency": "NGN"
}

Then persist that identity.

CREATE UNIQUE INDEX transfer_operation_unique
ON transfers(operation_id);

Retries can now become safe:

const previous = await transfers.findByOperationId(operationId);

if (previous) {
  if (previous.requestHash !== hash(input)) {
    throw new ConflictError(
      "Operation ID was already used with different data"
    );
  }

  return previous;
}

This gives you two useful behaviors:

same key + same payload
=> return previous result

same key + different payload
=> reject

That is much stronger than blindly returning:

Already exists

Reversals are better than rewriting history

Suppose a journal entry is wrong.

The tempting solution:

UPDATE journal_lines
SET debit = 0, credit = 100000
WHERE id = 9001;

Now you have a clean-looking current state.

But you lost historical truth.

A better model:

Journal #123
Debit Cash                 100,000
Credit Revenue             100,000

Reversal #456
Credit Cash                100,000
Debit Revenue              100,000

The original event still exists.

The correction exists separately.

They are linked.

type Journal = {
  id: string;
  reversedBy?: string;
  reverses?: string;
};

And the reversal command itself should be idempotent.

async function reverseJournal(id: string) {
  return db.transaction(async tx => {
    const original = await tx.journals
      .where({ id })
      .forUpdate()
      .first();

    if (!original) {
      throw new NotFoundError();
    }

    if (original.reversedBy) {
      return tx.journals.find(original.reversedBy);
    }

    const reversal = await createOppositeJournal(
      tx,
      original
    );

    await tx.journals.update(original.id, {
      reversedBy: reversal.id
    });

    return reversal;
  });
}

Retrying the reversal should not produce:

reverse
reverse
reverse
reverse

It should return the one reversal that already exists.

Financial history should be append-heavy

This pattern is useful far beyond journals.

Payments, refunds, voids, settlement corrections, and adjustments can all be represented as related events.

That makes the system much easier to explain later.

Why does this balance equal ₦42,300?

Instead of saying:

> Because that is the value currently stored in the balance column.

The system can explain:

Opening balance         +100,000
Purchase                 -35,000
Transfer                 -20,000
Fee                       -1,500
Refund                    +5,000
Adjustment                -6,200
--------------------------------
Current balance           42,300

That explainability becomes extremely important once support, finance teams, auditors, and users start asking questions.

Your browser should not decide how much money moves

Suppose a transfer has:

Amount:       10,000
Network fee:     100
Service fee:      50
Total debit:  10,150

Do not let the client invent the final debit.

This is weak:

const total = amount + networkFee + serviceFee;

await api.post("/transfer", {
  amount,
  total
});

Why trust total?

The client is not authoritative.

A stronger flow looks like this.

Step 1: Create a quote

POST /transfer-quotes
Content-Type: application/json

{
  "amount_minor": 1000000,
  "destination_id": "acct_123"
}

Server response:

{
  "quote_id": "quote_abc",
  "amount_minor": 1000000,
  "fee_minor": 15000,
  "total_debit_minor": 1015000,
  "expires_at": "2026-10-07T20:30:00Z"
}

Step 2: Display the quote

The UI renders what the server calculated.

Step 3: Authorize that exact debit

POST /transfers
Content-Type: application/json

{
  "quote_id": "quote_abc",
  "authorization": {
    "type": "pin",
    "proof": "..."
  }
}

The backend verifies the quote itself.

Not a client-provided total.

The server should always be able to explain exactly what amount the user authorized.

Minor units deserve boring code

Money bugs are often surprisingly boring.

Consider:

const minimum = 50000;

What does that mean?

₦50,000?

₦500?

50,000 kobo?

Without a convention, the code is ambiguous.

I prefer names like:

const minimumMinor = 50_000n;

Formatting happens only at the presentation edge.

function formatMoney(minor: bigint): string {
  return new Intl.NumberFormat("en-NG", {
    style: "currency",
    currency: "NGN"
  }).format(Number(minor) / 100);
}

Arithmetic stays integer-based:

const totalDebitMinor =
  amountMinor +
  serviceFeeMinor +
  networkFeeMinor;

Not:

const total = 100.1 + 20.2;

You do not want floating-point surprises anywhere near money.

Reconciliation is not cleanup

External systems introduce an uncomfortable state.

You send a transfer request.

Then:

Your request times out.

What happened?

You do not know.

That is different from failure.

This can be dangerous:

try {
  await sendTransfer(input);
} catch {
  transfer.status = "FAILED";
}

The external system may have accepted the transaction before the connection disappeared.

Now your local system says:

FAILED

while the external system later says:

SUCCESS

Then the user retries.

The original transfer eventually completes.

The retry completes too.

A better state model includes uncertainty.

type TransferStatus =
  | "PENDING"
  | "SUBMITTED"
  | "UNKNOWN"
  | "CONFIRMED"
  | "FAILED";

After an ambiguous timeout:

transfer.status = "UNKNOWN";

Then reconciliation decides the final state.

const external = await fetchTransferStatus(
  transfer.externalReference
);

switch (external.status) {
  case "successful":
    await settleExactlyOnce(transfer, external);
    break;

  case "failed":
    await markFailed(transfer);
    break;

  default:
    await scheduleAnotherCheck(transfer);
}

UNKNOWN is a valid state.

Pretending uncertainty does not exist is not.

Webhooks can arrive twice

Actually, assume they will.

This is dangerous:

app.post("/webhook", async req => {
  await creditAccount(req.body);
});

A webhook sender may retry because:

your server returned 500
the connection closed too early
they did not receive your acknowledgement
delivery is intentionally at-least-once

A safer approach:

await db.transaction(async tx => {
  const exists = await tx.webhookEvents.findUnique({
    where: {
      eventId: event.id
    }
  });

  if (exists) {
    return;
  }

  await applyFinancialEffect(tx, event);

  await tx.webhookEvents.create({
    data: {
      eventId: event.id,
      processedAt: new Date()
    }
  });
});

Plus a database constraint:

CREATE UNIQUE INDEX webhook_event_unique
ON webhook_events(event_id);

Again, the database is part of your correctness model.

Stale data can be worse than no data

One particularly interesting class of bugs happens when the application already has a value.

For example:

{
  "balance_minor": 0
}

The application sees that and assumes:

Balance is zero.

But that value may have been captured hours ago.

Meanwhile, a more recent confirmed event says funds exist.

The local value is not missing.

It is stale.

That can be more dangerous because it looks authoritative.

A better representation carries metadata.

type BalanceSnapshot = {
  amountMinor: bigint;
  observedAt: Date;
  source: "ledger" | "external-confirmation" | "cache";
};

Now the application can reason about evidence.

function chooseBalance(
  a: BalanceSnapshot,
  b: BalanceSnapshot
) {
  if (
    a.source === "external-confirmation" &&
    b.source === "cache"
  ) {
    return a;
  }

  return a.observedAt > b.observedAt ? a : b;
}

The real implementation may be more complicated.

The important lesson is simpler:

Presence does not imply authority.

Authorization belongs inside the domain

A UI might hide a reversal button.

That is not security.

This:

{user.canReverse && (
  <button>Reverse</button>
)}

is purely presentation.

The real check must happen server-side.

async function reverseEntry(user, entryId) {
  const entry = await entries.find(entryId);

  if (!user.permissions.includes("journal.reverse")) {
    throw new ForbiddenError();
  }

  if (entry.branchId !== user.branchId) {
    throw new ForbiddenError();
  }

  return reversalService.reverse(entry);
}

Notice that there are two separate questions:

Can this user reverse entries?

Can this user reverse this specific entry?

The second one is where many authorization bugs appear.

Lock rows in a deterministic order

Concurrency can also create deadlocks.

Suppose one transaction locks:

Account 8
then Account 12

while another locks:

Account 12
then Account 8

Both can wait on each other.

One useful mitigation is deterministic ordering.

const ids = [...new Set(accountIds)]
  .sort((a, b) => a.localeCompare(b));

const accounts = await tx.accounts
  .whereIn("id", ids)
  .orderBy("id")
  .forUpdate();

Every transaction tries to acquire the same resources in the same order.

Not every deadlock disappears.

But you remove an entire class of avoidable ones.

UI correctness is still correctness

Not every production bug is deep distributed-systems theory.

One issue looked approximately like this:

<div class="product-code text-xs">
  ABC-123
</div>

The data existed.

The backend returned it.

The test found it in the HTML.

But an old CSS rule contained:

.product-row .text-xs {
  display: none;
}

The automated test said:

PASS

The user said:

Where is the product code?

Both observations were technically correct.

That is why this test:

expect(html).toContain("ABC-123");

is not equivalent to:

await expect(
  page.getByText("ABC-123")
).toBeVisible();

Sometimes the correct layer to test is the browser.

A passing request does not always mean a successful operation

Another pattern worth separating is transport success from business success.

Suppose an API call returns HTTP 200:

{
  "status": "accepted"
}

That might only mean:

The request was accepted for processing.

It may not mean:

The financial operation is complete.

This distinction should exist in your own models too.

type OperationState =
  | "CREATED"
  | "AUTHORIZED"
  | "SUBMITTED"
  | "PROCESSING"
  | "SETTLED"
  | "FAILED";

Collapsing everything into:

success = true

removes useful information.

For asynchronous operations, state machines are usually more honest than booleans.

Think in commands, not just endpoints

A useful shift for consequential systems is to stop thinking only in HTTP routes.

Instead of:

POST /journal
POST /transfer
POST /refund

think:

PostJournal
SubmitTransfer
ReverseJournal
RefundPayment
SettleTransfer

A command can have:

an operation identity
authorization rules
invariants
state preconditions
side effects
an idempotency policy
an audit trail

For example:

type PostJournalCommand = {
  operationId: string;
  actorId: string;
  branchId: string;
  postingDate: string;
  lines: JournalLine[];
};

Then the command handler becomes the boundary that protects the operation.

async function handlePostJournal(
  command: PostJournalCommand
) {
  authorize(command);
  validate(command);

  return db.transaction(async tx => {
    return postJournalExactlyOnce(tx, command);
  });
}

The HTTP controller becomes much less interesting.

That is usually a good sign.

A useful retry rule

I increasingly think about operations using this simple question:

If I execute this again, what should happen?

There are only a few acceptable answers.

Read operation

Execute again
=> return current data

Idempotent command

Execute again with same identity
=> return previous result

Conflicting retry

Same identity + different payload
=> reject

Explicitly repeatable command

Execute again
=> intentionally create another operation

The dangerous answer is:

Execute again
=> maybe duplicate something, depends on timing

What I now test by default

For consequential features, I increasingly start with hostile questions.

What if the request arrives twice?

What if two workers run this simultaneously?

What if the external API succeeds but our request times out?

What if the webhook is redelivered?

What if the accounting period closes while the form is open?

What if the same idempotency key is reused with different data?

What if the user belongs to another branch?

What if the client modifies the fee?

What if stale local state disagrees with a newer confirmed event?

What if the correction command is executed twice?

What if the data exists but the UI hides it?

These are not exotic edge cases.

They are normal production behavior.

A practical checklist

When building something that affects money or operational truth, this is the checklist I increasingly reach for:

[ ] Is there a stable identity for this operation?

[ ] Can the same request safely execute twice?

[ ] Is uniqueness enforced in the database?

[ ] Are important state checks repeated at commit time?

[ ] Are concurrent writes considered?

[ ] Are locks acquired consistently?

[ ] Can historical financial data be edited?

[ ] Should correction be represented as a reversal instead?

[ ] Is the server authoritative for financial calculations?

[ ] Are money values stored using exact minor units?

[ ] Can asynchronous events be delivered more than once?

[ ] Is UNKNOWN represented separately from FAILED?

[ ] Is there a reconciliation path?

[ ] Can stale state override fresher evidence?

[ ] Are permissions checked server-side?

[ ] Is object, branch, or account scope enforced?

[ ] Can the system explain how the current balance was produced?

[ ] Are user-visible properties tested in the browser?

If several answers are uncomfortable, that is usually where the interesting engineering work begins.

The real feature is preserving truth

I used to think production hardening happened after the feature.

Build the feature.

Then make it reliable.

I increasingly think that is the wrong mental model.

For consequential systems, reliability is part of what the feature means.

A transfer that succeeds twice is not a working transfer feature.

A journal entry that can be posted into a closed period is not a working journal feature.

A balance that cannot explain its own history is not a trustworthy balance.

A reversal that rewrites history instead of recording correction is not a complete accounting workflow.

A permission system that only hides buttons is not authorization.

The implementation is not finished simply because the happy path works.

It is finished when the system can survive:

retries
timeouts
concurrency
duplicate delivery
stale state
partial failure
correction
unauthorized requests

without inventing a new version of the truth.

That is the part of production engineering I find increasingly interesting.

Not just making the software work.

Making sure it continues telling the truth when everything around it gets messy.

I’m Almost Done Building the Biggest Financial Platform I’ve Worked On

Williams Ashibuogwu — Sat, 05 Sep 2026 06:10:57 +0000

I kept telling myself I would write about this project when I finished it.

I'm still not completely finished.

There are a few user journeys I'm checking again, some UI things I keep noticing at annoying screen sizes, a performance problem I deliberately don't want to "fix" before measuring it properly, and a few pieces of technical debt I already know I'll be coming back for.

But I'm close.

And I think I'd rather write this now, while I'm still inside the build, than six months from now when my brain has conveniently turned the whole experience into a clean sequence of sensible engineering decisions.

Because it wasn't that clean.

I've been shipping almost every day on a financial platform through a pretty intense stretch of development.

I can't name the company, the product, or the financial infrastructure behind it, and there are obviously security and production details that don't belong in a public article.

But I can talk about what it has actually been like to build.

At one point, a repository snapshot looked roughly like this:

700+ commits
1,000+ tracked files
200,000+ tracked lines
3 applications
13 shared packages
500+ tests

I don't think I ever had a day where it felt like:

Today I am contributing to a 200,000-line system.

It felt more like:

Why is this route sending an authenticated user back to login?

Then:

This notification should open the thing it is notifying them about.

Then:

Why can this worker be delayed by marketing jobs?

Then:

This transfer timed out. Did the money actually move?

Then:

This looks terrible on a 390px screen.

Enough of those days happen and suddenly you have a platform.

When the project stopped being "an app"

The repository eventually settled into a TypeScript monorepo with three main applications:

apps/
  api/
  web/
  worker/

and shared packages handling things like:

packages/
  cache/
  config/
  database/
  documents/
  ledger/
  providers/
  public-api/
  queues/
  safety/
  security/
  shared/
  storage/
  tooling/

The frontend is React and TypeScript.

The HTTP layer is Fastify.

PostgreSQL is the primary database.

PgBouncer handles connection pooling.

Valkey sits behind caching and short-lived coordination.

BullMQ runs asynchronous workloads.

Around that are object storage, Caddy, systemd, structured logging, health checks, deployment tooling, API schemas, background workers, and the other things that become necessary once "run the server" stops being a deployment strategy.

None of those technology choices are particularly exotic.

The interesting part was what the product gradually demanded from them.

What started as one financial application eventually included customer onboarding, account approval, KYC, financial accounts, transfers, FX, cards, bill payments, statements, notifications, partner workflows, internal operations, support tooling, developer-facing APIs, public documentation, a PWA, and some separately privileged product areas.

Different users.

Different access boundaries.

Different kinds of state.

Different kinds of money movement.

Different failure modes.

One system.

At the beginning, I was still mostly thinking in features.

Build onboarding.

Build the dashboard.

Add transfers.

Add notifications.

Wire up the provider.

Make the mobile version work.

The problem is that every one of those features brought more than a screen with it.

Transfers brought retries, timeouts, provider state, reconciliation, authorization, limits, notifications, and audit.

Notifications brought preferences, retries, delivery channels, deep links, and several applications generating different kinds of events.

Partner features brought attribution, permissions, workflow state, and the problem of showing enough information to make a relationship useful without leaking private customer data.

Eventually the implementation had grown enough that I stopped trusting the shape of the original application.

Not because every part was bad.

That actually made the decision harder.

There is a point in a project where almost every problem still looks individually fixable.

A route is broken.

Fix it.

An authorization check is awkward.

Fix it.

One provider call is living in the wrong place.

Move it.

A page breaks on mobile.

Patch the CSS.

A workflow has a weird state.

Add another condition.

Every individual fix can be reasonable while the overall shape of the application gets worse.

Eventually I reached the point where I didn't trust enough of the assumptions underneath the implementation.

So I froze that development state.

I kept the infrastructure, but started the application layer again on a clean development environment.

Not because rewrites are fun.

They're not.

It was more:

I don't want to keep making stronger promises on top of a foundation I no longer fully trust.

The original implementation plan had 24 engineering sprints.

That eventually became 30.

Not because I suddenly wanted six more sprints, but because areas like partner operations, developer APIs, documentation, PWA behaviour, and final hardening turned out to be much larger domains when I tried to make them complete instead of merely present.

That gave me one of the first lessons from the project that really stuck:

Sometimes scope grows because people keep adding things. Sometimes scope grows because you finally understand what the original sentence actually implied.

Those are very different problems.

Rebuilding the foundation changed what I meant by "reliable"

One of the first clean-build deployments failed.

I like that story now.

I did not like it when it happened.

The useful part wasn't that the deployment failed.

The useful part was that the release machinery did what it was supposed to do.

The new state wasn't copied over the running application and left there half-working. The release stopped and the previous known-good state came back.

The real deployment tooling is more involved, but conceptually the release flow became something like:

release="$(create_release)"

build_into "$release"
run_checks "$release"

switch_atomically "$release"

if ! healthcheck; then
  rollback
fi

I had already written "rollback" into the architecture.

That failure was the moment I knew rollback actually existed.

There is a difference.

I also learned a smaller operational detail by watching enough deployments: the first health probe after switching a release doesn't always return 200 immediately.

Sometimes the process needs another attempt or two while it finishes warming up.

The first few times that happened, I treated it like something was broken.

Eventually the deployment tooling learned what normal startup behaviour actually looked like.

That's one of those boring things you don't really learn from drawing an architecture diagram.

Architecture tells you how you think the system behaves.

Operations tells you how it actually behaves.

That experience also made me stricter about apparently small changes.

At one point CI went red because Prettier wanted a multi-line import collapsed back onto one line.

That was basically the entire behavioural impact of the commit.

Before:

import {
  AppSessionLoader,
  PushPermissionDecision,
  RequestCard,
  Shell,
  icon,
} from "./MemberPortal";

After:

import { AppSessionLoader, PushPermissionDecision, RequestCard, Shell, icon } from "./MemberPortal";

No business behaviour changed.

The gate was still red.

So the formatting change got synced, built, deployed, and verified like everything else.

At the time I thought:

This is ridiculous.

Then I thought about the alternative.

You start deciding that some changes are too small to deserve the normal release process.

Then somebody has to decide what "small" means.

Then exceptions start accumulating.

So yes.

The one-line import got a proper deployment.

The Git setup behind all of this is less elegant than I'd like.

I deliberately separated the environment where commits are authored from the checkout that handles deployment and remote repository authentication.

That buys credential isolation.

It also means synchronization occasionally takes more thought than a normal one-checkout workflow should. I've had moments where the deployment checkout already contained an uncommitted copy of the exact change I was trying to pull as a commit, forcing me to verify the diffs matched before pulling cleanly.

It works.

I know why it exists.

I still want to simplify it.

That's debt. Not hidden debt. Just something useful enough to tolerate for now.

The bigger shift was that I started thinking about reliability less as:

Does this code work?

and more as:

What does the system do when this code doesn't get the outcome it expected?

That question became much more serious once money movement entered the picture.

Money movement forced me to stop thinking in success and failure

From the UI, a transfer looks like:

recipient
amount
confirm

The backend version is much less pleasant.

Suppose I submit a transfer to an external financial provider.

Then my connection disappears.

Did it fail?

Maybe.

Did the provider receive the request?

Maybe.

Did they execute it successfully and the response disappear somewhere between their system and mine?

Also maybe.

The one thing I definitely cannot do is immediately submit the transfer again.

A naïve version of the flow wants to look like this:

const result = await provider.transfer(input);

if (!result.ok) {
  throw new Error("Transfer failed");
}

return { status: "success" };

That code wants the world to have two states.

Success.

Failure.

Financial systems occasionally respond:

lol, no.

The application eventually needed stable operation IDs, idempotency keys, request hashes, provider references, and recoverable operation states.

The mental model became much closer to:

type OperationState =
  | "submitting"
  | "accepted"
  | "processing"
  | "checking"
  | "completed"
  | "failed"
  | "reversed"
  | "unknown"
  | "reconciling";

The state I care about most there is unknown.

Earlier in my career, I think I would have treated unknown as a failure to finish implementing the workflow.

Now I think it can be one of the most honest states in a distributed financial system.

If I genuinely do not know whether money moved yet, the application should not invent an answer because red or green is easier to render.

That same problem changed how I thought about webhooks.

A webhook implementation initially looks very simple:

app.post("/webhook", async (request) => {
  await updateTransaction(request.body);
  return { ok: true };
});

Then you spend five minutes thinking like someone trying to break it.

Who sent that body?

Has it been modified?

How old is it?

Have I seen this event before?

Is it from the right environment?

What if the provider sends it three times?

What if they retry because my server took too long to answer?

What if the webhook says a transaction completed but the amount or reference does not match what I expected?

Eventually the flow looked much more like:

receive
  ↓
validate request shape
  ↓
verify signature + timestamp
  ↓
reject replay
  ↓
persist event
  ↓
acknowledge quickly
  ↓
process asynchronously
  ↓
verify against provider when necessary
  ↓
apply financial effect

And reconciliation stopped pretending that every incoming event could immediately become success or failure.

A shortened, anonymised version of a pattern in the codebase looks like this:

type ReconcileOutcome =
  | { action: "processed"; postingId: string; duplicate: boolean }
  | { action: "skipped"; reason: string }
  | { action: "needs_review"; reason: string }
  | { action: "error"; reason: string };

That needs_review state matters.

For example, if I receive one reference and the authoritative lookup comes back with something else:

if (verifiedReference !== reference) {
  return {
    action: "needs_review",
    reason: "Provider verification reference mismatch",
  };
}

Don't guess.

Don't force it through because it probably belongs to that transaction.

Don't silently throw it away either.

Put it somewhere the system can reason about it safely.

The more I worked on this, the more I realised that a financial platform needs to be comfortable saying:

I have received information.

without automatically saying:

Therefore this is now financial truth.

Those are different statements.

And that distinction led directly to another architectural problem.

The more financial operations I added, the easier it would have been to let the external provider define the whole application.

Need an account?

Call their account endpoint.

Need a transfer?

Call their transfer endpoint.

Need FX?

Call that endpoint.

Need a card?

Same thing.

That is extremely productive at the beginning.

Then one day your controllers know provider-specific field names, your workers know provider-specific status strings, your frontend knows which provider can do what, and half your business logic is written in the vocabulary of a company you don't control.

I didn't want that.

So financial integrations ended up behind a provider boundary.

Conceptually:

application/domain logic
          ↓
provider capability + routing layer
          ↓
provider-specific adapter
          ↓
external infrastructure

An early implementation could easily have ended up with logic like:

if (currency === "USD") {
  showTransferButton();
}

Except "does the provider support USD?" is not actually a useful question.

USD for what?

Account issuance?

Holding a balance?

Inbound transfer?

Outbound transfer?

FX?

Cards?

Bills?

A provider can support one and not another.

Or expose something technically while that capability is not enabled for your account.

So support had to become contextual.

The provider layer can ask something closer to:

registry.supports("banking_rail", {
  country,
  currency,
  customerType,
});

with the underlying logic checking capability, geography, currency, customer type, and whether that declaration is enabled.

Something roughly like:

if (!declaration.enabled) return false;
if (declaration.capability !== capability) return false;

if (
  declaration.countries !== "ALL" &&
  !declaration.countries.includes(country)
) {
  return false;
}

if (
  declaration.currencies !== "ALL" &&
  !declaration.currencies.includes(currency)
) {
  return false;
}

return declaration.customerTypes.includes(customerType);

There is another detail I particularly like.

Once an existing financial relationship has been created with a particular provider, changing the routing preference for new relationships should not silently move that existing one somewhere else.

So existing relationships can be pinned.

New ones can follow the current routing rules.

Old ones continue using the provider they were actually created with.

That sounds obvious now.

It wasn't something I was thinking about before I had to design a system that could survive changing providers.

Once correctness mattered, infrastructure became part of product behaviour

The financial workflows created more asynchronous work.

Provider events.

Reconciliation.

Security messages.

Notifications.

Documents.

Exports.

Analytics.

At first, the normal queue question was:

Does this need to happen outside the HTTP request?

Useful question.

Not enough.

The more important question became:

What is this job allowed to wait behind?

Imagine thousands of marketing jobs enter the queue.

At the same time, somebody is trying to log in and needs a security email.

Should that security message sit behind the marketing campaign?

Obviously not.

But if they are all just "background jobs," you may have accidentally made exactly that decision.

So the queue layer stopped treating every job as equal.

A simplified version looks like:

const queueClass = {
  critical: [
    "security-email",
    "provider-webhooks",
    "financial-reconciliation",
  ],

  normal: [
    "transactional-email",
    "push-notifications",
    "documents",
  ],

  deferrable: [
    "exports",
    "analytics",
    "marketing-bulk",
    "maintenance",
  ],
};

They don't all get the same retry behaviour either:

const policies = {
  critical: {
    attempts: 8,
    backoff: { type: "exponential", delay: 1000 },
  },

  normal: {
    attempts: 5,
    backoff: { type: "exponential", delay: 2000 },
  },

  deferrable: {
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
  },
};

The specific numbers are not the interesting part.

The interesting part is that queue design became a product decision.

When the system is under pressure, what are we willing to make wait?

An analytics refresh?

Sure.

A login security message?

Much less so.

A financial reconciliation job?

Definitely not because somebody requested a large export.

That's how I think about queues now.

They aren't just a place to put work you don't want inside the request.

They encode priority.

They encode which delays the product considers acceptable.

The database went through a similar change in my head.

PostgreSQL stopped being just "where the data goes."

There are some bugs I don't want application-level if statements to be solely responsible for preventing.

Two requests trying to spend the same available amount.

Two workers processing the same event.

Two people trying to consume the last available unit of something.

An update based on stale state.

Several records that need to change together or not change at all.

So the database became part of the correctness model.

That meant actually using:

unique constraints;
foreign keys;
check constraints;
optimistic version columns;
atomic conditional updates;
row locks;
short transactions;
deterministic lock ordering;
and stronger isolation where a specific invariant justified it.

I knew what SELECT ... FOR UPDATE did before this project.

The project taught me something more useful:

when I actually want it.

Same with serializable transactions.

I knew they existed.

Now I understand much better why I don't want everything running at serializable isolation, and why a few particularly sensitive invariants might actually justify paying that cost.

Knowing the feature and knowing where it belongs are different levels of understanding.

That pattern kept repeating.

The system would grow.

A problem I previously understood academically would become a real operational constraint.

Then I would understand why the boring engineering primitive exists.

The visible feature was almost never the whole feature

Authentication started small too.

At one point, "authentication" basically meant:

login
logout
forgot password

Then the actual requirements arrived.

OTP.

Passkeys.

Device and session management.

Session rotation.

Step-up authentication.

Recovery.

Staff authentication.

Privileged staff authentication.

Trusted devices.

Revocation.

Different session policies.

Authorization became even more important because the system eventually had several application surfaces.

A customer.

A partner.

A developer.

Normal staff.

Privileged staff.

Someone can be allowed to see a record without being allowed to modify it.

A partner may need to know that a referral progressed without being allowed anywhere near that person's banking details.

A customer interface may correctly hide an operation while an API route underneath it accidentally still allows the request.

So I became increasingly annoying about one rule:

A hidden or disabled button is not authorization.

If someone is not allowed to perform an action, the server performing that action has to reject it.

Not the menu.

Not the React component.

Not the disabled state.

The actual boundary that changes or reveals the protected resource.

Frontend visibility is UX.

Authorization is policy.

Notifications followed the same pattern.

What began as "we need notifications" became in-app events, email, push, preferences, retries, deduplication, deep links, and failure handling across several application surfaces.

The useful model was to treat the internal notification as the durable record and email or push as delivery attempts around it.

A failed delivery does not undo the event that caused it, and it definitely does not change financial truth.

And while all of this was happening on the backend, the frontend was teaching me a different version of the same lesson:

software can be technically correct and still be bad.

The mobile topbar, for example, had several lives.

It went from:

full wordmark
      ↓
personalised greeting
      ↓
small icon

The greeting version wasn't a mock-up.

It actually loaded the member's name.

It had responsive typography.

It was built and styled.

Then I looked at it again and the smaller icon was cleaner.

So the greeting disappeared.

I used to think of that kind of work as wasted effort.

I don't anymore.

Sometimes the fastest way to know whether a visual idea works is to build the thing and look at it.

One of my favourite frontend bugs had nothing to do with code correctness.

A dashboard section was redesigned.

The code compiled.

Types passed.

Routes worked.

The layout worked.

Then I looked at the screen and realised that the same phrase appeared three times in the same panel.

Once as the eyebrow.

Once as the heading.

Once again on the button.

Perfectly valid software.

Terrible interface.

No type system is going to catch that.

No unit test is going to tell you your copy hierarchy looks stupid.

Sometimes the correct QA process is literally:

Why does this look stupid?

Mobile made this even more obvious.

There were pages that were technically responsive.

Nothing overflowed.

Every element was visible.

They were still bad mobile interfaces.

Authentication was the clearest example.

On desktop, more explanatory text and a wider composition can look good.

On a phone, especially inside an installed PWA, the same thing can feel like a desktop website somebody squeezed into a smaller rectangle.

So I started removing things.

Less surrounding chrome.

Less unnecessary copy.

Better viewport spacing.

Forms that actually sit comfortably between the screen edges.

Cleaner hierarchy.

OTP fields that look like OTP fields instead of a normal text input.

Navigation designed for thumbs instead of a desktop sidebar awkwardly collapsing.

That changed the question I ask.

Not:

Does this fit on mobile?

But:

If mobile were the only platform, would I have designed it this way?

If the answer is no, then I probably haven't actually designed the mobile version yet.

The same idea connects all of these problems.

A disabled button doesn't mean an action is forbidden.

A page fitting inside the viewport doesn't mean it is a good mobile experience.

A transfer button working doesn't mean the transfer journey is reliable.

The thing I could see was often only the beginning.

"Done" became a much harsher word

Near the end of the build, I started discovering a lot of things that existed but weren't actually complete.

A notification bell could exist without every product surface producing the right events.

A developer portal could look finished while credential rotation, scopes, idempotency, or webhook management were still incomplete.

An admin page could render perfectly while the underlying action lacked the right authorization, audit trail, or failure behaviour.

That was when my working definition of a complete feature became much harsher:

data model
backend action
authorization
validation
state transitions
audit
notifications
failure states
empty states
responsive behaviour
tests
documentation
operational controls

That definition is painful when somebody says:

But the screen is already there. Aren't we almost done?

Sometimes.

Sometimes the screen means you're 60% done.

Testing changed for me too.

The project has more than 500 tests, and substantial behavioural changes still get the appropriate suites.

But I stopped treating "run everything after everything" as synonymous with discipline.

For a presentation-only change, typecheck, build, and actual visual inspection may tell me more than hundreds of unrelated domain tests.

My rule became:

Pay for the verification that can realistically catch the class of mistake you just introduced.

If I changed a ledger invariant, visual QA isn't going to save me.

If I changed a heading from 18px to 20px, hundreds of domain tests probably aren't going to save me either.

And despite all of this, the project isn't magically debt-free because I'm near the end.

The main frontend bundle is still larger than I want it to be.

The bundler complains about it.

Correctly.

I have deliberately not responded by throwing lazy() around random components and declaring the performance problem solved.

There is a proper performance investigation queued.

The bundle is an obvious place to start, but I want to measure the actual user-visible latency before deciding what needs to change.

There is also an authenticated dashboard component that has accumulated too many responsibilities, an icon fallback that is resilient but can hide misspelled glyph names, and some operational cleanup I deliberately postponed until the feature pressure drops.

I have debt.

I know where it is.

I know why it exists.

I've become much more comfortable with the difference between:

unknown, accidental debt

and:

known, bounded debt that I deliberately chose not to fix today.

The second kind still needs paying.

But at least I know what I owe.

The part that changed how I engineer

I've tried to work out which part of the project had the biggest impact on me.

The transfer system?

The provider abstraction?

The ledger?

Security?

Partner workflows?

Developer APIs?

Deployment?

The PWA?

The queue architecture?

I don't think it is one feature.

The bigger job was repeatedly taking vague product statements and turning them into rules a system could actually enforce.

Take:

Users should be able to transfer money.

Okay.

Which users?

From which accounts?

At what account state?

Which currencies?

Which rails?

Which provider capabilities?

Which verification state?

What limits?

How is the recipient resolved?

How long is that resolution trusted?

Can somebody transfer to themselves?

What happens if they submit twice?

What happens if the connection disappears?

What if the provider actually processed the transfer?

What if the webhook arrives first?

What if it arrives twice?

What does the user see while we genuinely don't know?

What does support see?

What gets logged?

What gets audited?

Who gets notified?

What happens if a dependency is unavailable?

Then repeat that exercise for cards.

FX.

KYC.

Bill payments.

Notifications.

Partner referrals.

Developer applications.

Documents.

Authentication.

Administration.

At some point I realised that a huge part of my work had become finding every unanswered:

Okay, but what happens if...?

and making the system choose an answer deliberately.

That's probably the best description I have for the project.

Earlier in the build, I was much more likely to fix the visible problem.

Authenticated user gets kicked back to login?

Fix the redirect.

Button doesn't work?

Fix the button.

Mobile page looks broken?

Fix that page.

Later I became more annoying.

Why did an authorization failure become an authentication failure?

Does another route use the same middleware pattern?

Why was the button able to request an operation the server should have rejected anyway?

Is the responsive bug local to this component, or is the same broken primitive used on twelve other pages?

Somewhere in that process, the way I debug changed.

I moved from:

fix the bug

toward:

fix the invariant that allowed this class of bug.

The second one can take longer.

It tends to pay back.

There are definitely things I would do differently if I were starting again.

I would define important state machines much earlier.

Before the UI.

Before controllers.

Before three different parts of the application all have their own interpretation of what pending means.

I would establish provider boundaries earlier.

It is much easier to prevent provider-specific knowledge from spreading than to remove it later.

I would design mobile as its own interaction problem from day one.

I would establish stronger conventions around notifications, authorization, and operational states before several domains needed them.

I would define "done" much more harshly at the beginning.

And yes, I would simplify the Git setup.

Very much that last one.

I like the stack I'm using.

But I don't think "I learned Fastify" or "I used BullMQ" is what I'll remember from this project.

What I'll remember is how often I ended up asking:

Where does truth live?

Who owns this state?

Who is allowed to change it?

What happens if this runs twice?

What happens if it succeeds externally but fails internally?

Can I safely retry it?

If I can't retry it, how do I recover?

Does the database need to enforce this?

Should this piece of work be allowed to wait?

What evidence do we leave behind?

Can somebody else figure out what happened when I'm not there?

What does the user see while the system genuinely doesn't know yet?

I knew what idempotency was before this project.

I knew what queues were.

I knew what database locking was.

I knew what RBAC was.

I knew what webhooks were.

I knew what reconciliation meant.

I just understand why those things exist much better now.

And that's probably the part I'll remember longest.

I'm still inside the build.

I still find awkward UI.

I still find journeys where a state exists without a good recovery path.

I still open files and think:

Why did I let you get this big?

I still have the performance investigation waiting for me.

There are refactors I've deliberately postponed until the feature pressure drops.

But when I compare what I thought I was building at the beginning with what exists now, the difference is kind of absurd.

More than 700 commits.

More than 200,000 tracked lines.

Three applications.

Thirteen shared packages.

More than 500 tests.

And considerably more time thinking about failure than I expected when I started.

I started the project thinking mostly about how to implement features.

I'm finishing it thinking much more about what the system is actually allowed to promise.

That's a much more useful change than another framework on my CV.

I'm almost done.

For now, back to figuring out why that last thing is still broken.

If you've built systems where retries, partial failure, or uncertain state became real problems, I'd genuinely like to hear what changed in the way you engineer.