I’m Almost Done Building the Biggest Financial Platform I’ve Worked On

I kept telling myself I would write about this project when I finished it.

I'm still not completely finished.

There are a few user journeys I'm checking again, some UI things I keep noticing at annoying screen sizes, a performance problem I deliberately don't want to "fix" before measuring it properly, and a few pieces of technical debt I already know I'll be coming back for.

But I'm close.

And I think I'd rather write this now, while I'm still inside the build, than six months from now when my brain has conveniently turned the whole experience into a clean sequence of sensible engineering decisions.

Because it wasn't that clean.

I've been shipping almost every day on a financial platform through a pretty intense stretch of development.

I can't name the company, the product, or the financial infrastructure behind it, and there are obviously security and production details that don't belong in a public article.

But I can talk about what it has actually been like to build.

At one point, a repository snapshot looked roughly like this:

• 700+ commits
• 1,000+ tracked files
• 200,000+ tracked lines
• 3 applications
• 13 shared packages
• 500+ tests

I don't think I ever had a day where it felt like:

Today I am contributing to a 200,000-line system.

It felt more like:

Why is this route sending an authenticated user back to login?

Then:

This notification should open the thing it is notifying them about.

Then:

Why can this worker be delayed by marketing jobs?

Then:

This transfer timed out. Did the money actually move?

Then:

This looks terrible on a 390px screen.

Enough of those days happen and suddenly you have a platform.

When the project stopped being "an app"

The repository eventually settled into a TypeScript monorepo with three main applications:

apps/
  api/
  web/
  worker/

and shared packages handling things like:

packages/
  cache/
  config/
  database/
  documents/
  ledger/
  providers/
  public-api/
  queues/
  safety/
  security/
  shared/
  storage/
  tooling/

The frontend is React and TypeScript.

The HTTP layer is Fastify.

PostgreSQL is the primary database.

PgBouncer handles connection pooling.

Valkey sits behind caching and short-lived coordination.

BullMQ runs asynchronous workloads.

Around that are object storage, Caddy, systemd, structured logging, health checks, deployment tooling, API schemas, background workers, and the other things that become necessary once "run the server" stops being a deployment strategy.

None of those technology choices are particularly exotic.

The interesting part was what the product gradually demanded from them.

What started as one financial application eventually included customer onboarding, account approval, KYC, financial accounts, transfers, FX, cards, bill payments, statements, notifications, partner workflows, internal operations, support tooling, developer-facing APIs, public documentation, a PWA, and some separately privileged product areas.

Different users.

Different access boundaries.

Different kinds of state.

Different kinds of money movement.

Different failure modes.

One system.

At the beginning, I was still mostly thinking in features.

Build onboarding.

Build the dashboard.

Add transfers.

Add notifications.

Wire up the provider.

Make the mobile version work.

The problem is that every one of those features brought more than a screen with it.

Transfers brought retries, timeouts, provider state, reconciliation, authorization, limits, notifications, and audit.

Notifications brought preferences, retries, delivery channels, deep links, and several applications generating different kinds of events.

Partner features brought attribution, permissions, workflow state, and the problem of showing enough information to make a relationship useful without leaking private customer data.

Eventually the implementation had grown enough that I stopped trusting the shape of the original application.

Not because every part was bad.

That actually made the decision harder.

There is a point in a project where almost every problem still looks individually fixable.

A route is broken.

Fix it.

An authorization check is awkward.

Fix it.

One provider call is living in the wrong place.

Move it.

A page breaks on mobile.

Patch the CSS.

A workflow has a weird state.

Add another condition.

Every individual fix can be reasonable while the overall shape of the application gets worse.

Eventually I reached the point where I didn't trust enough of the assumptions underneath the implementation.

So I froze that development state.

I kept the infrastructure, but started the application layer again on a clean development environment.

Not because rewrites are fun.

They're not.

It was more:

I don't want to keep making stronger promises on top of a foundation I no longer fully trust.

The original implementation plan had 24 engineering sprints.

That eventually became 30.

Not because I suddenly wanted six more sprints, but because areas like partner operations, developer APIs, documentation, PWA behaviour, and final hardening turned out to be much larger domains when I tried to make them complete instead of merely present.

That gave me one of the first lessons from the project that really stuck:

Sometimes scope grows because people keep adding things. Sometimes scope grows because you finally understand what the original sentence actually implied.

Those are very different problems.

Rebuilding the foundation changed what I meant by "reliable"

One of the first clean-build deployments failed.

I like that story now.

I did not like it when it happened.

The useful part wasn't that the deployment failed.

The useful part was that the release machinery did what it was supposed to do.

The new state wasn't copied over the running application and left there half-working. The release stopped and the previous known-good state came back.

The real deployment tooling is more involved, but conceptually the release flow became something like:

release="$(create_release)"

build_into "$release"
run_checks "$release"

switch_atomically "$release"

if ! healthcheck; then
  rollback
fi

I had already written "rollback" into the architecture.

That failure was the moment I knew rollback actually existed.

There is a difference.

I also learned a smaller operational detail by watching enough deployments: the first health probe after switching a release doesn't always return 200 immediately.

Sometimes the process needs another attempt or two while it finishes warming up.

The first few times that happened, I treated it like something was broken.

Eventually the deployment tooling learned what normal startup behaviour actually looked like.

That's one of those boring things you don't really learn from drawing an architecture diagram.

Architecture tells you how you think the system behaves.

Operations tells you how it actually behaves.

That experience also made me stricter about apparently small changes.

At one point CI went red because Prettier wanted a multi-line import collapsed back onto one line.

That was basically the entire behavioural impact of the commit.

Before:

import {
  AppSessionLoader,
  PushPermissionDecision,
  RequestCard,
  Shell,
  icon,
} from "./MemberPortal";

After:

import { AppSessionLoader, PushPermissionDecision, RequestCard, Shell, icon } from "./MemberPortal";

No business behaviour changed.

The gate was still red.

So the formatting change got synced, built, deployed, and verified like everything else.

At the time I thought:

This is ridiculous.

Then I thought about the alternative.

You start deciding that some changes are too small to deserve the normal release process.

Then somebody has to decide what "small" means.

Then exceptions start accumulating.

So yes.

The one-line import got a proper deployment.

The Git setup behind all of this is less elegant than I'd like.

I deliberately separated the environment where commits are authored from the checkout that handles deployment and remote repository authentication.

That buys credential isolation.

It also means synchronization occasionally takes more thought than a normal one-checkout workflow should. I've had moments where the deployment checkout already contained an uncommitted copy of the exact change I was trying to pull as a commit, forcing me to verify the diffs matched before pulling cleanly.

It works.

I know why it exists.

I still want to simplify it.

That's debt. Not hidden debt. Just something useful enough to tolerate for now.

The bigger shift was that I started thinking about reliability less as:

Does this code work?

and more as:

What does the system do when this code doesn't get the outcome it expected?

That question became much more serious once money movement entered the picture.

Money movement forced me to stop thinking in success and failure

From the UI, a transfer looks like:

recipient
amount
confirm

The backend version is much less pleasant.

Suppose I submit a transfer to an external financial provider.

Then my connection disappears.

Did it fail?

Maybe.

Did the provider receive the request?

Maybe.

Did they execute it successfully and the response disappear somewhere between their system and mine?

Also maybe.

The one thing I definitely cannot do is immediately submit the transfer again.

A naïve version of the flow wants to look like this:

const result = await provider.transfer(input);

if (!result.ok) {
  throw new Error("Transfer failed");
}

return { status: "success" };

That code wants the world to have two states.

Success.

Failure.

Financial systems occasionally respond:

lol, no.

The application eventually needed stable operation IDs, idempotency keys, request hashes, provider references, and recoverable operation states.

The mental model became much closer to:

type OperationState =
  | "submitting"
  | "accepted"
  | "processing"
  | "checking"
  | "completed"
  | "failed"
  | "reversed"
  | "unknown"
  | "reconciling";

The state I care about most there is unknown.

Earlier in my career, I think I would have treated unknown as a failure to finish implementing the workflow.

Now I think it can be one of the most honest states in a distributed financial system.

If I genuinely do not know whether money moved yet, the application should not invent an answer because red or green is easier to render.

That same problem changed how I thought about webhooks.

A webhook implementation initially looks very simple:

app.post("/webhook", async (request) => {
  await updateTransaction(request.body);
  return { ok: true };
});

Then you spend five minutes thinking like someone trying to break it.

Who sent that body?

Has it been modified?

How old is it?

Have I seen this event before?

Is it from the right environment?

What if the provider sends it three times?

What if they retry because my server took too long to answer?

What if the webhook says a transaction completed but the amount or reference does not match what I expected?

Eventually the flow looked much more like:

receive
  ↓
validate request shape
  ↓
verify signature + timestamp
  ↓
reject replay
  ↓
persist event
  ↓
acknowledge quickly
  ↓
process asynchronously
  ↓
verify against provider when necessary
  ↓
apply financial effect

And reconciliation stopped pretending that every incoming event could immediately become success or failure.

A shortened, anonymised version of a pattern in the codebase looks like this:

type ReconcileOutcome =
  | { action: "processed"; postingId: string; duplicate: boolean }
  | { action: "skipped"; reason: string }
  | { action: "needs_review"; reason: string }
  | { action: "error"; reason: string };

That needs_review state matters.

For example, if I receive one reference and the authoritative lookup comes back with something else:

if (verifiedReference !== reference) {
  return {
    action: "needs_review",
    reason: "Provider verification reference mismatch",
  };
}

Don't guess.

Don't force it through because it probably belongs to that transaction.

Don't silently throw it away either.

Put it somewhere the system can reason about it safely.

The more I worked on this, the more I realised that a financial platform needs to be comfortable saying:

I have received information.

without automatically saying:

Therefore this is now financial truth.

Those are different statements.

And that distinction led directly to another architectural problem.

The more financial operations I added, the easier it would have been to let the external provider define the whole application.

Need an account?

Call their account endpoint.

Need a transfer?

Call their transfer endpoint.

Need FX?

Call that endpoint.

Need a card?

Same thing.

That is extremely productive at the beginning.

Then one day your controllers know provider-specific field names, your workers know provider-specific status strings, your frontend knows which provider can do what, and half your business logic is written in the vocabulary of a company you don't control.

I didn't want that.

So financial integrations ended up behind a provider boundary.

Conceptually:

application/domain logic
          ↓
provider capability + routing layer
          ↓
provider-specific adapter
          ↓
external infrastructure

An early implementation could easily have ended up with logic like:

if (currency === "USD") {
  showTransferButton();
}

Except "does the provider support USD?" is not actually a useful question.

USD for what?

Account issuance?

Holding a balance?

Inbound transfer?

Outbound transfer?

FX?

Cards?

Bills?

A provider can support one and not another.

Or expose something technically while that capability is not enabled for your account.

So support had to become contextual.

The provider layer can ask something closer to:

registry.supports("banking_rail", {
  country,
  currency,
  customerType,
});

with the underlying logic checking capability, geography, currency, customer type, and whether that declaration is enabled.

Something roughly like:

if (!declaration.enabled) return false;
if (declaration.capability !== capability) return false;

if (
  declaration.countries !== "ALL" &&
  !declaration.countries.includes(country)
) {
  return false;
}

if (
  declaration.currencies !== "ALL" &&
  !declaration.currencies.includes(currency)
) {
  return false;
}

return declaration.customerTypes.includes(customerType);

There is another detail I particularly like.

Once an existing financial relationship has been created with a particular provider, changing the routing preference for new relationships should not silently move that existing one somewhere else.

So existing relationships can be pinned.

New ones can follow the current routing rules.

Old ones continue using the provider they were actually created with.

That sounds obvious now.

It wasn't something I was thinking about before I had to design a system that could survive changing providers.

Once correctness mattered, infrastructure became part of product behaviour

The financial workflows created more asynchronous work.

Provider events.

Reconciliation.

Security messages.

Notifications.

Documents.

Exports.

Analytics.

At first, the normal queue question was:

Does this need to happen outside the HTTP request?

Useful question.

Not enough.

The more important question became:

What is this job allowed to wait behind?

Imagine thousands of marketing jobs enter the queue.

At the same time, somebody is trying to log in and needs a security email.

Should that security message sit behind the marketing campaign?

Obviously not.

But if they are all just "background jobs," you may have accidentally made exactly that decision.

So the queue layer stopped treating every job as equal.

A simplified version looks like:

const queueClass = {
  critical: [
    "security-email",
    "provider-webhooks",
    "financial-reconciliation",
  ],

  normal: [
    "transactional-email",
    "push-notifications",
    "documents",
  ],

  deferrable: [
    "exports",
    "analytics",
    "marketing-bulk",
    "maintenance",
  ],
};

They don't all get the same retry behaviour either:

const policies = {
  critical: {
    attempts: 8,
    backoff: { type: "exponential", delay: 1000 },
  },

  normal: {
    attempts: 5,
    backoff: { type: "exponential", delay: 2000 },
  },

  deferrable: {
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
  },
};

The specific numbers are not the interesting part.

The interesting part is that queue design became a product decision.

When the system is under pressure, what are we willing to make wait?

An analytics refresh?

Sure.

A login security message?

Much less so.

A financial reconciliation job?

Definitely not because somebody requested a large export.

That's how I think about queues now.

They aren't just a place to put work you don't want inside the request.

They encode priority.

They encode which delays the product considers acceptable.

The database went through a similar change in my head.

PostgreSQL stopped being just "where the data goes."

There are some bugs I don't want application-level if statements to be solely responsible for preventing.

Two requests trying to spend the same available amount.

Two workers processing the same event.

Two people trying to consume the last available unit of something.

An update based on stale state.

Several records that need to change together or not change at all.

So the database became part of the correctness model.

That meant actually using:

• unique constraints;
• foreign keys;
• check constraints;
• optimistic version columns;
• atomic conditional updates;
• row locks;
• short transactions;
• deterministic lock ordering;
• and stronger isolation where a specific invariant justified it.

I knew what SELECT ... FOR UPDATE did before this project.

The project taught me something more useful:

when I actually want it.

Same with serializable transactions.

I knew they existed.

Now I understand much better why I don't want everything running at serializable isolation, and why a few particularly sensitive invariants might actually justify paying that cost.

Knowing the feature and knowing where it belongs are different levels of understanding.

That pattern kept repeating.

The system would grow.

A problem I previously understood academically would become a real operational constraint.

Then I would understand why the boring engineering primitive exists.

The visible feature was almost never the whole feature

Authentication started small too.

At one point, "authentication" basically meant:

login
logout
forgot password

Then the actual requirements arrived.

OTP.

Passkeys.

Device and session management.

Session rotation.

Step-up authentication.

Recovery.

Staff authentication.

Privileged staff authentication.

Trusted devices.

Revocation.

Different session policies.

Authorization became even more important because the system eventually had several application surfaces.

A customer.

A partner.

A developer.

Normal staff.

Privileged staff.

Someone can be allowed to see a record without being allowed to modify it.

A partner may need to know that a referral progressed without being allowed anywhere near that person's banking details.

A customer interface may correctly hide an operation while an API route underneath it accidentally still allows the request.

So I became increasingly annoying about one rule:

A hidden or disabled button is not authorization.

If someone is not allowed to perform an action, the server performing that action has to reject it.

Not the menu.

Not the React component.

Not the disabled state.

The actual boundary that changes or reveals the protected resource.

Frontend visibility is UX.

Authorization is policy.

Notifications followed the same pattern.

What began as "we need notifications" became in-app events, email, push, preferences, retries, deduplication, deep links, and failure handling across several application surfaces.

The useful model was to treat the internal notification as the durable record and email or push as delivery attempts around it.

A failed delivery does not undo the event that caused it, and it definitely does not change financial truth.

And while all of this was happening on the backend, the frontend was teaching me a different version of the same lesson:

software can be technically correct and still be bad.

The mobile topbar, for example, had several lives.

It went from:

full wordmark
      ↓
personalised greeting
      ↓
small icon

The greeting version wasn't a mock-up.

It actually loaded the member's name.

It had responsive typography.

It was built and styled.

Then I looked at it again and the smaller icon was cleaner.

So the greeting disappeared.

I used to think of that kind of work as wasted effort.

I don't anymore.

Sometimes the fastest way to know whether a visual idea works is to build the thing and look at it.

One of my favourite frontend bugs had nothing to do with code correctness.

A dashboard section was redesigned.

The code compiled.

Types passed.

Routes worked.

The layout worked.

Then I looked at the screen and realised that the same phrase appeared three times in the same panel.

Once as the eyebrow.

Once as the heading.

Once again on the button.

Perfectly valid software.

Terrible interface.

No type system is going to catch that.

No unit test is going to tell you your copy hierarchy looks stupid.

Sometimes the correct QA process is literally:

Why does this look stupid?

Mobile made this even more obvious.

There were pages that were technically responsive.

Nothing overflowed.

Every element was visible.

They were still bad mobile interfaces.

Authentication was the clearest example.

On desktop, more explanatory text and a wider composition can look good.

On a phone, especially inside an installed PWA, the same thing can feel like a desktop website somebody squeezed into a smaller rectangle.

So I started removing things.

Less surrounding chrome.

Less unnecessary copy.

Better viewport spacing.

Forms that actually sit comfortably between the screen edges.

Cleaner hierarchy.

OTP fields that look like OTP fields instead of a normal text input.

Navigation designed for thumbs instead of a desktop sidebar awkwardly collapsing.

That changed the question I ask.

Not:

Does this fit on mobile?

But:

If mobile were the only platform, would I have designed it this way?

If the answer is no, then I probably haven't actually designed the mobile version yet.

The same idea connects all of these problems.

A disabled button doesn't mean an action is forbidden.

A page fitting inside the viewport doesn't mean it is a good mobile experience.

A transfer button working doesn't mean the transfer journey is reliable.

The thing I could see was often only the beginning.

"Done" became a much harsher word

Near the end of the build, I started discovering a lot of things that existed but weren't actually complete.

A notification bell could exist without every product surface producing the right events.

A developer portal could look finished while credential rotation, scopes, idempotency, or webhook management were still incomplete.

An admin page could render perfectly while the underlying action lacked the right authorization, audit trail, or failure behaviour.

That was when my working definition of a complete feature became much harsher:

data model
backend action
authorization
validation
state transitions
audit
notifications
failure states
empty states
responsive behaviour
tests
documentation
operational controls

That definition is painful when somebody says:

But the screen is already there. Aren't we almost done?

Sometimes.

Sometimes the screen means you're 60% done.

Testing changed for me too.

The project has more than 500 tests, and substantial behavioural changes still get the appropriate suites.

But I stopped treating "run everything after everything" as synonymous with discipline.

For a presentation-only change, typecheck, build, and actual visual inspection may tell me more than hundreds of unrelated domain tests.

My rule became:

Pay for the verification that can realistically catch the class of mistake you just introduced.

If I changed a ledger invariant, visual QA isn't going to save me.

If I changed a heading from 18px to 20px, hundreds of domain tests probably aren't going to save me either.

And despite all of this, the project isn't magically debt-free because I'm near the end.

The main frontend bundle is still larger than I want it to be.

The bundler complains about it.

Correctly.

I have deliberately not responded by throwing lazy() around random components and declaring the performance problem solved.

There is a proper performance investigation queued.

The bundle is an obvious place to start, but I want to measure the actual user-visible latency before deciding what needs to change.

There is also an authenticated dashboard component that has accumulated too many responsibilities, an icon fallback that is resilient but can hide misspelled glyph names, and some operational cleanup I deliberately postponed until the feature pressure drops.

I have debt.

I know where it is.

I know why it exists.

I've become much more comfortable with the difference between:

unknown, accidental debt

and:

known, bounded debt that I deliberately chose not to fix today.

The second kind still needs paying.

But at least I know what I owe.

The part that changed how I engineer

I've tried to work out which part of the project had the biggest impact on me.

The transfer system?

The provider abstraction?

The ledger?

Security?

Partner workflows?

Developer APIs?

Deployment?

The PWA?

The queue architecture?

I don't think it is one feature.

The bigger job was repeatedly taking vague product statements and turning them into rules a system could actually enforce.

Take:

Users should be able to transfer money.

Okay.

Which users?

From which accounts?

At what account state?

Which currencies?

Which rails?

Which provider capabilities?

Which verification state?

What limits?

How is the recipient resolved?

How long is that resolution trusted?

Can somebody transfer to themselves?

What happens if they submit twice?

What happens if the connection disappears?

What if the provider actually processed the transfer?

What if the webhook arrives first?

What if it arrives twice?

What does the user see while we genuinely don't know?

What does support see?

What gets logged?

What gets audited?

Who gets notified?

What happens if a dependency is unavailable?

Then repeat that exercise for cards.

FX.

KYC.

Bill payments.

Notifications.

Partner referrals.

Developer applications.

Documents.

Authentication.

Administration.

At some point I realised that a huge part of my work had become finding every unanswered:

Okay, but what happens if...?

and making the system choose an answer deliberately.

That's probably the best description I have for the project.

Earlier in the build, I was much more likely to fix the visible problem.

Authenticated user gets kicked back to login?

Fix the redirect.

Button doesn't work?

Fix the button.

Mobile page looks broken?

Fix that page.

Later I became more annoying.

Why did an authorization failure become an authentication failure?

Does another route use the same middleware pattern?

Why was the button able to request an operation the server should have rejected anyway?

Is the responsive bug local to this component, or is the same broken primitive used on twelve other pages?

Somewhere in that process, the way I debug changed.

I moved from:

fix the bug

toward:

fix the invariant that allowed this class of bug.

The second one can take longer.

It tends to pay back.

There are definitely things I would do differently if I were starting again.

I would define important state machines much earlier.

Before the UI.

Before controllers.

Before three different parts of the application all have their own interpretation of what pending means.

I would establish provider boundaries earlier.

It is much easier to prevent provider-specific knowledge from spreading than to remove it later.

I would design mobile as its own interaction problem from day one.

I would establish stronger conventions around notifications, authorization, and operational states before several domains needed them.

I would define "done" much more harshly at the beginning.

And yes, I would simplify the Git setup.

Very much that last one.

I like the stack I'm using.

But I don't think "I learned Fastify" or "I used BullMQ" is what I'll remember from this project.

What I'll remember is how often I ended up asking:

Where does truth live?

Who owns this state?

Who is allowed to change it?

What happens if this runs twice?

What happens if it succeeds externally but fails internally?

Can I safely retry it?

If I can't retry it, how do I recover?

Does the database need to enforce this?

Should this piece of work be allowed to wait?

What evidence do we leave behind?

Can somebody else figure out what happened when I'm not there?

What does the user see while the system genuinely doesn't know yet?

I knew what idempotency was before this project.

I knew what queues were.

I knew what database locking was.

I knew what RBAC was.

I knew what webhooks were.

I knew what reconciliation meant.

I just understand why those things exist much better now.

And that's probably the part I'll remember longest.

I'm still inside the build.

I still find awkward UI.

I still find journeys where a state exists without a good recovery path.

I still open files and think:

Why did I let you get this big?

I still have the performance investigation waiting for me.

There are refactors I've deliberately postponed until the feature pressure drops.

But when I compare what I thought I was building at the beginning with what exists now, the difference is kind of absurd.

More than 700 commits.

More than 200,000 tracked lines.

Three applications.

Thirteen shared packages.

More than 500 tests.

And considerably more time thinking about failure than I expected when I started.

I started the project thinking mostly about how to implement features.

I'm finishing it thinking much more about what the system is actually allowed to promise.

That's a much more useful change than another framework on my CV.

I'm almost done.

For now, back to figuring out why that last thing is still broken.

---

If you've built systems where retries, partial failure, or uncertain state became real problems, I'd genuinely like to hear what changed in the way you engineer.