I was asked to look at a fintech backend where something small but persistent kept happening. Every month when the finance team tried to close the books, the internal ledger and the payment processor's own records were off by a small amount. Not a dramatic amount, sometimes a few transactions, sometimes a few dollars here and there, but enough that nobody could confidently say the numbers were correct. And in a fintech, correct is not optional.
This is called reconciliation drift, and it is one of the quieter problems a payments backend can have. It does not throw an error. It does not crash anything. It just slowly stops being trustworthy, one small mismatch at a time, until someone finally notices during a monthly close and has to spend hours figuring out where things went wrong.
What was actually happening
Once I started digging, the cause was not one single bug. It was a pattern that shows up a lot in systems that grow organically over time, multiple different places in the code were allowed to change a transaction's status or a balance, and none of them were talking to each other.
A webhook handler updated a transaction when the payment provider confirmed it. A retry job also updated the same transaction if the original webhook was ever missed. And an admin tool, built later for handling support tickets, allowed a staff member to manually mark a transaction as settled if a customer complained. Three different code paths, three different points where the same number could be changed, and no single place that recorded why a change happened or which of these three paths actually caused it.
@Injectable()
export class TransactionsService {
async markSettled(transactionId: string) {
const transaction = await this.transactionRepo.findOne({
where: { id: transactionId },
});
transaction.status = 'settled';
await this.transactionRepo.save(transaction);
}
}
This method looked fine everywhere it was called. The problem was that it was called from three different places, and the transaction record itself kept no history of that. Once a transaction was marked settled, there was no way to tell whether it had happened once, correctly, or twice, incorrectly.
The fix, treat every change as an event, not an overwrite
The real fix was not patching each of those three code paths individually. It was changing how transactions record their own history in the first place. Instead of updating a transaction's status directly, every change becomes its own recorded entry, and the transaction's current state is something you calculate from that history, not something you overwrite.
@Entity()
export class TransactionEvent {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
transactionId: string;
@Column()
eventType: string;
@Column()
source: string;
@CreateDateColumn()
createdAt: Date;
}
@Injectable()
export class TransactionsService {
constructor(
private readonly eventRepo: Repository<TransactionEvent>,
) {}
async recordSettlement(transactionId: string, source: string) {
await this.eventRepo.save({
transactionId,
eventType: 'settled',
source,
});
}
async getCurrentStatus(transactionId: string): Promise<string> {
const events = await this.eventRepo.find({
where: { transactionId },
order: { createdAt: 'ASC' },
});
const latest = events[events.length - 1];
return latest ? latest.eventType : 'pending';
}
}
The source field is what makes debugging this kind of issue possible going forward. Once every change records where it came from, the webhook, the retry job, or the admin tool, you can immediately see if the same transaction was settled twice by two different paths, instead of just seeing a status that quietly changed with no explanation attached.
The fix, an automated job that actually compares your records against theirs
Recording history properly stops new mistakes from being invisible, but it does not catch drift that already exists, or drift caused by something entirely outside your own system, like a webhook that never arrived at all. For that, the backend needs to actively compare its own records against the payment provider's records on a regular schedule, rather than assuming they always agree.
NestJS's schedule module makes this straightforward to set up as a recurring job.
@Injectable()
export class ReconciliationService {
constructor(
private readonly transactionRepo: Repository<TransactionEvent>,
private readonly discrepancyRepo: Repository<ReconciliationDiscrepancy>,
private readonly providerClient: PaymentProviderClient,
) {}
@Cron('0 2 * * *')
async runDailyReconciliation() {
const providerRecords = await this.providerClient.getSettledTransactions();
const internalStatuses = await this.getAllInternalStatuses();
for (const record of providerRecords) {
const internalStatus = internalStatuses.get(record.transactionId);
if (internalStatus !== 'settled') {
await this.discrepancyRepo.save({
transactionId: record.transactionId,
issue: 'provider shows settled but internal record does not',
});
}
}
}
}
This job runs quietly every night, comparing what the provider says actually happened against what your own system believes happened. Anything that does not line up gets written down as a discrepancy, rather than staying hidden until someone stumbles onto it during a manual review weeks later.
@Entity()
export class ReconciliationDiscrepancy {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
transactionId: string;
@Column()
issue: string;
@Column({ default: false })
resolved: boolean;
@CreateDateColumn()
detectedAt: Date;
}
Keeping discrepancies in their own table, separate from transactions themselves, means the finance team has a single place to look every morning, a short list of exactly what needs attention, instead of having to compare two entire data sets by hand.
Making the discrepancies actually get noticed
A table full of unresolved mismatches is only useful if someone actually sees it. Once the discrepancy records existed, the next step was making sure the job did not just write quietly to a database that nobody checked.
@Injectable()
export class ReconciliationService {
constructor(
private readonly notificationService: NotificationService,
) {}
private async notifyIfDiscrepanciesFound(count: number) {
if (count > 0) {
await this.notificationService.alertFinanceTeam(
`Reconciliation found ${count} unresolved mismatches overnight`,
);
}
}
}
This is a small addition, but it is the difference between a system that quietly protects itself and one that quietly accumulates a growing pile of unresolved discrepancies nobody remembers to check.
The bigger picture
None of this required exotic tooling. A recorded event history instead of silent overwrites, a scheduled job that compares your truth against the provider's truth, and a clear place for discrepancies to land where someone will actually see them. NestJS made this easy to structure cleanly, a dedicated service for events, a dedicated service for reconciliation, and a built in scheduler that made the recurring job simple to wire up without reaching for a separate tool.
What this kind of review usually reveals is not one dramatic bug. It is a handful of small, reasonable seeming decisions made at different points in time, each one fine on its own, that quietly stopped agreeing with each other. Catching that early is a lot cheaper than discovering it during an audit.
If your team suspects your own numbers might not fully line up with what your payment provider believes happened, that gap is worth closing before it grows, and it is exactly the kind of review I would be glad to help with.
I am Peace Melodi, a backend software engineer. If you want your business to scale big, comfortably handling millions of users without breaking, with strong scalability and security in place, feel free to reach out.
LinkedIn: https://www.linkedin.com/in/melodi-peace-406494368
GitHub: https://github.com/PeaceMelodi
Top comments (6)
The nightly job only catches one direction, right? Provider says settled, internal doesn't. The opposite case, where the admin tool marked something settled that the provider never confirmed, seems like the more likely drift given how that tool gets used, and I don't see a check for it.
The piece scopes reconciliation around the failure mode caused by infrastructure gaps, missed webhooks, retries that never fire, since that's the one most teams hit first without realizing it. The admin override path is a separate category of risk entirely, a manual settlement with no provider confirmation behind it, and it needs its own check rather than being folded into the same job.
In a production setup I'd run the comparison in both directions, provider confirmed with no internal match, and internal marked settled with no provider match. The second one usually surfaces less often but tends to matter more once it does, since it means the ledger is trusting a human step instead of the source of truth.
Solid detail to bring up though, this is exactly the kind of distinction that separates a reconciliation job that looks complete from one that actually is.
The event-sourcing rewrite fixes attribution, but there's a race condition hiding in
getCurrentStatusthat'll reintroduce the exact drift you're trying to kill. If the webhook and the retry job both fire near-simultaneously — which is precisely when a missed-then-redelivered webhook happens — they both append asettledevent, and now the ledger double-counts. Recording thesourcelets you see the double settlement after the fact, but nothing in this code prevents it. You need an idempotency key on the event, something like a unique constraint on(transactionId, eventType, providerEventId), so the second write collapses into the first instead of becoming a second row.The reconciliation cron has a subtler blind spot too: it only checks one direction. You catch "provider says settled, we don't," but not "we say settled, provider has no record" — which is the shape a buggy admin tool or a retry against the wrong ID produces. In a real close, the money that goes missing from your side matters as much as the money that shows up unexpectedly, and a one-directional diff will quietly pass a ledger that's already wrong.
Event recording and duplicate prevention are two separate problems here, and this response was built to solve only the first one. Knowing which path caused a change is different from stopping two paths from writing the same change, and that second guarantee needs a unique constraint on transactionId, eventType, and providerEventId together, so a webhook and a retry racing each other collapse into one row at the database level instead of depending on application code to catch the timing.
The reconciliation direction is worth being precise about too. Provider confirmed with no internal match usually points to an infrastructure gap, a dropped webhook, a retry that never fired. Internal marked settled with no provider match is a different failure entirely, a manual override or a retry hitting the wrong id, and it needs its own explicit check rather than getting folded into the same job. A real close treats both directions as equally important, since money missing from your side costs just as much as money the provider has no record of.
Appreciate the depth here, comments like this are rare and they push the thinking further than the piece alone ever could.
@peacemelodi You're right to split those two problems apart — recording the causal path and preventing duplicates are genuinely orthogonal, and the rewrite as written only nails the first. Attribution tells you which command produced a ledger entry; it does nothing to stop the same command from being applied twice under a retry or a concurrent writer.
The fix I lean on is an idempotency key carried on the command itself, enforced at the write boundary — a unique constraint on (aggregate_id, idempotency_key) in the events table, so a duplicate append fails loudly instead of quietly double-crediting. That keeps dedup where the race actually lives (the append) rather than trying to reason about it after the fact in a projection. Optimistic concurrency on the aggregate version number handles the concurrent-writer case: two commands racing the same expected version, one wins, the loser retries against fresh state.
Good catch flagging it, though — I framed the rewrite around attribution because that was the bug in front of me, and I didn't make it explicit that idempotency is a separate control you still have to add. It should've been called out in the piece.
The idempotency key on the command itself is a solid way to enforce that boundary, a unique constraint on aggregate_id and idempotency_key at the write point, with optimistic concurrency handling the racing writer case. Worth having in a real implementation.
The piece itself was scoped specifically to attribution, tracing which path produced a given change, since that was the actual problem in front of me. Idempotency is its own separate concern with its own separate solution, and treating it as a natural extension of the piece rather than something the piece was ever meant to cover keeps the two properly distinct.
Thanks for commenting, appreciate you taking the time to dig into it.