The first version of my wallet kit had a stress-test endpoint. You called it, 50 threads tried to withdraw $10 from a $100 wallet, and it answered "passed": true.
That endpoint was honest about the one scenario it checked. The problem was everything it didn't check. So I replaced it with a real integration suite and one rule: write the test first, watch it fail in CI, then fix the code.
The suite now has 162 tests. Along the way it found three bugs that the demo never could. Here they are, with the code.
The setup
Every test gets its own fresh PostgreSQL 16 database. No in-memory provider, no mocks.
- Requests go through the real HTTP pipeline with
WebApplicationFactory, including the idempotency filter. - Connection pooling is off, so concurrent requests really use separate connections.
- Races start behind a common barrier, to push requests into the same moment.
- After every scenario, an
Audithelper checks every transaction (two entries, distinct wallets, zero sum, one currency) and every wallet: its balance equals the sum of its entries,BalanceAfteris the running sum, and sequence numbers have no gaps. That includes the settlement wallets.
The audit is the important part. A global SUM(Amount) = 0 can hide a transaction that is wrong in a way that balances out somewhere else.
Bug 1: A transfer could move money out of the settlement wallet
The kit models deposits and withdrawals as transfers between a user wallet and a per-currency system settlement wallet. Normal wallets cannot go negative. The settlement wallet can, because it represents money outside the system:
b.ToTable(t => t.HasCheckConstraint(
"CK_Wallets_NonNegativeBalance",
"\"Balance\" >= 0 OR \"IsSystem\" = TRUE"));
Deposits and withdrawals looked up the user wallet with !w.IsSystem. The transfer path did not check IsSystem on either side:
var fromWallet = lockedWallets.FirstOrDefault(w => w.Id == fromWalletId)
?? throw new DomainException($"Source wallet {fromWalletId} not found.", 404);
So a transfer from the settlement wallet to a user created a user balance of any size, recorded as a Transfer. A transfer to the settlement wallet was really a withdrawal, also recorded as a Transfer.
The public API didn't expose settlement wallet IDs, so you couldn't reach this without knowing one. That was luck, not a control. Any admin screen or report that shows a counterparty ID would have opened it.
It was found by reading the code during a review. The tests came first: a transfer in each direction with the settlement wallet's ID read from the database. In CI both failed with Expected 404; got 200.
The first fix rejected system wallets in transfers. A second change turned it into an allow-list, because a deny-list breaks the day someone adds a Fee or Adjustment transaction type:
// New transaction types must explicitly opt in to settlement access.
var allowsSystemWallet = type is TransactionType.Deposit or TransactionType.Withdrawal;
var fromWallet = lockedWallets.FirstOrDefault(
w => w.Id == fromWalletId && (!w.IsSystem || allowsSystemWallet))
?? throw new DomainException($"Source wallet {fromWalletId} not found.", 404);
The response is the same 404 as a missing wallet, so the API doesn't confirm that an ID belongs to a settlement wallet.
Bug 2: Big numbers came back as a 500
Amounts are stored as numeric(18,4), so the largest value is 14 integer digits. The code validated currency precision (two decimals for USD, none for JPY) but never checked the range.
An amount, or a resulting balance, beyond numeric(18,4) went all the way to PostgreSQL. It failed there with error 22003 (numeric field overflow), and the client got a 500. The settlement wallet has no lower bound, so its balance can leave the range too. A related case: an oversized withdrawal came back as "insufficient funds", which was the wrong reason.
The edge-case tests covered precision, trailing zeros, zero and negative amounts, unsupported and lower-case currencies, the largest valid amount, and values past the limit. On the first CI run, 17 of them failed.
The fix validates the amount, then checks the resulting balance before adding. The check is written as a subtraction, so even an extreme decimal can't overflow during the check itself:
public const decimal MaximumBalanceMagnitude = 99999999999999.9999m;
if (signedAmount > MaximumBalanceMagnitude - Balance ||
signedAmount < -MaximumBalanceMagnitude - Balance)
throw new DomainException("Resulting wallet balance exceeds the supported range.");
Now these are 400s. Balance and sequence don't change, nothing is written, and the idempotency key can be reused after a correction.
Bug 3: Two instances migrating at once
This one surprised me most, because it came from code I thought was already covered.
When the kit moved from EnsureCreated to EF Core migrations, I followed the documentation: since EF Core 9, Migrate takes a database-wide lock, so concurrent instances are protected. I added a test that raced two migrators on an empty database. It passed.
Then I added a second migration, and the same test failed. One migrator tried to add a column that the other had already added.
The diagnosis from the failure: with EF Core 10.0.4 and Npgsql 10.0.3, migrations commit one by one, and the provider's lock on the history table lasts only for a migration's transaction. Between two migrations another migrator can read stale history and start applying work that's already done. With one migration there is no "between", so the test passed.
The fix holds a session-level advisory lock across the whole migration call:
await db.Database.OpenConnectionAsync(ct);
try
{
await db.Database.ExecuteSqlRawAsync("SELECT pg_advisory_lock(71853240619243801)", ct);
await db.Database.MigrateAsync(ct);
}
finally
{
// Unlock, then close the connection (a crash releases the lock with the session).
}
The lesson I took: a concurrency test that passes with one migration says nothing about two. It's worth re-running such tests whenever their inputs grow, not only when the code changes.
In v3.1 migrations also moved out of application startup into a separate --migrate step that runs as an owner database role. The API now connects as a least-privilege runtime role and refuses to start while migrations are pending. The advisory lock still guards the migrate step.
What changed in how I work
- A failing test before the fix. A bug fix starts with a test that I've watched fail, in CI where possible, and only then does the fix go in. A test I never saw fail doesn't prove much.
- Claims point to tests. The README no longer says "deadlock-free" or "zero double-spending". Each guarantee names its mechanism, the tests behind it, and what those tests don't show.
- Limits are written down. The tests don't cover crash recovery, authorization (the kit has none yet), or every possible interleaving. Synchronized races increase contention, but they don't prove no deadlock can ever happen.
Thanks to the comments
Two hardening steps in v3.1 came from the discussion under my previous article:
- a read-only reconciliation query that uses
LAGto report the exact ledger row where a wallet's chain breaks; - a runtime database role without UPDATE, DELETE or TRUNCATE on the ledger, so history stays protected even if the append-only triggers are disabled.
A test now disables the triggers as the owner and confirms that the runtime role is still refused.
The kit
BulletproofWalletKit is the full source: the .NET 10 and PostgreSQL wallet engine, the 162-test suite, migrations, Docker Compose, decision records and an upgrade guide.
What's the bug your own tests found that you'd never have caught by hand? I'd like to hear it in the comments.
This article was drafted with AI assistance. The code, the bugs and the test results are from the kit's repository, and I reviewed every claim.
Top comments (0)