DEV Community

Cover image for 162 Integration Tests Found 3 Real Bugs in My .NET Wallet Kit
Harut Petrosyan
Harut Petrosyan

Posted on Edited on

162 Integration Tests Found 3 Real Bugs in My .NET Wallet Kit

The first version of my wallet kit had a stress-test endpoint. You called it, 50 threads tried to withdraw $10 from a $100 wallet, and it answered "passed": true.

That endpoint was honest about the one scenario it checked. The problem was everything it didn't check. So I replaced it with a real integration suite and one rule: write the test first, watch it fail in CI, then fix the code.

The suite now has 162 tests. Along the way it found three bugs that the demo never could. Here they are, with the code.

The setup

Every test gets its own fresh PostgreSQL 16 database. No in-memory provider, no mocks.

  • Requests go through the real HTTP pipeline with WebApplicationFactory, including the idempotency filter.
  • Connection pooling is off, so concurrent requests really use separate connections.
  • Races start behind a common barrier, to push requests into the same moment.
  • After every scenario, an Audit helper checks every transaction (two entries, distinct wallets, zero sum, one currency) and every wallet: its balance equals the sum of its entries, BalanceAfter is the running sum, and sequence numbers have no gaps. That includes the settlement wallets.

The audit is the important part. A global SUM(Amount) = 0 can hide a transaction that is wrong in a way that balances out somewhere else.

Bug 1: A transfer could move money out of the settlement wallet

The kit models deposits and withdrawals as transfers between a user wallet and a per-currency system settlement wallet. Normal wallets cannot go negative. The settlement wallet can, because it represents money outside the system:

b.ToTable(t => t.HasCheckConstraint(
    "CK_Wallets_NonNegativeBalance",
    "\"Balance\" >= 0 OR \"IsSystem\" = TRUE"));
Enter fullscreen mode Exit fullscreen mode

Deposits and withdrawals looked up the user wallet with !w.IsSystem. The transfer path did not check IsSystem on either side:

var fromWallet = lockedWallets.FirstOrDefault(w => w.Id == fromWalletId)
    ?? throw new DomainException($"Source wallet {fromWalletId} not found.", 404);
Enter fullscreen mode Exit fullscreen mode

So a transfer from the settlement wallet to a user created a user balance of any size, recorded as a Transfer. A transfer to the settlement wallet was really a withdrawal, also recorded as a Transfer.

The public API didn't expose settlement wallet IDs, so you couldn't reach this without knowing one. That was luck, not a control. Any admin screen or report that shows a counterparty ID would have opened it.

It was found by reading the code during a review. The tests came first: a transfer in each direction with the settlement wallet's ID read from the database. In CI both failed with Expected 404; got 200.

The first fix rejected system wallets in transfers. A second change turned it into an allow-list, because a deny-list breaks the day someone adds a Fee or Adjustment transaction type:

// New transaction types must explicitly opt in to settlement access.
var allowsSystemWallet = type is TransactionType.Deposit or TransactionType.Withdrawal;
var fromWallet = lockedWallets.FirstOrDefault(
        w => w.Id == fromWalletId && (!w.IsSystem || allowsSystemWallet))
    ?? throw new DomainException($"Source wallet {fromWalletId} not found.", 404);
Enter fullscreen mode Exit fullscreen mode

The response is the same 404 as a missing wallet, so the API doesn't confirm that an ID belongs to a settlement wallet.

Bug 2: Big numbers came back as a 500

Amounts are stored as numeric(18,4), so the largest value is 14 integer digits. The code validated currency precision (two decimals for USD, none for JPY) but never checked the range.

An amount, or a resulting balance, beyond numeric(18,4) went all the way to PostgreSQL. It failed there with error 22003 (numeric field overflow), and the client got a 500. The settlement wallet has no lower bound, so its balance can leave the range too. A related case: an oversized withdrawal came back as "insufficient funds", which was the wrong reason.

The edge-case tests covered precision, trailing zeros, zero and negative amounts, unsupported and lower-case currencies, the largest valid amount, and values past the limit. On the first CI run, 17 of them failed.

The fix validates the amount, then checks the resulting balance before adding. The check is written as a subtraction, so even an extreme decimal can't overflow during the check itself:

public const decimal MaximumBalanceMagnitude = 99999999999999.9999m;

if (signedAmount > MaximumBalanceMagnitude - Balance ||
    signedAmount < -MaximumBalanceMagnitude - Balance)
    throw new DomainException("Resulting wallet balance exceeds the supported range.");
Enter fullscreen mode Exit fullscreen mode

Now these are 400s. Balance and sequence don't change, nothing is written, and the idempotency key can be reused after a correction.

Bug 3: Two instances migrating at once

This one surprised me most, because it came from code I thought was already covered.

When the kit moved from EnsureCreated to EF Core migrations, I followed the documentation: since EF Core 9, Migrate takes a database-wide lock, so concurrent instances are protected. I added a test that raced two migrators on an empty database. It passed.

Then I added a second migration, and the same test failed. One migrator tried to add a column that the other had already added.

The diagnosis from the failure: with EF Core 10.0.4 and Npgsql 10.0.3, migrations commit one by one, and the provider's lock on the history table lasts only for a migration's transaction. Between two migrations another migrator can read stale history and start applying work that's already done. With one migration there is no "between", so the test passed.

The fix holds a session-level advisory lock across the whole migration call:

await db.Database.OpenConnectionAsync(ct);
try
{
    await db.Database.ExecuteSqlRawAsync("SELECT pg_advisory_lock(71853240619243801)", ct);
    await db.Database.MigrateAsync(ct);
}
finally
{
    // Unlock, then close the connection (a crash releases the lock with the session).
}
Enter fullscreen mode Exit fullscreen mode

The lesson I took: a concurrency test that passes with one migration says nothing about two. It's worth re-running such tests whenever their inputs grow, not only when the code changes.

In v3.1 migrations also moved out of application startup into a separate --migrate step that runs as an owner database role. The API now connects as a least-privilege runtime role and refuses to start while migrations are pending. The advisory lock still guards the migrate step.

Update, 9 Nov 2026. A reader asked whether the suite covers one migrator failing while another waits for the advisory lock. It didn't, so I added two integration tests. They are in the repository now (164 tests) and will ship with the next kit release:

  • A failing migrator releases the lock to the one waiting behind it. A gate connection holds the advisory lock while two migrators queue behind it on a database with incompatible history. The test first confirms in pg_locks that both are waiting, then opens the gate. Both migrators must fail with the same clear error, schema and data must stay unchanged, and the lock must be free afterwards.
  • A waiting migrator takes over when the lock holder dies. The test kills the backend that holds the lock with pg_terminate_backend. PostgreSQL releases the session lock, and the waiting migrator applies every migration exactly once.

Neither test found a new bug, which is the answer I was hoping for. Thanks to Bob's Media for the question.

What changed in how I work

  • A failing test before the fix. A bug fix starts with a test that I've watched fail, in CI where possible, and only then does the fix go in. A test I never saw fail doesn't prove much.
  • Claims point to tests. The README no longer says "deadlock-free" or "zero double-spending". Each guarantee names its mechanism, the tests behind it, and what those tests don't show.
  • Limits are written down. The tests don't cover crash recovery (beyond a migrator dying while it holds the lock), authorization (the kit has none yet), or every possible interleaving. Synchronized races increase contention, but they don't prove no deadlock can ever happen.

Thanks to the comments

Two hardening steps in v3.1 came from the discussion under my previous article:

  • a read-only reconciliation query that uses LAG to report the exact ledger row where a wallet's chain breaks;
  • a runtime database role without UPDATE, DELETE or TRUNCATE on the ledger, so history stays protected even if the append-only triggers are disabled.

A test now disables the triggers as the owner and confirms that the runtime role is still refused.

What's the bug your own tests found that you'd never have caught by hand? I'd like to hear it in the comments.

This article was drafted with AI assistance. The code, the bugs and the test results are from the kit's repository, and I reviewed every claim.

Top comments (7)

Collapse
 
bobsmedia profile image
Bob's Media •

The migration race appearing only after a second migration is a useful reminder to vary the history, not just the number of concurrent callers. A single migration can miss the gap between transactions. Does the suite also cover one migrator failing while another waits for the advisory lock?

Collapse
 
harut_petrosyan profile image
Harut Petrosyan •

Thanks, that's exactly the lesson. With a single migration there is no gap between EF's per-migration transactions, so adding more concurrent callers never exposed the race. It only showed up once the history had a second migration.

To your question: partly. The suite covers two separate cases today:

  • Two migrators start together on an empty database. Both succeed, and each migration is applied exactly once.
  • A migrator fails on incompatible history (duplicate or missing references, unbalanced entries, and a few more). The test then checks that the advisory lock is free again.

What it doesn't cover yet is your case, where the failure happens while a second migrator is already blocked on the lock. I'm adding two tests for it:

  1. Two migrators wait behind the lock on bad history. Each must fail with the same clear error and leave no partial schema behind.
  2. The connection holding the lock is killed while another migrator waits. The waiter must take over and apply every migration exactly once.

Thanks for the nudge, I'll update the post when they're in.

Collapse
 
harut_petrosyan profile image
Harut Petrosyan •

Both tests are in now (164 in total). I added a short update at the end of the post with what each one checks. Thanks again.

Collapse
 
bobsmedia profile image
Bob's Media •

Thanks for following up and adding both tests. The killed-connection case is especially useful: it checks that the waiting migrator can recover and finish the work, beyond checking that the lock gets released. Very cool of you to share the results.

Thread Thread
 
harut_petrosyan profile image
Harut Petrosyan •

Thanks, Bob! Glad you found the killed-connection case helpful. Both tests will ship with the next release of the kit.

Collapse
 
challan116ux profile image
challan116-ux •

The audit helper is the real lesson here: a global zero-sum check can pass while individual transactions are wrong in ways that cancel out, and only the per-wallet, per-entry invariants catch it. Starting races behind a common barrier is the other detail worth stealing — concurrency bugs that need requests to land in the same moment will hide from any suite that starts threads politely one at a time. It mirrors document-flow testing closely: the happy-path file always parses, and the value is in the checks that verify every line still reconciles after the ugly cases run.

— Chris, founder of SignalEDI

Collapse
 
harut_petrosyan profile image
Harut Petrosyan •

Appreciate the deep dive, Chris! Exactly right about starting races behind a common barrier. Without it, the requests tend to reach PostgreSQL one after another, and the race never really happens. In the tests, each racing request also goes over HTTP with its own DbContext and its own physical database connection (pooling is off).

On the audit: the global sum is still worth checking, but it's the per-transaction and per-wallet replay that catches errors that cancel out. Settlement wallets are included too, because leaving them out could hide a partial write.

Curious how this looks on your side: in EDI, do you reconcile every line of a document, or the totals per batch?