DEV Community: Vadim Toptunov

I audited 100 third-party VS Code extensions. 19 ship known-CVE deps

Vadim Toptunov — Tue, 10 Nov 2026 15:00:00 +0000

Live Server — 84 million installs, the top row of the results below — bundles websocket-driver 0.7.4 and gets caught by CRITICAL GHSA-xv26-6w52-cph6 / CVE-2026-54466. It is the loudest finding in a scan of the top 100 third-party extensions on the VS Code Marketplace, but not the only one.

Every finding below is at the evidence level "Present": the vulnerable (name, version) pair is inside the shipped .vsix bundle. I haven't shown that any of them is reachable or exploitable; the levels are explained further down. The point of the post is not "these extensions can be hacked." It is that VS Code gives users no interface to see the dependency surface of what they install, and I could not find a publicly documented Marketplace mechanism that re-checks published bundles against advisories that appear after release.

Why I ran this

Two reasons.

Workspace Trust mitigated the easy extension attacks. Since VS Code 1.57 (May 2021), an extension that does not declare capabilities.untrustedWorkspaces support is disabled when you open a repo marked untrusted. The classic "clone a malicious repo, press F5, get RCE via .vscode/settings.json" class is now gated behind the user's trust decision. The gate is not absolute — a user who clicks "Trust" through habit re-opens the surface, extensions that declare support run anyway, and the check does not reach dependency CVEs — but the easy path is meaningfully smaller than it was four years ago.

But what is under the hood? VS Code extensions ship their own node_modules inside the .vsix bundle. Whatever the maintainer npm installed the day they published is frozen in. Advisories published after that release do not propagate backwards; I could not find a publicly documented mechanism in the Marketplace or in VS Code that tells either the maintainer or the user "an advisory now applies to something you shipped three months ago." That's the frozen-bundle problem.

I wanted to measure it.

Method

Reproducible in ~15 minutes with stdlib Python. No scanner install required.

Snapshot: Marketplace and OSV queried on 2026-11-08. Install counts and advisory data are from that date; both change over time.

1. Pull the top 200 extensions from the Marketplace extensionquery API, sorted by install count.

2. Filter the sample. Drop first-party publishers (ms-*, vscjava, VisualStudioExptTeam, MicrosoftDocs, redhat, Microsoft, MS-CEINTL), GitHub, and anything whose name or display title contains language-pack, icon-theme, material-theme, vscode-icons, theme, keybindings, or cheatsheet. From 200 raw entries this kept 116; I took the top 100 by install count. The precise phrase is "top 100 non-first-party, non-theme code extensions by install count on 2026-11-08", not "top 100 extensions on the Marketplace".

3. Download each .vsix from the /vspackage endpoint. A .vsix is a zip. Gentle pacing at 1 req/sec. All 100 downloads succeeded (~1.6 GB total).

4. Enumerate every node_modules/<pkg>/package.json inside extension/ after unzipping. For the top 100 this gave 1,357 nested dependency manifests, of which 935 were unique by (name, version) (712 unique package names, many shipped in two or more extensions at the same version). 64 of the 100 extensions had zero shipped node_modules entries — some genuinely have no runtime deps, most are built with esbuild/webpack/rollup into a single JS file, and the scanner is blind to CVEs inside those bundles. See Limitations.

5. Batch-query OSV.dev at /v1/querybatch — 500 queries per request, free, no auth. OSV covers GHSA, so every npm advisory shows up. Collect every matched advisory.

6. Score and rank. For each (extension, dependency, advisory) row I keep the worst severity and compute score = installs × severity_weight with CRITICAL=10, HIGH=5, MODERATE=2, LOW=1. The results table below is sorted by install count — most-shipped at the top. The score column is in the companion CSV for anyone who wants to re-rank.

Affected means: at least one (dependency, version) pair inside the extension bundle has a matching OSV advisory of any severity. It does not imply the code path is loaded at runtime, that user input can reach it, or that any impact is realised.

Six Python scripts, 527 lines, stdlib only, plus diff_runs.py for comparing two runs. Three APIs: Marketplace's extensionquery, Marketplace's /vspackage, OSV's querybatch. No auth. No quota.

Results

Three different counts, because the numbers answer three different questions:

19 / 100 sampled extensions had at least one (dependency, version) inside their bundle matching an OSV advisory.
103 unique (extension, dependency) pairs with ≥1 matching advisory. A pair can match multiple advisories — a dep may have several open advisories or be below more than one fix threshold — so this is less than the total advisory matches.
149 distinct OSV record IDs across those 103 pairs — all GHSAs; alias-overlap check against each record's CVE aliases found no duplicate underlying vulnerability (i.e. 149 records = 149 underlying issues for this sample, with 2 of the 149 having no CVE alias). One advisory often appears in many extensions (lodash, minimatch, braces, brace-expansion and semver show up most often), so advisory-id-count and pair-count diverge.
Severity breakdown of the 103 pairs, each counted once at its worst-severity advisory: 12 CRITICAL, 61 HIGH, 24 MODERATE, 6 LOW.

Sorted by install count on the snapshot date. "Worst sev" is the maximum severity across all matched advisories inside that extension's bundle.

#	Installs	Extension	Worst sev	Lead deps (abridged)
1	84,254,316	`ritwickdey.LiveServer`	CRITICAL	`websocket-driver 0.7.4`, `braces`, `lodash`, `js-yaml`, +14 more
2	42,727,107	`formulahendry.code-runner`	HIGH	`braces`, `picomatch` (ReDoS class)
3	19,651,537	`christian-kohler.path-intellisense`	HIGH	`json5`, `minimatch`, `brace-expansion`
4	18,267,232	`batisteo.vscode-django`	HIGH	`toml`
5	15,805,395	`xdebug.php-debug`	HIGH	`@xmldom/xmldom`, `moment`, `brace-expansion`
6	15,603,346	`octref.vetur`	HIGH	`lodash`, `js-yaml`, `cross-spawn`, +3 more
7	15,369,970	`EditorConfig.EditorConfig`	HIGH	`brace-expansion`, `minimatch`
8	13,517,697	`oderwat.indent-rainbow`	CRITICAL	`decompress` (zip-slip), `koa`, +4 more
9	12,451,186	`cschlosser.doxdocgen`	CRITICAL	`simple-git`, `moment`
10	10,683,614	`HookyQR.beautify`	CRITICAL	`minimist` (proto pollution), +4 more
11	10,027,714	`Zignd.html-css-class-completion`	CRITICAL	`form-data`, `json-schema`, `lodash`, +3 more
12	9,742,813	`naumovs.color-highlight`	HIGH	`brace-expansion`, `minimatch`
13	8,170,674	`DotJoshJohnson.xml`	CRITICAL	`lodash`, `minimist`, `xmldom`, +3 more
14	7,479,892	`platformio.platformio-ide`	CRITICAL	`tar` (symlink override), +5 more
15	6,766,249	`CoenraadS.bracket-pair-colorizer-2`	CRITICAL	`minimist`, `json5`
16	6,696,727	`Oracle.oracle-java`	HIGH	`brace-expansion`, `fast-uri`
17	6,140,258	`zobo.php-intellisense`	HIGH	`brace-expansion`, `cross-spawn`, `minimatch`
18	5,887,595	`James-Yu.latex-workshop`	MODERATE	`glob`, `minimatch`
19	5,517,213	`steoates.autoimport`	HIGH	`lodash`

Full per-extension, per-advisory breakdown with versions and OSV IDs is in SUMMARY.md and DATASET.csv in the companion repo.

Levels of evidence

The headline number is Present, not Exploitable. The distinction matters, and most writeups in this space blur it.

Level	What has been established
Present	A vulnerable `(name, version)` pair is in the shipped bundle
Loaded	The extension's runtime actually loads this copy of the module
Reachable	A code path that reaches the vulnerable code is exercised during normal use
Exploitable	An adversary-controlled input reaches the vulnerable code in a reproduced PoC

Every number in the Results table is Present. For a few deps I believe Loaded is very likely from static reading, but I did not instrument the extensions to prove it. Reachable and Exploitable are not claimed for any row.

Publishing at Present is still worth it: the frozen-bundle pattern is a measurable property of the ecosystem, and 19 / 100 is a baseline you can compare future runs against. Upgrading individual rows to higher evidence levels is per-extension work, not a batch job.

Limitations

Three things to flag before anyone cites a number from this post (and the evidence level above: every row is Present, nothing more):

The scanner cannot see dependencies inside bundled JS. It walks extension/node_modules/<pkg>/package.json. An extension built with esbuild, webpack, or rollup into a single JS file contains its dependencies' code but no per-dep package.json, so the scanner records zero deps for it. 64 of the 100 sampled extensions had zero shipped node_modules entries (a measured count). Some genuinely have no runtime deps; the rest are most likely bundled (I did not individually inspect each build output), and whatever CVEs are riding inside those bundles are invisible to this method. So 19 / 100 is the observed rate of extensions in this sample where this manifest-based inventory matched at least one OSV advisory — not a lower bound on vulnerable code, and not an estimate of overall prevalence of vulnerable code in VS Code extensions. Covering the bundled case properly needs an SBOM at build time, which the Marketplace does not require.
The sample is top-by-installs, not top-by-risk. A niche extension with 50k installs might ship worse code than a top-10 extension and wouldn't appear in this cut. The chosen sample serves the "most-shipped bundles" question; a risk-weighted sample would look different.
One snapshot is a data point, not a trend. The repo pins 2026-11-08 under runs/ and ships diff_runs.py for quarterly comparisons. The next re-run is scheduled for 2027-02-08.

A closer look at Live Server

The highest-install row in the table. Tracing it as far as the static scan allows.

ritwickdey.LiveServer 5.7.10 ships extension/node_modules/live-server/ — the live-server npm package, which the extension wraps as its HTTP + live-reload server. live-server depends on faye-websocket, which depends on websocket-driver 0.7.4 — present under extension/node_modules/websocket-driver/.

GHSA-xv26-6w52-cph6 / CVE-2026-54466 — "websocket-driver: Message corruption via abuse of protocol length headers". Severity critical per the GitHub Advisory API. Vulnerable range: < 0.7.5; first patched version: 0.7.5. Advisory published on 2026-07-15 (cross-checked against the GitHub Advisory API on 2026-11-08).

The chronology is the interesting part. Live Server v5.7.10 was last updated on the Marketplace on 2026-02-27 — about five months before the GHSA on websocket-driver was published. The maintainer wasn't ignoring a known advisory. The advisory simply did not exist yet. A reasonable npm install on 2026-02-27 picks up websocket-driver 0.7.4, which was clean at the time. Five months later an advisory is filed against that version. The bundle on disk does not change, and I could not find anything user-visible in the Marketplace that surfaces the new risk.

Reachability without a PoC: faye-websocket is live-server's WebSocket layer, and the live-reload channel is the extension's headline feature, so this copy of websocket-driver is very likely loaded when Live Server is running. I did not instrument the extension to confirm, and I did not attempt a crafted-frame reproduction — the finding sits at Present and likely Loaded, no higher. If you do the dynamic work, the repo has everything needed to reproduce the inventory and I'll link verified follow-ups.

Static takeaway: an extension ranked 8th by installs on the unfiltered Marketplace on 2026-11-08 (and 1st in this filtered third-party sample) ships a CRITICAL advisory on a module that sits under its headline feature, four months after the advisory was filed, with nothing user-visible in the install flow flagging it.

Three observations from the shape of this list

Install count is not a maintenance signal. 84M installs doesn't mean anyone upstream is watching the advisory feed on your behalf. It means a lot of people downloaded the thing once. The maintainer may or may not be active — in Live Server's case they shipped a release about eight months before this snapshot, and the vulnerable dep is still there because the advisory postdates the release.

The still-on-disk-after-dormancy problem. Bracket Pair Colorizer 2 was last updated on the Marketplace in March 2022. VS Code has shipped native bracket pair colorization in the core editor since 2021 (editor.bracketPairColorization.enabled). The extension has accumulated roughly 6.7M Marketplace installs; the public count doesn't say how many active copies remain, but every active copy still loads on VS Code startup if enabled and still ships minimist with a CRITICAL prototype-pollution advisory. VS Code does not uninstall dormant extensions on your behalf.

Lots of overlap on the same tired deps. brace-expansion, minimatch, braces, lodash, semver show up across most of the affected extensions. One upstream fix, one maintainer release, several vulnerable extensions become clean. Maintainers usually don't notice until someone emails them.

The size side of the problem

Not every extension ships a big bundle, but the ones that do carry a lot of dead surface along for the ride. Live Server's unpacked extension/ directory is 30 MB, and almost all of that is extension/node_modules/. The actual Live Server glue code — the TypeScript that bridges live-server into the VS Code extension API — is tiny; the bulk is transitive dependencies the npm resolver dragged in.

Bundling with esbuild or webpack can substantially reduce unused code — whether it removes a specific vulnerable path depends on the dependency graph, side effects, dynamic imports, and build configuration. A transitive dep that is called stays in the bundle, and the vulnerable code path inside it still ships, but bundling typically shrinks both the bundle and the matched-advisory count. Measure the audit output before and after; do not assume the number went to zero.

For comparison: Code Runner's whole unpacked tree is 1.8 MB (1.2 MB of node_modules). Still shipping two HIGH advisories, but far less dead surface by volume.

What VS Code could do

OSV's /v1/querybatch is free. The Marketplace already runs malware scans on .vsix uploads. Running OSV against the shipped node_modules of a .vsix on upload — exactly what the scripts in the repo do — would surface the top-20 findings on this list in an afternoon. This same approach runs into the bundled-JS blind spot (see Limitations); covering bundled extensions properly needs an SBOM at build time, which vsce could require as a publisher input.

More importantly, the same scan could run periodically against already-published .vsix artefacts and notify the publisher when a new advisory applies. A count surfaced on the public install page would let users factor it into their choice. That's the backfill mechanism I could not find in the public Marketplace feature set at the snapshot date — if one exists internally, I'd update the post.

What maintainers can do

For anyone shipping a VS Code extension right now:

Audit the final .vsix, not just the source tree. npm audit against your working copy reports what's in your lockfile — a source-tree check, not an inventory of what shipped. For extensions that ship unpacked node_modules: vsce package, unzip, enumerate every (name, version) under extension/node_modules/*/package.json (and the @scope/name variant), query OSV. That's an artifact-level inventory for the unpacked part — exactly what audit_extensions.py in the repo does. For extensions you ship as a bundled JS file this inventory misses your deps entirely: generate an SBOM at build time (e.g. CycloneDX from your bundler's manifest), keep provenance linking the SBOM to the published .vsix, and audit the SBOM alongside the artifact. Neither path proves anything about reachability or exploitability on its own.
Bundle with esbuild or webpack. Shipped node_modules directories are often large and contain code the extension never calls. Tree-shaking typically reduces both size and the matched-advisory count. Not a guarantee — vulnerable paths inside deps you do call still ship. Measure before/after.
Re-audit already-published releases on a schedule. The chronology in the Live Server case — advisory published five months after the release — isn't unusual. A quarterly re-check of your already-published .vsix against OSV catches advisories that appear after you ship. Treat it as the backfill the Marketplace does not provide.

What users can do

Honest answer: not much, today.

Don't install extensions you don't use. It's the one step that removes the dependencies entirely.
Prefer extensions whose Marketplace page shows a recent "last updated" date. A long-stale date is a yellow flag. Recent activity isn't a clean bill of health — Live Server was updated about eight months before this snapshot and still ships a CRITICAL — but a long gap is clearly worse.
Keep Workspace Trust ON. It does not address dependency CVEs, but it gates the much worse "arbitrary RCE on repo open" class behind an explicit prompt.

Running it yourself

query_marketplace.py     87 lines — Marketplace extensionquery
filter_publishers.py     49 lines — drop first-party + themes
download_vsix.py         61 lines — /vspackage, 1 req/sec
audit_extensions.py     137 lines — unpack + OSV batch
rank_findings.py        113 lines — enrich with GHSA severity
summarize.py             80 lines — collapse + markdown
                        ---
                        527 lines — stdlib only

diff_runs.py            108 lines — compare two snapshots

Running it quarterly against the top 100 is a useful ambient check. Running against the top 500 is the same with an hour-long download. I'll probably do both and track the shape over the next year — next snapshot already scheduled for 2027-02-08.

Full per-extension breakdown with every (dep, version, advisory) row, the CSV dataset and the scripts are in the companion repo:

https://github.com/vadimtoptunov/kassi-vsix-audit

If you re-run this against a different sample or a later snapshot and the shape of the list has moved, send me the diff — I'll link it.

UI tests that stayed green on a broken app, and what it took to make them fail

Vadim Toptunov — Mon, 02 Nov 2026 17:00:00 +0000

I had 34 UI tests, 135 switchable bugs, a mock backend that answered in 120 ms, a price feed that ticked every second, and a minus sign that wasn’t a hyphen. Not that I needed all of that to test one banking app. But once you’ve built a serious collection of bugs, you want to switch every one of them on and see what happens.

On the clean build, all 34 tests were green. That’s nice, but it only says the app works. It says nothing about whether the tests would notice if it didn’t.

So I started breaking the app on purpose and watching which tests went red. The first results weren't great. Nine of the 34 tests never went red, whatever I broke. And when I switched on the 23 money bugs one at a time, 18 of them got through the whole suite without a single red test.

Below are the four that taught me the most, and what I changed in the tests. None of it is specific to my app or my framework. I'd bet most UI suites have at least one of these.

How the app gets broken

The app is ChaosBank, a banking app I built to practise mobile testing on (intro post). By default its code is correct. It also has 135 named bugs, and each one can be switched on at launch:

if Defects.isActive(.pnlSign) {
    // the small, deliberate bug
} else {
    // the correct implementation
}

The tests are the ChaosBank × KassiOS reference suite, written on KassiOS, my layer over XCUITest (the post about it). A bug is switched on through the test runner's environment, so the tests themselves stay the same:

TEST_RUNNER_CB_INJECT_DEFECTS=pnlSign xcodebuild test-without-building …

If this sounds like mutation testing, it is, just at a different level. Tools like PIT, Stryker and Muter change your code automatically thousands of times and rerun the unit tests after each change. UI tests are far too slow for that, so ChaosBank's bugs are written by hand, at the level a user would notice: a loss shown as a gain, a total that skips an account. The suite runs once per bug. If some test goes red, the bug is "killed". If everything stays green, it "survived".

A wrong number is still a number

This was the first test that stayed green with every bug on at once:

func test_portfolio_showsTotalAndPnl() {
    launchUnlocked(tab: "portfolio")
    onScreen(PortfolioScreen.self) { p in
        p.totalValue.assertVisible()
        p.pnl.assertPresent()
        p.list.assertExists()
    }
}

One of those bugs, pnlSign, shows TSLA's loss as a gain. The test didn't care. It checks that the numbers are on the screen, and a wrong number is on the screen just as well as a right one. To be fair, the name says exactly that: "shows total and P&L". It never promised they'd be correct.

The fix should have been one line. TSLA is held at a loss, so its P&L should have a minus in it:

func test_portfolio_tslaLoss_isRenderedAsLoss() {
    launchUnlocked(tab: "portfolio")
    onScreen(PortfolioScreen.self) { p in
        p.holdingPnl("TSLA").assertLabelContains("−")
    }
}

It took two detours.

The first: on the clean build the test failed with "does not exist", even though portfolio.holding.TSLA.pnl was right there in the view code. I dumped the accessibility tree and saw that every text in the holdings list had the identifier of the list itself. It turns out that in SwiftUI, .accessibilityIdentifier on a container gets passed down to its children and replaces their own identifiers, unless the container is marked .accessibilityElement(children: .contain). The row identifiers had never reached XCUITest at all.

That explained the old test. It could only check presence, because there was no way to find the value. I looked for the same pattern in the rest of the app and found it in 14 more places: the login gate, the card, the market rows, the transaction rows, the confirm sheets. The tests had quietly worked around nearly all of them. The CVV, for example, was found by looking for a text containing "CVV" among everything on the card. Fixing the app took one modifier per container, and after that I could delete 81 lines of workarounds from the tests.

The second detour: my first version checked for "-" and failed on the clean build with expected label containing '-' but found '−$229.84'. ChaosBank formats money with U+2212, the real minus sign. It looks like a hyphen but it's a different character. If you check a formatted value, check it for the characters the formatter actually puts there.

€100 was the wrong amount to test with

transferRoundsUp takes €101 from your account when you send €100.40. The original transfer test sent €100. With €100, a correct transfer and a rounded-up one are the same €100, so no check in the world could have caught this with that input.

There's a second bug on the same flow, transferDebitsWrongAccount, which takes the money from USD instead of EUR. That one is easy to catch, as long as the test looks at EUR at all. The old test only checked that the confirm sheet closed.

The new test sends an amount with cents, and also checks the account that shouldn't change:

/// A €100.40 transfer debits exactly €100.40 from EUR and leaves USD alone.
/// Catches `transferRoundsUp` and `transferDebitsWrongAccount`.
func test_transfer_debitsExactlyTheAmountFromEUR() {
    launchUnlocked()
    var eurBefore = Double.nan, usdBefore = Double.nan
    onScreen(HomeScreen.self) { home in
        eurBefore = home.account("EUR").lastNumber()
        usdBefore = home.account("USD").lastNumber()
        home.transferAction.tap()
    }
    onScreen(TransferScreen.self) { transfer in
        transfer.recipientField.typeText("Alex")
        transfer.amountField.typeText("100.40")
        transfer.continueButton.tap()
        transfer.confirmButton.tap()
        transfer.confirmButton.within(timeout: 20).waitUntilGone()
    }
    onScreen(HomeScreen.self) { home in
        XCTAssertEqual(home.account("EUR").lastNumber(), eurBefore - 100.40, accuracy: 0.005)
        XCTAssertEqual(home.account("USD").lastNumber(), usdBefore, accuracy: 0.005)
    }
}

Every number was right except the total

totalValueOmitsHolding leaves ETH out of the portfolio total. Each holding shows the right value. The total is formatted correctly. It's just the wrong sum, and none of my tests ever compared the total with the holdings.

So I wrote that test: read the holdings, add them up, compare with the total. It failed on the clean build every now and then. Prices in ChaosBank tick, and reading four labels one after another can catch two different ticks. The values have to be read at the same moment:

func test_portfolio_total_equalsSumOfHoldings() throws {
    launchUnlocked(tab: "portfolio")
    let ids = ["AAPL", "TSLA", "ETH"].map { "portfolio.holding.\($0).value" }
    var labels: [String: String] = [:]
    onScreen(PortfolioScreen.self) { p in labels = p.readTexts(ids + ["portfolio.totalValue"]) }
    let sum = try ids.map { try XCTUnwrap(numbers(in: try XCTUnwrap(labels[$0])).first) }.reduce(0, +)
    let total = try XCTUnwrap(numbers(in: try XCTUnwrap(labels["portfolio.totalValue"])).first)
    XCTAssertEqual(total, sum, accuracy: 0.02)
}

readTexts reads all the labels from one snapshot of the screen. Once I had it, I wrote a few more checks of the same kind: the Home total against its three accounts at the app's exchange rates, a holding's value against the market price, and the loan's monthly payment against the advertised rate. On a money screen, a lot of what can go wrong is in how the numbers relate to each other, not in any single number.

The double tap that never raced

exchangeDoubleSubmit removes the guard that stops you from tapping Execute twice, so a fast double tap makes two exchanges. I had a test that taps twice. It stayed green.

There were two reasons. The mock backend answers in about 120 ms, and two separate synthesized taps arrive further apart than that. So the first exchange had already finished when the second tap came in, the guard had nothing to stop, and two exchanges in a row is also what two deliberate taps would give you. XCUIElement.doubleTap() wouldn't help either: it's one system gesture, and the button turns it into one action. It can't race anything.

What worked was slowing the network down. ChaosBank's dev menu has a "Slow" network setting (a network condition, not a bug), which keeps the request in flight for 3 seconds. The test turns it on, taps twice on the same spot without checking the button in between, and then counts rows in the history instead of checking that the sheet closed:

onScreen(ExchangeScreen.self) { ex in
    ex.amountField.typeText("100")
    ex.executeButton.rapidTap(times: 2)
    ex.executeButton.within(timeout: 20).waitUntilGone()
}
onScreen(HomeScreen.self) { $0.seeAllActivity.tap() }
onScreen(TransactionsScreen.self) { tx in
    tx.count.assertValue(closeTo: countBefore + 1, tolerance: 0.001)
}

With the guard in place there's one new row, without it there are two. The same trick caught the double charge on transfers and the double buy on orders.

Whole groups that turned nothing red

In the first run, the localization and flaky groups didn't turn a single test red, and security turned one. Nothing was wrong with the tests there. There just weren't any tests for those things. You don't see that from a green run on the clean build. You only see it when you switch the bugs on.

Most of the tests I added for them are short:

under the German locale, numbers group as 1.234.567,89 and the euro sign comes last;
a transfer amount typed as 1,000.50 parses correctly;
a wrong OTP is rejected, and three wrong ones lock the step;
the card number is masked except for the last four digits;
the session token is in the Keychain (the dev menu reports where it is);
a deep link opened while the app is locked doesn't skip the login screen.

Two bugs no test can catch on this data

Of the 23 money bugs, 21 are caught now. The other two, detailChangeWrongBase and pnlPercentAbsOnly, look exactly like the correct code. The first computes a percentage against the wrong base, but on the simulated price feed both bases give the same number. The second drops the sign from a P&L percentage, but the seeded portfolio is up overall, so the sign is positive anyway.

In mutation testing these are called equivalent mutants. Here they're only equivalent on this particular data. I didn't write tests for them, but they're a good reminder that the test data decides which bugs you can see in the first place.

Where the suite is now

There are 71 tests. Every test that's meant to catch a bug says which one in a doc comment:

/// A €100.40 transfer debits exactly €100.40 from EUR and leaves USD alone.
/// Catches `transferRoundsUp` and `transferDebitsWrongAccount`.

A script collects these lines into a list, and kassios mutants switches on each listed bug and runs its tests, one bug at a time. Before that it runs all the tests on the clean build, and they all have to pass. All 59 bugs on the list are caught. That's 59 of ChaosBank's 135. The performance and network bugs have no tests yet, and most of the UI and state ones aren't covered by any test.

There's one problem I haven't fixed. On a slower CI machine running two simulators at once, two of the 59 got through. One was the deep link test. It waits a fixed 2 seconds and then checks that the login screen is still there. Under load, the app most likely hadn't handled the link within those 2 seconds even with the bug on, so the test passed. A test that checks "nothing happened" by waiting a while will pass whenever the machine is too slow for anything to happen yet. The fix is to wait for a sign that the link was handled, and that's still on my list.

Trying this on your own app

You don't need ChaosBank or KassiOS for any of this. Pick one screen with numbers on it. In a local build, break one thing on purpose: flip a sign, drop an item from a total, round an amount. Run your tests. If nothing goes red, you know which test to write next. And before you trust a new test, make it fail once on a build where the thing it checks is broken.

If you'd rather have something ready to break, every ChaosBank bug is listed in exercises.json, with the screen, the locators and what the broken build does. You can switch one on from the command line:

xcrun simctl launch booted VadimToptunov.ChaosBank \
  -ChaosBankStartUnlocked 1 -ChaosBankTab portfolio -ChaosBankDefects pnlSign

Nobody has touched the performance and network bugs yet, if you're looking for somewhere to start. And I'd like to hear which of these mistakes you've seen in your own tests, and which ones I haven't run into yet.

KassiOS: a Kaspresso-style layer for XCUITest, tested against a broken bank

Vadim Toptunov — Wed, 28 Oct 2026 16:12:26 +0000

My background is QA automation, mostly mobile and mostly Appium. When I decided to learn native iOS UI testing properly, XCUITest looked like the obvious choice: it's Apple's own framework, it needs no automation server, and it works with the accessibility hierarchy directly. Then a simple login turned into this:

let email = app.textFields["login.email"]
XCTAssertTrue(email.waitForExistence(timeout: 10), "Email field did not appear")
email.tap()
email.typeText("test@example.com")

let loginButton = app.buttons["login.submit"]
XCTAssertTrue(loginButton.waitForExistence(timeout: 10), "Login button did not appear")
loginButton.tap()

XCUITest gives you primitives and leaves waiting, retries, screen objects and reporting to you. In most projects I've seen, that gap gets filled by a waitAndTap() extension, then a few more, until there's a small framework that nobody tests. I decided to write that framework on purpose and see where it went. That became KassiOS.

The ideas come from Kaspresso, which I liked on Android: a test describes what the user does, the framework handles the waiting, and every step shows up in the report. The API is my own, and underneath it's still plain XCUITest in your UI-test target, with no dependencies.

What a test looks like

onScreen(LoginScreen.self) { login in
    login.email.typeText("test@example.com")
    login.password.typeText("secret")
    login.loginButton.tap()
}
onScreen(HomeScreen.self) { $0.welcome.assertVisible() }

final class LoginScreen: KassScreen {
    lazy var email = textField("login.email")
    lazy var password = secureTextField("login.password")
    lazy var loginButton = button("login.submit")

    override var onLoad: [KassElement] { [email, loginButton] }
}

Before the block runs, KassiOS checks the elements in onLoad. If the app ended up on a different screen, the test fails right there, in the "On LoginScreen" step, instead of on whichever field it happened to touch first. Every onScreen block is an activity in Xcode's report, so a red run reads as a list of steps.

The test is shorter, but that isn't what I was after. I wanted the waiting and retrying to live in one place, in the framework, so that each test doesn't carry its own copy.

Making one tap trustworthy

The first version of tap() waited for the element and tapped it. Most of the work since then went into the cases where that isn't enough. Today a single login.loginButton.tap(), within one time budget:

waits for the element to exist and be hittable. Internal retries share the budget, so three attempts at a 10-second action take 10 seconds, not 30;
scrolls to the element if it's off screen, and dismisses the keyboard if it covers the target. It uses Done or Hide for that, never Return, because Return can submit a form;
taps once. Only the preconditions are retried. A framework that taps again "just in case" can cause a double submit, or hide a real one;
finds the element again on every attempt. A KassElement stores how to find the element (() -> XCUIElement), not the element itself, so if SwiftUI rebuilds the view, the next attempt gets the new one.

Some actions also check what they did. setSwitch(on:), toggle(), replaceText and adjustPicker read the element back and fail if its state didn't change. Without it, a switch whose tap lands on its label, next to the control, returns as if it had worked, and the test can pass with the switch still off.

When something does fail, the report gets a screenshot of the screen at that moment and the full accessibility tree, and the message tries to say why:

KassiOS: text 'welcom' — assertVisible failed: does not exist
  ↳ element not found in the current hierarchy
  ↳ did you mean: 'welcome', 'Welcome!'?

Waiting has a limit too. A retry that forgives too much turns a broken app into a green test, so in KassiOS waiting only gives the UI time to reach a state. When the budget runs out, the test fails with the last reason the framework saw. Each of these behaviors can be turned off in KassConfig.

Checking it against a broken bank

At some point I had tidy abstractions, a lot of unit tests, and no idea whether any of it caught bugs. A green UI test tells you the app did what the test checked. It doesn't tell you the test would turn red if the bug it was written for showed up.

To find out, I built ChaosBank, a banking app with bugs you switch on at launch (previous post), and a reference suite that tests it through KassiOS. Every test in the suite has the same contract:

clean app        →  test passes
enable a defect  →  the same test fails

The first full run had 34 tests (Xcode 27.1, iPhone 17 simulator, M1 Max), and it was humbling:

test_portfolio_showsTotalAndPnl stayed green with every defect on, including one that shows a loss as a gain. It checked that the numbers were on screen, not that they were right.
The localization and flaky defect groups didn't turn a single test red. The suite had no locale or race checks at all.
Red runs were much slower than green ones. Most of the extra time went into my own "did you mean?" hint, which queried about 150 elements one by one. Reading a single hierarchy snapshot instead took the failing transfer test from 85 seconds to 23.

The suite also found a bug in KassiOS. After I added the read-back, one test failed like this:

KassiOS: textField 'order.limitPriceField' — replaceText('179.00') failed:
  typed '179.00' but the field reads '179.009.07'

The price field was right-aligned and pre-filled with 189.07. The tap at its centre put the cursor in the middle of the text, the deletes removed only 18, and the new value went in front of 9.07. The previous version cleared fields the same way. It just didn't look, so the test would have failed a line later on a wrong order total, with nothing pointing at the field. Now KassiOS taps the trailing edge before deleting.

What the defects added

Several checks came from defects that slipped through. Take successToastMissing: the transfer goes through and the sheet closes, but the success toast never appears. A test that waits for the sheet to close stays green. Catching it needed a different kind of wait:

/// Catches `successToastMissing`.
func test_transfer_happyPath_completes() {
    launchUnlocked()
    onScreen(HomeScreen.self) { $0.transferAction.tap() }
    onScreen(TransferScreen.self) { transfer in
        transfer.recipientField.typeText("Alex")
        transfer.amountField.typeText("100")
        transfer.continueButton.tap()
        transfer.confirmButton.tap()
        transfer.successToast.assertAppears(within: 2)
        transfer.confirmButton.within(timeout: 20).waitUntilGone()
    }
}

The toast is on screen for about 1.4 seconds. The normal budget is built for "wait until it's ready", and assertAppears polls a short window instead, for "catch it while it's there". A few more that I kept needing:

// A double submit: two taps on the same spot, no re-check in between.
order.placeButton.rapidTap(times: 2)

// A value that must not move while nothing happens.
home.totalBalance.assertUnchanged(for: 3)

// Prices that tick: read related values from one hierarchy snapshot.
let t = home.readTexts(["home.total", "home.account.EUR", "home.account.USD"])

// The right number of rows can still be the wrong rows.
history.cells().assertNoDuplicates()
history.cells().assertEach("is money-in") { row in
    guard row.readLabel().contains("+") else { throw KassError("expected a money-in row") }
}

verifyAll runs every assertion in its block and reports all the mismatches on a screen, not only the first. For conditions you compose yourself there's eventually, continuously(during:), anyOf and retry(times:), as in Kaspresso.

Validation defects tend to need the same form filled with many inputs, so there's a parameterized runner as well:

func test_login_validatesTheEmail() {
    parameterized([("a@b.c", true), ("bad", false), ("", false)], name: { $0.0 }) { email, valid in
        relaunch()   // a clean app for every case
        onScreen(LoginScreen.self) { login in
            login.email.replaceText(email)
            login.loginButton.tap()
            if valid {
                onScreen(HomeScreen.self) { $0.welcome.assertVisible() }
            } else {
                login.error.assertVisible()
            }
        }
    }
}

Each case is its own activity in the report, named by the name closure, so a red run says the "bad" case failed. A failing case doesn't stop the rest. Swift Testing has @Test(arguments:), but in Xcode 27.1 it only has a unit-test template, and the UI-test templates are still XCTest. I use Swift Testing for model and view-model logic and KassiOS for the flows, in separate targets.

One defect at a time

Running the suite against groups of defects tells you that something went red. It doesn't tell you which defect each test catches, or which defect nobody catches. For that, KassiOS has a command-line tool, kassios mutants. You list the defects and the tests each one must turn red, and it runs them one defect at a time:

mutant                 verdict
zeroAmountAccepted     SURVIVED  (still green: …/TransferTests/test_transfer_zeroAmount_disablesContinue)
balanceAfterAdds       killed
...
58 of 59 killed

--clean first runs every listed test with no defect on, because a test that's already red "catches" everything. In the reference suite the list is generated from a /// Catches `id` comment above each test, like the one on the toast test above, and CI fails if the generated file is out of date.

The suite has since grown from 34 tests to 71, and all 59 defects on the list are caught. Running them one by one takes about 70 minutes on my Mac. With --parallel 3, which runs three defects at once on clones of the simulator, it takes 28. Which tests stayed green the longest, and what it took to make them fail, is the next post.

A few other parts

kassios lint reads screen objects before anything runs and flags those without onLoad or with computed identifiers. Apple's accessibility audit can run from a test.
kassios agent start starts a small loopback-only Mac process for things a test can't do from inside the simulator: grant permissions, freeze the status bar, set a location. It picks its own token and port and tells the test where to find them. Without it, those calls skip with a message instead of hanging.
KassHTTP arranges state through the app's API, so a transfer test doesn't have to create an account through the UI first.
webView(), link and links() reach into a WKWebView. HTML has no accessibility identifiers, so web elements are found by label.
With recordVideoOnFailure and the agent running, a failed test also gets a video of the simulator screen. Allure and JUnit reporters are built in.

Scenarios, network stubs, snapshot tests and the rest are in the README and the Guide.

Who it's for, and who it isn't

I think KassiOS fits if:

your XCUITest suite has outgrown its helpers. There's a waitAndTap(), a base test class and page objects, and every new flaky test gets its own sleep;
a silent failure is expensive in what you test. Forms, payments and settings are where a tap that missed or a field that kept part of its old value lets a test pass on a broken app;
your team knows Kaspresso from Android and wants the same structure on iOS: screens, steps, scenarios, Allure;
you want to know what your suite actually catches. kassios mutants only needs the app to switch a defect on at launch, through a launch argument or an environment variable.

It's probably not for you if:

the suite is small. A dozen tests over a few screens are fine on raw XCUITest with a couple of extensions that every iOS developer can read;
you need one suite for iOS and Android. Appium and Maestro do that, through a server or a runtime of their own. KassiOS is iOS only, Swift inside the app's Xcode project;
you need the app's own idle state. EarlGrey runs inside the app and knows when it's idle. KassiOS stays out of process and polls. It has a synchronizer hook, with an example EarlGrey adapter in the repo;
you want nothing between a failure and its line. KassiOS actions call XCTFail internally and return the element, so with continueAfterFailure = true a chain keeps going after a failure;
you'd rather stay with Apple's tools than depend on a third-party layer with one maintainer.

None of this fixes flakiness that comes from the app, the test data or the environment. When something behaves oddly, you still need to know what XCUITest does underneath. Building KassiOS taught me more about that than writing tests would have.

Built with AI, checked like QA

I built KassiOS with a lot of help from AI coding tools. I'm a QA engineer, not a Swift framework developer, and that help let me go much further than I would have alone. It also meant checking everything: 172 unit tests, 59 UI tests against a small SwiftUI demo app in the KassiOS repo, the ChaosBank reference suite, and SwiftLint, all in CI on every pull request. The cursor bug above was found by the ChaosBank suite, not by me reading the code.

Trying it

.package(url: "https://github.com/VadimToptunov/KassiOS.git", from: "1.10.0")

Add it to your UI-test target, make the test class a KassTestCase instead of XCTestCase, and write one screen object, for example for the screen your flakiest test starts on. Plain XCUITest calls keep working next to it. The public API has followed semantic versioning since 1.0.

The command-line tools (the agent, the linter and the mutation runner) are one Homebrew formula, prebuilt, so there's nothing to build:

brew install vadimtoptunov/tap/kassios
kassios agent start

The library doesn't need them; install them when you want one of those features.

If you try it, I'd like to hear what feels unnecessary, what's missing, and where it gets in your way. Issues and PRs are welcome.

ChaosBank app — 131 switchable bugs to practice mobile testing on

Vadim Toptunov — Fri, 23 Oct 2026 15:01:45 +0000

For a while now I've been playing with different approaches to mobile UI test automation: plain XCUITest and Espresso/Compose, Appium, Maestro, page objects versus small DSLs, different ways of waiting and of dealing with flaky tests. Pretty quickly I ran into a boring but real problem: what do I actually run all this against?

The open-source apps I found mostly fell into two groups. Some only exist for Android, which doesn't help much when you want to compare approaches across both platforms. Others are intentionally simple: a login screen, a list, a details page. They're great for learning how to drive a simulator, but they don't have many of the bugs I've seen cause trouble in real projects, like:

a loss rendered as a gain,
a double-tap that sends a transfer twice,
1,000.50 parsed as 1.0005 because the device locale is German,
a stale network response overwriting fresh state,
a CVV printed right there on the card face.

A fintech app seemed like a good place for that kind of bug, since money, auth, live data, network timing and security all meet on a handful of screens. So I ended up building my own. Two, actually, one per platform, where the bugs are the feature. They're planted on purpose, written down, and you can switch each one on and off.

ChaosBank-iOS — SwiftUI (plus a UIKit build): https://github.com/VadimToptunov/ChaosBank-iOS
ChaosBank-Android — Kotlin + Jetpack Compose (plus an XML build): https://github.com/VadimToptunov/ChaosBank-Android

Both are MIT-licensed. Functionally it's a small neobank with a stock/crypto broker on top, loosely in the style of Revolut and Robinhood, and it currently has 131 named defects, each with a short written exercise.

I started it as a test bench for my own experiments, but it might also be useful as a practice ground if you're just getting into mobile development or mobile QA. Below is how it's put together, how I'd try using it, and a few things I'm still not sure about.

What it looks like

It's a working app rather than a mock-up. There's a web login (deliberately inside a WKWebView / WebView, more on that below), then OTP, a 6-digit passcode and Face ID. After that: a multi-currency home screen, transfers, currency exchange, transaction history with search and pagination, a markets screen with ticking prices, an order ticket, a portfolio with live P&L, and card controls.

The Android app tries to be a 1:1 port: same screens, same defects, same locator names.

Keeping the correct code and the bugs apart

This is the decision most of the project depends on.

The obvious way to make a buggy app is to just write buggy code. For what I wanted, that didn't work well: you can't easily tell whether a test caught the bug or failed for some unrelated reason, and the "correct" behaviour only exists in the author's head. When comparing frameworks, I wanted to know exactly what each run was supposed to catch.

So the production logic is written to be correct (money in Decimal, correct order pricing, correct balances), and each bug is a small override behind one check:

if Defects.isActive(.doubleCharge) {
    // small, isolated buggy override
} else {
    // the correct reference implementation
}

if (Defects.isActive(DefectId.doubleCharge)) {
    // small, isolated buggy override
}

What this gives you is a simple way to check that a test really guards something:

Run the test against the clean profile. It should pass.
Relaunch with the defect switched on. It should fail.
If it doesn't fail, the test probably isn't checking what you think it is.

Here's how that looks on screen. Same build, same screen. On the left is clean; on the right, two security defects are on:

This one is less obvious. Have a look at TSLA:

On the left, TSLA is down $290.56 and shown in red. On the right (pnlSign), the same loss shows up as a green +$290.56. All the elements are still there, so a test that only checks that they exist will pass. You need an assertion on the value to notice it.

And one more: is the total on the right the sum of the accounts?

It isn't. The account cards scroll horizontally, and the third one (GBP) is just off screen. With homeTotalOmitsAccount on, the total quietly leaves it out: €16,626.11 instead of €17,379.28. A check like "the total is a positive number" won't catch that.

Locators stay put

The second rule is that a defect changes behaviour or values, but not the accessibility identifiers. All IDs live in one file (Core/A11y.swift on iOS, core/A11y.kt on Android), and the strings are the same on both platforms: card.cvv is card.cvv everywhere.

The idea is that a suite written against clean should keep finding its elements when defects are switched on, and fail on the assertions instead.

That's also what I needed for my own experiments. You can point XCUITest, Espresso/Compose, Appium or Maestro at the same screens and compare how they deal with changed behaviour, how they wait, and how readable the tests end up, without the comparison being mostly about chasing moved locators.

Same bug, same ID, two platforms:

Switching bugs on

Defects are set by configuration, so you don't need a separate build for each one. There are three ways to do it.

1. Launch arguments. This is handy for CI and quick runs:

# iOS
xcrun simctl launch booted VadimToptunov.ChaosBank \
  -ChaosBankDefects pnlSign -ChaosBankStartUnlocked 1 -ChaosBankTab portfolio

# Android
adb shell am start -n com.vadimtoptunov.chaosbank_android/.MainActivity \
  -e CHAOSBANK_DEFECTS pnlSign -e CHAOSBANK_START_UNLOCKED 1 -e CHAOSBANK_TAB portfolio

You can enable a single defect, a whole profile (ui, validation, security, network, flaky…), or a difficulty bundle (beginner, middle, senior). StartUnlocked skips the auth flow so you land straight on the screen you care about. It only changes where you start, not how the app behaves.

2. The in-app developer menu. Long-press the build badge in the top-right corner. From there you can switch profiles, toggle individual defects, simulate offline/slow/flaky networking, flip RTL, and browse the exercises.

3. Prebuilt variants. Xcode build configurations (product flavors on Android) such as ChaosBank Flaky install next to the clean app under their own name. That can be useful if someone wants to do manual exploratory testing without touching launch arguments.

The exercises

Each defect has a matching entry in exercises.json:

{
  "id": "IOS-MON-02",
  "title": "Loss displayed as gain",
  "difficulty": "junior",
  "category": "money",
  "defects": ["pnlSign"],
  "launchArgument": "-ChaosBankDefects pnlSign",
  "expectedClean": "Loss shows negative / loss color.",
  "expectedBuggy": "Loss shows positive / gain color.",
  "task": "Open Portfolio and assert TSLA's P&L renders as a loss (negative, red).",
  "keyLocators": ["portfolio.holding.TSLA.pnl", "portfolio.pnl"]
}

Right now there are 12 junior, 97 middle and 22 senior exercises across ten categories: money, state, security, UI, validation, localization, networking, accessibility, concurrency and performance. The difficulty labels are my own guess and are probably off in places.

The junior ones are things like "a zero-amount transfer is allowed" or "the full card number isn't masked". The senior ones are mostly about timing: double-submits, a late response overwriting fresh state, animations that never quite settle.

If you're starting out in mobile QA

This is roughly the order I'd try things in. It's just one way to do it.

1. No code at first. Install the beginner build and poke around. Try not to read the defect list first. Write down what you find, then compare your notes with exercises.json. It's a fairly honest way to see what you notice and what you miss.

2. A first automated test. Pick a junior exercise and write a plain XCUITest for it. Something like:

func test_portfolio_lossIsRenderedAsLoss() {
    let app = XCUIApplication()
    app.launchArguments += ["-ChaosBankStartUnlocked", "1", "-ChaosBankTab", "portfolio"]
    app.launch()

    let pnl = app.descendants(matching: .any)["portfolio.holding.TSLA.pnl"]
    XCTAssertTrue(pnl.waitForExistence(timeout: 5))
    XCTAssertFalse(pnl.label.contains("+"), "A loss must not be shown as a gain: \(pnl.label)")
}

Or something similar with Compose on Android:

@Test
fun card_doesNotShowCvv() {
    val intent = Intent(ApplicationProvider.getApplicationContext(), MainActivity::class.java)
        .putExtra("CHAOSBANK_START_UNLOCKED", "1")
        .putExtra("CHAOSBANK_TAB", "card")
    ActivityScenario.launch<MainActivity>(intent)

    composeRule.onNodeWithTag("card.cvv").assertDoesNotExist()
}

These are rough sketches, not a finished suite. Run each one on clean (green), then with -ChaosBankDefects pnlSign or CHAOSBANK_DEFECTS=cardCvvVisible (red). If it stays green, that's worth digging into.

3. Timing and flakiness. The flaky profile has race conditions pinned to a seed. It's a decent place to see why sleep(2) tends to backfire. The network profile plus "slow" in the dev menu is another one.

4. Hybrid screens. The web login sits inside a WebView on purpose. By default it's opaque: Appium only sees NATIVE_APP, so you reach the fields through the native accessibility tree. With -ChaosBankInspectableWeb 1 (Android: -e CHAOSBANK_INSPECTABLE_WEB 1), a WEBVIEW_… context shows up and you can switch_to.context(...) into it. It's a small thing, but it tends to be confusing the first time you run into it in a real app.

There's also a hand-written UI suite for the iOS app: ChaosBank ⨯ KassiOS. It's another experiment of mine, a small Kaspresso-style DSL over XCUITest called KassiOS. It passes on clean and goes red on each injected defect. It might be more useful to look at after writing a few tests of your own.

If you're starting out in mobile development

The apps are also a small codebase with no third-party dependencies, so they might be worth reading. A few parts I spent the most time on:

Money. Decimal everywhere, except for one deliberate spot: the roundingDrift defect routes a single calculation through Double, so you can see what that does.
Determinism. The simulated price feed and the race-condition "coin flips" come from a seeded RNG (SplitMix64), so a failing run can be reproduced. Live Yahoo Finance quotes are available, but opt-in, because they make tests non-deterministic.
A mock backend as an actor, with timeouts, retries, offline mode and out-of-order responses.
Testable view models. Logic-layer coverage is around 97% on iOS and 95% on Android, checked in CI. The UI is left to UI tests.
Two UI stacks. The UIKit (UITableView) and XML (RecyclerView) builds exist because some bugs only really happen there, like a recycled cell showing the previous row's amount.

One exercise: pick a defect, find its Defects.isActive check, and read the correct else branch next to the buggy one. Another: try adding a defect of your own. The developer's guide has the steps. Note that CI compares the exercise catalogs of the two repos, so a new defect has to land on both platforms.

Using it for take-home assignments

This one I didn't plan for, but it seems to fit: test assignments for hiring, or onboarding someone onto a team. Since the bug set is just configuration, you can build a variant around what your team cares about:

For a security-focused role, use the security build: unmasked card data, an expired OTP that's still accepted, a deep link that skips the auth gate, a token in UserDefaults.
For a team fighting flaky tests, use flaky or senior, and ask for a suite that stays stable across, say, ten runs in a row.
For a junior role, use beginner and ask for a written bug report before any code.
For something narrower, pass your own list with -ChaosBankDefects / CHAOSBANK_DEFECTS.

Profile builds install as a separate app with their own name (ChaosBank Security, ChaosBank Flaky…), so a candidate can get a single APK or simulator build. The clean build gives the reviewer a baseline to run the submitted tests against.

One caveat: the catalog is public, so a candidate can look up the defect list. I'd probably treat it as open-book and focus the review on how the tests are written (waits, value assertions, how findings are explained) rather than on how many bugs were found.

A few things I learned

"Locators stay put" was harder than I expected. Some realistic bugs, like a recycled cell keeping a stale accessibility ID, are about locators. I ended up making those separate, clearly labelled defects instead of exceptions to the rule.
Randomness works better as an opt-in. Live prices are fun, but any assertion on a price or P&L stops meaning much when the number changes every second. So the seeded feed is the default.
Keeping two codebases in sync needs automation. Doing it by hand drifts. A CI script now compares the two exercise catalogs.
The clean build has to really be clean. If clean has an accidental bug, the "green on clean, red on buggy" idea falls apart. That's the main reason the reference suite exists.

Limitations

iOS needs a Mac with Xcode 26+ and an iOS 17+ simulator. Android needs JDK 21 and API 29+.
There's a reference UI suite for iOS, but none yet for Android. Espresso/Compose, Appium and Maestro suites are on the roadmap.
It's a sandbox. The "bank" lives in memory and the card number is a test one.
I'm one person maintaining two apps, so some defects are probably less realistic than others.

Links

iOS app: https://github.com/VadimToptunov/ChaosBank-iOS
Android app: https://github.com/VadimToptunov/ChaosBank-Android
User guide for QA/SDET: docs/USER_GUIDE.md
Reference UI suite: https://github.com/VadimToptunov/ChaosBank-KassiOS-Tests

Next on my list are Espresso/Compose, Appium and Maestro suites against the same screens, to see how the approaches compare in practice. If you give the apps a try, I'd be glad to hear which defects were hard to catch, which ones feel unrealistic, and which bugs from your own work are missing. Comments, issues and PRs are all welcome.