Vadim Toptunov

Posted on Oct 23 AI-assisted

ChaosBank app — 131 switchable bugs to practice mobile testing on

#testing #ios #android #beginners

For a while now I've been playing with different approaches to mobile UI test automation: plain XCUITest and Espresso/Compose, Appium, Maestro, page objects versus small DSLs, different ways of waiting and of dealing with flaky tests. Pretty quickly I ran into a boring but real problem: what do I actually run all this against?

The open-source apps I found mostly fell into two groups. Some only exist for Android, which doesn't help much when you want to compare approaches across both platforms. Others are intentionally simple: a login screen, a list, a details page. They're great for learning how to drive a simulator, but they don't have many of the bugs I've seen cause trouble in real projects, like:

a loss rendered as a gain,
a double-tap that sends a transfer twice,
1,000.50 parsed as 1.0005 because the device locale is German,
a stale network response overwriting fresh state,
a CVV printed right there on the card face.

A fintech app seemed like a good place for that kind of bug, since money, auth, live data, network timing and security all meet on a handful of screens. So I ended up building my own. Two, actually, one per platform, where the bugs are the feature. They're planted on purpose, written down, and you can switch each one on and off.

ChaosBank-iOS — SwiftUI (plus a UIKit build): https://github.com/VadimToptunov/ChaosBank-iOS
ChaosBank-Android — Kotlin + Jetpack Compose (plus an XML build): https://github.com/VadimToptunov/ChaosBank-Android

Both are MIT-licensed. Functionally it's a small neobank with a stock/crypto broker on top, loosely in the style of Revolut and Robinhood, and it currently has 131 named defects, each with a short written exercise.

I started it as a test bench for my own experiments, but it might also be useful as a practice ground if you're just getting into mobile development or mobile QA. Below is how it's put together, how I'd try using it, and a few things I'm still not sure about.

What it looks like

It's a working app rather than a mock-up. There's a web login (deliberately inside a WKWebView / WebView, more on that below), then OTP, a 6-digit passcode and Face ID. After that: a multi-currency home screen, transfers, currency exchange, transaction history with search and pagination, a markets screen with ticking prices, an order ticket, a portfolio with live P&L, and card controls.

The Android app tries to be a 1:1 port: same screens, same defects, same locator names.

Keeping the correct code and the bugs apart

This is the decision most of the project depends on.

The obvious way to make a buggy app is to just write buggy code. For what I wanted, that didn't work well: you can't easily tell whether a test caught the bug or failed for some unrelated reason, and the "correct" behaviour only exists in the author's head. When comparing frameworks, I wanted to know exactly what each run was supposed to catch.

So the production logic is written to be correct (money in Decimal, correct order pricing, correct balances), and each bug is a small override behind one check:

if Defects.isActive(.doubleCharge) {
    // small, isolated buggy override
} else {
    // the correct reference implementation
}

if (Defects.isActive(DefectId.doubleCharge)) {
    // small, isolated buggy override
}

What this gives you is a simple way to check that a test really guards something:

Run the test against the clean profile. It should pass.
Relaunch with the defect switched on. It should fail.
If it doesn't fail, the test probably isn't checking what you think it is.

Here's how that looks on screen. Same build, same screen. On the left is clean; on the right, two security defects are on:

This one is less obvious. Have a look at TSLA:

On the left, TSLA is down $290.56 and shown in red. On the right (pnlSign), the same loss shows up as a green +$290.56. All the elements are still there, so a test that only checks that they exist will pass. You need an assertion on the value to notice it.

And one more: is the total on the right the sum of the accounts?

It isn't. The account cards scroll horizontally, and the third one (GBP) is just off screen. With homeTotalOmitsAccount on, the total quietly leaves it out: €16,626.11 instead of €17,379.28. A check like "the total is a positive number" won't catch that.

Locators stay put

The second rule is that a defect changes behaviour or values, but not the accessibility identifiers. All IDs live in one file (Core/A11y.swift on iOS, core/A11y.kt on Android), and the strings are the same on both platforms: card.cvv is card.cvv everywhere.

The idea is that a suite written against clean should keep finding its elements when defects are switched on, and fail on the assertions instead.

That's also what I needed for my own experiments. You can point XCUITest, Espresso/Compose, Appium or Maestro at the same screens and compare how they deal with changed behaviour, how they wait, and how readable the tests end up, without the comparison being mostly about chasing moved locators.

Same bug, same ID, two platforms:

Switching bugs on

Defects are set by configuration, so you don't need a separate build for each one. There are three ways to do it.

1. Launch arguments. This is handy for CI and quick runs:

# iOS
xcrun simctl launch booted VadimToptunov.ChaosBank \
  -ChaosBankDefects pnlSign -ChaosBankStartUnlocked 1 -ChaosBankTab portfolio

# Android
adb shell am start -n com.vadimtoptunov.chaosbank_android/.MainActivity \
  -e CHAOSBANK_DEFECTS pnlSign -e CHAOSBANK_START_UNLOCKED 1 -e CHAOSBANK_TAB portfolio

You can enable a single defect, a whole profile (ui, validation, security, network, flaky…), or a difficulty bundle (beginner, middle, senior). StartUnlocked skips the auth flow so you land straight on the screen you care about. It only changes where you start, not how the app behaves.

2. The in-app developer menu. Long-press the build badge in the top-right corner. From there you can switch profiles, toggle individual defects, simulate offline/slow/flaky networking, flip RTL, and browse the exercises.

3. Prebuilt variants. Xcode build configurations (product flavors on Android) such as ChaosBank Flaky install next to the clean app under their own name. That can be useful if someone wants to do manual exploratory testing without touching launch arguments.

The exercises

Each defect has a matching entry in exercises.json:

{
  "id": "IOS-MON-02",
  "title": "Loss displayed as gain",
  "difficulty": "junior",
  "category": "money",
  "defects": ["pnlSign"],
  "launchArgument": "-ChaosBankDefects pnlSign",
  "expectedClean": "Loss shows negative / loss color.",
  "expectedBuggy": "Loss shows positive / gain color.",
  "task": "Open Portfolio and assert TSLA's P&L renders as a loss (negative, red).",
  "keyLocators": ["portfolio.holding.TSLA.pnl", "portfolio.pnl"]
}

Right now there are 12 junior, 97 middle and 22 senior exercises across ten categories: money, state, security, UI, validation, localization, networking, accessibility, concurrency and performance. The difficulty labels are my own guess and are probably off in places.

The junior ones are things like "a zero-amount transfer is allowed" or "the full card number isn't masked". The senior ones are mostly about timing: double-submits, a late response overwriting fresh state, animations that never quite settle.

If you're starting out in mobile QA

This is roughly the order I'd try things in. It's just one way to do it.

1. No code at first. Install the beginner build and poke around. Try not to read the defect list first. Write down what you find, then compare your notes with exercises.json. It's a fairly honest way to see what you notice and what you miss.

2. A first automated test. Pick a junior exercise and write a plain XCUITest for it. Something like:

func test_portfolio_lossIsRenderedAsLoss() {
    let app = XCUIApplication()
    app.launchArguments += ["-ChaosBankStartUnlocked", "1", "-ChaosBankTab", "portfolio"]
    app.launch()

    let pnl = app.descendants(matching: .any)["portfolio.holding.TSLA.pnl"]
    XCTAssertTrue(pnl.waitForExistence(timeout: 5))
    XCTAssertFalse(pnl.label.contains("+"), "A loss must not be shown as a gain: \(pnl.label)")
}

Or something similar with Compose on Android:

@Test
fun card_doesNotShowCvv() {
    val intent = Intent(ApplicationProvider.getApplicationContext(), MainActivity::class.java)
        .putExtra("CHAOSBANK_START_UNLOCKED", "1")
        .putExtra("CHAOSBANK_TAB", "card")
    ActivityScenario.launch<MainActivity>(intent)

    composeRule.onNodeWithTag("card.cvv").assertDoesNotExist()
}

These are rough sketches, not a finished suite. Run each one on clean (green), then with -ChaosBankDefects pnlSign or CHAOSBANK_DEFECTS=cardCvvVisible (red). If it stays green, that's worth digging into.

3. Timing and flakiness. The flaky profile has race conditions pinned to a seed. It's a decent place to see why sleep(2) tends to backfire. The network profile plus "slow" in the dev menu is another one.

4. Hybrid screens. The web login sits inside a WebView on purpose. By default it's opaque: Appium only sees NATIVE_APP, so you reach the fields through the native accessibility tree. With -ChaosBankInspectableWeb 1 (Android: -e CHAOSBANK_INSPECTABLE_WEB 1), a WEBVIEW_… context shows up and you can switch_to.context(...) into it. It's a small thing, but it tends to be confusing the first time you run into it in a real app.

There's also a hand-written UI suite for the iOS app: ChaosBank ⨯ KassiOS. It's another experiment of mine, a small Kaspresso-style DSL over XCUITest called KassiOS. It passes on clean and goes red on each injected defect. It might be more useful to look at after writing a few tests of your own.

If you're starting out in mobile development

The apps are also a small codebase with no third-party dependencies, so they might be worth reading. A few parts I spent the most time on:

Money. Decimal everywhere, except for one deliberate spot: the roundingDrift defect routes a single calculation through Double, so you can see what that does.
Determinism. The simulated price feed and the race-condition "coin flips" come from a seeded RNG (SplitMix64), so a failing run can be reproduced. Live Yahoo Finance quotes are available, but opt-in, because they make tests non-deterministic.
A mock backend as an actor, with timeouts, retries, offline mode and out-of-order responses.
Testable view models. Logic-layer coverage is around 97% on iOS and 95% on Android, checked in CI. The UI is left to UI tests.
Two UI stacks. The UIKit (UITableView) and XML (RecyclerView) builds exist because some bugs only really happen there, like a recycled cell showing the previous row's amount.

One exercise: pick a defect, find its Defects.isActive check, and read the correct else branch next to the buggy one. Another: try adding a defect of your own. The developer's guide has the steps. Note that CI compares the exercise catalogs of the two repos, so a new defect has to land on both platforms.

Using it for take-home assignments

This one I didn't plan for, but it seems to fit: test assignments for hiring, or onboarding someone onto a team. Since the bug set is just configuration, you can build a variant around what your team cares about:

For a security-focused role, use the security build: unmasked card data, an expired OTP that's still accepted, a deep link that skips the auth gate, a token in UserDefaults.
For a team fighting flaky tests, use flaky or senior, and ask for a suite that stays stable across, say, ten runs in a row.
For a junior role, use beginner and ask for a written bug report before any code.
For something narrower, pass your own list with -ChaosBankDefects / CHAOSBANK_DEFECTS.

Profile builds install as a separate app with their own name (ChaosBank Security, ChaosBank Flaky…), so a candidate can get a single APK or simulator build. The clean build gives the reviewer a baseline to run the submitted tests against.

One caveat: the catalog is public, so a candidate can look up the defect list. I'd probably treat it as open-book and focus the review on how the tests are written (waits, value assertions, how findings are explained) rather than on how many bugs were found.

A few things I learned

"Locators stay put" was harder than I expected. Some realistic bugs, like a recycled cell keeping a stale accessibility ID, are about locators. I ended up making those separate, clearly labelled defects instead of exceptions to the rule.
Randomness works better as an opt-in. Live prices are fun, but any assertion on a price or P&L stops meaning much when the number changes every second. So the seeded feed is the default.
Keeping two codebases in sync needs automation. Doing it by hand drifts. A CI script now compares the two exercise catalogs.
The clean build has to really be clean. If clean has an accidental bug, the "green on clean, red on buggy" idea falls apart. That's the main reason the reference suite exists.

Limitations

iOS needs a Mac with Xcode 26+ and an iOS 17+ simulator. Android needs JDK 21 and API 29+.
There's a reference UI suite for iOS, but none yet for Android. Espresso/Compose, Appium and Maestro suites are on the roadmap.
It's a sandbox. The "bank" lives in memory and the card number is a test one.
I'm one person maintaining two apps, so some defects are probably less realistic than others.

Links

iOS app: https://github.com/VadimToptunov/ChaosBank-iOS
Android app: https://github.com/VadimToptunov/ChaosBank-Android
User guide for QA/SDET: docs/USER_GUIDE.md
Reference UI suite: https://github.com/VadimToptunov/ChaosBank-KassiOS-Tests

Next on my list are Espresso/Compose, Appium and Maestro suites against the same screens, to see how the approaches compare in practice. If you give the apps a try, I'd be glad to hear which defects were hard to catch, which ones feel unrealistic, and which bugs from your own work are missing. Comments, issues and PRs are all welcome.

DEV Community