DEV Community

Cover image for KassiOS: a Kaspresso-style layer for XCUITest, tested against a broken bank
Vadim Toptunov
Vadim Toptunov

Posted on AI-assisted

KassiOS: a Kaspresso-style layer for XCUITest, tested against a broken bank

#testing #ios #swift #opensource

My background is QA automation, mostly mobile and mostly Appium. When I decided to learn native iOS UI testing properly, XCUITest looked like the obvious choice: it's Apple's own framework, it needs no automation server, and it works with the accessibility hierarchy directly. Then a simple login turned into this:

let email = app.textFields["login.email"]
XCTAssertTrue(email.waitForExistence(timeout: 10), "Email field did not appear")
email.tap()
email.typeText("test@example.com")

let loginButton = app.buttons["login.submit"]
XCTAssertTrue(loginButton.waitForExistence(timeout: 10), "Login button did not appear")
loginButton.tap()
Enter fullscreen mode Exit fullscreen mode

XCUITest gives you primitives and leaves waiting, retries, screen objects and reporting to you. In most projects I've seen, that gap gets filled by a waitAndTap() extension, then a few more, until there's a small framework that nobody tests. I decided to write that framework on purpose and see where it went. That became KassiOS.

The ideas come from Kaspresso, which I liked on Android: a test describes what the user does, the framework handles the waiting, and every step shows up in the report. The API is my own, and underneath it's still plain XCUITest in your UI-test target, with no dependencies.

What a test looks like 

onScreen(LoginScreen.self) { login in
    login.email.typeText("test@example.com")
    login.password.typeText("secret")
    login.loginButton.tap()
}
onScreen(HomeScreen.self) { $0.welcome.assertVisible() }
Enter fullscreen mode Exit fullscreen mode 
final class LoginScreen: KassScreen {
    lazy var email = textField("login.email")
    lazy var password = secureTextField("login.password")
    lazy var loginButton = button("login.submit")

    override var onLoad: [KassElement] { [email, loginButton] }
}
Enter fullscreen mode Exit fullscreen mode

Before the block runs, KassiOS checks the elements in onLoad. If the app ended up on a different screen, the test fails right there, in the "On LoginScreen" step, instead of on whichever field it happened to touch first. Every onScreen block is an activity in Xcode's report, so a red run reads as a list of steps.

The test is shorter, but that isn't what I was after. I wanted the waiting and retrying to live in one place, in the framework, so that each test doesn't carry its own copy.

Making one tap trustworthy

The first version of tap() waited for the element and tapped it. Most of the work since then went into the cases where that isn't enough. Today a single login.loginButton.tap(), within one time budget:

  • waits for the element to exist and be hittable. Internal retries share the budget, so three attempts at a 10-second action take 10 seconds, not 30;
  • scrolls to the element if it's off screen, and dismisses the keyboard if it covers the target. It uses Done or Hide for that, never Return, because Return can submit a form;
  • taps once. Only the preconditions are retried. A framework that taps again "just in case" can cause a double submit, or hide a real one;
  • finds the element again on every attempt. A KassElement stores how to find the element (() -> XCUIElement), not the element itself, so if SwiftUI rebuilds the view, the next attempt gets the new one.

Some actions also check what they did. setSwitch(on:), toggle(), replaceText and adjustPicker read the element back and fail if its state didn't change. Without it, a switch whose tap lands on its label, next to the control, returns as if it had worked, and the test can pass with the switch still off.

When something does fail, the report gets a screenshot of the screen at that moment and the full accessibility tree, and the message tries to say why:

KassiOS: text 'welcom' — assertVisible failed: does not exist
  ↳ element not found in the current hierarchy
  ↳ did you mean: 'welcome', 'Welcome!'?
Enter fullscreen mode Exit fullscreen mode

Waiting has a limit too. A retry that forgives too much turns a broken app into a green test, so in KassiOS waiting only gives the UI time to reach a state. When the budget runs out, the test fails with the last reason the framework saw. Each of these behaviors can be turned off in KassConfig.

Checking it against a broken bank

At some point I had tidy abstractions, a lot of unit tests, and no idea whether any of it caught bugs. A green UI test tells you the app did what the test checked. It doesn't tell you the test would turn red if the bug it was written for showed up.

To find out, I built ChaosBank, a banking app with bugs you switch on at launch (previous post), and a reference suite that tests it through KassiOS. Every test in the suite has the same contract:

clean app        →  test passes
enable a defect  →  the same test fails
Enter fullscreen mode Exit fullscreen mode

The first full run had 34 tests (Xcode 27.1, iPhone 17 simulator, M1 Max), and it was humbling:

  • test_portfolio_showsTotalAndPnl stayed green with every defect on, including one that shows a loss as a gain. It checked that the numbers were on screen, not that they were right.
  • The localization and flaky defect groups didn't turn a single test red. The suite had no locale or race checks at all.
  • Red runs were much slower than green ones. Most of the extra time went into my own "did you mean?" hint, which queried about 150 elements one by one. Reading a single hierarchy snapshot instead took the failing transfer test from 85 seconds to 23.

The suite also found a bug in KassiOS. After I added the read-back, one test failed like this:

KassiOS: textField 'order.limitPriceField' — replaceText('179.00') failed:
  typed '179.00' but the field reads '179.009.07'
Enter fullscreen mode Exit fullscreen mode

The price field was right-aligned and pre-filled with 189.07. The tap at its centre put the cursor in the middle of the text, the deletes removed only 18, and the new value went in front of 9.07. The previous version cleared fields the same way. It just didn't look, so the test would have failed a line later on a wrong order total, with nothing pointing at the field. Now KassiOS taps the trailing edge before deleting.

What the defects added

Several checks came from defects that slipped through. Take successToastMissing: the transfer goes through and the sheet closes, but the success toast never appears. A test that waits for the sheet to close stays green. Catching it needed a different kind of wait:

/// Catches `successToastMissing`.
func test_transfer_happyPath_completes() {
    launchUnlocked()
    onScreen(HomeScreen.self) { $0.transferAction.tap() }
    onScreen(TransferScreen.self) { transfer in
        transfer.recipientField.typeText("Alex")
        transfer.amountField.typeText("100")
        transfer.continueButton.tap()
        transfer.confirmButton.tap()
        transfer.successToast.assertAppears(within: 2)
        transfer.confirmButton.within(timeout: 20).waitUntilGone()
    }
}
Enter fullscreen mode Exit fullscreen mode

The toast is on screen for about 1.4 seconds. The normal budget is built for "wait until it's ready", and assertAppears polls a short window instead, for "catch it while it's there". A few more that I kept needing:

// A double submit: two taps on the same spot, no re-check in between.
order.placeButton.rapidTap(times: 2)

// A value that must not move while nothing happens.
home.totalBalance.assertUnchanged(for: 3)

// Prices that tick: read related values from one hierarchy snapshot.
let t = home.readTexts(["home.total", "home.account.EUR", "home.account.USD"])

// The right number of rows can still be the wrong rows.
history.cells().assertNoDuplicates()
history.cells().assertEach("is money-in") { row in
    guard row.readLabel().contains("+") else { throw KassError("expected a money-in row") }
}
Enter fullscreen mode Exit fullscreen mode

verifyAll runs every assertion in its block and reports all the mismatches on a screen, not only the first. For conditions you compose yourself there's eventually, continuously(during:), anyOf and retry(times:), as in Kaspresso.

Validation defects tend to need the same form filled with many inputs, so there's a parameterized runner as well:

func test_login_validatesTheEmail() {
    parameterized([("a@b.c", true), ("bad", false), ("", false)], name: { $0.0 }) { email, valid in
        relaunch()   // a clean app for every case
        onScreen(LoginScreen.self) { login in
            login.email.replaceText(email)
            login.loginButton.tap()
            if valid {
                onScreen(HomeScreen.self) { $0.welcome.assertVisible() }
            } else {
                login.error.assertVisible()
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Each case is its own activity in the report, named by the name closure, so a red run says the "bad" case failed. A failing case doesn't stop the rest. Swift Testing has @Test(arguments:), but in Xcode 27.1 it only has a unit-test template, and the UI-test templates are still XCTest. I use Swift Testing for model and view-model logic and KassiOS for the flows, in separate targets.

One defect at a time

Running the suite against groups of defects tells you that something went red. It doesn't tell you which defect each test catches, or which defect nobody catches. For that, KassiOS has a command-line tool, kassios mutants. You list the defects and the tests each one must turn red, and it runs them one defect at a time:

mutant                 verdict
zeroAmountAccepted     SURVIVED  (still green: …/TransferTests/test_transfer_zeroAmount_disablesContinue)
balanceAfterAdds       killed
...
58 of 59 killed
Enter fullscreen mode Exit fullscreen mode

--clean first runs every listed test with no defect on, because a test that's already red "catches" everything. In the reference suite the list is generated from a /// Catches `id` comment above each test, like the one on the toast test above, and CI fails if the generated file is out of date.

The suite has since grown from 34 tests to 71, and all 59 defects on the list are caught. Running them one by one takes about 70 minutes on my Mac. With --parallel 3, which runs three defects at once on clones of the simulator, it takes 28. Which tests stayed green the longest, and what it took to make them fail, is the next post.

A few other parts

  • kassios lint reads screen objects before anything runs and flags those without onLoad or with computed identifiers. Apple's accessibility audit can run from a test.
  • kassios agent start starts a small loopback-only Mac process for things a test can't do from inside the simulator: grant permissions, freeze the status bar, set a location. It picks its own token and port and tells the test where to find them. Without it, those calls skip with a message instead of hanging.
  • KassHTTP arranges state through the app's API, so a transfer test doesn't have to create an account through the UI first.
  • webView(), link and links() reach into a WKWebView. HTML has no accessibility identifiers, so web elements are found by label.
  • With recordVideoOnFailure and the agent running, a failed test also gets a video of the simulator screen. Allure and JUnit reporters are built in.

Scenarios, network stubs, snapshot tests and the rest are in the README and the Guide.

Who it's for, and who it isn't

I think KassiOS fits if:

  • your XCUITest suite has outgrown its helpers. There's a waitAndTap(), a base test class and page objects, and every new flaky test gets its own sleep;
  • a silent failure is expensive in what you test. Forms, payments and settings are where a tap that missed or a field that kept part of its old value lets a test pass on a broken app;
  • your team knows Kaspresso from Android and wants the same structure on iOS: screens, steps, scenarios, Allure;
  • you want to know what your suite actually catches. kassios mutants only needs the app to switch a defect on at launch, through a launch argument or an environment variable.

It's probably not for you if:

  • the suite is small. A dozen tests over a few screens are fine on raw XCUITest with a couple of extensions that every iOS developer can read;
  • you need one suite for iOS and Android. Appium and Maestro do that, through a server or a runtime of their own. KassiOS is iOS only, Swift inside the app's Xcode project;
  • you need the app's own idle state. EarlGrey runs inside the app and knows when it's idle. KassiOS stays out of process and polls. It has a synchronizer hook, with an example EarlGrey adapter in the repo;
  • you want nothing between a failure and its line. KassiOS actions call XCTFail internally and return the element, so with continueAfterFailure = true a chain keeps going after a failure;
  • you'd rather stay with Apple's tools than depend on a third-party layer with one maintainer.

None of this fixes flakiness that comes from the app, the test data or the environment. When something behaves oddly, you still need to know what XCUITest does underneath. Building KassiOS taught me more about that than writing tests would have.

Built with AI, checked like QA

I built KassiOS with a lot of help from AI coding tools. I'm a QA engineer, not a Swift framework developer, and that help let me go much further than I would have alone. It also meant checking everything: 172 unit tests, 59 UI tests against a small SwiftUI demo app in the KassiOS repo, the ChaosBank reference suite, and SwiftLint, all in CI on every pull request. The cursor bug above was found by the ChaosBank suite, not by me reading the code.

Trying it 

.package(url: "https://github.com/VadimToptunov/KassiOS.git", from: "1.10.0")
Enter fullscreen mode Exit fullscreen mode

Add it to your UI-test target, make the test class a KassTestCase instead of XCTestCase, and write one screen object, for example for the screen your flakiest test starts on. Plain XCUITest calls keep working next to it. The public API has followed semantic versioning since 1.0.

The command-line tools (the agent, the linter and the mutation runner) are one Homebrew formula, prebuilt, so there's nothing to build:

brew install vadimtoptunov/tap/kassios
kassios agent start
Enter fullscreen mode Exit fullscreen mode

The library doesn't need them; install them when you want one of those features.

If you try it, I'd like to hear what feels unnecessary, what's missing, and where it gets in your way. Issues and PRs are welcome.

Links

Top comments (0)