Our Journey to Automated Testing on Android

With all the different options available to do automated testing for Android Apps, it's easy to get overwhelmed or lost in the process. At least we did. After many not-properly tested projects, we figured out a framework that works for us currently. It consists of two types of tests that can run in CI on every PR: Unit Tests and Robolectric UI Tests.

1. Robolectric UI tests

To test our app from UI perspective, we create behaviour tests that we run on JVM using Robolectric. Not having to start up an Android Emulator for these tests is a big plus, as they can run alongside our unit tests and are therefore significantly faster than emulator tests.

A typical test (usually per screen) may look like that:

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
@Config(application = TestApplication::class, qualifiers = MyDeviceQualifiers.MEDIUM_PHONE)
class DetailScreenTest : KoinTest {

    @get:Rule
    val composeTestRule = createAndroidComposeRule<MainActivity>()

    @BeforeTest
    fun setup() {
        startKoin {
            androidContext(...)
            val testModule = module { single<Api> { FakeApi() } }
            modules(myModules + testModule)
        }
    }

    @Test
    fun editDetailScreen() = runTest {
        launchDetailScreen(composeTestRule) {
            clickEdit()
        } verify {
            editDetailIsPresent()
        }
    }

We're using koin as dependency injection framework. In the setup function we want to inject the modules our app needs as well as some dependencies we want to overwrite. In order that these can run in a controlled fashion, we replace all dependencies that would usually do network operations etc. with fake ones. Note: Usually our Application would call startKoin but here we're passing a empty TestApplication to @Config so that we can manually inject test parameters. You can also create Rules that abstract this logic to use in multiple different tests.

To make the tests readable, we use Robot classes that abstract away all the composeTestRule calls. The robot for the test above looks like this:

class DetailRobot(private val rule: ComposeContentTestRule) {

    fun clickEdit() {
        val string = RuntimeEnvironment.getApplication().getString(R.string.edit)
        rule.onNodeWithText(string).performClick()
    }

    infix fun verify(block: DetailVerification.() -> Unit): DetailVerification {
        return DetailVerification(rule).apply(block)
    }

    class DetailVerification(private val rule: ComposeContentTestRule) {

        fun editDetailIsPresent() {
            val title = RuntimeEnvironment.getApplication().getString(R.string.edit_screen_title)
            rule.onNodeWithText(title).assertIsDisplayed()
        }
    }
}

The benefit of creating Robot classes is that you can use them in further tests that require going further down your app. In this example, we are using the ListRobot to open up the list and clicking on the first item in this list (which results in opening the detail screen)

fun launchDetailScreen(rule: ComposeContentTestRule, block: DetailRobot.() -> Unit): DetailRobot {
    return DetailRobot(rule).apply {
        launchListScreen(rule) { // <-- thats from another Robot
            clickFirstItem()
        }
        block()
    }
}

These UI tests test a large portion of our app's UX and help us find issues quite quickly, as they run on every PR alongside the other tests mentioned. Since you cannot see what Robolectric is doing, we found it helpful to use Roborazzi to debug our tests visually.

2. ViewModel tests (unit tests)

In most cases our ViewModels deliver everything in a single State, therefore it's quite handy test doing actions while observing the behaviour of this state. We're using cashapp/turbine for that. Also here, when instantiating a viewModel we provide Fake classes to get similar results every time we run the tests.

class NavigateScreenViewModelTest {

    private lateinit var viewModel: NavigateViewModel
    private val locationProvider = LocationProvidingFake()

    @BeforeTest
    fun setup() {
        viewModel = NavigateViewModel(
            dispatcher = UnconfinedTestDispatcher(),
            locationProvider = locationProvider,
        )
    }

    @Test
    fun state_whenDoSomeAction_myFieldIsTrue() = runTest {
        viewModel.state.test {
            val initialState = awaitItem()
            assert(initialState.myField == false)

            viewModel.doSomeAction()

            val state = awaitItem()
            assert(state.myField == true)
        }
    }
}

3. Unit tests

For everything else thats business logic, network logic we're using simple unit tests that test one or multiple classes and try to focus on testing behaviour. A helpful library to create parameterized tests here is cashapp/burst.

These tests may look quite different. From simple validation checks like below to more complex ktor-mock-server tests.

    @Test
    fun validateDriverLicense_whenInvalid_returnsInvalid(
        license: String = burstValues("1234567890123", "ABC-123"),
    ) {
        val result = PersonalIdentityValidator.validateDriverLicense(license)
        result.shouldBeTypeOf<ValidationResult.Invalid>()
}

With this setup of tests that can all run with a single command and don't need fancy emulator setup we find that we're getting the most useful results and seem to create new tests more often than before, when our system was unclear and hard to use. Not to say that the current system is perfect, but it definitely helps us to get a solid foundation and making it easier to understand and maintain.

In CI, we're using kotlinx-kover to give us an idea of the code coverage and more importantly see which code is currently not covered with any test.