DEV Community

Cover image for Maestro Flakiness: Source Code Analysis
Om Narayan
Om Narayan

Posted on • Originally published at devicelab.dev on

Maestro Flakiness: Source Code Analysis

#testing #mobile #opensource #automation

Maestro markets itself as a test framework that "embraces the instability of mobile applications." But what does that actually mean in code? I dug into the source to find out.

The Marketing Promise

From Maestro's documentation:

"UI elements will not always be where you expect them, screen tap will not always go through, etc. Maestro embraces the instability of mobile applications and devices and tries to counter it."

"No need to pepper your tests with sleep() calls. Maestro knows that it might take time to load the content and automatically waits for it (but no longer than required)."

Sounds great. Let's see what the code actually does.

1. Element Finding: The Hardcoded 17-Second Timeout

When you write tapOn: "Login", Maestro doesn't look once and fail. It polls continuously. But for how long?

Source: Orchestra.kt

class Orchestra(
    private val maestro: Maestro,
    private val lookupTimeoutMs: Long = 17000L,        // Hardcoded: 17 seconds
    private val optionalLookupTimeoutMs: Long = 7000L, // Hardcoded: 7 seconds for optional
)
Enter fullscreen mode Exit fullscreen mode

What this means:

  • Every element lookup waits up to 17 seconds by default
  • Optional elements wait 7 seconds
  • You cannot change this per-command

Want to wait 30 seconds for a slow API response? Too bad. Want to fail-fast in 3 seconds for performance testing? Also no.

2. The Polling Mechanism: Simple but Rigid

Source: MaestroTimer.kt

fun <T> withTimeout(timeoutMs: Long, block: () -> T?): T? {
    val endTime = System.currentTimeMillis() + timeoutMs

    do {
        val result = block()
        if (result != null) {
            return result
        }
    } while (System.currentTimeMillis() < endTime)

    return null
}
Enter fullscreen mode Exit fullscreen mode

This is a tight polling loop with no configurable delay between iterations. It just hammers the view hierarchy until it finds the element or times out.

Compare to Appium's FluentWait where you can set custom timeout, custom polling interval, and exceptions to ignore.

3. Tap Retries: Only 2 Attempts, Non-Configurable

Maestro has a clever feature: if a tap doesn't change the UI, it retries. But how many times?

Source: Maestro.kt

private fun getNumberOfRetries(retryIfNoChange: Boolean): Int {
    return if (retryIfNoChange) 2 else 1  // That's it. Just 2.
}
Enter fullscreen mode Exit fullscreen mode

Two retries. Hardcoded. Not configurable.

And the explicit retry command in YAML? Also capped:

Source: Orchestra.kt

private const val MAX_RETRIES_ALLOWED = 3
Enter fullscreen mode Exit fullscreen mode

So even if you write retry: 10 in your YAML, you get 3 maximum.

4. Screenshot Diff Threshold: Magic Number 0.5%

When Maestro decides whether a tap "worked," it compares screenshots pixel-by-pixel:

Source: Maestro.kt

private const val SCREENSHOT_DIFF_THRESHOLD = 0.005  // 0.5% pixel difference
Enter fullscreen mode Exit fullscreen mode

If less than 0.5% of pixels changed, Maestro assumes the tap failed and retries.

Problems:

  • A small spinner animation might trigger false "success"
  • A full-screen color change of 0.4% would be considered "no change"
  • You can't adjust this threshold

5. Wait For App To Settle: Fixed 200ms Polling

After interactions, Maestro waits for the UI to stabilize:

Source: ScreenshotUtils.kt

repeat(10) {  // Poll up to 10 times
    val hierarchyAfter = viewHierarchy(driver)
    if (latestHierarchy == hierarchyAfter) {
        val isLoading = latestHierarchy.root.attributes
            .getOrDefault("is-loading", "false").toBoolean()
        if (!isLoading) {
            return hierarchyAfter
        }
    }
    latestHierarchy = hierarchyAfter

    MaestroTimer.sleep(MaestroTimer.Reason.WAIT_TO_SETTLE, 200)  // Fixed 200ms
}
Enter fullscreen mode Exit fullscreen mode

Hardcoded values:

  • 10 iterations maximum
  • 200ms between polls
  • Total maximum wait: 2 seconds

Fast app? You're wasting 200ms per poll. Slow app? 2 seconds might not be enough.

6. waitUntilVisible: 10 Seconds, Take It or Leave It

Source: Maestro.kt

private fun waitUntilVisible(element: UiElement): ViewHierarchy {
    var hierarchy = ViewHierarchy(TreeNode())
    repeat(10) {  // 10 attempts
        hierarchy = viewHierarchy()
        if (!hierarchy.isVisible(element.treeNode)) {
            LOGGER.info("Element is not visible yet. Waiting.")
            MaestroTimer.sleep(MaestroTimer.Reason.WAIT_UNTIL_VISIBLE, 1000)  // 1 second
        } else {
            LOGGER.info("Element became visible.")
            return hierarchy
        }
    }
    return hierarchy
}
Enter fullscreen mode Exit fullscreen mode

10 polls × 1 second = 10 seconds maximum. Not configurable.

7. Platform Differences: Android vs iOS

Maestro handles settling differently per platform:

Android (AndroidDriver.kt):

// Checks if window is still updating
val windowUpdating = blockingStubWithTimeout.isWindowUpdating(...)
Enter fullscreen mode Exit fullscreen mode

iOS (IOSDriver.kt):

// Uses screenshot comparison to detect animation end
val didFinishOnTime = waitUntilScreenIsStatic(SCREEN_SETTLE_TIMEOUT_MS)
Enter fullscreen mode Exit fullscreen mode

Different approaches, different reliability characteristics, same lack of configurability.

Summary: The Hardcoded Reality

Parameter Value Configurable?
Element lookup timeout 17,000ms ❌ No
Optional element timeout 7,000ms ❌ No
Tap retry attempts 2 ❌ No
Max retry command 3 ❌ No
Screenshot diff threshold 0.5% ❌ No
Settle polling interval 200ms ❌ No
Settle max iterations 10 ❌ No
waitUntilVisible timeout 10,000ms ❌ No

The Verdict

Maestro's "built-in flakiness handling" is real—it does more than raw XCUITest or Espresso. But it's a one-size-fits-all solution with hardcoded values.

The code is clean and the approach is sound. The problem is the lack of escape hatches. When the defaults don't work for your app, you're stuck.

This isn't necessarily bad—it's a trade-off for simplicity. But the marketing implies more intelligence and adaptability than the code delivers.

Key Source Files

If you want to explore further:

We're Not Just Pointing Out Problems

We love Maestro's YAML syntax—it's the best thing to happen to mobile test automation in years. Simple, readable, version-control friendly.

But the execution engine has real limitations. Hardcoded timeouts. No configurability. Platform inconsistencies.

So we're building something to fix it.

An open-source engine that runs Maestro YAML tests on Appium's battle-tested infrastructure. Configurable timeouts. Real device support. No magic numbers.

Watch this space.

Learn More

Top comments (0)