By October, our Maestro end-to-end testing setup for the mobile app looked well covered. Every one of the 99 routes expected to have automated device coverage had a flow in the repository. Then we traced what ran in CI.
Only 28 routes were reached by flows that CI executed. The other 71 depended on quarantined flows. That gap mattered more than the repository inventory: our release decisions needed evidence from tests that actually ran on mobile devices.
As the engineering manager, I needed to know why the existing flows weren't producing evidence before asking the team to write more.
Follow the call graph
Our Maestro end-to-end tests were assembled in layers. A leaf flow covered a product journey. Shared subflows handled repeated steps. Suite aggregates assembled journeys for a run. The cloud jobs called those aggregates.
That last connection mattered. A new leaf could be well written, reviewed, and tagged, yet never execute if no aggregate called it. A tag on the leaf did not put it into a cloud job. The path to evidence was:
CI job → suite aggregate → leaf flow → product route
If any edge was missing, the route was absent from that run. We began reviewing coverage through this path. Shared setup and suite entrypoints were useful, but they were not independent mobile journeys.
There was a second trap. A route could have a leaf flow that we had quarantined because it was unstable. That route looked covered in a repository search, but its flow supplied no evidence in the normal CI run. Our route audit kept those two states separate: a flow exists and a flow runs.
Make the run small enough to finish
Getting a flow into CI did not automatically make the suite useful. One flaky suite could stop later suites from producing results. We split nightly execution into a job per suite and platform so one failure would not erase the rest of the run.
Then a nine-leaf suite hit Maestro Cloud's 15-minute limit. Some leaves spent roughly two minutes just setting up their state. We split long suites into groups of no more than three leaves, keeping the jobs short enough to return evidence.
That constraint changed how I looked at coverage. A test that times out as part of an oversized batch is about as helpful to a release decision as a test that never starts.
Check reachability before review
We added a fast static check for the Maestro test definitions. It checked syntax, file paths, app identifiers, selectors that could not resolve, and whether a leaf was reachable from a CI entrypoint. It ran in the normal lint step, before anyone waited for a device run. Exceptions needed an explicit reason.
Device execution still told us whether a flow worked. The static check caught an earlier failure: work that looked like coverage in a pull request but had no route into the actual test run.
I stopped asking how many test files we had and started asking which release decisions those files could support. The audit left 71 routes with a clear next job: make their quarantined flows reliable and connect them to an executed suite. At least we were working from the run, rather than from the directory listing.
The fuller account, including the audit and rollout details, is in my Beyond the QA Bottleneck case study.
Top comments (0)