DEV Community: Dmitrii Morozov

Measurements with MetricKit

Dmitrii Morozov — Sat, 27 Apr 2024 21:47:09 +0000

Intro

Sometimes you need to measure something in the app. It can be different things like app launch time or just custom time spent between certain points of execution. One of the most convenient tools in this regard is MetricKit framework. With its help, you can measure different performance and diagnostic metrics, monitor regressions and identify problems in your app. The article describes the basic setup, output data format and possible options to process it. Also, it describes custom measurements and events reporting with MetricKit.

Getting started

Let’s start with a simple example: how to measure your app’s launch time?

Setup

First, we need to define an object responsible for handling info from the framework and implement MXMetricManagerSubscriber. This protocol contains two optional methods:
optional func didReceive(_ payloads: [MXMetricPayload])
optional func didReceive(_ payloads: [MXDiagnosticPayload])
In this article we will focus only on the first one that provides metrics.
According to docs:

The system calls this method at most once per day. It’s safe to process the payload on a separate thread.

In real life you can open an app every day but the callback will be called once in three days, it only guarantees to call the method no more than once a day. in my experience, it is called on average once every 2-3 days.
Let’s create a separate class responsible for metrics reporting:




import MetricKit

final class MetricReporter: MXMetricManagerSubscriber {
    func didReceive(_ payloads: [MXMetricPayload]) {
        // TODO: Report metrics
    }
}

Those methods are called in the background so they don’t affect app performance. Later I will explain how to parse data from those metrics. Now we need to have this object during the whole app lifecycle so let’s add it to AppDelegate.swift:



@main
class AppDelegate: UIResponder, UIApplicationDelegate {
    private lazy var metricReporter = MetricReporter()
    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        MXMetricManager.shared.add(metricReporter)
        return true
    }
}

So now we are ready to try payload example.

Payload overview

To get a sample for MetricKit we can use Xcode Debug -> Simulate MetricKit Payloads, but it is important to use a real device, for a simulator this option is disabled.

Every MXMetricPayload contains different instances of MXMetric subclasses such as MXAppLaunchMetric, MXAppExitMetric, MXMemoryMetric and others. All of them in turn contain different sets of following classes and structures:

Measurement structure represents value including information about units. This is how it looks like in the example of peak memory usage:



"peakMemoryUsage" : "200000 kB"

MXHistogram class represents the number of times the measured value falls into a specific range of possible values. This is how it looks in json form in the example of the launch time of the app:



"histogrammedTimeToFirstDrawKey" : {
      "histogramNumBuckets" : 3,
      "histogramValue" : {
        "0" : {
          "bucketEnd" : "1010 ms",
          "bucketCount" : 50,
          "bucketStart" : "1000 ms"
        },
        "1" : {
          "bucketEnd" : "2010 ms",
          "bucketCount" : 60,
          "bucketStart" : "2000 ms"
        },
        "2" : {
          "bucketEnd" : "3010 ms",
          "bucketCount" : 30,
          "bucketStart" : "3000 ms"
        }
      }
    }

MXAverage class represents average value including information about sample count and standard deviation. This is how it looks like in the example of average pixel luminance:



"averagePixelLuminance" : {
    "averageValue" : "50 apl",
    "standardDeviation" : 0,
    "sampleCount" : 500
  }

Processing

There are two different approaches for metrics data processing: preprocess data on a device and send it or send data as it is. Both have disadvantages and advantages, let’s take a look at the details.

Processing on device

Process data on mobile and send result values to your analytics solution. In general, you need to transform metrics-specific data structures into simpler ones, for example extract average value from MXHistogram data structure. A great example of this approach can be found here. Just a few important things to remember:

Remember to check includesMultipleApplicationVersions field to filter out reports containing data from different versions
Remember to filter out abnormal values with isNan , isNormal or isFinite checks. In the previously mentioned example author used isNan but you can also utilise isFinite or isNormal, but remember that zero is not a normal number

In this case, you select on mobile the exact data you want to send for analyzing. Unfortunately, if you make any mistake on this step it would be quite hard to catch since you don’t have access to raw data.

Offload processing computation off mobile devices

Offload processing computation off mobile devices and send data as it is for processing. In this great example the author uses a web service to process metric payloads. The main advantages of this method are:

All raw data is available so if later you discover any issues with your processing you can fix it without any consequences
At any point in time you can access data from the past

The main disadvantage here is the higher complexity.

Custom metrics

Apart from predefined measurements MetricKit also supports custom measurements and event tracking with mxSignpost

Custom measurements

This is how you can do it on the example of applying some heavy operation:



func apply() {
    // create log handler
    let handle = MXMetricManager.makeLogHandle(category: "ApplyCategory")
    mxSignpost(.begin, log: handle, name: "ApplyTrace")
    // critical code section begins
    // ...
    // critical code section ends
    // end measuring
    mxSignpost(.end, log: handle, name: "ApplyTrace")
}

func cancel() {
    let handle = MXMetricManager.makeLogHandle(category: "CancelCategory")
    mxSignpost(.begin, log: handle, name: "CancelTrace")
    // ...
    mxSignpost(.end, log: handle, name: "CancelTrace")
}

Though it looks pretty simple there are several important things to highlight:

It is important to specify the same name in mxSignpost calls, otherwise, you will not get results in a report
There is special note about this API usage:

The system limits the number of custom signpost metrics saved to the log in order to reduce on-device memory overhead. Limit the use of custom metrics to critical sections of code.

As a result, you get something similar to the following piece in result payloads:



{
  "signpostMetrics" : [
    {
      "signpostIntervalData" : {
        "histogrammedSignpostDurations" : {
          "histogramNumBuckets" : 2,
          "histogramValue" : {
            "0" : {
              "bucketCount" : 1,
              "bucketStart" : "0 ms",
              "bucketEnd" : "99 ms"
            },
            "1" : {
              "bucketCount" : 1,
              "bucketStart" : "100 ms",
              "bucketEnd" : "199 ms"
            }
          }
        },
        "signpostCumulativeCPUTime" : "262 ms",
        "signpostAverageMemory" : "64433 kB",
        "signpostCumulativeLogicalWrites" : "748 kB"
      },
      "signpostCategory" : "ApplyCategory",
      "signpostName" : "ApplyTrace",
      "totalSignpostCount" : 2
    },
    {
      "signpostIntervalData" : {
        "histogrammedSignpostDurations" : {
          "histogramNumBuckets" : 1,
          "histogramValue" : {
            "0" : {
              "bucketCount" : 81,
              "bucketStart" : "0 ms",
              "bucketEnd" : "99 ms"
            }
          }
        },
        "signpostCumulativeCPUTime" : "295 ms",
        "signpostAverageMemory" : "211037 kB",
        "signpostCumulativeLogicalWrites" : "168 kB"
      },
      "signpostCategory" : "CancelCategory",
      "signpostName" : "CancelTrace",
      "totalSignpostCount" : 81
    }
  ]
}

As you can see in the attached payload there is histogram data that allows you to collect information about the execution time for specific code sections.

Events tracking

Apart from begin and end, you can use event OSSignpostType and in this case in a payload you will receive a number of times this event occurred. Here is an example:



let handle = MXMetricManager.makeLogHandle(category: "TestViewController")

override func viewDidLoad() {
    super.viewDidLoad()
    mxSignpost(.event, log: handle, name: "viewDidLoad")
}

override func viewWillAppear() {
    super.viewWillAppear()
    mxSignpost(.event, log: handle, name: "viewWillAppear")
}

With this code we can get the following payload part:



"signpostMetrics" : [
    {
      "signpostCategory" : "TestViewController",
      "totalSignpostCount" : 5,
      "signpostName" : "viewDidLoad"
    },
    {
      "signpostCategory" : "TestViewController",
      "totalSignpostCount" : 5,
      "signpostName" : "viewWillAppear"
    }
  ]

As you can see this can be utilised as an event-collecting tool to build various types of event funnels

Conclusion

MetricKit provides wide options of reporting tools to monitor:

Predefined metrics like application launch time, network usage, memory usage, etc.
Custom metrics recorded with mxSignpost for critical code sections
Events tracking with mxSignpost for critical events

These options can be utilised in various scenarios such as performance related improvements or event funnels.

Discovering UI Performance testing with XCTest. Navigation performance.

Dmitrii Morozov — Sun, 21 May 2023 19:19:28 +0000

One of the most important components of app performance is the performance of navigation transitions between different screens. Applying new changes to a screen without proper testing can significantly degrade this aspect of an app’s performance. In this article, I will show how to build performance tests to address this problem. This article has some references to my previous article about scrolling performance and I recommend to read it first.

Setup

I prepared the demo project. This is a simple app that shows you a list of plants, you can select a plant and open the details screen. For all measurements I used MacBook M2 Pro with Ventura 13.0 and iPhone 13 Pro with iOS 16.0, XCode version is 14.2.

Here is the test I used for testing. It tests navigation performance when opening the details screen from the main screen:

import XCTest

final class NavigationPerformanceTests: XCTestCase {
    let app = XCUIApplication()

    func testNavigationTransition() {
        app.launch()
        let measureOptions = XCTMeasureOptions()
        measureOptions.invocationOptions = [.manuallyStop]

        measure(metrics: [XCTOSSignpostMetric.navigationTransitionMetric], options: measureOptions) {
            app.staticTexts["Monstera"].tap()
            app.navigationBars.buttons.element(boundBy: 0).tap()
            stopMeasuring()
        }
    }
}

As I found out in the previous article there are limited metrics available on a simulator compared with a real device. To understand if this method is applicable I will test it both on a simulator and a real device.

Testing

Firstly we need to set up baselines for our test. I described the process in detail here. After baseline setup we are ready to go. To simulate delay I used this:

/* x is number of microseconds, for example usleep(10000)
blocks current thread for 10000 microseconds
or 0.01 second and usleep(1000000) blocks for 1 second */

usleep(x)

I did not notice any difference between different launches in contrast to measuring scroll performance so there is only one try for each delay value. Following hitch-related metrics are not relevant in this case and during testing value for them is always zero:

Hitch Time Ratio (NavigationTransition): 0.000 ms per s
Hitches Total Duration (NavigationTransition): 0.000 ms
Number of Hitches (NavigationTransition): 0.000 hitches

I ignored these values during testing. For delay values I tried to find the pivot point when tests start to fail. Following table demonstrates test results:

Results show the following:

Duration is the most sensitive metric during measuring navigation performance.
Simulator results are very similar to real device results.
Tests start to fail both for a simulator and a real device with a delay equal to or more than 50 ms
Frame Count and Frame Rate are not predictable in the described case and hardly usable

Conclusion

The described method is an efficient tool to detect navigation performance issues in early development stages. It has some constraints such as using the same device model for test recording and testing, but even with a simulator it can detect delays that are not noticeable for a human. In my opinion, this can be a great way of controlling crucial flows performance in your app.

Useful links

Writing and running performance tests

Discovering UI Performance testing with XCTest. Scrolling performance.

Dmitrii Morozov — Wed, 10 May 2023 22:42:19 +0000

Mobile app performance is a critical aspect of the user experience. Poor performance can lead to frustration, negative reviews, and ultimately, loss of users. During app development it is crucial to detect performance issues as early as possible. In 2020 Apple introduced new tools to build Performance tests for UI interactions and currently this is one of recommended methods of improving your apps performance. In this article I tested these tools and described my experience with them. This article is focused on scrolling performance specifically.

Setup

I prepared the demo project. This is simple app which shows you a list of plants, you can select a plant and open details screen:

For all measurements I used MacBook M2 Pro with Ventura 13.0 and iPhone 13 Pro with iOS 16.0, XCode version is 14.2.
Here is the test I used for testing (it is based on content from here). It tests scrolling performance on the main screen of the app.



final class ScrollingPerformanceTests: XCTestCase {
    let app = XCUIApplication()

    func testScrollAnimationPerformance() {
        app.launch()
        let collection = app.collectionViews.firstMatch
        let measureOptions = XCTMeasureOptions()
        measureOptions.invocationOptions = [.manuallyStop]

        measure(metrics: [XCTOSSignpostMetric.scrollingAndDecelerationMetric], options: measureOptions) {
            collection.swipeUp(velocity: .fast)
            stopMeasuring()
            collection.swipeDown(velocity: .fast)
        }
    }
}

Test runner performs 5 iterations of measure (there is also additional first iteration when measuring does not happen) and shows average values. The number of iterations can be changed via XCTMeasureOptions object. Also for the test scheme I disabled all diagnostic features and switched to release configuration as mentioned here.
It is time to start our testing, let’s launch it on a simulator.

Unfortunately the simulator supports only Duration metric. You have to use a real device to get all available metrics. Let’s try it with a real device.

There are five new metrics: frame count, frame rate, number of hitches, hitches total duration, hitches time ratio. While frame count and frame rate are more or less obvious, there are three metrics mentioning the term “hitch”. According to this: a hitch is any time a frame appears on screen later than expected. Thus hitch time ratio is the total hitch time in an interval divided by its duration.

Baseline setup

Now for our performance tests to work we need to set up baselines. To set a baseline you need to tap on the performance measurement status icon (gray icon with a dot) and tap on Set Baseline button. Also on that screen you can see performance of the test through different iterations:

After you set a baseline you will see a new folder with .baseline suffix inside of xcshareddata folder (it is inside of .xcodeproj package). Inside this folder you can see two files: Info.plist and file with generated name and .plist suffix. First file stores information about a device used to record baseline. That will be used to compare test results, you can’t use test results with baseline if baseline was recorded on another device model. For our case the device model is iPhone 14,2 (iPhone 13 Pro), device model for simulator is the device model of the current computer. Second file stores class names and baseline values.
During baseline recording I encountered with two issues:

In most of the cases the test fails because of exceeding maximum standard deviation during one of the iteration for hitch related metrics (Number of Hitches, Hitches Total Duration and Hitches Time Ratio).
Test fails sometimes because of exceeding max allowed deviation for average value (this also relates only to above mentioned metrics).

Those issues show that currently performance tools don’t work perfectly with hitches related metrics and we have to search for a workaround to make it work stably. As a workaround for the first case I increased maximum standard deviation to 400% to basically ignore this check and focus only on average values. For the second case I did 5 runs to see if the current baseline is stable and if the test fails I accepted test values as a baseline, after that test had become stable and it was green without any changes in code.

Testing

For the testing purposes I put following line right before cellForItemAt in collection view data source:



/* x is number of microseconds, for example usleep(10000)
blocks current thread for 10000 microseconds
or 0.01 second and usleep(1000000) blocks for 1 second */

usleep(x)

For test delays I tried to find turning values when test starts to fail. Here is the table of results:

This table uses following abbreviations: HTR — Hitch Time Ratio, HTD — Hitch Total Duration, NoH — Number of Hitches, FR — Frame Rate, FC — Frame Count.

From the table you see that tests start to fail on 25ms delay and all tries fail when there is 75 ms delay which I believe is a good result. Also as you can see from the table metrics related to hitches are more sensitive than others: frame rate and frame count failed only on 100ms when hitches are noticeable visually.

Conclusion

To sum up, tests are quite sensitive and easy to build. Unfortunately mandatory use of real devices imposes restrictions, to automate the process you need a device farm. One time fails because of max standard deviation without any code changes impairs efficiency. Also worth mentioning that baseline recording can be complicated if we talk about using CI.
In my opinion this tool is powerful and easy to use, it is great for controlling scrolling performance in your app. It also can be automated as a part of your release process along with your other unit and UI tests. But there are still a lot of things to improve, first of all in terms of stability and I hope to see such improvements in the next version of Xcode.

Useful links:
Eliminate animation hitches with XCTest
Ultimate application performance survival guide
Explore UI animation hitches and the render loop