DEV Community

Cover image for Catching Production Jank You Can’t Reproduce: ProfilingManager System Triggers 🔴
Android Expert
Android Expert

Posted on Originally published at Medium

Catching Production Jank You Can’t Reproduce: ProfilingManager System Triggers 🔴

The ticket reads the same way every time. “Checkout feels sticky after a while. Only on some phones.” QA scrolls through the screen on three devices and sees nothing. Your crash dashboard is green. Your frame-time percentiles moved by a hair. And a handful of users keep telling you the app stutters.

Aggregated metrics tell you that something is slow. They never tell you why. The why lives in a trace: which thread was blocked, what the CPU was doing, which of your own sections was running when the frame missed its deadline. The usual problem is that you cannot start a recorder before a stutter you did not predict.

ProfilingManager system triggers remove that constraint. You tell the platform which events interest you, and when one fires on a real user's device, the OS hands your app a Perfetto trace of what led up to it. This post walks through the whole loop in Kotlin: trigger types, registration, the result callback, upload, privacy limits, and reading the result afterwards. It also covers the part the documentation does not shout about: there is no built-in "jank" trigger, so for jank you build a small bridge.

why-system-triggers

How system triggers work

A normal profiling request starts a recording at the moment you ask. Trigger-based capture inverts that. The OS periodically runs a background trace on its own schedule, and when a registered event fires, it saves the recent portion of that trace into your app’s storage.

Three details of this model shape every design decision that follows:

  1. The background trace is not continuous. To protect battery and performance, the system starts it at randomized times. If nothing happens to be recording when your trigger fires, there is nothing to snapshot. Triggers are explicitly not guaranteed.
  2. The snapshot is a window, not a full history. The trace behaves like a ring buffer, so what you get is the most recent activity before the event, and it may not span the entire recording if the buffer filled.
  3. Results come back through one door. Output from system-triggered profiling is delivered only to a listener registered with registerForAllProfilingResults. There is no per-request callback because you never made a request.

Know your API levels before writing code

The class itself arrived in API 35, but triggers came later and keep growing, so version gating is part of the design, not an afterthought:

  • API 36 (Android 16): addProfilingTriggers, TRIGGER_TYPE_ANR, TRIGGER_TYPE_APP_FULLY_DRAWN, per-trigger rate limiting.
  • 36.1 (Android 16 QPR2): requestRunningSystemTrace, addAllProfilingTriggers, TRIGGER_TYPE_APP_REQUEST_RUNNING_TRACE, and the three user-initiated-kill triggers.
  • API 37 (Android 17): TRIGGER_TYPE_COLD_START, TRIGGER_TYPE_KILL_EXCESSIVE_CPU_USAGE, TRIGGER_TYPE_OOM, TRIGGER_TYPE_ANOMALY, TRIGGER_TYPE_APP_COMPAT.

Minor SDK releases are checked with SDK_INT_FULL, not SDK_INT. This is the pattern the Android team documents for QPR releases:

import android.os.Build

fun isAtLeastBaklava() = Build.VERSION.SDK_INT >= Build.VERSION_CODES.BAKLAVA

fun isAtLeastBaklava1() =
    isAtLeastBaklava() &&
        Build.VERSION.SDK_INT_FULL >= Build.VERSION_CODES_FULL.BAKLAVA_1
Enter fullscreen mode Exit fullscreen mode

Most of your users will not be on these versions yet. Treat trigger profiling as a sampling instrument over the newest slice of your fleet, and keep that in mind when you read the data later.

The trigger catalog, and the gap in it

trigger-catalog

Two observations from that catalog matter more than the rest.

Most triggers deliver a snapshot of the already-running trace. That is the valuable part: it contains the seconds before the event. TRIGGER_TYPE_COLD_START is the exception, because it starts a fresh trace plus stack sampling as early as possible, then runs until you call reportFullyDrawn() or a default timeout of five seconds elapses. TRIGGER_TYPE_OOM delivers a Java heap dump instead of a trace.

None of them is named “jank”. The closest built-ins are:

  • TRIGGER_TYPE_ANR, which fires after the system identifies an ANR but before it would attempt to kill the app. It does not mean the app was killed. This catches the extreme end of jank, where the main thread is blocked for seconds.
  • TRIGGER_TYPE_APP_FULLY_DRAWN, which fires after Activity.reportFullyDrawn() on a cold start. This is startup slowness, a form of jank users feel immediately.
  • TRIGGER_TYPE_APP_REQUEST_RUNNING_TRACE, which fires when your code calls requestRunningSystemTrace. This is the door for everything in between, and we use it in the jank section below.

Registering triggers

Registration has two parts that must happen on every process start: a global result listener, and the trigger list. Register the listener first. If your process died while a long collection was running, the system finishes and saves the profile, and then reports it to the next listener you register.

import android.app.Application
import android.os.Build
import android.os.ProfilingManager
import android.os.ProfilingTrigger
import androidx.annotation.RequiresApi
import java.util.concurrent.Executor
import java.util.concurrent.Executors

object ProfilingBootstrap {

    // Never the main executor: the callback copies files, and the platform
    // may reuse this executor for its own housekeeping.
    private val resultExecutor: Executor = Executors.newSingleThreadExecutor { runnable ->
        Thread(runnable, "profiling-results").apply { priority = Thread.MIN_PRIORITY }
    }

    fun install(app: Application, outbox: TraceOutbox) {
        if (!isAtLeastBaklava()) return
        val manager = app.getSystemService(ProfilingManager::class.java) ?: return

        manager.registerForAllProfilingResults(resultExecutor) { result ->
            outbox.accept(result)
        }
        manager.addProfilingTriggers(buildTriggers())
    }

    @RequiresApi(Build.VERSION_CODES.BAKLAVA)
    private fun buildTriggers(): List<ProfilingTrigger> = buildList {
        add(trigger(ProfilingTrigger.TRIGGER_TYPE_APP_FULLY_DRAWN, cooldownHours = hours(6)))
        add(trigger(ProfilingTrigger.TRIGGER_TYPE_ANR, cooldownHours = hours(1)))
        if (isAtLeastBaklava1()) {
            add(trigger(ProfilingTrigger.TRIGGER_TYPE_APP_REQUEST_RUNNING_TRACE, cooldownHours = hours(1)))
        }
    }

    // A period of 0 means no app-provided rate limiting: useful while debugging only.
    private fun hours(release: Int) = if (BuildConfig.DEBUG) 0 else release

    @RequiresApi(Build.VERSION_CODES.BAKLAVA)
    private fun trigger(type: Int, cooldownHours: Int): ProfilingTrigger =
        ProfilingTrigger.Builder(type)
            .setRateLimitingPeriodHours(cooldownHours)
            .build()
}
Enter fullscreen mode Exit fullscreen mode

Call it from Application.onCreate, not from an activity:

class ShopApp : Application() {
    lateinit var outbox: TraceOutbox
        private set

    override fun onCreate() {
        super.onCreate()
        outbox = TraceOutbox(this)
        ProfilingBootstrap.install(this, outbox)
    }
}
Enter fullscreen mode Exit fullscreen mode

A few notes on the design:

  • One registration per trigger type. The platform keeps a single registration for each type, so build the list once instead of appending on every screen. If your configuration changes at runtime (say, a remote flag), clearProfilingTriggers() or removeProfilingTriggersByType lets you rebuild cleanly.
  • setRateLimitingPeriodHours is your own cooldown, applied on top of the platform's limits. It is the minimum wait before the same trigger may deliver again, and a period of 0 turns it off. Without it, one noisy trigger such as ANR on a bad day can spend the whole budget before a rarer, more valuable one gets a turn.
  • Cold start needs a real “done” signal. APP_FULLY_DRAWN only fires if something calls reportFullyDrawn(). If you use Compose, the ReportDrawnWhen helpers from androidx.activity.compose make that explicit; otherwise call it when the first meaningful content is interactive.

The result callback

The callback receives a ProfilingResult. Five properties matter: errorCode, errorMessage, resultFilePath (null unless the request succeeded), triggerType (TRIGGER_TYPE_NONE when a trigger did not start the profile), and tag (the caller-defined tag, possibly null). Model that as a small sealed type so the rest of your code never touches raw integers:

import android.os.ProfilingResult
import java.io.File

sealed interface TraceDelivery {
    data class Ready(val file: File, val triggerType: Int, val tag: String?) : TraceDelivery
    data class Denied(val errorCode: Int, val reason: String?) : TraceDelivery
}

fun ProfilingResult.toDelivery(): TraceDelivery {
    val path = resultFilePath
    return if (errorCode == ProfilingResult.ERROR_NONE && path != null) {
        TraceDelivery.Ready(File(path), triggerType, tag)
    } else {
        TraceDelivery.Denied(errorCode, errorMessage)
    }
}
Enter fullscreen mode Exit fullscreen mode

The failure codes are worth counting rather than logging and forgetting. ERROR_FAILED_RATE_LIMIT_PROCESS and ERROR_FAILED_RATE_LIMIT_SYSTEM tell you your budget (or the whole device's) was spent. ERROR_FAILED_NO_DISK_SPACE and ERROR_FAILED_POST_PROCESSING mean the trace existed but was lost. Their ratio to successes is your delivery health.

The first job of the successful path is to take custody of the file. The platform stores profiles in a directory it manages, and the executor you pass to registerForAllProfilingResults may also be used for housekeeping such as cleaning up old delivered profiles. So do not hold a path and come back to it hours later:

import android.content.Context
import androidx.work.Constraints
import androidx.work.ExistingWorkPolicy
import androidx.work.NetworkType
import androidx.work.OneTimeWorkRequestBuilder
import androidx.work.WorkManager
import androidx.work.workDataOf
import java.io.File
import java.util.UUID

class TraceOutbox(private val context: Context) {

    private val dir = File(context.filesDir, "trace-outbox").apply { mkdirs() }

    // Runs on the single "profiling-results" thread.
    fun accept(result: ProfilingResult) {
        when (val delivery = result.toDelivery()) {
            is TraceDelivery.Denied -> ProfilingStats.recordDenied(delivery.errorCode)
            is TraceDelivery.Ready -> keep(delivery)
        }
    }

    private fun keep(delivery: TraceDelivery.Ready) {
        pruneOldFiles()
        val copy = File(dir, "${UUID.randomUUID()}-${delivery.file.name}")
        val copied = runCatching { delivery.file.copyTo(copy) }.isSuccess
        if (!copied) return ProfilingStats.recordCopyFailed()
        runCatching { delivery.file.delete() } // best effort: the platform may own cleanup

        val request = OneTimeWorkRequestBuilder<TraceUploadWorker>()
            .setConstraints(
                Constraints.Builder()
                    .setRequiredNetworkType(NetworkType.UNMETERED)
                    .setRequiresBatteryNotLow(true)
                    .build()
            )
            .setInputData(
                workDataOf(
                    TraceUploadWorker.KEY_PATH to copy.absolutePath,
                    TraceUploadWorker.KEY_TRIGGER to delivery.triggerType,
                    TraceUploadWorker.KEY_TAG to delivery.tag,
                )
            )
            .build()

        WorkManager.getInstance(context)
            .enqueueUniqueWork("trace-${copy.name}", ExistingWorkPolicy.KEEP, request)
    }

    private fun pruneOldFiles(maxAgeMs: Long = 3L * 24 * 60 * 60 * 1000, maxFiles: Int = 5) {
        val now = System.currentTimeMillis()
        val files = dir.listFiles().orEmpty().sortedByDescending { it.lastModified() }
        files.forEachIndexed { index, f ->
            if (index >= maxFiles || now - f.lastModified() > maxAgeMs) f.delete()
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Notice the bounded outbox. A phone that stays offline for a week should not slowly fill with traces. Cap by count and by age.

Bridging the gap: triggering on jank

Here is the part you have to build. The idea is to keep your frame-level detector (the Jetpack JankStats library, or FrameMetrics directly), and when it observes a severe stall, ask the platform for the trace that ended at that moment.

jank-detection-bridge

import android.os.ProfilingManager
import android.os.SystemClock
import androidx.metrics.performance.FrameData
import androidx.metrics.performance.JankStats

class JankTraceRequester(
    private val manager: ProfilingManager,
    private val now: () -> Long = SystemClock::elapsedRealtime,
    private val minFrameMs: Long = 200,        // ignore ordinary dropped frames
    private val cooldownMs: Long = 10 * 60_000, // one request per episode, roughly
) : JankStats.OnFrameListener {

    private var lastRequestAt: Long? = null

    override fun onFrame(volatileFrameData: FrameData) {
        if (!volatileFrameData.isJank) return
        val frameMs = volatileFrameData.frameDurationUiNanos / 1_000_000
        if (frameMs < minFrameMs) return

        val t = now()
        val last = lastRequestAt
        if (last != null && t - last < cooldownMs) return
        lastRequestAt = t

        val screen = volatileFrameData.states.firstOrNull { it.key == "screen" }?.value
        manager.requestRunningSystemTrace("jank:${JankScreen.from(screen).wire}")
    }
}

enum class JankScreen(val wire: String) {
    Home("home"), Search("search"), Checkout("checkout"), Other("other");

    companion object {
        fun from(raw: String?): JankScreen = entries.firstOrNull { it.wire == raw } ?: Other
    }
}
Enter fullscreen mode Exit fullscreen mode

Wiring it up in an activity:

class MainActivity : ComponentActivity() {

    private var jankStats: JankStats? = null

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        // setContent { ... }   // the window needs a decor view before tracking starts

        if (isAtLeastBaklava1()) {
            getSystemService(ProfilingManager::class.java)?.let { manager ->
                jankStats = JankStats.createAndTrack(window, JankTraceRequester(manager))
            }
        }
    }

    override fun onResume() {
        super.onResume()
        jankStats?.isTrackingEnabled = true
    }

    override fun onPause() {
        jankStats?.isTrackingEnabled = false
        super.onPause()
    }

    override fun onDestroy() {
        jankStats = null
        super.onDestroy()
    }
}
Enter fullscreen mode Exit fullscreen mode

JankStats calls the listener on the thread that delivers frame data (the frame-metrics thread on API 24 and up), so there is no executor to manage and the callback must stay cheap: compare, maybe make one call, return. The isTrackingEnabled toggles follow the library's own lifecycle guidance.

Three judgment calls are baked into that code, and they are yours to tune:

  • The threshold. The 200 ms floor is my choice, not a platform value. Ordinary dropped frames are common, and every request spends rate-limit budget. Start with severe stalls and lower the floor only once you can see how many results you actually receive.
  • The cooldown. One request per episode is plenty. A stall that lasts a second produces dozens of janky frames, and you want exactly one trace of it.
  • Detection latency. The snapshot reflects the trace up to the time of the request, and frame data reaches your listener slightly after the frame finishes. The cause is in the recent past, so the sooner you request, the better your trace covers it.

The result arrives in the same global listener as everything else, with triggerType == ProfilingTrigger.TRIGGER_TYPE_APP_REQUEST_RUNNING_TRACE and the tag you passed. Since the method returns nothing, that listener is also how you learn whether a request was honored.

From the file to your backend

trigger-to-perfetto-pipeline

The upload side is deliberately boring, and that is the goal. It should be impossible for tracing to hurt the user’s data plan or battery.

import android.content.Context
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
import java.io.IOException

class TraceUploadWorker(
    context: Context,
    params: WorkerParameters,
) : CoroutineWorker(context, params) {

    override suspend fun doWork(): Result {
        val file = inputData.getString(KEY_PATH)?.let(::File) ?: return Result.failure()
        if (!file.exists()) return Result.success() // already handled or pruned
        if (file.length() > MAX_BYTES) {
            file.delete()
            return Result.failure()
        }

        val api = (applicationContext as ShopApp).traceApi
        val meta = TraceMeta(
            triggerType = inputData.getInt(KEY_TRIGGER, -1),
            tag = inputData.getString(KEY_TAG),
            appVersion = BuildConfig.VERSION_NAME,
            sdk = android.os.Build.VERSION.SDK_INT,
        )

        return withContext(Dispatchers.IO) {
            try {
                if (api.upload(file, meta)) {
                    file.delete()
                    Result.success()
                } else {
                    retryOrGiveUp(file)
                }
            } catch (_: IOException) {
                retryOrGiveUp(file)
            }
        }
    }

    private fun retryOrGiveUp(file: File): Result =
        if (runAttemptCount < MAX_ATTEMPTS) {
            Result.retry()
        } else {
            file.delete()
            Result.failure()
        }

    companion object {
        const val KEY_PATH = "path"
        const val KEY_TRIGGER = "trigger"
        const val KEY_TAG = "tag"
        private const val MAX_ATTEMPTS = 5
        private const val MAX_BYTES = 40L * 1024 * 1024
    }
}
Enter fullscreen mode Exit fullscreen mode

On the wire, keep a few rules:

  • Compress and bound. Traces compress well. Reject anything over a hard size limit on the device and on the server.
  • Attach context, not identity. App version, OS version, trigger type, tag, and a coarse device class are what let you group traces later. A user ID is not.
  • Upload on unmetered, charging-friendly conditions. The constraints in the outbox above make that the default. A slow trace on a slow day is still worth less than a surprise data charge.
  • Delete on success. The device should not accumulate your diagnostics.

Privacy limits

privacy-boundary

The platform does real work here. Traces from ProfilingManager are redacted on the device before delivery: activity from other processes is folded into one combined view, so you see how busy the rest of the system was without seeing what it was doing. The file lands in your app's private storage.

But the boundary is exactly the callback. After that, everything is your responsibility, and the redaction does not cover what you put into the trace:

  • Custom trace sections. If you instrument with androidx.tracing, every section name travels in the file. A name like "loadCart user=$userId" is a user identifier inside a diagnostic upload. Keep names static:
  import androidx.tracing.trace

  // Leaks an identifier into every uploaded trace.
  suspend fun loadCartBad(userId: String) = trace("loadCart user=$userId") { /* ... */ }

  // Static name; put the variable part in your own logs, not the trace.
  suspend fun loadCart() = trace("loadCart") { /* ... */ }
Enter fullscreen mode Exit fullscreen mode
  • Tags. The tag you pass to requestRunningSystemTrace is stored with the result. That is why the earlier example maps screens through an allow-list enum instead of forwarding arbitrary strings.
  • Consent and disclosure. A trace of your process is diagnostic data about a person’s device use. Treat it that way: decide whether it needs consent in your markets, and make sure your privacy policy and any store data-disclosure forms describe it. Your legal team, not this post, decides the specifics.
  • Retention and access. Keep retention short. Limit who can open traces, since even redacted traces reveal usage patterns.

Reading the trace afterwards

reading-a-jank-trace

When a trace arrives, resist the urge to hunt for the longest slice first. Ask three questions in order, because each narrows what the next one can mean:

  1. Was the system busy? The combined OtherProcesses lane shows how much CPU the rest of the device consumed. A crowded CPU explains slowness without implicating your code. It also tells you not to chase a bug you cannot fix.
  2. What was the main thread’s state? Running means your code was executing. Runnable means it wanted the CPU and was waiting for a core, which points to contention or priority. Sleeping means it was waiting on something: a lock, a binder call, I/O.
  3. Which of your slices owns the time? Only now do your own named sections matter. A long computation, or a long wait inside a section, is the actual culprit.

For local inspection, open the file in the Perfetto UI at ui.perfetto.dev. For fleet analysis, load many traces into Perfetto's trace processor and query them. This SQL (adjust the package pattern; I wrote it against Perfetto's standard tables and you should try it in the UI before building on it) lists the longest main-thread slices:

select s.name, s.dur / 1e6 as ms
from slice s
join thread_track tt on s.track_id = tt.id
join thread t using (utid)
join process p using (upid)
where p.name like 'com.example.shop%'
  and t.tid = p.pid
order by s.dur desc
limit 10;
Enter fullscreen mode Exit fullscreen mode

On a device, profiles live under your app’s files/profiling/ directory. Treat the exact file name as an implementation detail and always use resultFilePath: names for on-demand requests include your tag, while trigger-based files encode the trigger type (the API reference shows profile_trigger-type-2_<datetime>.perfetto-trace as the example for an ANR). On a debuggable build you can pull it with adb, and the platform's debug mode can also retain unredacted traces for local analysis.

A single trace is an anecdote. The value comes from grouping: by trigger type, by tag, by OS version and device class, and then by the dominant cause you classified using the three questions. Ten traces that all show Sleeping inside the same binder call are a finding. One trace is a hint.

Failure modes that bite in production

field-failure-modes

Three of those deserve more than a row.

1. Missing traces are not missing events

missing-trace-reasons

Because the background trace starts at random times and budgets are finite, the number of traces you receive is not the number of incidents. If you chart “ANR traces per day,” you are charting a mixture of incident rate, sampling luck, and rate-limit headroom. Report rates against a denominator you control (sessions, or a count of the triggers you requested), and never compare raw trace counts across OS versions that sample differently.

2. Testing in a lab hides the rate limiter

Locally, the limiter and the random background trace make triggers feel flaky. The platform provides a testing mode for system triggers: one command pins a background trace on for your package and lets every one of its triggers pass the system-level limiter.

adb shell device_config put profiling_testing system_triggered_profiling.testing_package_name com.example.shop
Enter fullscreen mode Exit fullscreen mode

That mode does not touch the cooldown you set with setRateLimitingPeriodHours, so use a period of 0 (no app-provided limiting) in debug builds, as the bootstrap code above does.

You may also see a second flag, rate_limiter.disabled true, described in the debug docs as turning off the app and system limiters. For system triggers it is not needed, and the platform source suggests it only governs on-demand requestProfiling() calls, so do not rely on it when testing triggers.

Use these settings on your own devices only. If you only ever validate with the limiters bypassed, you will ship a feature that behaves very differently in the field, and you will not notice until a quiet week of data.

3. Handlers that quietly disable triggers

The OOM trigger only works if any custom Thread.UncaughtExceptionHandler you install passes the exception through to the default handler. A "log and swallow" crash reporter can silently turn it off.

val default = Thread.getDefaultUncaughtExceptionHandler()
Thread.setDefaultUncaughtExceptionHandler { thread, throwable ->
    MyCrashReporter.record(throwable)
    default?.uncaughtException(thread, throwable) // do not skip this
}
Enter fullscreen mode Exit fullscreen mode

A rollout checklist

  • Gate everything on API 36 (and SDK_INT_FULL for 36.1 features); assume most devices do nothing.
  • Register the listener and triggers in Application.onCreate, listener first.
  • Give every trigger its own cooldown so one noisy event cannot starve the rest.
  • Copy results out of the platform directory immediately, on a background thread.
  • Upload through WorkManager with network and battery constraints, a size cap, and deletion on success.
  • Keep trace section names static and tags allow-listed.
  • Count denials by error code so you can see delivery health.
  • Analyze by cause and by rate, never by raw trace count.
  • Enable gradually (a small percentage of eligible users) and watch the outbox size and upload failure rate first.

🔚 Closing thought

The shift trigger-based capture brings is simple to state and subtle to apply: you stop asking users to reproduce problems and start letting problems submit their own evidence. The platform solves the hardest part, capturing the seconds before an event you cannot predict. What remains is engineering discipline around it: honest expectations about sampling, careful handling of what the trace contains, and a bridge for the jank the catalog does not name.

Sources

📱 Go Beyond Using Jetpack Compose

If you’re building on Android, understanding what happens under the hood separates developers who use Compose from those who master it. I highly recommend “Mastering Jetpack Compose Internals”. It’s a deep, architecture-first walkthrough of the composition tree, the slot table, snapshot state, and the runtime that powers modern Android UI — capped off with a full case study building a real app called Mosaic.

Top comments (0)