The dream of on-device Generative AI is finally a reality. With the introduction of Gemini Nano and Google’s AICore, developers can now run Large Language Models (LLMs) directly on a user's smartphone. No more latency-heavy API calls to the cloud, no more massive server costs, and no more privacy concerns regarding data leaving the device. It feels like magic—until the device starts to heat up, the UI begins to stutter, and the operating system aggressively kills your background processes.
Deploying GenAI on-device introduces a fundamental engineering conflict that we call the Performance Paradox. On one hand, we want maximum throughput to provide a snappy, "human-like" conversational experience. On the other hand, we are operating within a passively cooled, battery-constrained environment where the laws of thermodynamics are non-negotiable.
In this guide, we will dive deep into the architecture of on-device AI, explore the critical metrics you need to track, and implement a thermal-aware orchestration system in Kotlin to ensure your app remains a "good citizen" of the Android ecosystem.
(This article is based on the ebook On-Device GenAI with Android Kotlin)
The Performance Paradox: Why Mobile is Different
In the cloud, scaling is a matter of spinning up more A100 GPUs and ensuring the data center’s industrial cooling systems are humming. If a model is slow, you throw more compute at it. On Android, your "data center" is a glass-and-metal sandwich in a user's pocket.
When a Neural Processing Unit (NPU) or GPU runs at peak utilization to generate tokens, it generates concentrated heat. Unlike a PC, an Android device has no fans. It relies on passive heat dissipation. Once the System on Chip (SoC) reaches a critical thermal threshold, the Android kernel triggers Thermal Throttling. This is a defensive mechanism that aggressively lowers clock speeds to prevent hardware damage or physical discomfort for the user.
For developers, this creates a volatile performance environment. A benchmark run at "cold boot" (when the device is cool) will yield significantly better results than a benchmark run after five minutes of continuous usage. Understanding this volatility is the cornerstone of professional AI development on mobile.
The Architecture of AICore: Model-as-a-Service
Google’s strategic decision to move Gemini Nano into AICore—a system-level service—rather than bundling it as a library within your APK, is a game-changer for performance. To understand why, let’s look at the "Room Database" analogy.
Just as you wouldn't want every single feature module in your app to maintain its own separate SQLite connection and migration logic, you cannot have every AI-enabled app loading its own 2GB+ LLM into RAM. If five different apps used their own local copy of Gemini Nano, the device would run out of memory (OOM) almost instantly.
AICore acts as a System Provider Model, offering three primary benefits:
- Memory Deduplication: AICore ensures only one instance of the model weights is loaded into the system's shared memory (using
ionordmabuf). This prevents the Android OOM killer from nuking your background processes. - Hardware Abstraction: AICore abstracts the complexity of NPU/GPU drivers. It dynamically determines whether to run an operation on the TPU, the GPU via OpenCL/Vulkan, or the CPU via Neon instructions, based on the current thermal state.
- Seamless Updates: By decoupling the model from the app, Google can update model weights or the inference engine via Play System Updates. You don't have to push a new APK just because the model got 5% more efficient.
The Three Pillars of AI Benchmarking
When we talk about performance in GenAI, traditional "execution time" is a useless metric. We need to decompose performance into three AI-centric metrics:
1. Time to First Token (TTFT)
TTFT measures the latency from the moment the user hits "Send" to the moment the first character appears on the screen. This is dominated by the Prompt Processing (Prefill) phase.
- The Technical Reality: The model must process the entire input context before it can predict the first token.
- The UX Impact: High TTFT makes the app feel "frozen." If your TTFT is over 1 second, you need a loading state or a "thinking" animation.
2. Tokens Per Second (TPS)
Once the first token is generated, the model enters the Autoregressive (Decoding) phase. TPS measures the steady-state generation speed.
- The Technical Reality: This is where the NPU is doing the heavy lifting, predicting one token at a time.
- The UX Impact: Human reading speed is roughly 5–10 tokens per second. If your TPS drops below 5, the experience feels sluggish and frustrating.
3. Memory Pressure (Peak RSS)
On-device LLMs are memory-hungry. We track the Resident Set Size (RSS) to see how much physical RAM is occupied.
- The Technical Reality: If an AI task pushes the system into a "Low Memory" state, Android will kill background apps (like the user's music player).
- The UX Impact: Your app might be fast, but if it makes the user's Spotify crash, they will uninstall it.
The Thermal Loop: How Android Fights Back
Thermal management in Android is not a binary "on/off" switch; it is a gradient. Think of it like CameraX. When you record 4K video, the camera might drop from 60fps to 30fps to prevent overheating. AICore does the same thing.
The Thermal Loop works in five stages:
- Compute Spike: You send a massive prompt to Gemini Nano. The NPU hits max frequency.
- Heat Accumulation: The SoC temperature rises rapidly.
- Thermal HAL Trigger: The Android Thermal Hardware Abstraction Layer (HAL) detects a threshold breach (e.g.,
THERMAL_STATUS_MODERATE). - Frequency Scaling (DVFS): Dynamic Voltage and Frequency Scaling kicks in, lowering the clock speed of the NPU.
- Performance Degradation: Your TPS drops from 15 t/s to 6 t/s.
As a developer, you cannot stop this loop, but you can monitor it and react to it.
Implementation: Building a Performance & Thermal Monitor
To capture these metrics without slowing down the system (the "observer effect"), we leverage Kotlin’s non-blocking primitives. We will use callbackFlow to listen to thermal changes and StateFlow to update the UI.
The Performance Monitor Logic
import android.content.Context
import android.os.PowerManager
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*
/**
* Data class to encapsulate the performance snapshot of a single inference request.
*/
data class InferenceMetrics(
val ttftMs: Long = 0,
val averageTps: Double = 0.0,
val peakMemoryMb: Long = 0,
val thermalStatus: String = "Normal"
)
class AiPerformanceMonitor(private val context: Context) {
private val powerManager = context.getSystemService(Context.POWER_SERVICE) as PowerManager
/**
* A Flow that streams the current thermal status of the device.
* Converts the Android Callback API into a modern Kotlin Flow.
*/
val thermalStatusFlow: Flow<Int> = callbackFlow {
val listener = PowerManager.OnThermalStatusChangedListener { status ->
trySend(status)
}
powerManager.addThermalStatusListener(listener)
trySend(powerManager.currentThermalStatus)
awaitClose { powerManager.removeThermalStatusListener(listener) }
}.flowOn(Dispatchers.Default)
/**
* Measures the performance of an inference call.
*/
suspend fun <T> measureInference(
block: suspend () -> T
): Pair<T, InferenceMetrics> = withContext(Dispatchers.Default) {
val runtime = Runtime.getRuntime()
val startMemory = (runtime.totalMemory() - runtime.freeMemory()) / (1024 * 1024)
val startTime = System.currentTimeMillis()
// Execute the AI task
val result = block()
val endTime = System.currentTimeMillis()
val totalDuration = endTime - startTime
val endMemory = (runtime.totalMemory() - runtime.freeMemory()) / (1024 * 1024)
val metrics = InferenceMetrics(
ttftMs = 0, // In a real app, capture this from the first emitted token
averageTps = 0.0, // Calculate based on token count / duration
peakMemoryMb = endMemory - startMemory,
thermalStatus = getStatusString(powerManager.currentThermalStatus)
)
Pair(result, metrics)
}
private fun getStatusString(status: Int): String = when (status) {
PowerManager.THERMAL_STATUS_NONE -> "Cool"
PowerManager.THERMAL_STATUS_MODERATE -> "Moderate"
PowerManager.THERMAL_STATUS_SEVERE -> "Severe"
PowerManager.THERMAL_STATUS_CRITICAL -> "Critical"
else -> "Unknown"
}
}
Advanced Strategy: Thermal-Aware Orchestration
In a production-grade app, you shouldn't just watch the performance drop; you should change your strategy. This is called Thermal-Aware Orchestration.
If the device is "Cool," use the highest precision model and the GPU. If the device reaches "Moderate" heat, switch to a quantized model or add "cooling gaps" (delays) between inference calls to let the hardware rest.
The Thermal Orchestrator Implementation
sealed class InferenceStrategy {
object HighPerformance : InferenceStrategy() // Max NPU usage
object PowerSaver : InferenceStrategy() // CPU only, slower but cooler
object EmergencyCooling : InferenceStrategy() // Stop inference, notify user
}
class ThermalOrchestrator(
private val monitor: AiPerformanceMonitor,
private val scope: CoroutineScope
) {
private val _currentStrategy = MutableStateFlow<InferenceStrategy>(InferenceStrategy.HighPerformance)
val currentStrategy: StateFlow<InferenceStrategy> = _currentStrategy.asStateFlow()
init {
monitor.thermalStatusFlow
.onEach { status ->
_currentStrategy.value = when {
status >= PowerManager.THERMAL_STATUS_SEVERE -> InferenceStrategy.EmergencyCooling
status >= PowerManager.THERMAL_STATUS_MODERATE -> InferenceStrategy.PowerSaver
else -> InferenceStrategy.HighPerformance
}
}
.launchIn(scope)
}
suspend fun executeAiTask(prompt: String, aiRepo: AIInferenceRepository): String {
return when (val strategy = _currentStrategy.value) {
is InferenceStrategy.EmergencyCooling -> {
"Device too hot. Please wait a moment."
}
is InferenceStrategy.PowerSaver -> {
// Add a cooling gap to reduce SoC strain
delay(200)
aiRepo.runInference(prompt, useGpu = false)
}
is InferenceStrategy.HighPerformance -> {
aiRepo.runInference(prompt, useGpu = true)
}
}
}
}
Common Pitfalls to Avoid
Even with a great monitoring system, developers often fall into these three traps:
- Ignoring the "Warm-up" Effect: The very first time you run an AI model, it’s slow. The system is loading weights into RAM and compiling GPU kernels. Never use the first run as your benchmark. Perform 2-3 "warm-up" runs and discard them before recording data.
- Main Thread Blocking: AI inference is the definition of a CPU-intensive task. If you run it on
Dispatchers.Main, your UI will freeze, and Android will trigger an ANR (Application Not Responding) dialog. Always useDispatchers.Default. - Memory Leaks in Callbacks: When using
PowerManagerlisteners, always ensure you unregister them in theawaitCloseblock of your Flow. Failing to do so will leak the entire ViewModel or Activity context.
Conclusion: Being a Good Citizen
The future of Android development is AI-native, but that doesn't mean we can ignore the hardware. By treating on-device GenAI as a resource-constrained system service rather than a local library, we can build apps that are both powerful and responsible.
Benchmarking TTFT and TPS gives you the data you need to optimize the user experience. Implementing a Thermal Orchestrator ensures that your app doesn't become the reason a user's phone feels like a hot brick. As we move toward more complex on-device models, the developers who master the balance between "Maximum Throughput" and "Thermal Stability" will be the ones who define the next generation of mobile experiences.
Let's Discuss
- How are you currently handling long-running AI tasks on Android to prevent the device from overheating?
- Do you think users prefer a slower, more consistent AI response or a fast response that might trigger thermal throttling midway through?
Author's Note: This post is part of a series on Modern Android AI Development. If you found this technical deep-dive useful, consider sharing it with your engineering team.
The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the ebook
On-Device GenAI with Android Kotlin: Mastering Gemini Nano, AICore, and local LLM deployment using MediaPipe and Custom TFLite models. You can find it here: Leanpub.com
Check also all the other programming & AI ebooks with python, typescript, c#, swift, kotlin: Leanpub.com
Android Kotlin & AI Masterclass:
Book 1: On-Device GenAI. Mastering Gemini Nano, AICore, and local LLM deployment using MediaPipe and Custom TFLite models.
Book 2: Edge AI Performance. Optimizing hardware acceleration via NPU (Neural Processing Unit), GPU, and DSP. Advanced quantization and model pruning.
Book 3: Android AI Agents. Building autonomous apps that use Tool Calling, Function Injection, and Screen Awareness to perform tasks for the user.
Top comments (0)