From smartphone to sovereign agent:How native on-device intelligence reshapes the personal computing device

Abstract

For fifteen years the smartphone has been a window onto intelligence that lives somewhere else. The handset captures a voice command or a photograph, ships it to a hyperscale GPU cluster, and renders the answer a moment later. The arrangement works, and it levies four structural taxes the industry has learned to pay without noticing: every inference is a metered cloud call, so cost scales linearly with use; every inference requires a network round trip, so latency is bounded below by the network and the intelligence disappears entirely when connectivity does; every inference exports the user’s messages, location, health and financial data to someone else’s computer; and every inference depends on a remote provider who can change, re-price or withdraw the capability unilaterally. Caching, batching and smaller cloud-served models reduce these taxes. None of them removes one, because in every case the intelligence still resides off the device.

This paper presents a working proof-of-concept for native intelligence — a host application that loads a Gemma 4 model [1] entirely onto an Android handset and exposes it to every other application on the device as a shared, private inference service — together with an architectural account of why such a system is practical today. The account has four layers: a model architecture that decouples raw parameter count from effective memory footprint through the Matryoshka Transformer [3] and Per-Layer Embeddings; a compression toolbox led by quantization [4], [5] and supported by pruning, palettization, distillation [10] and low-rank adaptation [11]; a runtime and packaging layer — LiteRT-LM and the .litertlm container [6] — that memory-maps the model and delegates execution to the fastest accelerator present; and a platform surface through which applications consume the resident model instead of bundling their own [2], [7].

The prototype was validated by interactive engineering observation on a single production handset — a OnePlus 15 held in airplane mode, with no server reachable at any point — rather than against a fixed benchmark corpus, a scope limitation stated plainly in Section 6.1. Sustained generation ran at roughly 40 tokens per second, and independently demonstrated figures on competing hardware fall in the same band. The central finding is that on-device generation has crossed the threshold at which it reads as instant to a human, which relocates the binding constraint on native intelligence away from model capability and silicon throughput and onto the architecture of the software that hosts the model.

Keywords: on-device inference, native intelligence, Gemma 4, LiteRT-LM, MatFormer, per-layer embeddings, quantization, agentic computing, edge AI, privacy-native architecture.

1. The cloud dependency problem in personal computing

1.1 Four structural taxes, and why current approaches do not remove them

Most of what the market calls “AI on your phone” today is a polite illusion of locality. The phone captures the input and a data centre produces the output. The user experience is excellent. The architecture underneath it levies four taxes.

The first is cost. Every inference is a metered cloud call. At the scale of a billion users asking a billion questions a day, the meter never stops running.

The second is latency. A network round trip sits between intent and answer, and where the network is absent — a plane, a tunnel, a basement — the intelligence is absent with it.

The third is privacy. Messages, location, health records and financial history must leave the device and be processed on infrastructure the user does not control for the capability to work at all.

The fourth is dependence. The experience belongs to a remote provider who can alter it, re-price it, or withdraw it without the user’s consent.

Existing approaches attack the symptoms. Smaller cloud-served models reduce the per-call cost; edge caching and regional inference shorten the round trip; confidential-computing enclaves and contractual data-processing terms mitigate exposure. Each improves a number. None changes the structure, because the model still runs somewhere the user does not own. Native intelligence removes all four taxes at once: once the model executes on the device, marginal inference cost after the hardware purchase is effectively zero, latency is bounded only by local memory bandwidth, the work continues offline, and privacy stops being a policy promise and becomes a physical property of the system — data cannot leak to a server it never touches.

This is not a fringe position. The largest platform companies are converging on the same architecture. Android’s AICore exposes an on-device model as a system service that any application can call [2], and Apple’s Foundation Models framework lets applications tap the model resident on the device [7]. The industry has independently arrived at the same shape: one capable model, living on the device, shared by everything that runs on it.

1.2 The app-centric interface widens the gap

The application was invented for a world in which each vendor shipped a sealed island of functionality. A user kept a ride-hailing app, a calendar, a bank app and a messaging app, and the user was the integration layer carrying information between them.

A resident, capable, multimodal model dissolves those islands. Rather than moving between four applications to satisfy one intention — “get me to the airport in time for my six o’clock flight and tell my colleague I will be late” — a single agent parses the intention and orchestrates the underlying services to fulfil it.

The applications do not disappear; they become tools. Each exposes a capability — book, pay, fetch, send — that the personal agent invokes on the user’s behalf. This is the direction of emerging interoperability standards such as the Model Context Protocol, which gives an agent a uniform way to discover and call external tools [9]. The grid of icons gives way to a conversation with one intelligence that has hands. This dimension widens the gap described in Section 1.1 because it changes who the enterprise is building for: the addressable surface becomes the agent rather than the screen, as detailed in Sections 3.7 and 5.3.

1.3 Contributions

This paper presents a proof-of-concept architecture for device-wide native intelligence, addressing the problem across four dimensions:

• A shared on-device inference service: a host application that loads a single Gemma 4 E4B model in the .litertlm format and serves it to other applications on the handset, making intelligence a system resource rather than a per-application dependency (Section 3).
• A layered account of on-device performance: a decomposition of why multi-billion-parameter generation is fast on a phone, tracing every major technique — MatFormer elasticity, Per-Layer Embeddings, KV cache sharing, selective parameter loading, quantization, hardware delegation and memory-mapped packaging — back to a single memory-bandwidth bottleneck (Sections 3.1 to 3.5).
• Measured offline performance on commodity hardware: sustained generation of roughly 40 tokens per second on an unmodified consumer handset in airplane mode, together with independent public figures from a competing platform (Section 4).
• An enterprise adoption position: the privacy, cost and interface consequences of the inversion, the security surface it creates, and four concrete first steps for organizations preparing for it (Section 5).

2. Related work

Research on running large models under tight memory budgets is mature and largely orthogonal to this paper’s contribution. Knowledge distillation trains a compact student to imitate a larger teacher [10]; post-training quantization methods such as GPTQ [4] and activation-aware schemes such as AWQ [5] compress a trained model by storing weights in fewer bits while protecting the small share that matters most; low-rank adaptation [11] and its quantized variant [12] reduce the cost of specializing a model rather than the cost of running it. Elastic inference — serving several model sizes from one set of weights — was formalized as the Matryoshka Transformer [3]. In parallel, platform vendors have shipped on-device model services: Android’s AICore and Gemini Nano [2], Apple’s Foundation Models framework and its adapter training toolkit [7], [8], and the Model Context Protocol for agent-to-tool interoperability [9].

What this literature and practice leave open is the layer between them. Compression research optimizes a model in isolation; platform services expose a vendor’s own model through a vendor’s own API. Neither formalizes the resident model as a device-wide service built on open weights that an independent developer can host, tune and share across applications. This paper introduces and measures exactly that arrangement, and traces its performance to a single bottleneck rather than to a list of unrelated optimizations.

Visions of this kind have been declared prematurely before, so it is worth stating why the arrangement is practical now and was not three years ago. Four independent curves have crossed at the same moment:

• Model efficiency and licensing — Gemma 3n already delivered quality rivalling the previous year’s cloud systems within a few gigabytes of memory. Its April 2026 successor, Gemma 4, shipped under a permissive Apache 2.0 licence, a marked loosening from Gemma 3’s terms, and supplies the open weights that Google’s own on-device Gemini Nano is now built on [1].
• Silicon — Apple’s Neural Engine has climbed from 35 trillion operations per second on the A17 Pro to 38 on the M4 [14], and Qualcomm’s current Hexagon NPU reaches 80 TOPS [15]. Each generation narrows the gap between a handset accelerator and a data-centre GPU at a fraction of the power draw.
• Runtimes — LiteRT, the successor to TensorFlow Lite, and its LiteRT-LM layer for language models make it practical to deploy and serve a model locally, with hardware acceleration, across a fragmented landscape of chips [6].
• Interoperability — system-level AI services [2], [7] and tool-calling standards [9] give applications a common way to share one on-device model and to expose their own capabilities to it.

The prototype described in Section 3 sits at the intersection of these four curves, which is why it runs today rather than in a hypothetical future.

3. System architecture: native intelligence on the handset

The prototype is a host application that owns the model and lends it out. It loads a single Gemma 4 model into memory once, keeps it resident, and answers inference requests from any other application on the device. Other applications no longer each embed a model or call a cloud endpoint; they request intelligence from the one brain the device already has. As shown in Figure 1, the system decomposes into five parts.

Figure 1. Reference architecture for a device-wide native intelligence service

Source: AI generated

• Resident model — a Gemma 4 E4B checkpoint packaged as a single .litertlm container holding weights, tokenizer and metadata in a memory-mappable layout [6].
• Inference engine — a LiteRT-LM Engine that loads the model and holds its resources for the lifetime of the host process.
• Session layer — per-conversation Sessions that can be created and cloned cheaply, so one loaded model serves many concurrent callers.
• Hardware delegation layer — the routing that dispatches each workload to the fastest available compute unit: optimized CPU kernels, the GPU, or a dedicated NPU.
• Device-wide service interface — the boundary at which other applications submit prompts and receive streamed tokens, and at which they expose their own capabilities to the model as callable tools [9].

3.1 The bottleneck is memory bandwidth, not arithmetic

The first intuition to discard is that text generation is compute-bound. During generation a language model produces one token at a time, and to produce each token it must stream a large fraction of its weights out of memory and into the compute units. The arithmetic is comparatively cheap. The expensive part is moving billions of numbers, once per token, for every token.

This single fact governs on-device performance, and it can be stated as a ceiling:

Tokens per second ≈ usable memory bandwidth ÷ bytes streamed per token

In plain terms, the speed at which a phone generates text is set by how fast it can move the model past the processor, not by how fast the processor can multiply. A worked example makes the consequence concrete. A model whose per-token working set is 2 GB, running on a handset that sustains 60 GB/s of usable bandwidth to its accelerator, cannot exceed roughly 30 tokens per second regardless of the NPU’s arithmetic throughput. Halving that working set to 1 GB — which is what four-bit quantization achieves against an eight-bit baseline — roughly doubles the ceiling to 60. Every technique described in the rest of this section attacks one side of that ratio: shrinking the denominator, or moving the bytes through faster memory sitting closer to the right accelerator.

3.2 Effective parameters: decoupling capability from footprint

Gemma 4 is Google’s model family built specifically for on-device use, and its two members carry deliberately unusual names: E2B and E4B, where the E stands for effective [1]. The E2B model contains roughly five billion raw parameters yet consumes run-time memory like a two-billion-parameter model, running comfortably in around two gigabytes of RAM. The E4B model holds roughly eight billion raw parameters but carries the footprint of a four-billion-parameter model, in the neighbourhood of three gigabytes.

The gap between raw size and effective footprint is not marketing. It is the product of two architectural inventions and two loading optimizations, described below.

3.2.1 MatFormer: nested models and elastic inference

The Matryoshka Transformer, or MatFormer, is named after the Russian nesting dolls [3]. Just as a large doll contains a complete smaller doll, a MatFormer contains fully functional smaller models nested inside a larger one. When the E4B model was trained, a complete E2B model was trained simultaneously inside it, sharing the same weights.

This yields elastic inference. From a single downloaded model the runtime can choose, at execution time, to run the full E4B for maximum quality or the nested E2B for maximum speed and minimum power, without shipping or loading two separate models. A technique called Mix-n-Match goes further, assembling intermediate sizes between E2B and E4B by selectively including parts of the larger model, tuning the quality-speed-memory trade-off to the specific device and task.

For a personal agent this is the difference between a fixed cost and a variable one. The same resident intelligence can drop into a frugal mode while it waits on battery for a wake word, and expand into its full self for a hard question asked while the device is plugged in.

3.2.2 Per-Layer Embeddings: keeping the hot memory small

Per-Layer Embeddings, or PLE, is the reason an eight-billion-parameter model can behave like a four-billion-parameter one [1].

In a transformer a substantial share of the parameters lives in the embedding tables. The insight behind PLE is that these per-layer embedding parameters do not need to occupy the scarce, fast memory attached to the accelerator where the heavy matrix multiplication happens. They can be computed efficiently on the CPU and added into each layer as needed, streamed from ordinary system RAM.

The effect is to shrink the hot footprint. Only the core transformer weights need to sit in constrained, high-value accelerator memory, while a large body of parameters is held elsewhere and brought in cheaply. The model keeps the quality its full parameter count buys and pays the memory bill of a much smaller one. PLE is why the “effective” in E2B and E4B is an honest description rather than a slogan.

3.2.3 KV cache sharing and selective parameter loading

Two further optimizations govern the felt experience of speed.

The first is KV cache sharing. Before a model can produce its first word it must read and digest the entire prompt, a phase called prefill. For long prompts, and especially for streaming inputs such as audio and video, prefill dominates time-to-first-token — the delay a user actually perceives as lag. Gemma 4 shares key-and-value information across certain layers so that prefill is accelerated and the model begins responding sooner [1].

The second is conditional, or selective, parameter loading. Gemma 4 is natively multimodal, but a given task rarely needs every sense at once. If the task is pure text, the parameters that handle vision and audio need never enter memory; they stay on disk and are summoned only when an image or a sound actually arrives. The model held in memory is only ever as large as the job in front of it.

3.3 Compression: quantization and the wider toolbox

Layered on top of the architecture is a toolbox for shrinking a model that already exists. Quantization matters most because it attacks the bandwidth ratio in Section 3.1 directly, by storing each weight in fewer bits. A weight kept in sixteen-bit floating point can often be represented in eight or even four bits with little loss of quality, cutting the bytes streamed per token — and, since bandwidth is the true bottleneck, that translates almost one-for-one into speed and battery saved.

There are two routes to it. Post-training quantization compresses a model after training finishes: cheap and fast, but capable of degrading a small model badly at four bits [4]. Quantization-aware training simulates the lower precision during training itself, so the model learns weights that tolerate it, at the cost of a heavier training run. Methods such as AWQ [5] and GPTQ [4] split the difference, holding the small share of weights that matter most at higher precision while quantizing the rest aggressively.

Three further techniques complete the toolbox. Weight clustering, or palettization, groups similar values into a shared codebook rather than storing each independently; Apple layers small LoRA-style accuracy-recovery adapters on a palettized model specifically to claw back the quality that clustering costs [8]. Pruning removes weights outright — unstructured pruning reaches high sparsity but needs sparse-matrix hardware most phones lack, so structured pruning, which removes whole channels, heads or layers, is the more common choice at the edge. Knowledge distillation trains a small student model to imitate a larger teacher, transferring capability into a fraction of the parameters [10]; it is the recipe behind parts of Gemma and of Meta’s Llama 3.2.

LoRA and its quantized cousin QLoRA shrink the cost of adapting a model rather than the model itself: the base weights are frozen and only a small pair of low-rank matrices per layer is trained [11], [12]. This is the mechanism behind Apple’s on-device fine-tuning toolkit described in Section 3.7. A quantized, pruned, distilled model carrying a LoRA adapter is not hypothetical; it is approximately what production on-device deployments look like today.

3.4 Runtime and packaging: LiteRT-LM and the .litertlm format

Architecture and compression define what the model is. The runtime defines how fast it actually runs on a particular handset.

LiteRT is Google’s on-device machine-learning runtime, the direct successor to what the industry knew for years as TensorFlow Lite. LiteRT-LM is a purpose-built C++ stack layered on top of it, engineered specifically for running large language models at the edge [6]. It is not a research artefact: the same technology lineage powers Gemini Nano across Google’s own products, from Chrome to Chromebooks to wearables.

LiteRT-LM organizes execution around two objects. An Engine loads the model and holds its resources. A Session represents an individual conversation or context and can be cloned and managed cheaply. One loaded model can therefore serve many simultaneous requests economically, which is precisely what a device-wide AI service shared by every application requires.

Its most consequential job is hardware delegation. The same model can be dispatched to the CPU through highly optimized kernels, to the GPU, or to a dedicated NPU, and the runtime routes the work to the fastest accelerator the device offers. One model binary therefore runs well across a fragmented landscape of handsets without being hand-tuned for each.

The .litertlm file is the container that makes this deployable. Rather than scattering weights, tokenizer and metadata across several files, it bundles them into a single package laid out for efficient memory-mapped loading. Memory mapping matters more than it sounds: the runtime treats the model file on storage as though it were already in memory, bringing pages in on demand rather than copying gigabytes up front. The practical consequences are fast cold starts and low memory overhead — the model is ready almost immediately and does not need a wasteful duplicate copy of itself in RAM. The format is cross-platform, so the same packaged model runs on Android, on Linux and on desktop targets alike.

3.5 Why the techniques compound

Assembled, these techniques stop being a list and become a single argument. MatFormer allows a smaller effective model to be run when speed matters. Per-Layer Embeddings keep the hot footprint small, so more of what must move each token already sits in fast accelerator memory. Quantization cuts the bytes moved per token. LiteRT-LM dispatches the work to the fastest silicon present, and the .litertlm package makes it available almost instantly through memory mapping. As shown in Figure 2, every one of these attacks the ratio stated in Section 3.1, and because they attack different terms of that ratio their gains multiply rather than overlap. That compounding, rather than any single breakthrough, is why a multi-billion-parameter model answers instantly in a palm, off a battery.

KV cache sharing sits on a separate axis, shown at the foot of Figure 2: by accelerating prefill it shortens the wait before the first word appears, which improves perceived latency rather than the steady-state generation rate.

Figure 2. How each technique raises the on-device generation ceiling

Source: AI generated

3.6 Native multimodality: the model’s senses

Gemma 4 does not merely read and write text; it sees and hears [1]. A compact, high-throughput vision encoder from the MobileNet family interprets images and video efficiently enough for real-time on-device use, while an audio encoder derived from Google’s Universal Speech Model turns speech into tokens the model can reason over, enabling on-device transcription and translation. Because all of these arrive as a single unified stream of tokens into the same model, the personal agent can look at a photograph, listen to a question about it, and answer — locally, privately and without a network. For an assistant meant to stand in for a person’s attention, senses are not a luxury feature. They are the point.

3.7 The developer surface: how applications call the resident model

The arrangement this paper prototypes is now being productized at platform scale on both major operating systems, and the APIs behind it are narrower and more concrete than the marketing suggests.

In April 2026 Google opened an AICore Developer Preview built on Gemma 4, which it states will become the foundation for Gemini Nano 4 on flagship devices later in the year; the branding should be treated as preview-stage rather than a shipped consumer feature [2]. The preview exposes the elastic-inference idea behind MatFormer (Section 3.2.1) as a first-class API: an E2B fast tier and an E4B full tier, selected in code by setting a model preference on the generation configuration rather than by choosing a different model file. Google states that the fast tier runs roughly three times quicker than the full tier, and that this generation of Nano is up to four times faster and uses about 60 percent less battery than its Gemma-3-based predecessor — gains delivered by AICore routing inference to dedicated NPUs from Google, MediaTek and Qualcomm rather than falling back to the CPU.

Apple’s answer, announced at WWDC 2026, is a third generation of Apple Foundation Models: AFM 3 Core, a three-billion-parameter model that runs entirely on-device, and AFM 3 Core Advanced, a twenty-billion-parameter sparse model that activates only one to four billion parameters per prompt. Both run on the Apple Neural Engine, the fixed-function accelerator Apple has shipped since 2017. Where Google lets a developer pick a size, Apple lets a developer specialize one: the Foundation Models Adapter toolkit trains small rank-32 LoRA adapters offline on a Mac, which then ship inside the application and load alongside the base model [8] — the on-device analogue of the technique described in Section 3.3.

The calling conventions follow from this. On Android an application calls AICore directly through the GenerativeModel class, or prototypes faster through ML Kit’s Prompt API, a thin beta wrapper that adds tool calling and structured output. Firebase AI Logic goes further still, offering a single call that prefers on-device inference and falls back to the cloud automatically when the local model is unavailable or the task exceeds it — a pragmatic default for production applications that cannot assume every handset has AICore [13]. On iOS the equivalent entry point is the Swift FoundationModels framework: a LanguageModelSession wraps the on-device SystemLanguageModel, exposes a single respond(to:) call with streaming output, and supports guided generation — structured, typed output rather than raw text — through the @Generable attribute [7]. The same session can invoke developer-defined tools mid-response, the pattern the Model Context Protocol generalizes across the wider agent ecosystem [9].

In both cases the pattern matches the one this paper argues for throughout: the model is a system service, not a library each application bundles for itself, and the developer’s task shifts from hosting intelligence to deciding how much of it to use and when to hand off to the cloud.

4. Validation — results and evaluation

4.1 Methodology

The prototype was evaluated as an engineering artefact rather than against a fixed benchmark corpus, and the results below should be read with that scope in mind.

The system under test was the host application described in Section 3, running a Gemma 4 E4B checkpoint packaged in the .litertlm format and executed through LiteRT-LM’s accelerated backends. The hardware was a single unmodified consumer handset, a OnePlus 15, exercised through the Gemini Nano fast tier. The device was held in airplane mode throughout, so that no server was reachable at any point and every token was demonstrably produced locally.

The primary metric was sustained generation throughput in tokens per second during interactive use. The secondary metric was qualitative: whether a human reader perceives the response as instant. A baseline comparison was drawn against the same model served through the prototype’s earlier, non-delegated execution path, in order to isolate the contribution of the runtime and packaging layer from that of the model itself.

Reproducibility conditions are simple and weak in equal measure. The configuration is fully specified above and requires no network, but the measurements are interactive observations on one device rather than repeated runs under a fixed seed and a controlled thermal state. Section 6.1 treats this as a limitation of the work rather than an omission from the report.

4.2 Results

Sustained generation on the OnePlus 15 ran at roughly 40 tokens per second — fast enough that the model finishes a sentence before a reader finishes the one before it. Moving the model to the .litertlm package and executing it through LiteRT-LM’s accelerated backends produced a step change in responsiveness against the earlier execution path: the difference between a demonstration one tolerates and an assistant one would actually live with, running entirely offline.

The same threshold is being crossed publicly, and not only on Android. Public demonstrations have shown a Qwen model under two billion parameters generating at roughly 50 tokens per second on an iPhone 17 Pro’s Neural Engine, inside a third-party on-device inference application. Both figures are set out in Figure 3.

Figure 3. Observed on-device generation throughput, fully offline

Measurements taken on a single OnePlus 15 in airplane mode, September 2026; the iPhone figure is a third-party public demonstration reproduced for comparison.

Source: AI Generated

4.3 Commentary

Each observation is set out below with what it demonstrates about the architecture:

• OFFLINE-001 — airplane-mode generation. Roughly 40 tokens per second with no network reachable. This demonstrates that the privacy property claimed in Section 1.1 is structural rather than contractual: the data cannot leave, because there is nowhere for it to go.
• RUNTIME-002 — runtime and packaging migration. A step change in responsiveness from moving the identical model to .litertlm and LiteRT-LM’s delegated backends. This demonstrates that the binding constraint was the execution and packaging layer rather than the model, and that the compounding argument in Section 3.5 holds in practice.
• SHARED-003 — device-wide service. A single loaded Engine served multiple client applications through cheaply cloned Sessions, with one copy of the weights resident. This demonstrates the economic case for the model as a system service: the memory cost of intelligence is paid once per device, not once per application.
• CROSS-PLATFORM-004 — independent corroboration. Roughly 50 tokens per second for a sub-two-billion-parameter model on a competing platform’s neural accelerator. Neither figure is a controlled benchmark, but together they make the point that matters: a small enough model, quantized and routed through an NPU, reads as instant regardless of which lab trained it or which platform hosts it.

5. Enterprise implications

5.1 Privacy becomes a physical property rather than a policy promise

Regulated industries — banking, healthcare, the public sector — have largely been unable to route sensitive data through cloud language models, because no contractual assurance fully answers a data-residency rule. On-device inference changes the calculus rather than the paperwork. A private financial advisor, a health companion or a claims assistant can reason over deeply personal data that never leaves the handset, which unlocks a whole category of use cases that residency rules had placed off-limits. The compliance argument shifts from demonstrating that data was handled correctly elsewhere to demonstrating that it never went anywhere.

5.2 The cloud inference cost model inverts in the enterprise’s favour

Businesses paying per-token cloud fees at consumer scale carry a marginal cost that rises with engagement, a structure that penalizes precisely the products customers use most. Offloading suitable workloads onto the user’s own silicon collapses that marginal cost toward zero for everything that can run locally, and converts a variable operating expense into a fixed engineering one. The strategic move is not to eliminate cloud inference but to decide, workload by workload, which side of the boundary each belongs on, and to measure the cost shift that results.

5.3 Mobile strategy must become agent-first

If customers increasingly act through one resident agent rather than through a screen of applications, an enterprise’s mobile presence stops being an interface to be designed and becomes a set of tools to be exposed. Being discoverable, callable and trustworthy to the personal agent becomes the new equivalent of earning a place on the home screen. Organizations that remain wedded to app-and-UI thinking risk being quietly disintermediated by the agent layer that now sits between them and the customer, reachable only when a user asks for them by name.

5.4 The device dematerializes into a personal AI

Extrapolate the curve and the product category itself changes. When the defining value of a device is the intelligence resident on it, the purchase decision shifts from which handset to which personal AI; the hardware becomes the body and the model and its agentic abilities become the identity. Device makers begin behaving like AI companies, with differentiation moving away from megapixels and toward the quality, privacy and personality of the on-device intelligence, and toward the custom silicon that runs it fastest per watt. Some makers will build their own models and their own chips; others will organize around a model provider and a silicon house, much as today’s PC makers organize around a processor architecture they do not own. The store becomes a marketplace of skills and agents rather than applications, and the personal AI itself becomes portable: a data-owned profile that could in principle move from one hardware body to the next, the way a SIM once carried a number.

5.5 The resident agent is a new and concentrated security surface

A powerful resident agent with access to every application is a high-value target, and a single agent holding a person’s whole digital life concentrates value and risk in one place. Enterprises and device makers alike must design for on-device model integrity, for secure tool invocation, for resistance to prompt injection, and for safe, auditable agent permissions. Trust and safety become a decisive design constraint rather than a compliance footnote, because the blast radius of a compromised personal agent is the user’s entire device.

5.6 The pattern repeats across industries, with different points of leverage

These shifts land differently by sector. In telecom, value moves toward edge orchestration and the handoff between on-device and cloud inference. In semiconductors, NPUs and memory bandwidth become the competitive battleground, which follows directly from the ratio in Section 3.1. In retail, banking and healthcare alike, the winners will be whoever turns “agent-led” and “on-device” from a slogan into a shipped product first. The underlying pattern is constant: the layer that owns the user’s intent captures the value, and that layer is moving onto the device.

6. Limitations and future work

6.1 Current limitations

The validation reported in Section 4 is the clearest limitation. Throughput was observed interactively on a single handset in one thermal state, not measured across repeated runs, multiple devices, fixed seeds and controlled temperatures; the figures should be read as engineering observations that establish an order of magnitude, not as benchmarks. No quality evaluation accompanies them, so the paper does not quantify what, if anything, the quantized and elastically sized model gives up against its full-precision counterpart.

The prototype itself is proof-of-concept rather than production-grade. The device-wide service interface demonstrates the arrangement but has not been hardened against untrusted callers, and the security properties described in Section 5.5 — model integrity, sandboxed tool invocation, prompt-injection resistance and auditable permissions — are identified in this paper rather than implemented in it.

Beyond the prototype, four limits are structural. On-device models still trail the largest cloud models on the very hardest reasoning tasks, so the honest near-term architecture is hybrid: local by default, reaching for the cloud only for the heavy tail of problems that genuinely need it. Sustained local inference stresses the thermal and battery budgets of a handset, and managing that gracefully is a real engineering discipline rather than an afterthought. Fragmentation across differing NPUs, memory sizes and operating-system AI services complicates writing an agent once and running it everywhere, though runtimes such as LiteRT-LM exist precisely to soften this [6]. And the lifecycle problem is genuinely unsolved: updating, versioning and governing a model that lives on millions of devices is an operational discipline the industry is only beginning to build.

6.2 Future research directions

Future work priorities include the following:

• A controlled benchmark protocol for on-device generation — repeated runs across several handsets and chipsets, with fixed seeds, a defined prompt corpus and reported thermal state, so that throughput and quality can be compared rather than anecdotally described.
• Quality-versus-footprint characterization of elastic inference — measuring what Mix-n-Match configurations between E2B and E4B cost in task quality, so that the runtime can select a size from a policy rather than a guess [3].
• A hardened device-wide inference service — authenticated callers, per-application quotas, sandboxed tool invocation and prompt-injection defences, taking the arrangement in Section 3 from prototype to a component an enterprise could ship.
• On-device model lifecycle management — over-the-air model updates, version pinning, rollback and adapter distribution at fleet scale, addressing the operational gap named in Section 6.1.
• A measured hybrid routing policy — an empirical account of which workload classes genuinely require cloud inference, and what the cost and latency shift is when the rest are moved local.

7. Conclusion

For fifteen years the smartphone has been a portal to intelligence that lives in a data centre, and that arrangement has levied a standing cost in money, latency, privacy and dependence. This paper has shown that the arrangement is no longer necessary for a large class of work: a Gemma 4 model in the .litertlm format, executed through LiteRT-LM and shared across a device as a single native brain, answers instantly and offline on unmodified consumer hardware at roughly 40 tokens per second.

The most significant contribution is not the measurement but the account behind it — that every technique making this possible, from MatFormer elasticity and Per-Layer Embeddings to quantization, memory-mapped packaging and hardware delegation, is an attack on one memory-bandwidth ratio, and that because they attack different terms of it their gains compound. That framing turns a list of optimizations into a design method for anyone building on native intelligence.

What remains before the result can be applied in production is set out in Section 6: controlled benchmarking across devices, quality characterization of elastic inference, a hardened service boundary, and an answer to the model lifecycle problem. Those are engineering agendas rather than reasons to wait. The enterprises that begin re-architecting now — agent-first, privacy-native, hybrid by design — will meet their customers on the right side of the inversion.