I did not take a traditional path into AI. There was no Kaggle phase, no PhD in machine learning, no journey that started with importing TensorFlow in a Jupyter notebook. My path started with an oscilloscope probe on a PCB trace, reading firmware documentation at 2 AM, and writing code that ran on hardware with no operating system at all.
Today I work on on-device LLM inference: running large language models on mobile phones using LiteRT. And the thing I keep realizing is that almost everything I learned in embedded systems applies directly to this work, in ways that are not obvious unless you have lived in both worlds.
This is about the mental models that transfer, the gaps between disciplines, and why edge AI needs people who understand both the model and the metal.
The Career Arc Nobody Plans
My career does not look like a straight line. It looks like a series of right angles.
Embedded systems and firmware. I started writing code that talked directly to hardware: bare-metal C, no RTOS, no abstraction. You learn a particular discipline when your debugging tool is a logic analyzer because there is no printf. You learn to reason about what the hardware is doing at every cycle because there is nothing between you and the silicon.
My first real taste of that was in my second year of undergrad. For a campus quiz event I built a buzzer system on a Texas Instruments chip: several teams wired in, whoever pressed first fired an interrupt, and that team's number lit up on a seven-segment display. It worked flawlessly on a development board. Then I moved it to a PCB I had laid out myself - a big deal at that stage - and it simply stopped working. I lost a day or two to it before the cause surfaced: the dev board had quietly taken care of some pull-up/pull-down resistors for me, and I had not read the chip's datasheet closely enough to carry them over to my own board. There was no printf to fall back on. I found it by reading the spec line by line and tracing every path on the PCB, and I fixed it before the event - the buzzer ran all three days without a glitch. It was the first thing I had built that felt like a real product.
Android WiFi: the HAL layer. I moved into Android's WiFi stack, working in the Hardware Abstraction Layer. The HAL is the interface between Linux kernel drivers and the Android framework. In the WiFi case, the framework talks to the vendor chipset HAL for chip and interface management, and to WPA Supplicant - itself exposed through its own separate HAL interface - for authentication and connection management, all hidden behind a common HIDL/AIDL contract so the Android framework above does not need to know which Qualcomm, Broadcom, or MediaTek radio is installed. This was my first deep exposure to the pattern of hiding hardware specifics behind a stable API contract - the same pattern I would later recognize in LiteRT's delegates.
Silicon-level AI: FP8 optimization on the Sohu chip. Then I joined a silicon company and worked at the root level on FP8 quantization for the first Sohu chip. FP8, 8-bit floating point, is a data format designed specifically for deep learning inference and training. Unlike INT8 quantization, which maps floating-point values to 8-bit integers with a scale factor, FP8 retains a floating-point representation with a sign bit, exponent bits, and mantissa bits. The two common FP8 variants are E4M3 (4 exponent bits, 3 mantissa bits, better for inference because it has higher precision) and E5M2 (5 exponent bits, 2 mantissa bits, better for training because it has wider dynamic range). Working at the chip level meant understanding how transformer model operations, such as attention, feed-forward layers, and normalization, map onto actual silicon: the data paths, the accumulator widths, the memory bandwidth requirements. I saw a transformer not as a Python abstraction but as a pattern of multiply-accumulate operations flowing through physical hardware.
Robotics. I worked on a robotics team where real-time constraints were not aspirational but existential. If your control loop misses its deadline, the robot does not "respond slowly." It crashes into a wall.
Competitive apps benchmarking (my day job today). Now I work in benchmarking: measuring how apps perform against their competitors, on real hardware, under conditions a customer actually cares about. That means automating performance measurement, defining the benchmarking goals with the customer, and building out entire labs of devices and toolchains to run it all. I have written a lot of the tooling around that ecosystem myself, end-to-end platforms that span everything from selecting the right device and installing the apps to the dashboards a customer opens to see the results. Measuring honestly, and building the infrastructure to measure at all when none exists off the shelf, is the daily work. (If that sounds familiar from earlier in this series, it should - it is the same discipline behind how I benchmarked Redacto.)
On-device AI. With that background, I was able to run LLM inference on mobile phones. Along with my team, Edge Artists, I built Redacto at a hackathon, an on-device PII redaction app running Gemma on Snapdragon hardware (it later won the Qualcomm x Google LiteRT Developer Hackathon). Edge AI is the direction I want to move into full-time next. And through all of it, I keep having the same thought: I have seen this problem before.
Mental Models That Transfer
The reason embedded systems experience translates so well to edge AI is not because the technologies are the same. They are not. It is because the constraints are the same, and constraint-driven thinking is a transferable skill.
Memory Hierarchies
In embedded systems, you think constantly about where data lives. Is it in L1 cache or L2? Is it in SRAM or DRAM? How many cycles does a cache miss cost? You design data structures and access patterns around the memory hierarchy because the performance difference between a cache hit and a main-memory fetch can be 100x.
On-device AI has the exact same problem at a different scale. A 2.59 GB model needs to fit in the memory of a phone that has, say, 8 GB of RAM total, but the operating system, background apps, and the camera service are all competing for that RAM. The model weights need to stream through the compute units in a pattern that keeps the hardware fed. If your model's working set exceeds the NPU's local memory, you are constantly shuttling data back and forth across a memory bus, and throughput collapses.
Same constraint. Different scale. Same way of thinking about it.
Hardware Abstraction Layers
The Android WiFi HAL defines a contract: the framework gets an IWifiChip and calls createStaIface(), and the HAL implementation translates that into whatever vendor-specific sequence the underlying chipset requires. The framework does not know whether it is talking to a Qualcomm QCA6490 or a Broadcom BCM4389. It does not need to.
LiteRT's delegate pattern is the same architecture. The inference engine defines a contract: Prepare() the subgraph, Invoke() it, collect the output. The GPU delegate, the NPU delegate (via QNN or the NNAPI), and the XNNPACK CPU delegate each implement this contract differently, translating the same high-level graph operations into hardware-specific instruction sequences. The application code does not know whether the matrix multiplications are running on Adreno shader cores or Hexagon vector units.
When I first read the LiteRT delegate documentation, I did not see a new concept. I saw the HAL pattern with different nouns.
Dispatch Mechanisms
In embedded systems, interrupt dispatch is a core concept. A hardware event fires an interrupt. The interrupt controller routes it to the correct handler based on priority and type. You write vector tables, you manage priorities, you handle preemption.
LiteRT's operation dispatch works the same way. The framework has a graph of operations. Each operation gets dispatched to the appropriate hardware backend: NPU for supported ops, GPU for others, CPU as fallback. The partitioner decides which ops go where, based on what each backend supports and what the performance characteristics are. Operations that the NPU cannot handle get "fallen through" to the GPU or CPU, just like interrupts that a specialized handler cannot process get routed to a default handler.
The vocabulary is different. The mechanism is the same.
Real-Time Constraints
Robotics taught me to think in latency budgets. You have X milliseconds to read the sensor, run the algorithm, and command the actuator. If you exceed that budget, the system does not degrade gracefully: it fails.
On-device LLM inference has its own latency budgets. Time-to-first-token (TTFT) determines whether the user perceives the response as instant or laggy. On the Snapdragon 8 Elite running Gemma through our app, NPU TTFT was around 92ms and GPU TTFT around 366ms. (These come from a single directional benchmarking session on one Galaxy S25 Ultra, not a rigorous multi-run study, so treat them as ballpark rather than reproducible figures.) That is the difference between "the app feels responsive" and "the user wonders if it crashed." It is not a robotics control loop, but the thinking is identical: you have a time budget, the hardware determines whether you can meet it, and if you cannot, you need a different approach.
Debugging Without Visibility
In embedded firmware, you often cannot set a breakpoint. There is no stdout. You have a logic analyzer, maybe a JTAG probe, and you reconstruct what happened from signal traces and register dumps. You develop a skill for reasoning about system behavior from indirect evidence.
On-device AI debugging is remarkably similar. There is no interactive profiler attached to the NPU. You have logcat output, ADB shell commands, and timing measurements. When inference produces wrong results, you cannot step through the model's execution on the NPU the way you would step through code in a debugger. You reason from outputs, from timing anomalies, from the gap between expected and observed behavior. You look at the same kind of indirect evidence and build the same kind of mental models about what the hardware is doing.
What Embedded Engineers See That ML Engineers Miss
I have worked alongside ML engineers who are brilliant at model architecture, training pipelines, and evaluation. But when the model moves to a real device, there are things they tend not to see.
The hardware is not infinite. In the cloud, if you need more memory, you provision a bigger instance. On a phone, 8 GB is 8 GB. If your model plus the OS plus the foreground app exceed available RAM, the OS kills your process. There is no swap file that gracefully handles the overflow. Embedded engineers internalize this constraint from day one: you always know exactly how much memory you have and how much you are using.
Silent failures are worse than crashes. In embedded systems, a hard fault is visible. It triggers an exception handler. You can diagnose it. The dangerous failures are the ones where the system keeps running but produces wrong results: a corrupted sensor reading that looks plausible, a timing error that only manifests under load. On-device AI has the same problem. A model that outputs confidently wrong text is harder to catch than one that throws an exception. Quantization can introduce subtle accuracy degradation that does not trigger any error. The model keeps producing output. It is just slightly wrong.
The deployment environment is hostile. Phones overheat. Thermal throttling reduces clock speeds mid-inference. Background processes compete for resources. The OS can kill your process to reclaim memory. The DSP state may not clean up properly between inference calls. These are not edge cases: they are the normal operating environment. Embedded engineers are used to designing for hostile environments. ML engineers who have only worked in controlled cloud settings sometimes underestimate how many things can go wrong between "the model works on my laptop" and "the model works reliably on a user's phone."
What ML Engineers See That Embedded Engineers Miss
Fair is fair. The transfer is not one-directional. There are things ML engineers understand intuitively that embedded engineers have to learn.
Probabilistic systems behave differently than deterministic ones. Embedded engineers expect deterministic behavior: same input, same output, every time. LLMs are stochastic. The same prompt can produce different outputs. This is not a bug. The randomness (temperature, top-k sampling) is a feature that makes the output more natural. Learning to evaluate a system where "correct" is a distribution, not a single value, requires a genuine shift in thinking.
"Good enough" accuracy is a valid engineering target. In firmware, a CRC check either passes or fails. A packet is either delivered or lost. In ML, you are working with precision, recall, F1 scores, perplexity. A model that gets 94% accuracy might be perfectly acceptable for the use case. An embedded engineer's instinct is to chase the remaining 6%, but in ML, that last 6% might require 10x more compute for diminishing returns. Learning where to stop optimizing is a skill.
The model IS the program. This is the hardest shift. In embedded systems, you write explicit logic: if-else chains, state machines, lookup tables. The program does exactly what you coded. In LLM-based applications, the model's behavior is shaped by training data and prompts, not by explicit rules. Prompt engineering is programming, just in a domain where the "compiler" is a neural network and the "instruction set" is natural language. This requires a fundamentally different approach to system design.
Why Edge AI Needs People at the Intersection
The gap between "model works in Python on a cloud GPU" and "model runs reliably on a phone's NPU" is not primarily an ML problem. It is a hardware-software integration problem.
Getting a transformer model onto an NPU requires understanding quantization (how does FP8 vs INT8 affect this model's accuracy on this specific hardware?), memory layout (how do you tile the attention computation to fit in the NPU's local memory?), dispatch (which operations run on the NPU and which fall back to CPU, and what is the cost of that transition?), and system-level behavior (what happens when the OS throttles the NPU because the phone is warm?).
This is exactly the skill set of people who have worked at hardware-software boundaries. People who have written HALs, debugged firmware, optimized data paths through silicon, and designed systems under real-time constraints.
The edge AI space is growing. Models are getting smaller and more capable. Hardware is getting more specialized. The APIs and tooling are maturing. But the bottleneck is not the models or the hardware. The bottleneck is people who understand both the model and the metal: who can look at a transformer architecture and see the memory access patterns, who can look at an NPU spec sheet and understand what it means for model design.
If you come from embedded systems, from firmware, from silicon, you already have half the picture. The model side can be learned. The hardware intuition cannot be taught in a bootcamp.
And if you come from ML, consider looking down the stack. Understand the hardware your models run on. Learn what a cache miss costs, why memory bandwidth matters, what happens when your model's working set exceeds the accelerator's local memory. The best edge AI engineers I have worked with are the ones who can move fluidly between the model and the metal.
The intersection is where the interesting problems are.
Related in this series of "Edge AI from the Trenches"
- Why My LLM Runs 4x Faster on Hardware I Had Never Heard Of: the purpose-built silicon that embedded engineers will recognize as familiar territory
- FP32, INT4, and Everything Between - What I Learned About Precision on Mobile: the precision-compression trade-off at the heart of on-device deployment
- The Deployment Boundary: Why On-Device AI Breaks After Colab: the concrete deployment failures that require both ML and hardware expertise
Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.
Last updated: July 2026
20th of 23 posts in the "Edge AI from the Trenches" series
Top comments (0)