OAK: The "Linux Kernel" for AI Agents, Running 100% On-Device

Anyone shipping AI into the enterprise hits the same wall: the customer needs data to stay on-prem, zero latency, zero leakage — but most agents run in the cloud.

OAK (Open Agent Kernel, 474 stars, Apache-2.0, C++) attacks that from the bottom. Its pitch: the Linux kernel for AI agents — build agents that run 100% on-device.

The problem it attacks

On-device agents are the last mile for regulated industries — government, finance, healthcare — where data leaving the device is a hard red line. But running agents on-device fights compute, power, and latency. OAK wants to be the substrate that makes it tractable:

• No cloud dependency — the agent runs fully local, data never leaves the device;
• No network latency — local inference, no round-trip to a server;
• No data leakage — sensitive data is never uploaded.

The hard number: sub-100ms inference latency on Qualcomm NPU — a strong figure for on-device agents.

Three signals it's worth watching

• It targets the hardest pain in edge AI. "Data stays on-prem" is a hard requirement, not a preference, in regulated sectors. Whoever makes on-device agents run well owns the "last mile" of AI deployment.
• The "kernel" positioning is smart. Plenty build agent frameworks (orchestration, tool-calling) and models; almost nobody builds the on-device runtime kernel. OAK claims that empty slot — like Linux, doing the low-level, infrastructure-grade, unglamorous work others skip.
• Pure C++, pure open source. Apache-2.0, no commercial cloud lock-in — the critical property for enterprises deploying in Xinchuang / domestic-NPU environments.

The honest caveat

It's very early — Alpha. 474 stars, 13 forks, and the README says plainly: core kernel works, APIs are unstable, contributions welcome. That means:

• Don't expect a production-ready kernel today;
• It targets CPU + Qualcomm NPU, not all hardware;
• APIs will shift, so anything you integrate now may need revisiting.

It's for teams researching on-device agent architecture or doing edge-AI pre-study, not a turnkey tool.

I've localized the README and core docs to Chinese: https://github.com/yangshun2005/MasterAgent-cn

If you find this project useful, a star on the original repo supports the author's ongoing maintenance.