I Built Chunks: an On-Device AI That Sees My Mac Screen and Acts Without Stealing My Mouse

Rishideep Mandavilli — Tue, 03 Nov 2026 18:46:13 +0000

This is a submission for the MLH x DEV Writing Challenge

What I Built

Chunks is a native macOS menu-bar and notch companion. You talk to it, it looks at your screen, explains what's going on, and can act on your apps for you. And it does all of that without grabbing your cursor or sending your screen to a cloud server.

I built it because the "computer use" agents I tried annoyed me. They warp your physical mouse, so you can't touch your laptop while they work. They take 30 to 60 seconds per step. They ship full-screen screenshots off to someone else's backend. Meanwhile, normal AI chat sits in a browser tab and knows nothing about the app you're stuck in.

I wanted something that felt like a friend sitting next to me: it sees what I see, points at the right button, and does the boring clicks only when I say yes.

How it works

Accessibility-first actions, not fake mouse events. Chunks uses the macOS Accessibility API (AXUIElement) through a driver I call AXTDriver. It finds controls by meaning (app="Notes" role=button name="New Note"), climbs the accessibility tree to find something pressable, checks focus before typing, and fires AXUIElementPerformAction. My real mouse never moves. A smooth Bezier "pointer trail" shows you where it's acting instead.
Two brains. A fast cloud path uses Gemini Flash with my own key. Requests are hedged in parallel at 3.5s to cut tail latency, with "store": false. A fully offline path runs Gemma 4 (gemma4:e2b-it-qat and gemma4:e4b-it-qat) locally through Ollama on Apple Silicon. A full vision turn finishes in about 2.3 seconds with nothing leaving the laptop.
One coordinate system to rule them all. Retina scaling and the AppKit vs CoreGraphics origin flip caused my worst bugs. So the model always reasons on a 0–1000 grid, and a single CoordinateSpace layer maps that to device pixels, window coordinates and "Circle-to-Explain" crops.
Privacy and safety by design. There's zero backend. Keys live in the macOS Keychain. Memory is a 2 KB local markdown file, and a SecretSweeper scrubs API keys, JWTs and passwords before anything is written. A ConsentGate asks before any state-changing action, and a PolicyGate blocks risky commands. One rule I'm proud of: something Chunks infers from the screen can never overwrite something I said out loud.

Stack: Swift 6, AppKit, SwiftUI, ScreenCaptureKit, ApplicationServices (AXUIElement), AVFoundation, Speech.framework, Ollama, Gemini API.

It's backed by a dependency-free suite of 710+ unit tests and live OS probes (--probe-local, --probe-ax, --probe-semantic).

Demo

60-second demo: https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED_W06cSpv/view?usp=sharing

Code: https://github.com/Rishideep-Mandavilli/chunks

Partner Technologies

I entered the Gemma track, and Gemma is what makes the "private" part real. Running Gemma 4 2B and 4B quantized models through Ollama meant I could do screen understanding fully offline on a MacBook. A few things I learned:

Streaming /api/chat as NDJSON with think: false made the difference between "usable" and "waiting".
The 2B model is fast enough for quick "what is this?" questions. The 4B model is noticeably better at picking the right UI element when the screen is busy.
Keeping prompts tied to the normalized grid made small models far more reliable at pointing at things.

Gemini Flash stays as an optional fast cloud mode, but Chunks works end to end without it.

Hackathon Experience

I built Chunks for Hacktoberfest Hack Day × AI in Vizag on November 1, 2026, in the Gemma use-case track. Honestly, I didn't present it as well as I wanted, and Chunks didn't place. But I did win the special prize for my LinkedIn post about the day, which was a nice surprise.

What I'll remember most is the energy of a room full of people building with open models on a Sunday, and the moment Chunks typed into Notes while my hands were off the trackpad. Next time I'll practise the demo as hard as I practised the code.

What's next: a signed release so anyone can try it, and better small-model accuracy on crowded screens.