This is a submission for the Built with Google Gemini: Writing Challenge
What I Built with Google Gemini
What if AI didn't live inside a browser window, but felt like a native, context-aware extension of your operating system?
That's the question behind Pointer, an agentic desktop assistant that reimagines how you interact with AI. Instead of copy-pasting code and context back and forth, Pointer lives seamlessly in your OS, triggered via global hotkeys. You press a shortcut, and it instantly captures your active context, reading your text selection or analyzing a screenshot of your screen. Whether you want it to intelligently type out code inline, answer questions via an overlay window at your cursor, or perform system tasks, Pointer acts as a digital co-pilot sitting right beside you.
At the core of Pointer is a Google Gemini-powered orchestrator built using the Google Agent Development Kit (ADK). Gemini processes the high-level intent, reasons about your screen state via multimodal vision capabilities, and routes tasks to specialized sub-agents. This was a deliberate choice: Gemini’s multimodal vision and text processing make it exceptionally well-suited for understanding screen context on the fly and taking meaningful action.
One of the things we're most proud of is the accessibility of the system. Pointer is free to get started; users simply bring their own Gemini API key (or another provider's key). No subscription wall, no lock-in. This lowers the barrier for developers, students, and curious builders who want to plug AI into their daily workflows without committing upfront.
Pointer won recognition at multiple hackathons, including an MLH-sponsored event, validating that this vision of a new AI interface resonates well beyond our own excitement.
Demo
📋 Join the waitlist at getpointer.tech
What I Learned
Building Pointer was as much a lesson in systems design as it was in AI engineering.
The tech stack — Rust (Tauri), TypeScript (React), and Python — was intentional but demanding. Python handles the heavy lifting for OS interactions through libraries like pynput and PyObjC: managing global hotkeys, capturing screenshots, and intelligently simulating inline typing over accessibility APIs. Tauri and Rust govern the lightweight native windowing, letting us spawn transparent React-driven overlays instantly exactly where your cursor is. TypeScript handles the snappy UI interactions. Getting these runtimes to communicate cleanly, passing high-res screenshots and state over WebSockets without bottlenecks, was genuinely hard.
The biggest unexpected lesson was around OS-level permissions and packaging. Getting a Python backend packaged via PyInstaller into a Tauri macOS application to listen to system-wide keyboard events and simulate typing requires navigating Accessibility permissions, entitlements, and sandboxing rules that aren't always well-documented. We spent a surprising amount of time just on the permission handshake before a single pixel moved.
On the AI side, thinking in terms of agentic loops and tool calling, parsing the user's intent to route out to specific integrations (like our Google Calendar, SQL database endpoints, or Email generators) rather than single-shot prompts, was a significant mental shift. The Google ADK framework helped structure reliable routing between our toolsets.
Google Gemini Feedback
What worked well: Gemini's vision + language multimodal performance was the backbone of everything. Being able to pass a screenshot on a hotkey press and get a structured, actionable response about the screen's content was smooth and surprisingly reliable out of the box. The straightforward API key flow also meant we could onboard testers quickly without any complex billing setup.
Where we needed more support: Latency in the agentic loop was occasionally noticeable. When Pointer captures the screen and routes through an orchestrator agent to decide on an action, even a few hundred milliseconds of model response time compounds into a feeling of sluggishness. Streaming partial decisions or having a lighter "fast path" model for simpler routing actions would be a huge improvement. Better documentation around ADK telemetry and rate limits for agentic use cases (multiple rapid tool calls) would also help builders working in this space.
Overall, Gemini felt like the right engine for Pointer, powerful enough to understand context visually, flexible enough to fit into our polyglot architecture, and accessible enough that anyone can try it out.
Top comments (0)