Not because it's bad. Because it has no idea what you were doing five minutes ago.
I'd have five tabs open, half a research trail in my head, and the second I opened a chat with Claude or ChatGPT, I was starting from zero again. Re-explaining myself to my own AI tools got old fast. So I built something to fix it.
Clippy Vision is a fully local screen-memory assistant. It passively watches your windows, clipboard, and screenshots, and lets you ask questions about your own activity, "what was I reading about X earlier," and it pulls the real answer from your history instead of asking you for context.
Everything runs on your machine. No cloud calls, nothing leaves your device.
The problem with the naive approach
The obvious first design is: screenshot everything, run every frame through a vision model, store the output. That melts your GPU fast and burns inference on a lot of frames that haven't meaningfully changed since the last one.
So the actual pipeline needed to be smarter about when to spend compute:
3-tier classification pipeline: Frames go through rules first, then heuristics, and only escalate to the LLM/vision layer if they actually need it. Most frames never touch the vision model at all. In my own benchmarking (900 screenshots over 3 days of real activity), this cut LLM inference calls by 37%.
Perceptual hashing + union-find deduplication: Near-identical frames (you didn't switch windows, nothing on screen actually changed) get caught and skipped before they'd otherwise get reprocessed. This gave another 23% reduction in vision model workload on top of the tiering. The union-find approach for grouping duplicate frames was actually a suggestion I got while researching this part, not something I'd have reached for on my own, but it turned out to be a clean fit for the problem.
Progressive memory summarization: Raw activity history balloons fast if you just keep everything forever. Summarization keeps the stored history usable without needing to store every raw frame indefinitely.
A ReAct agent with SQL-generating tools on top: This is the part that actually answers your questions. Instead of just dumping raw history at an LLM, there's an agent that can generate and run SQL queries against the stored activity to pull exactly what's relevant to what you asked.
Tech Stack
Ollama for local inference
Qwen3-VL-4B for the vision layer
SQLite for storage
MCP server integration
Electron for the desktop app
Where it's at now
It's been my daily driver for a while now, it genuinely opens before Claude or ChatGPT most days for me at this point. I shipped it publicly a few days ago, it's open source, and it's already gotten its first outside contributor (thank you, if you're reading this).
There's still a real backlog of open problems if you want to dig in: markdown isn't rendered properly in the chat UI yet, there's no delete button for conversations, prefetched context can overwhelm answers in some cases, and Linux/Mac support isn't there yet (Windows-only for now).
If you're curious, poke around, open an issue, or check out existing ones. Repo's here: https://github.com/protocorn/clippy-vision
Curious what other people are doing here
If anyone's built something in a similar space (local vision pipelines, continuous screen monitoring, agent-based retrieval over your own activity), I'd genuinely like to compare notes on how you handled the compute/freshness tradeoff. It's the part of this I spent the most time on and I'm sure there are better approaches I haven't tried yet.
Top comments (0)