I Open Sourced Caddy: An Invisible Desktop AI Assistant Built on Backboard.io
It’s called Caddy:
an invisible desktop AI assistant that lives on your machine, watches context, transcribes audio locally for speed, and lets you chat against what’s happening on your screen.
GitHub:
https://github.com/chrisk60331/caddy
What it is
Caddy is a desktop overlay built on Backboard.io.
It can:
- capture screenshots
- run OCR on what’s on screen
- transcribe live audio locally
- send context into Backboard
- let you chat with that context using 17,000+ models
- keep the thread alive with persistent memory
So instead of copy-pasting between tabs like an animal, you just ask questions in context.
Why I built it
A lot of “desktop AI” tools still feel slow, fake, or weirdly disconnected from the actual workflow.
I wanted something that felt:
- fast
- local where it matters
- model-flexible
- memory-native
- actually useful while you work
So I forked Cluely, added local transcription, wired it into Backboard’s memory, and made it easy to use as a real-time desktop assistant.
That changed the feel of the product immediately.
Less lag.
More context.
Way better experience.
Why local transcription matters
If audio has to take a field trip before becoming usable context, the experience dies.
Caddy uses local transcription so it stays quick and responsive.
That means better support for:
- meetings
- calls
- live research
- screen-based workflows
- follow-up questions while context is still fresh
Fast input makes the assistant actually feel intelligent.
Why Backboard.io
Backboard gives this thing real leverage.
Caddy uses Backboard for:
- LLM routing
- memory
- contextual chat
- access to 17,000+ models
That means you’re not trapped in one provider or one brittle workflow.
You get a desktop-native interface on top of a massive model layer with memory baked in.
That combo is nasty.
Stack
A quick look under the hood:
- Electron for the invisible desktop overlay
- React frontend
- Flask backend
- Local Whisper for transcription
- Tesseract OCR
- Backboard SDK for LLMs + memory
Core features
- Invisible overlay — translucent, always-on-top, low-friction
- Screenshot analysis — capture anything on screen and get answers fast
- Audio intelligence — live transcription and analysis
- Contextual chat — ask follow-ups with memory intact
- Model selector — switch across providers via Backboard
- Screen watch — periodic OCR + analysis of what’s on screen
- Cross-platform — macOS, Windows, Linux
Who it’s for
Caddy is for people who live on their computers and want AI that works in the flow of actual work:
- developers
- researchers
- operators
- founders
- power users
- anyone tired of tab-hopping into prompt boxes all day
Why I open sourced it
Because desktop AI is still early, and a lot of the interesting stuff should be inspectable, hackable, and remixable.
Also because open source is still the cleanest way to pressure the market.
If someone wants to build on this, improve it, fork it, or weaponize the idea in a better direction — good.
That’s the point.
Try it
Repo:
https://github.com/chrisk60331/caddy
If you’re building with Backboard.io, or just want a fast open source desktop AI assistant with local transcription and memory, give it a spin.
If you improve it, I’d love to see where you take it.
Top comments (0)