This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
Wisp — A Quiet Desktop Spirit for Linux
What I Built
Most AI assistants live in a browser tab or a separate application. You open them, explain what you're doing, provide context, and wait for an answer.
I wanted to explore a different idea.
What if AI didn't need to be opened every time you needed it? What if it simply lived on your desktop?
Meet Wisp, a floating AI desktop companion for Linux that understands your workspace, observes what's happening on your screen, and provides contextual assistance when you need it.
Inspired by the theme Ghost in the Machine, Wisp is designed to feel less like another application and more like a quiet presence inside your operating system.
Wisp appears as a transparent, floating ghost overlay that stays out of your way. You can interact with it, change its appearance, switch between friendly and after-hours modes, and control its visual effects.
But the interesting part is what happens behind that little ghost.
How Wisp works
Imagine you're working on a programming assignment, debugging an error, reading documentation, or navigating an unfamiliar application.
Instead of taking screenshots, switching tabs, and explaining your entire situation to an AI assistant, you can:
- Bring Wisp to your desktop. It stays available as a lightweight floating companion.
- Start a screen-sharing session. You explicitly choose when Wisp is allowed to see your screen.
- Let Wisp understand the context. A locally running vision model analyzes screen content and generates observations.
- Continue working. Wisp monitors active-window changes and periodically refreshes its understanding of your workspace.
- Receive contextual assistance. When something relevant needs attention, Wisp can communicate through its floating overlay. You can also ask questions about your screen.
The goal is to reduce the friction between encountering a problem and getting useful assistance.
Wisp isn't designed to constantly interrupt you or demand your attention. It is designed to be present when needed and invisible when not.
What makes Wisp different?
- A native desktop presence: A transparent floating ghost overlay with click-through behavior, visual effects, and interaction modes.
- Contextual awareness: Tracks active-window changes and maintains recent observations instead of treating every interaction as an isolated prompt.
- Local vision intelligence: Uses Ollama and a vision-capable model to understand screen content.
- User-controlled observation: Screen capture only starts through an explicitly approved sharing session.
- Persistent background workflows: Temporal manages observation cycles, retries, pending questions, cooldowns, and workflow history.
- Linux integration: Supports screen capture through desktop portals on Wayland and primary-display capture on X11.
The central idea is simple: an AI companion that understands what you're doing without making you explain everything from scratch.
Demo
Explore the project: Wisp on GitHub
The repository README includes a screenshot of Wisp's screen-sharing interface, along with setup instructions and architecture documentation.
Code
GitHub Repository: https://github.com/GhananilShirpurkar/wisp
Wisp is released under the MIT License.
The repository contains the complete application source, architecture documentation, setup guide, tests, and CI workflows.
How I Built It
Wisp is built around one important architectural decision: AI inference happens locally, not through a cloud AI API.
Rather than building a simple chatbot interface around an API call, I built a system that coordinates desktop interaction, screen capture, vision inference, and background workflows.
Technology Stack
| Technology | Responsibility |
|---|---|
| Electron | Desktop application and floating ghost overlay |
| Node.js | Application and worker runtime |
| TypeScript | Core application and workflow logic |
| Ollama | Local AI inference |
| Vision-capable model | Screen understanding and visual analysis |
| Temporal | Durable workflow orchestration |
| Linux desktop portals and PipeWire | Wayland screen capture |
| X11 utilities | Display capture and window metadata |
System Architecture
Wisp separates four major responsibilities:
1. Desktop Interface
The Electron application manages the floating ghost, user interactions, screen-sharing controls, visual effects, and communication with the background worker.
2. Screen Capture
When the user explicitly enables screen sharing, Wisp captures screen frames through the appropriate Linux integration.
On Wayland, it uses the desktop portal and PipeWire. On X11, it captures the primary display.
3. Local AI Intelligence
Captured frames are passed to a local worker through an authenticated loopback connection.
The worker sends the images to Ollama, where a vision-capable model analyzes the screen and produces structured observations.
No cloud AI inference service is required.
4. Workflow Orchestration
Temporal manages the observation lifecycle and background execution.
It handles retries, pending questions, cooldowns, rolling observations, and workflow history rollover. This allows Wisp to maintain context across repeated observation cycles rather than relying on isolated requests.
The Observation Loop
Wisp uses a recurring observation cycle:
- Checks active-window metadata every five seconds.
- Triggers vision analysis when the active window changes, when 45 seconds have elapsed, or when a question is pending.
- Maintains a rolling 30-minute window of current observations.
- Uses workflow cooldowns to avoid unnecessary interruptions.
- Rolls workflow history forward periodically to keep long-running execution manageable.
This is what turns Wisp from a screenshot-to-prompt utility into a context-aware desktop system.
Why Does Open Innovation Matter?
Wisp's core idea would lose much of its value if it depended entirely on a proprietary cloud AI service.
The product is built around the idea that your desktop context should remain under your control.
Open-source AI and local inference make that possible.
With Ollama, Wisp can use locally available vision models without being tied to a single proprietary provider. Developers can experiment with different models, inspect the inference pipeline, modify the architecture, and build on top of the project.
There is also a deeper reason.
Screen content can contain source code, private messages, documents, personal information, and sensitive work. Sending all of that to an external AI service introduces a dependency that conflicts with the product's privacy goals.
Wisp processes screen images locally. On Linux, screenshot bytes remain in memory during processing and are not stored in Temporal workflow history. Window titles, questions, and model observations are retained in local workflow history.
The project also makes the boundaries explicit: users control screen-sharing sessions, and stopping a session prevents future captures, although an inference operation already in progress may finish.
Open innovation makes it possible to build an AI experience where the user has more control over the model, the infrastructure, and their data.
Your AI should understand your workspace without requiring your workspace to leave your machine.
The Spirit of It
I didn't want to build another assistant that occupies a window, waits for a prompt, and disappears when you close it.
I wanted to explore a different relationship between humans and AI.
Something that feels like it belongs on your desktop.
Imagine working late at night, switching between your editor, terminal, and documentation. A little ghost quietly floats in the corner. It doesn't demand attention. It doesn't interrupt every action. But when you need help understanding something on your screen, it's already there.
That is the experience I wanted to build.
Wisp takes the idea of a ghost in the machine quite literally: a small, intelligent presence living inside your operating system, observing only when you allow it, understanding through local AI, and helping without taking over your workspace.
Not another chatbot. Not another browser tab. A quiet desktop spirit.
Top comments (0)