I built the assistant I clicked on as a kid. It haunts me now!

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

I was five or six the first time a piece of software looked back at me.

It lived on the family PC in our dining room. I was supposed to be learning CorelDRAW, but when I got bored, I played around.

Its name was Clippy.

Clippy sat on its little mat, moving its eyes around while I worked. I clicked it on purpose just to make it dance. My mum liked it too, right up until she had serious work to do. Then she'd push it aside.

That sums up the original. It was charming, and it knew nothing. It could tell you "it looks like you're writing a letter", but it couldn't actually see what you were doing.

So I built Paperclip: a doodled bedsheet ghost that lives in the corner of my Mac and actually watches.

• Every 5 seconds, it checks which window I'm in.
• When the window changes, a vision model running on my laptop looks at the screen and works out what I'm doing.
• If I'm stuck on the same error for 20 minutes, it offers help. Now and then, it suggests something about whatever's on screen.
• After midnight, it gets possessed: red ink, limp arms, one pupil drifting on its own, and lines like Where are you going to?.
• Hover over it, and it runs from your cursor. In possessed mode, it stops running and follows you.
• You can't kill it. Force-quit it, and it comes back, remembering what it was watching.

The name is the joke. The ghost is my own drawing, not a paperclip. It's the ghost of the old assistant, back from the dead with eyes this time.

Demo

The video below shows all of Paperclip's effects

Code

Bennykillua / paperclip-ghost

A ghost that haunts your Mac. Watches your screen with a local vision model (Gemma via Ollama), comes back when you kill it (Temporal), and nothing leaves your machine.

Paperclip

A desktop ghost for macOS that watches what you're doing and butts in. It runs a local vision model through Ollama, so nothing about your screen leaves your machine.

What you need

• A Mac
• Node.js 20 or newer
• Homebrew (setup offers to install the rest with it)

Setup installs the Temporal CLI and Ollama if they're missing, with your permission.

Install and use

Four steps. Setup does the rest.

git clone https://github.com/Bennykillua/paperclip-ghost.git
cd paperclip-ghost
npm run setup              # checks the tools, asks for a model, builds, starts the agents
npm run ghost:test -- all  # see every effect once

Settings live in .env. .env.example lists them with notes. Setup writes VISION_MODEL for you, and Sentry stays off until you add SENTRY_DSN.

The command-line tools are listed in Brewfile. brew bundle installs them.

Then give macOS the two permissions in Permissions and run npm run ghost:start…

View on GitHub

How I Built It

The stack: Gemma 4 (E2B) through Ollama for vision, Temporal to keep the ghost alive, Sentry to trace each glance, and Electron for the window. It's all TypeScript, and it all runs locally.

The fastest model wasn't the right one

I tested four open vision models on my M5 Pro (24 GB). My first test was accidentally rigged. I sent the same screenshot three times, so the later runs did little, and each model produced a different amount of text. I threw that test out and reran it fairly: a fresh screenshot each run and the same output limit for every model.

Model	Avg time	Tokens/s
moondream	0.50s	172
gemma4:e2b	2.69s	77
qwen2.5vl:3b	4.73s	57

Moondream won on speed and lost on everything else. Its description of my screen was "a code editor window with a black background". A ghost can't haunt you with that. Gemma read actual file names and what I was doing, so Gemma won.

The biggest speed win came from asking for less. Instead of a paragraph, the model now returns a small JSON object: {app, activity, detail, notable}. Output dropped from about 131 tokens to about 54. Across 187 real calls, the vision step now takes 1.09s at the median and 1.26s at p95.

One glance takes about 1.5 seconds and costs $0

Step	Typical time
Read the active window	190–325ms
Screenshot	170ms
Resize to 1280px	~50ms
Gemma looks at it	~1,090ms

Even so, it doesn't look every 5 seconds. Reading the window title is cheap and exact, so it does that on every tick. It only calls the model when the window changes, or every 45 seconds otherwise.

A ghost you can't kill

Each step of a glance (read the window, take a screenshot, call the model, decide, speak) is a Temporal activity with its own timeout and retries. The last 30 minutes of what the ghost has seen live in the workflow's state, so they survive a crash. Every hour the workflow starts afresh and carries its memory over, so its history doesn't grow forever. Then launchd keeps all three parts running: the Temporal server, the worker, and the ghost window.

const { describeScreen } = proxyActivities<typeof activities>({
  startToCloseTimeout: '30s',
  retry: { maximumAttempts: 4, initialInterval: '1s', backoffCoefficient: 2 },
});

I tested it:

• kill -9 in the middle of a vision call. launchd restarted the worker. Temporal retried the same call as attempt 2, which finished in 567ms. Memory went from 68 to 73 observations, with the oldest one unchanged.
• Restarting the Temporal server. 238 observations before, 238 after.
• Pointing it at a dead Ollama. Every call made exactly 4 attempts, then gave up cleanly.

After a reinstall, the worker started before the Temporal server was ready and crashed. launchd brought it back, and it carried on from the same memory. Nobody was watching. That's the point.

Watching the watcher with Sentry

Each glance is one invoke_agent span in Sentry, with every step nested under it. The vision call is a gen_ai.chat span with the model name, input and output tokens (on average 499 in and 54 out), and a cost of 0.

The scariest thing I found all week wasn't the ghost. Sentry's SDK records AI inputs and outputs by default. For most apps, that's handy for debugging. For this one, it would have sent a description of my screen to a server every few seconds, which is exactly what I built Paperclip not to do. So it's switched off, and everything else gets stripped before sending:

Sentry.init({
  dsn,
  defaultIntegrations: false,
  dataCollection: { genAI: { inputs: false, outputs: false } },
  maxBreadcrumbs: 0,
  beforeSend: (event) => scrub(event),
  beforeSendTransaction: (event) => scrub(event),
});

Sentry sees how long the ghost took to look, and nothing about what it saw.

Why Does Open Innovation Matter?

Honestly, an AI that watches my screen makes me uneasy. It also excites me. I'm only comfortable running it because it never leaves my laptop.

The closed version of this project streams your screen live to someone else's server. The open version is:

• Private: the screenshots, window titles, and model replies stay on my Mac. Even the telemetry is timings only.
• Free to run: each glance costs $0, however many times it looks.
• Swappable: I tried four models in one evening by changing one line. If a better small vision model comes out next month, Paperclip gets better that day.
• Offline: no internet needed. I even bundled the font so the window never calls out.
• Unkillable: Clippy disappeared because someone else decided it should. This one lives in a repo anyone can fork.

The Spirit of It

The first time it spoke to me on its own, it was a little scary, and I liked it. That's roughly how I felt about Clippy at six.

When I showed it to a friend, they laughed and said it reminded them of Clippy, and that the jokes were funny. That was the whole goal.

My mum used to push Clippy out of the way when she had real work to do. You can push this one away too. It just comes back.

Prize Categories

• Best Use of Gemma: Gemma 4 (E2B) is the eyes, picked over three other open models on speed and accuracy.
• Best Use of Temporal: Every glance is a durable workflow that survives kill -9, server restarts, and a dead model server.
• Best Use of Sentry Agent Tracing: invoke_agent and gen_ai.chat spans with tokens, latency, and $0 cost, and no content.

I built this solo