DEV Community

Cover image for I’m building a local AI desktop companion that sees your screen — and you can help shape it
southy404
southy404

Posted on

I’m building a local AI desktop companion that sees your screen — and you can help shape it

Most AI tools feel disconnected.

They don’t see your screen.
They don’t understand what you're doing.

So I built one that does.


Meet OpenBlob

OpenBlob desktop AI companion showing animated blob avatar, floating UI, and context-aware interaction on Windows desktop

An open-source, local-first desktop AI companion for Windows that doesn’t just respond — it lives on your desktop.

👉 GitHub: https://github.com/southy404/openblob

It can:

  • understand what app you’re using
  • analyze screenshots
  • help inside games, apps, and browsers
  • react visually with an animated companion
  • and yes… even play hide and seek with you

The problem with current AI assistants

Most tools today are:

  • cloud-dependent
  • context-blind
  • static
  • not fun to use

They don’t feel like part of your system.


🧠 It understands context

OpenBlob looks at:

  • active window
  • app name
  • window title

So if you’re in a game, it knows.
If you're debugging, it adapts.

This is where things start to feel different.


🖼 It can see your screen

You can take a screenshot and it will:

  • extract visible text
  • detect what you're looking at
  • generate a real search query
  • explain what's going on
Screenshot → OCR → context → reasoning → answer
Enter fullscreen mode Exit fullscreen mode

Still a bit rough — but already very usable.


🎮 It actually helps inside games

Instead of:

alt-tab → google → guess

You can:

  • screenshot
  • let it detect the game
  • get a real answer

This alone changes how you play.


🤖 Multi-model AI (local-first)

Runs via Ollama with:

  • text models
  • vision models
  • fallback system

No cloud required.


🎨 It feels alive

The companion:

  • has moods (idle, thinking, love, sleepy)
  • reacts to interaction
  • can be “petted”
  • dances when music is playing

Small details, big difference.


🎮 The weird part (my favorite)

Hide and Seek mode

You can literally say:

“let’s play hide and seek”

And it will:

  • hide somewhere on your screen
  • peek occasionally
  • wait until you find it

Sounds dumb.

Feels surprisingly real.


⚡ New UI (WIP)

  • CTRL + SPACE to open
  • floating companion
  • instant interaction

Inspired by tools like Raycast / Arc — but alive.

⚠️ still slightly buggy


🧪 Screenshot assistant (work in progress)

  • fast snipping
  • instant processing
  • contextual answers

Works — but not perfect yet.


Why open source?

Because this shouldn’t belong to one company.

This kind of system should be:

  • transparent
  • hackable
  • community-built

Philosophy

  • local-first
  • context > prompt
  • playful + useful
  • build in public

Current state

Early stage.

  • evolving fast
  • sometimes buggy
  • lots of experiments

If you want to join

This project is wide open.

You can:

  • contribute features
  • improve UI
  • experiment with AI
  • build plugins

👉 https://github.com/southy404/openblob


Final thought

I don’t think the future of AI is chat.

I think it’s something that:

lives with you, understands your environment, and evolves

That’s what I’m trying to build.

Top comments (1)

Collapse
 
benjamin_nguyen_8ca6ff360 profile image
Benjamin Nguyen

Great project! I