DEV Community: Wenwei Lin

Pip: a desktop pet that notices you, not just your words

Wenwei Lin — Wed, 11 Nov 2026 04:24:16 +0000

This is a submission for the Presage SmartSpectra SDK Challenge.

What I Built

Pip is a little octopus who lives in a tide pool in the corner of my desktop. I can talk to it, and it talks back.

Most AI on my desktop works for me, and it only knows what I type or say. Pip is the other half. It notices me: whether I'm smiling or frowning, how often I blink, roughly how fast my heart and breath are going. When I told it "no" with a little smile, it answered: "Funny thing is, you're smiling a little while you say that."

What Pip does:

Reacts to your face, even when you're quiet. Smile and hearts pop up. Look surprised and it jumps. Frown and it frowns back. Nod and it nods. Blink and it blinks back. Stare at it long enough and it gets shy.
Talks with you. You speak normally; Pip listens, thinks for a moment, and answers out loud with its mouth moving. Every reply starts with a small action (happy, shy, nod, comfort…) that Pip acts out.
Hears more than words. Along with what you said, Pip's brain gets a short note of what the camera saw while you said it: expression, blink count, heart and breathing rate, and whether they went up or down.
Breathes with you. The water in its pool rises and falls with your chest, and it glows faintly with each heartbeat.

It's a pet, not a tool. It can't open files, read the page I'm on, or browse, and that's on purpose. Agents that read your screen and use your computer already exist, and I didn't want to build yet another harness that I'd never use. I built the one piece I hadn't seen: an agent that can read the person in front of it. The idea is that this piece could later plug into other desktop pets and agents, rather than living only inside Pip. So this is a small demo of that one ability.

The octopus is just one look. The sensing doesn't care what the pet is: a fish, a cat, a little robot, whatever you like. Any of them gets the same new senses, your expressions, your heartbeat, your breathing, and can respond to them in its own way. What I'm really adding is more ways for desktop AI to perceive the person; Pip is one way to show it.

Demo

In the video I'm reading DEV. Pip can't see the page; it only sees me. I react to the posts, and Pip reacts to me. A few things it said, unscripted:

"Hello hello! You look a bit surprised to see me."
"I'm good, just bobbing about in my pool. You're blinking a lot though."
"Funny thing is, you're smiling a little while you say that."
"I can chat, keep you company, and sometimes notice things like a smile or a frown. I can't use your computer or look stuff up yet though."

It isn't perfect: a couple of times it didn't catch what I said, and once it told me my message got cut off. Heartbeat and breathing are there (the glow and the water) but subtle on screen.

Code

wenwei-lin / tidepal

Tidepal

Pip is a desktop pet that notices you, not just your words: a little octopus who lives in a tide pool on your desktop. You can talk to it.

Voice assistants only hear your words. Pip also sees how you look while you say them: it uses the SmartSpectra SDK to read your face, blinks, breathing and heartbeat through an ordinary webcam, and passes that along with what you said. So when you say "I'm fine" with a frown, Pip can notice.

Talk — speak normally. Pip listens (the ring around the pool follows your voice), thinks, and answers out loud, moving its mouth.
Face — Pip blinks when you blink, blushes when you smile, jumps when you look surprised, nods along when you nod, and gets shy if you stare.
Breath — the water in Pip's pool rises and falls with your chest; Pip glows with your heartbeat.
…

View on GitHub

Electron app. Setup is in the README: npm install, put your keys in .env, npm start.

How It Responds to Human Signals

The SmartSpectra Node SDK runs inside the Electron renderer and reads the webcam on the device. I request the breathing, cardio and face metric groups and decode the protobuf stream.

Signal	What Pip does
Expressions (happy, surprise, sad, angry)	Mirrors you in real time; tells the chat model how you looked while speaking
Blinking	Blinks back; blink count goes to the chat model
Talking	Tilts its head and listens
Face landmarks	Eyes follow you; nod and shake detection; the "stare" reaction
Breathing trace and rate	The water level; rate and trend go to the chat model
Pulse trace and rate	The heartbeat glow; rate and trend go to the chat model (stable readings only)

The conversation loop:

camera ─► SmartSpectra (on device) ─► face, blinks, heart, breathing ─┐
                                                                      ├─► DeepSeek V4.1 Flash ─► reply + action tag
mic ─► cut at pauses ─► speech-to-text (SenseVoice) ─► your words ────┘              │
                                                       Pip acts out the tag, macOS speaks the reply

Things I learned on the way:

The chest matters. Breathing needs your shoulders and chest in view, which fights with how close we sit to a laptop. Expressions and pulse come through fine from the face alone, so Pip leans on those and only nags about the chest when it really needs breathing.
Raw breathing is jumpy. Drawn as-is, the water jerked up and down. I remove the slow drift, let the size of the swing adapt over a few seconds, and move the water with a soft spring. Now it glides.
"Face present" should come from expressions. Landmarks sometimes stop while the SDK waits for a better view; expression scores keep coming, so I count those as "I can see you."
Only stable numbers. Heart rate jumps around for the first seconds. Pip shows "measuring" until the SDK marks the reading stable, and only stable readings reach the chat model.

Prize Categories

Most Creative or Delightful Use.

Consent and Data Handling

Consent. On first launch Pip explains, in plain words, what the camera and microphone are for, and asks. "Just looking" never turns either on. A button turns the camera off any time. A small mirror shows exactly what the camera sees.
Video stays on the computer. The SmartSpectra SDK processes frames locally. Nothing is recorded, saved or uploaded. The SDK contacts Presage only to check the key and download its models.
SDK telemetry is off (enableTelemetry: false).
Voice. When you pause, the piece you just said is sent to a speech-to-text service (SiliconFlow SenseVoice) and turned into text. Audio is not saved. The mic is ignored while Pip is thinking or speaking.
AI service. Your words, plus a one-line description of what the camera saw (for example "face while speaking: frowning; heart rate about 81/min, faster than earlier"), go to DeepSeek V4.1 Flash to write Pip's reply. Nothing else is sent.
Retention. The conversation lives in memory and is gone when you quit. Only your consent choice and language are saved locally.
Not medical. Pip is told never to diagnose or interpret your numbers, and to say it's just a little octopus if you ask about health.

What's next

Pip is a pet on purpose. But the same "eyes for the person" could sit next to other agents:

Coding agents. Codex already has desktop pets: little companions that float over your windows and react to what the agent is doing, "thinking" or "task done". They look at the work. A pet with Pip's eyes could also look at the person: notice you frowning at a diff, or grinning when the tests pass, and respond to that. Coding agents like Codex or Claude Code already have the computer and the web; add how you're doing, and an agent could slow down and explain more when you look lost. I don't know of anyone doing this yet, and it seems like a fun interaction to try.
Speaking practice. When you practise a language or a talk, the moments you frown or hesitate are the parts you haven't figured out yet. Pip could mark them and go over them with you afterwards.
Coaching. A coach could get a small timeline of your expressions next to the transcript, to see which questions landed and which didn't.
Calm moments. A short breathing break where the tide follows your breath. I built a version of this and set it aside for now.

My personal agent is just a folder

Wenwei Lin — Wed, 11 Nov 2026 01:42:13 +0000

If you've wondered what a "personal agent" looks like in practice, this is mine: no app, no server, just a folder of Markdown files and a coding agent pointed at it. At the end there's a folder layout and the rules file, so you can start your own.

Yesterday I was a bit tired and almost skipped the gym. Before heading out, I asked the AI offhand: "If I train back today, which exercises should I do?" It said that if I was tired I should go lighter, taking 20 to 30 percent off last time's weights, then lined up four exercises, three sets each. With the plan in front of me, I actually wanted to go.

In the evening I told it, one by one, what I'd eaten at each meal and how much each cost, and finally asked: "How much did I spend today? Did I get enough protein?" It worked out how much protein I was still short, reminded me to have a scoop of protein powder, logged the spending in my ledger, and checked it against this month's budget while it was at it.

I didn't type a word of this. I said it all into my phone's voice input. Small things that used to take me ages to decide, I now say in a sentence and hand over.

What does these things for me is my personal agent. It isn't an app, just a folder on my computer. I open it in Claude Code, and the agent works inside it.

What it looks like

Fitness is only one of its modules. The whole folder looks roughly like this:

I described this diagram to it, and it drew it. The folder also opens directly in Obsidian: the AI writes in it, and I read in Obsidian. Files link to each other with wikilinks, so one click takes you to the source, and you can see everywhere a note is referenced.

I think four parts are useful to everyone:

Daily records. Notes, journals, recordings, photos and conversations with AI, all filed by date, along with the books and articles I've read. This is the AI's memory of me.
Knowledge. I borrowed the idea of Luhmann's slip box: ideas we've settled on after discussion go into a wiki, one by one. What matters most is sources and indexes: every entry links back to where it came from, and indexes the AI generates tie them together.
About me. What I've done, my personality, my experience and projects. When I write a résumé or introduce myself, this is where the material comes from.
People around me. The AI doesn't know what has happened between me and each person, and transcripts often get names wrong; one teacher's name came out spelled two or three different ways. So I keep a file for each person: who they are, how their name is written, and what we've been through. There are over thirty so far.

Everything else is modules, added as needed. Everyone lives differently, so everyone needs different modules:

Fitness. What I trained each time, the weights and sets, an exercise library, a training plan, and whether I'm getting enough protein each day.
Writing. A list of topics I want to write, am writing and have written; one folder per blog post; and a style guide that gains a rule each time I finish revising a piece.
Games. If you play Genshin Impact, you could make a Genshin folder for character builds, team comps and resource planning, plus a guide noting which game version it's for. Mine is for an auto-battler I play, with line-ups and counters.

When I need something new, I make a new folder and write its rules at the top of its main file. The fitness and spending modules came about a few days ago when I casually said "I'd like to log this every day", and it built them on the spot.

The more I use it, the more I notice tasks with fixed steps that keep coming back, so I distil them into skills: a write-up of the steps, sometimes with a script. The ones worth keeping show up while doing the work: this will come up again, and it'd be useful elsewhere too. There are seven now, such as converting recording transcripts to Markdown, importing ebooks, making Xiaohongshu cards (Xiaohongshu is a Chinese lifestyle social app), and pushing articles into my WeChat Official Account's drafts. How to write Xiaohongshu copy could be distilled the same way later; writing it down is what keeps it consistent.

Starting from zero: moving the past in

For the folder to be useful, it needs enough material. But at first it was empty.

I didn't write any scripts to move things in. I just gave the AI my Notion and flomo API keys, let it pull everything itself, and watched. Later I gave it access to the whole computer to dig through old records, and installed Claude Code on my old university laptop too. Personal projects, little demos, my graduation project and lab reports: it turned them all up, one by one.

Once it finds things, it also files them in the right places. When it found which zip file held my graduation project's code, it added that to my university experience while it was there.

My dated records now go back to June 2021, more than 1,200 days. That's the foundation of how it knows me. Later I'd like to photograph my handwritten journals and bring them in too, to see how I've changed over the years.

What I use it for

Fitness and meals are only the most everyday uses. Over the past month I've also used it for:

Job hunting. Revising my résumé, preparing for interviews, and reviewing afterwards which questions I didn't answer well. It knows what I've done, and it can dig up experiences from old material that even I had forgotten.
My blog. Publishing posts, switching frameworks, going live, adding analytics: I did all of it together with it.
An information officer. It filters my information sources and picks out what's worth reading. I wrote a whole post about that.¹
Diagrams. I describe one and it draws it. In university, lab reports ran to dozens of pages, and whenever I needed a figure I had to switch tools and drag flowcharts together bit by bit in Visio. It tires me out just remembering it.
Writing. It drafts and I edit. This post was written that way too.
Understanding my feelings. Anxiety after an interview, going round in circles after a decision: I say it out loud, and it helps me take it apart bit by bit. I've also asked it to find things people have said in praise of me.

Reading and learning new fields deserve a few more words.

Reading

Reading has become a conversation. As I read, I tell the AI what I'm thinking: a line that struck me, a word I don't know, a question the page raises. It annotates along with me, explains things, and fills in the background.

Sometimes it draws a map. When I was reading Paris from Bloom's Literary Places, I didn't know much about the city. It just drew me a map of Paris with the places from the book, so I didn't have to go searching websites myself.

A book also leads somewhere else. Napoleon came up while I was reading about Paris, and I realised I didn't really know what he had done. So I asked, and he went onto my list of things I want to understand.

All of these conversations are saved in the book's own folder, next to the book. When I finish, we go through my notes a few at a time and build a mind map together, each node linking back to its place in the book. Reading used to end when I closed the book; now what I thought along the way stays.

Turning the vague into the definite

When I want to understand an unfamiliar field, I often feel I know nothing and can't tell where I stand. Actually I know some of it; I just can't put it into words. Once I'm afraid, I can't break things down. AI happens to be good at structure. It splits the field into a few steps, I check myself against each one, and I can see how far I've come.

At my previous job I always felt I didn't understand manufacturing. Then I had it break a factory into six parts: taking orders, design and process, planning, purchasing and warehousing, production, and quality control and shipping. Checking against them, I found I understood some parts and genuinely didn't understand others. It wasn't as scary as I'd thought. What drops you into worry all at once is often just the vagueness.

My thinking used to be scattered. It can find things scattered all over, piece them together, and think with me about what they might add up to.

It also keeps a complete record of how something got done. I used to note down only the final conclusion; how I got there was mostly lost. Now every step of how I reached a conclusion is recorded. This post went through several drafts, and each time, what I said and why I changed it is saved as a conversation.

Why a folder

Before this, I built several versions, and they left me two lessons:

I stored memories but didn't know what to do with them. At first I wrote my own scripts to store memories and built a little agent on top of models like Kimi and Zhipu. Later I gave my rawest journals to an AI and had it summarise what kind of person I am, hoping other agents could use the summary. I had memories and a summary, but not much information, and I hadn't figured out what to use them for. It felt hollow, with no loop closing back.
Don't build the tools yourself. After that I built my own agent with all sorts of tools; it could discuss books with me as I read, which was a step forward. But it wasn't powerful enough, the interface wasn't pretty, and I had to maintain the tools and the folder myself. In the end it was never as convenient as Claude Code or Codex.

After a few tries I realised what matters most is what I actually use it for. So in this version the agent is just off-the-shelf Claude Code or Codex. I only look after the folder's structure and put my attention on using it. The idea came from a talk by ciaoye.² Everything in the folder is plain text, so whatever model or tool I switch to, my material stays in my own hands.

Start your own

You don't need my exact layout. This is the skeleton I'd start from:

Origin/    raw material, kept as-is
  Life/YYYY/YYYY-MM/YYYY-MM-DD/   notes, recordings, photos, chats with AI
  Library/                        books, articles, podcasts, talks
Self/      who I am, what I've done, what I want, people around me
Wiki/      ideas settled after discussion, written like an encyclopedia
Modules/   state and lists: fitness, reading shelf, projects, blog...
Index/     AI-generated indexes, can be rebuilt anytime

I wrote the agent just one manual, AGENTS.md, at the root of the folder. It describes the layout above, plus a few rules. The ones that matter most:

Keep original material exactly as it is. No rewriting, no polishing, no silent corrections. If something must be fixed, keep the original and say what changed.
Be clear about who is speaking. My words, outside sources and the agent's words are kept apart. The AI's replies are not my views.
When unsure, write "to be confirmed". Never guess dates, speakers, names or numbers.
Link back to the source. Every summary links to the original note or recording, so I can always check.
Keep the rules short. When changing a rule, delete or merge old ones first, so the file fits on a screen or two.

The last one I learned the hard way: rules that keep growing get ignored.

What I've learned using it

My biggest takeaway is to trust it, and hand over as much as I can of the things that take me ages to decide. I used to use a workout app, searching for exercises and tapping through them one by one. Now it's more like having a coach: it makes the plan, I just train, and I keep an eye on details like not using momentum. Bookkeeping is the same. No need to build an app; I say one sentence in the chat, and it's logged in Markdown and CSV. I learned this from Lan Xiaohuan, who gave AI his genetic and exercise data for health analysis.³

Logging my meals is something I could never manage before. In 2022 I saw a traditional Chinese medicine doctor, an elderly woman, who told me to write down my three meals every day. I found it a hassle and didn't want to do it at all. Looking back, she was right: only by writing down what you eat can you see how to adjust. Now one sentence logs it, and after a while I naturally get a sense of how to eat and live more healthily.

Handing things over doesn't mean not caring at all.

What it pieces together is sometimes off: I did one thing and it recorded another. When I notice, I have it go back and fix it at the source. Every entry links to what I originally said, so the source can always be found. On the whole I can trust it, and I proofread the details. After all, only I know best what I've been through.

It also won't know what I want on its own. It could only make that training plan after I told it how many times a week I train and which muscle groups to rotate. I have to think clearly first about how I want to use it, then see whether it can do it.

And plans can't keep up with reality. That day at the start, the AI had my back workout neatly planned, and it talked me out the door. At the gym, every back machine was taken, so I trained shoulders instead. (Next time I could ask for a backup plan.) However well it plans, it can't know whether the machines are free.

That has to do with where it lives. My personal agent is a local folder on my Mac, not in the cloud. I can only use it at my computer; once I'm in the gym, I'm on my own. I think that's fine. I plan with it at home and make my own calls on the spot. I don't need AI with me every moment.

I've always had a picture in mind: Ineffa from Genshin Impact. She's a robot Aino built, a kind of housekeeper who plans and organises Aino's life, like splitting the budget into money for the workshop and money for snacks. In April this year I told the AI I wanted to build an assistant like her. In May it struck me that she wasn't like that from the start either; she went through many generations of trial and error before becoming who she is. My personal agent is the same. The earlier versions were trial and error, and with this one it finally has the beginnings of a life assistant.

How would you use it?

My way isn't necessarily right. As I see it, personal agents are still in an in-between stage. Everyone is experimenting, and that in itself is a lot of fun.

Products like Muse live in the cloud, connected to email, calendar and LinkedIn, usable anywhere, anytime. One writer took it on a trip to the US and got a meeting with someone they'd wanted to meet.⁴ But products like that mostly keep your data inside their own walls. If I want to dig out a few ideas from long ago and look at them together, or let a coding agent read them too, it isn't easy. Mine only works at home, but it remembers my own past, and it's all plain text that any tool can read.

I can't say what it will grow into. Seeing Muse, though, I'd like to learn from how it keeps tracking things, and add scheduled tasks to the folder so it can keep an eye on what I've been following lately. How exactly to do it, I'm still thinking about.

If you've built something similar, or keep your life in a folder some other way, I'd love to hear how you set it up.

This post was first published on my blog. I wrote it in Chinese and translated it with AI help.

Wenwei Lin, "Letting an agent rebuild my information diet", 2026-11-05, https://www.wenwei.site/en/writings/information-sources ↩
ciaoye, a talk on a personal OS (in Chinese), https://ciaoye.github.io/agent-self-tech-talk/ ↩
Hosted by Yu Yi dev, with Lan Xiaohuan as guest, "Conversation with Lan Xiaohuan: inside AI, don't force old theories onto a new reality" (in Chinese), podcast AI Lens, 2026-09-02, https://www.xiaoyuzhoufm.com/episode/6a967e42f03e74ee6b0332ad ↩
Xiaoxin, "Ten days travelling in the US: Muse helped me get a meeting with economist Tyler Cowen" (in Chinese), WeChat Official Account "Agent Universe", 2026-11-02, https://mp.weixin.qq.com/s/PMLJvEg4-bmXGxBSIfwnZA ↩