I built an AI that only works when my phone is in my pocket

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Every "go outside" app I have tried wants me to look at it once I'm outside: a map, a streak, a badge, a feed. I wanted the opposite: an app that is finished with me before I reach the front door.

Drift is a walk with no destination. You choose how long and a mood (quiet, colors, old things, sounds). Gemma 4, running in your browser, writes a short set of playful rules. An open-weight voice reads them to you, one at a time, while the phone sits in your pocket:

"Trace the outline of the oldest brick you can find with your hand."
"Follow the faint scent that seems to remember rain."
"Stand still until the distant, muted sound settles into your bones."

(Those are real rules Gemma wrote during testing.)

The idea is borrowed from the dérive (French for "drift"), a practice from the 1950s Situationists in Paris. They walked cities by following rules instead of routes, to notice what a destination hides. Seventy years later we have a better excuse never to look up, so I gave the rules to an AI and took the screen away.

When you get home, you talk for a minute about what you noticed. Whisper transcribes it on the device, and Gemma writes it up as an entry in a field notebook. Each entry leaves one line of memory, so the next drift leans toward what you enjoyed.

There is one number Drift actually cares about: how many seconds your screen was on during the walk.

Meet Pip 🦔, who keeps you company before and after. Pip watches your finger, falls asleep while you walk, and gets star-eyed if you barely looked at your phone.

I took it outside

I opened Drift on my iPhone in Safari, and Gemma 4 wrote me a drift called The Quiet Current, right there on the phone. Then I went out and walked for about ten minutes.

I noticed things I hadn't noticed before: how the leaves were moving, and how the wind felt when it touched me. It was hot out, but there was a cool edge underneath the heat. It's October, and even in the sun I could feel a little of the cold that's coming. It was a very engaging feeling, and honestly I can't quite put it into words.

Then I came home and looked at what Drift had recorded, and it caught me out:

Drift	The Quiet Current
Device	iPhone, Safari, Gemma 4 running on the phone
Actually walked	about 10 minutes
Walk the app counted	2 minutes (I ended the drift early and kept walking)
Screen on	2 min 5 sec, the whole time it was counting
Notebook entry	"Thank you."

Two honest lessons from one short walk:

• I didn't lock my phone. I kept Drift open in my hand, so the screen-time counter did exactly its job and recorded that the screen was on the whole time. The best part of the walk happened after I stopped looking at the app. That's the whole thesis of Drift, demonstrated on its own author.
• My notebook entry just said "Thank you." That's a well-known Whisper failure: when it hears silence or almost nothing, it outputs "Thank you." (it was trained on a lot of subtitles). Drift saved that as my notebook entry. It now checks whether a recording is too short or too quiet, drops Whisper's silence phrases, and asks you to try again. There's a test for it with real Whisper.

Next time the phone goes in my pocket.

What a drift actually looks like

Here is a real one, written by Gemma 4 E2B running on WebGPU in my browser. I picked 30 minutes and the mood old things. Gemma wrote this in 11.5 seconds:

Echoes in the Cobblestones

1. Find the object with the deepest shadow beneath a large leaf.
2. Listen intently to the rhythm of the oldest wood.
3. Touch the texture of the wall that feels the longest.
4. Follow the faint scent that seems to remember rain.
5. Stand still until the distant, muted sound settles into your bones.
6. Allow your gaze to simply rest on the most enduring shape nearby.

None of these need a map, and none of them could be followed while looking at a phone. That's the point. You can't do "touch the texture of the wall that feels the longest" from a screen.

Every time you press Prepare the rules are new. Other drifts Gemma wrote while I was building: The Echo Chamber Wander, Chromatic Descent, Dust & Echoes.

Then Kokoro reads them in a calm voice, with a soft chime before each rule and minutes of silence after it. The rendered walk for The Echo Chamber Wander came out at 31 minutes of audio, from a single file. On the deployed site I watched the counter tick from "Starting" to "Rule 1 of 6" while the audio played, with the page doing nothing else.

	Time on my laptop (Apple GPU, WebGPU)
Gemma 4 writes a drift	11.5 s
Gemma 4 writes a notebook entry	2.2 s
First-time model download (2 GB)	~11–14 min on my connection
Network needed during the walk	none

Demo

🌿 Try it: drift-three-nu.vercel.app · Code: github.com/codeswithroh/drift. I've run it on an iPhone (Safari) and a MacBook (Chrome). It needs WebGPU: Chrome or Edge, or Safari 26+. The first visit downloads about 2.5 GB of models, so use Wi-Fi.

Choose	Prepare	Walk	Screen time	Notebook

Code

codeswithroh / drift

A walk with no destination. On-device Gemma writes the rules, Kokoro reads them, your phone stays in your pocket.

Drift

A walk with no destination. An open-weight AI writes the rules, an open-weight voice reads them, and your phone stays in your pocket.

Try it · How it works · Run it · Tests

---

Why

Every "go outside" app I have used wants me to look at it outside: a map, a feed, a badge. Drift takes the opposite approach. You spend about 30 seconds on the screen before you leave, and then the phone goes in your pocket.

It is based on the dérive (French for "drift"), a 1950s Situationist practice: walk a city by following playful rules instead of a destination, and notice what the route would have hidden. Drift asks Gemma to invent those rules for you, has an open-weight voice read them, and counts how many seconds your screen was on while you walked.

Features

• 🌿 Rules, not routes. Gemma writes a fresh drift for…

View on GitHub

How I Built It

Drift is a static web app: Vite and TypeScript, no framework, and no backend at all. Every model runs on your device.

Job	Open model	How it runs
Write the rules + the notebook	Gemma 4 E2B (web build, 2 GB)	MediaPipe LLM Inference on WebGPU
Read the rules aloud	Kokoro-82M	kokoro-js, WebGPU or WASM
Listen to your debrief	Whisper base	transformers.js, WebGPU or WASM

The trick: the whole walk is one audio file

My first idea was to speak each rule on a timer. That doesn't work: a locked phone suspends a web page's timers and background GPS, so the app goes to sleep with you.

Audio, though, keeps playing. So Drift does all the work before you leave. Gemma writes, say, 7 rules. Kokoro speaks each one. Then I stitch them into a single WAV file: a soft two-note chime, the spoken rule, then minutes of silence while you walk, all repeated.

for (const [i, step] of drift.steps.entries()) {
  const raw = await tts.generate(step.say, { voice: 'af_heart', speed: 0.95 });
  if (i > 0) push(chime());
  push(resample(toMono(raw.audio), raw.sampling_rate, 16000));
  push(new Float32Array(Math.round(step.walkSeconds * 16000))); // walk
}

You press play, lock the phone, and the next 30 minutes need no wake-ups, no GPS and no signal. The screen-time counter is just the Page Visibility API: every second the page is visible during the walk counts against you.

Things that broke (and what I learned)

I wrote end-to-end tests with Playwright that drive the real app with real models: real Gemma, real Kokoro, and a fake microphone playing recorded speech so Whisper has something to transcribe. They found bugs I would have shipped:

• Gemma 4 returned an empty string. MediaPipe passes your prompt to the model as-is, and Gemma 4 only answers inside its chat template (<|turn>user … <turn|><|turn>model). Without it, every drift silently fell back to my built-in rules. One line fixed it, and it was invisible until a test asserted that the title wasn't the fallback title.
• The 2 GB model didn't fit. Some browsers cap a site's storage below 2 GB. My cache.put() failed quietly, so the app re-downloaded Gemma on every visit. Now it checks navigator.storage.estimate() first and tells you honestly when the model can't be kept offline. The walk itself still never needs a network.
• Small models can't add up. Gemma's durations rarely total the 30 minutes you asked for, so Drift rescales the silences to fit exactly. Gemma also loved copying my example rules word for word, so the prompt now asks it to invent its own.
• The notebook made things up. My tests play a recorded debrief through a fake microphone: "I followed a yellow bicycle past a bakery. Then I stood under a huge old tree and listened to the wind in the leaves." Whisper transcribed it correctly. But with the smaller laptop fallback model (Gemma 3 4B via Ollama), the notebook entry came back mentioning "the coarse, warm texture of the brick wall against my palm." I never said anything about a wall. Gemma 4 had the opposite problem at first: it copied the transcript word for word. The prompt now asks for a rewrite in a naturalist's voice and forbids inventing sights. A notebook that lies about your own walk is worse than no notebook.
• "Thank you." My first real walk produced a notebook entry that was just Whisper's silence hallucination (see above). Silent recordings are now rejected before they reach the notebook.
• "Rule 5 of 4." The progress counter included the intro and outro clips. A classic.

Pip

Pip is drawn as two 3×3 sprite sheets (nine head directions, nine expressions), adapted from the MIT-licensed page-mascot by Kamran Ahmed. I ported the React component to plain TypeScript and gave Pip a mood for every screen. It's a small thing, but it makes a 2 GB download feel like waiting with a friend.

Why Does Open Innovation Matter?

Drift could not exist as a closed-API app, for three reasons.

1. It has to work with no signal. The walk happens where the network doesn't reach: parks, trails, the back streets the rules send you down. With an open-weight model the rules are written on your phone before you leave, and the walk needs nothing at all.

2. Your voice stays yours. The debrief is a recording of you, outside, saying what you noticed. Sending that to someone else's server to be transcribed and summarized would make me uneasy, so with Whisper and Gemma in the browser it never leaves the phone. There's no account, no API key, and no database. There is nothing to leak because there is nothing to collect.

3. It costs nothing to run. Drift is a static site. Every walk is computed by the walker's own device, so 10 users or 10,000 cost me the same: zero. A closed-model version would need a backend, a bill, and eventually a subscription. That's a strange thing to charge for in an app whose whole point is using your phone less.

Open models also let me make design choices a hosted API never would: swap Gemma for a smaller model on older phones, pick the voice, change exactly how the notebook is written, and read the source of every layer when something breaks (see "Gemma 4 returned an empty string").

Prize Categories

• Best Use of Gemma. Gemma 4 E2B runs entirely in the browser on WebGPU via MediaPipe. It writes every drift, writes every notebook entry, and keeps the memory that shapes the next walk.

---

Thanks to the Gemma, LiteRT, Kokoro, Whisper and transformers.js teams for making all of this run in a browser tab, and to Guy Debord for the idea, 70 years early. 🌿

Comments

[Ivan Annovazzi]

The screen-on seconds metric is such a good inversion, most walk apps would've made it a streak with a map attached. Now I'm dying to know how it handles a cheater though, if I pocket the phone and sit on a bench the whole time, does it notice you stopped moving?

[Rohit Purkait]

Well, currently, no. I have not put any sort of working tracking mechanism in it. This was built to really make you feel conscious about nature and its surroundings. So, even if you are just sitting on a bench, yet consciously observing nature and seeing things around, even that is amazing.

[Harun - solo dev]

the "single wav file" hack to keep the browser awake while the phone is locked is a masterclass in constraint-driven engineering. most devs would try to fight the os with messy background geolocation apis or wake locks, but stitching the audio with silence is brilliantly pragmatic.

your honesty about the whisper "thank you" hallucination and the gemma 4 chat template edge cases is exactly why open-weight local inference is superior. you can actually read the source, trace the bug, and fix the prompt contract instead of just guessing why a black-box api failed.

the screen-on time metric is the perfect inversion of typical engagement-bait apps. as someone building an ai tool specifically for $150 android phones on 3g networks, i deeply respect any architecture that actively tries to reduce screen time and network dependency.

curious about one edge case: for users on older devices or slower networks where the 2gb webgpu model download might timeout or fail the storage estimate, do you have a graceful fallback (like a smaller wasm model or a pre-baked text-only mode), or is webgpu a hard requirement for the core loop?

massive respect for the e2e playwright tests catching those silent failures. this is exactly how you build tools for the real world. 🐯🌿

[Rohit Purkait]

Thanks a lot for the appreciation and for everything that you have said. I was testing on one of my older devices, and I felt the same issue. I have indeed put a fallback model, which would directly work like a smaller WASM model. I'm thinking of making it much more efficient so that the massive 2 GB burden doesn't stay.

[Harun - solo dev]

that’s exactly the right path. the 2gb webgpu download is a massive barrier for older devices and spotty networks, so a lightweight wasm fallback is the perfect bridge.

even if the fallback is just for simpler tasks or a slightly degraded (but still functional) experience, it keeps the core loop alive without forcing the user to abandon the app entirely. accessibility is all about graceful degradation.

honestly, watching you iterate on this in real-time based on actual device testing is exactly why i love this community. keep pushing the boundaries of what’s possible in the browser, rohit. can’t wait to see how the optimized version turns out! 🐯🌿

[Rohit Purkait]

Thanks a lot for such kind words, and again, appreciate it for your feedback. I absolutely love breaking the boundaries and trying to build something that would really help everyone in their journey. Really appreciate it.

[ssapable]

"Timers sleep, audio keeps playing, so do all the work before you leave" is the kind of trick that only shows up when someone actually tried to ship the thing, and the table where your own first walk records 2:05 of screen time, the whole time it was counting, is the most honest user test in this challenge. (Also, your Pip and my cartoon's Pip share a name, which I didn't plan.) I turned the post into an 84-second hand-drawn cartoon: the map-streak-badge-feed trap, the rules Gemma wrote, the one-audio-file walk, and the "Thank you." notebook entry. No strings, yours to use or ignore, on a plain page with no sign-up:

ssap-pa.github.io/scripttoon-samples/

If any detail is off, say which shot and I'll re-render it. One question: the one line of memory each entry leaves, does it go into the next Prepare prompt verbatim, or does Gemma summarize it first?

[Rohit Purkait]

What you have created is just exceptional. I just saw the video. It was just so good.

Thanks a lot for creating this. I really love it. And yeah, definitely, Gemma summarizes it first.

[ssapable]

That made my evening, thank you. And "Gemma summarizes it first" is the right call: a verbatim line would drag the whole walk back into the next prompt. Enjoy the next drift, phone in the pocket this time.

[Rohit Purkait]

Yeah, absolutely. And you just made my evening. By the way, I made a tweet about your video. Let's connect on Twitter as well, if you are active there.

x.com/codeswithroh/status/21077803...