This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.
What I Built
I spend way too much time at my computer. Between coding, experimenting with AI models, and starting new projects when I have a perfectly good collection of unfinished ones already, I don't exactly have a shortage of reasons to stay indoors.
So when I saw that the theme for this week's Hacktoberfest challenge was Touch Grass, I immediately thought it would be fun to build an AI-powered game that encourages people to get away from their computers.
Which is a little ironic, considering how many hours I then spent at my computer building it. But anyway.
This is Side Quest!
Side Quest is a little outdoor adventure game powered by Google's Gemma model. The idea is pretty simple: you tell the game a little about where you are and what it's like outside, and Gemma comes up with a small quest for you to complete in the real world.
Maybe you're walking through a park on a fall afternoon. Maybe you're just hanging out in your backyard. Gemma takes your surroundings, the season, weather, and time of day into account and gives you something to go find.
You head outside, find whatever it asked for, and take a picture. Then Gemma examines your photo to decide whether you've actually completed the quest.
When you successfully complete a quest, you hatch a little creature inspired by whatever you found.
Of course I had to add a creature collection. I love games where you can discover and collect things, and I thought it would be great to have something to show for all those little outdoor adventures.
Plus, I liked the idea of making people pay attention to things they might normally walk right past. An interesting looking leaf, an unusual rock, a shape in a tree. The kinds of things you might notice if you weren't staring at your phone.
Yes, I realize the irony of using your phone to play a game that's supposed to get you to stop staring at your phone.
How it works
The gameplay is pretty straightforward:
- Set the scene. Tell Gemma where you are, what the weather is like, what season it is, and what time of day. You can choose from the options or describe your location yourself.
- Get your quest. Gemma creates a little challenge based on your surroundings.
- Go find it! Head outside, explore a bit, and take a photo of whatever you discover.
- Get judged by Gemma. The model examines your photo to see whether it matches the challenge. If it doesn't, it'll tell you why, and you can give it another shot.
- Hatch a creature. Complete the quest and get a new little companion for your collection.
The creatures were especially fun to work on.
I didn't want to just make ten pre-designed creatures and randomly hand one out whenever someone completed a quest. That felt a little boring.
Instead, Gemma looks at the photo and describes characteristics that could inspire a creature. The app then uses those traits to draw a little SVG character.
The photo also determines a consistent seed, so submitting the same image produces the same creature.
It's not generating a whole new image through an API. The creatures are actually drawn by the app, using the details Gemma provides.
I love this because your reward is connected to something you really found outside. And honestly, seeing what kind of little creature is going to hatch is half the fun.
There's also a collection where you can look through your creatures alongside the original photos. It's basically a nature journal, except everything has adorable little faces.
I also added a Scavenger Hunt mode!
Instead of getting one quest at a time, Gemma gives you a list of things to find during a walk. You can take photos of everything and submit the whole collection for judging.
I wanted Side Quest to be something that kids and adults could both enjoy. Something you could play with your family, or just by yourself when you need an excuse to get away from your desk for a while.
It's not trying to be a fitness app. There are no step counts, exercise goals, or leaderboards.
It's just a little game that makes going outside and exploring your surroundings more interesting.
Demo
Play Side Quest: https://side-quest.pinkpixel.dev
You can try Side Quest right in your browser.
For desktop, you'll need a recent Chrome or Edge browser with WebGPU support. Click Wake Gemma, and the model will load directly into your browser.
One thing to be aware of: the initial model download is around 3.4 GB. So you'll probably want to be on Wi-Fi for that first visit. Once it's downloaded, the model is cached in your browser.
The desktop version is great for trying everything out, but obviously, carrying your computer outside to photograph a leaf isn't exactly practical.
That's where the mobile experience comes in.
You can use your computer as a home base that runs Gemma while you take your phone outside to complete quests.
Just sign in with GitHub on your computer, leave Gemma running in the home-base tab, and scan the pairing QR code with your phone.
Then you're ready to go exploring.
Your phone handles the quests, photos, and creature collection while your computer does all the heavy AI work.
I also made sure the phone can hold onto your photos when you don't have a connection. They'll sit in a local queue and send when you're back online.
Code
GitHub: https://github.com/pinkpixel-dev/side-quest
Side Quest is open source under the Apache 2.0 license.
The repo includes the React frontend, Gemma inference worker, quest generation and creature logic, Cloudflare Worker backend, optional private home-base server, and tests.
If you'd like to run the basic solo version locally, it's pretty simple:
npm install
npm run dev
Then open http://localhost:5173 in a desktop browser that supports WebGPU.
The README has more detailed instructions for setting up the hosted features, account authentication, and the private phone-to-desktop option.
How I Built It
Putting Gemma to work
For the AI side of things, I used Gemma 4 E2B, Google's open-weight multimodal model, running with Transformers.js and WebGPU.
I'm using the 4-bit ONNX version, onnx-community/gemma-4-E2B-it-ONNX, which loads in a web worker so the model isn't blocking the main UI.
I've been having a lot of fun experimenting with models running directly in the browser lately, so I was excited to have an excuse to build something a little different with one.
And Gemma isn't just there to generate some flavor text. It's actually responsible for the main parts of the game.
It handles:
- Quest generation: Creating challenges based on your surroundings, weather, season, and time of day.
- Photo judging: Looking at your submitted image and deciding whether you actually found what the quest asked for.
- Creature traits: Describing features inspired by your photo that the app can use to draw your new creature.
- Scavenger hunts: Coming up with multiple things to find and evaluating the collected photos.
I especially like that Gemma evaluates your photo and describes the creature together. Your discovery actually influences what you get as a reward.
For the rest of the stack, I used React 19, TypeScript, Vite 8, and Tailwind CSS 4.
IndexedDB handles the local storage for quests, photos, completed finds, and creatures. And the creatures themselves are generated as SVGs, so there's no need for a separate image generation model or service.
The part that got complicated
There was one pretty obvious problem with my whole idea.
I wanted people to use this game while walking around outside. But the Gemma model runs best with a desktop GPU.
Not exactly ideal for a mobile game.
Naturally, I decided to see if I could get it running on my iPhone 12 anyway.
That experiment ended pretty quickly.
The browser made it about 500 MB into downloading the 3.4 GB model before deciding it had had enough and killing the tab.
So, there goes that idea.
I could have switched to a cloud AI API and called it a day. But I really wanted to keep Gemma running locally, and I wasn't ready to give up on that part.
So I came up with a different approach.
Let the phone do the exploring and let the computer do the thinking.
The desktop browser loads Gemma and acts as a home base. Your phone connects to it and becomes the lightweight interface you actually carry around.
When you submit a photo, it gets sent back to your home base, where Gemma evaluates it and generates the creature traits. The result then comes back to your phone.
The model never has to run on the phone, and there's no cloud AI inference service involved.
I also added some quest pre-generation. The home base prepares a little pool of quests and scavenger hunts ahead of time, so you don't have to stand around waiting for Gemma to think of something every time you want a new one.
If you change your surroundings or conditions, the old quests get discarded and new ones are generated.
And because I wanted the game to work when someone loses their signal, the mobile app saves unsent finds in IndexedDB and retries them once the connection returns.
What started as a simple little outdoor game was suddenly turning into a whole phone-to-desktop AI system.
You know how these things go. One little feature leads to another, and before you know it, you've built three extra systems just to make the original idea work.
Making the setup less ridiculous
My original solution used a local Node server and a temporary Cloudflare tunnel to connect the phone to the desktop.
It worked, and I actually kept that option in the project. You can still run it with npm run home if you prefer a private, self-hosted setup.
But then I started thinking about what it would be like for someone who just wanted to try the game.
Imagine telling someone:
"Hey, I made this cute little game where you go outside and find things. First, install Node, download some dependencies, run a server, start a tunnel, and pair your phone."
Yeah. No.
That's a lot of work just to go look for an interesting rock.
So I added a hosted option to make the whole process easier.
It uses a Cloudflare Worker to relay messages between your phone and the home-base browser, D1 for account and quest data, and R2 for photo storage.
You sign in with GitHub on your desktop, scan a temporary QR code with your phone, and you're connected.
Your creature collection can also sync between your signed-in devices.
The important part is that Cloudflare isn't running the AI model. Gemma still runs locally in your desktop browser using your GPU. The hosted services just handle communication and storage.
I also wanted to be clear about what happens to your data, because I think that's important when talking about local AI.
- Solo mode: Your photos and game data stay in your browser's local storage.
- Hosted mode: Photos go through the Cloudflare Worker and are stored in R2. You can delete your account, including its associated photos and records.
- Private home-base mode: Your photos stay on your own computer instead of being stored in the hosted service.
- Location: The game doesn't track your GPS coordinates. If you enter a location, it's just text used to help Gemma create an appropriate quest.
So while the AI inference is local, I'm not going to pretend that every part of the hosted experience is completely offline.
A few things I learned along the way
One thing that surprised me about this project was how little of the actual work involved getting Gemma to generate text.
That part was fun, of course. But once I got into the phone-to-desktop setup, I found myself spending a lot more time dealing with state synchronization, queued requests, connection handling, and making sure everything still behaved properly when one device went offline.
Basically, all the less exciting stuff that has to work before you can enjoy the exciting stuff.
I also ran into a particularly annoying issue with Cloudflare Worker requests.
My tests were passing, but when I tried making requests through the actual Vite development proxy, I kept getting 403 errors.
It turned out that the test harness and development proxy were presenting different combinations of request URLs and Origin headers. My origin validation was fine in the tests, but it was rejecting legitimate requests in the browser.
The fix was to account for the request host or explicitly configured application origin, and to make sure I was testing through the actual development proxy rather than relying entirely on the test harness.
A good reminder that passing tests don't necessarily mean everything works in the real world.
I used Vitest for testing, including Wrangler's test harness with local D1 and R2 bindings for the Cloudflare Worker.
Not quite as exciting as hatching adorable little creatures, but I suppose the plumbing has to work too.
Why Does Open Innovation Matter?
This is actually one of my favorite things about building projects with open-weight models.
I love experimenting with AI, especially when I can run models locally and figure out interesting ways to use them.
And I think projects like Side Quest are a great example of why having access to those models matters.
Sure, I could have built this game using a hosted vision API. It probably would have been easier in some ways.
But then every quest, every photo evaluation, and every creature would mean another request to an external AI service.
With Gemma, I can run the model directly on my own hardware, experiment with how it behaves, and build an entire little game around it without paying for every interaction.
For the core solo experience, there's no account required, no external AI API, and no server processing your photos. The model runs right in your browser.
And because the model is available to run myself, I can actually experiment with it.
I can change the prompts, adjust the structured outputs, fine-tune how quests are judged, and try different approaches to generating creature traits. I could even experiment with other compatible models in the future.
I'm not limited to whatever features a hosted API happens to expose.
But I think the most interesting part of this project is how that freedom influenced the design.
When I realized Gemma wasn't going to run on my phone, I didn't have to abandon local inference entirely.
Instead, I found a way to use my desktop browser as a tiny personal AI server while my phone handles the actual gameplay.
And I think that's pretty awesome.
It also makes me wonder what else you could build using that same approach. A powerful local model running on one device, with lightweight interfaces connecting to it from somewhere else. There's a lot of room to experiment there.
Of course, local AI isn't without its limitations. The model download is pretty hefty, you need hardware that can run it, and the optional hosted features still involve cloud services.
But open-weight models give developers more choices about how to build things, and I love that.
Not everything needs to be another app that sends a prompt to a paid API and displays the response.
Sometimes you can get a little more creative with it.
And in this case, I used an AI model to build a game that encourages people to spend less time interacting with AI.
I still find that pretty funny.
Prize Categories
- Best Use of Gemma: Gemma 4 E2B powers the core gameplay, including context-aware quest generation, multimodal photo judging, scavenger hunts, and creature trait generation. It runs locally in the browser using Transformers.js and WebGPU.
Final Thoughts
I had a lot of fun building this one!
It turned out to be a bit more involved than I originally expected, especially once I decided that the game really needed to work on a phone. But figuring out how to make that happen while keeping Gemma running locally was probably the most interesting part of the whole project.
And I genuinely love the little creatures. I could probably spend a lot of time just seeing what kinds of things Gemma comes up with from different photos.
I also think it's nice to build something that's just meant to be fun.
Not every AI project needs to automate your entire life, make you more productive, or solve some enormous problem.
Sometimes it's perfectly fine to build something that just gets people to go outside and take pictures of leaves.
If you give Side Quest a try, I'd love to hear what you think! And especially what kind of creatures you end up hatching.
Top comments (1)
tr.ee/dev-to