MUSE: An AI Scavenger Hunt That Forces You to Put Your Phone in Your Pocket

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

I spend way too many hours staring at screens, and almost every AI app I’ve seen lately tries to keep you there—scrolling through generated text, tweaking chatbot prompts, or staring at loading spinners.

For this challenge, I wanted to build the opposite. I made MUSE, a mobile scavenger hunt app built on React Native and Expo where the whole point is to put your phone away.

The basic loop:

1. You hit start, and MUSE gives you something specific to notice outside—not generic stuff like "find a dog," but physical textures and environmental quirks. Things like "Find weathered timber where the grain is peeling" or "Spot manmade symmetry in a public space."
2. You put your phone in your pocket. This was rule number one for me. You walk around your street, campus, or park looking with your actual eyes.
3. Once you spot it, you pull the phone out, snap a quick photo through the camera viewfinder, and the AI acts as an impartial referee.
4. If it matches, you get some XP, and your finding dynamically seeds the next clue so your walk turns into a connected trail.

The goal was simple: make the phone interaction take 5 seconds so the walk outside takes 20 minutes.

Demo

Here is how the flow looks on device—from opening base camp to snapping an object in the wild:

Base Camp & Mission HUD	Viewfinder & Verification
1. Pick up an observation quest & pocket the phone	2. Frame the real-world find in the reticle

Quest Mission Details	AI Verdict & XP
3. Difficulty tier & environmental criteria	4. Multimodal feedback on what was spotted

• GitHub Repository: github.com/lazyshrey/muse
• Visual Walkthrough: SCREENSHOTS.md in the repo has the full captures.

Code

lazyshrey / muse

Real-world sensory scavenger hunt & outdoor exploration mobile game powered by multimodal AI vision.

✦ M U S E

See the World Differently · Put Your Phone Away

Real-world sensory scavenger hunt & outdoor exploration mobile game powered by multimodal AI vision on React Native & Expo.

Explore the Visual Showcase (Screenshots & In-Game Captures)

About • Philosophy • Features • Architecture • Game Loop • Missions • Tech Stack • Quick Start • Safety • License

---

About MUSE

MUSE is an inverted mobile application that uses artificial intelligence to get you off your screen and into the real world.

Unlike conventional AI apps designed to maximize indoor screen time and passive scrolling, MUSE issues contextual observation quests. Players receive an objective, put their phone into their pocket, wander physical spaces (parks, streets, campuses, hiking trails), spot hidden nuances in reality, and verify their findings by capturing a photo with the viewfinder.

"The AI interaction should be short; the real-world interaction should be…

View on GitHub

The codebase is MIT licensed. It's built with React Native 0.86, Expo SDK 57, and TypeScript.

How I Built It

The local Gemma experiment (and why I switched to Gemini)

When I first sat down to write this, my plan was pure on-device open-source: bundle an open-weight vision model (like Gemma 2B or an on-device multimodal runtime via LiteRT/TFLite) directly into the app so it could run with zero internet.

That idea hit a wall pretty fast during actual testing:

1. The phone turned into a hand-warmer: Trying to load multi-gigabyte multimodal weights into phone memory caused massive RAM spikes and drained battery quickly. When you're out walking, the last thing you want is your phone dying after 15 minutes.
2. The 12-second awkward pause: On a typical phone, local multimodal inference took anywhere from 8 to 15+ seconds per frame. Standing on a public sidewalk pointing your camera at a tree stump waiting for an on-device model to chug through tokens felt terrible and completely ruined the outdoor flow.
3. React Native edge limits: Cross-platform native bindings for on-device multimodal LLMs in React Native are still experimental and heavy to bundle reliably across different Android and iOS devices without bloating the APK into hundreds of megabytes.

The pragmatic solution:

Instead of forcing a slow local model that made the app frustrating to use, I separated the AI logic into a modular provider interface (AIProviderManager):

• Primary referee (Gemini Flash API): I use the Google AI Studio endpoint running Gemini Flash for multimodal verification. The app grabs the frame with expo-camera, compresses it down on-device with expo-image-manipulator, and sends it with a strict JSON schema. The verdict comes back in under a second. You get an instant confirmation, pocket your phone, and keep moving.
• Pluggable local provider (GemmaLocalProvider): I kept the architecture skeleton ready so that once smaller, optimized mobile vision models (like quantized Gemma variants on LiteRT) become practical on phones, it can be swapped in without touching the rest of the game logic.
• Offline fallback (FallbackProvider): If you're on a trail with zero cell reception, the app doesn't just crash or lock you out—it falls back to an offline heuristic mode so you can still log finds and keep your expedition going.

Why Does Open Innovation Matter?

To me, this project highlighted two big things about open-source AI:

1. It lets us design for humans instead of engagement: Most venture-backed closed AI products want you hooked on their screen—more chat turns, more token consumption, more attention captured. Having open models and open frameworks means independent builders can build software that respects human presence and literally pushes people away from the screen.
2. Freedom to adapt as hardware catches up: Because the prompts, structured JSON schemas, and quest mechanics were written around open-weights concepts, I'm not locked into a proprietary ecosystem. The moment on-device multimodal hardware on mid-range phones can do 500ms vision inference locally, I can flip one switch in AIProviderManager and take the whole game completely offline.

My Agent Session

Saved and documented with DevRelay. Transcript covering the AI provider manager, camera pipeline, and Expo setup: [Link to session or DevRelay embed].

Prize Categories

• Hacktoberfest Open-Source AI Challenge: Week 1 (Touch Grass)