This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
Before We Scroll: I Built an AI That Turns “Just 5 More Minutes” Into Something Real
There is a tiny moment that happens several times every day.
You have 5 minutes before a game starts.
15 minutes before you need to leave.
You are waiting for someone.
You are standing in a park with friends.
And almost automatically, the phone comes out.
Not because there is nothing else to do — but because figuring out what else to do right now takes more effort than opening another feed.
That was the problem I wanted to solve.
So I built Before We Scroll.
Less scrolling. More doing.
Before We Scroll takes whatever context you already have — text, voice, a photo of your surroundings, or a combination of them — and turns that moment into three short, practical real-world missions.
And after building it, I did the most important test:
I actually took it outside.
What I Built
Before We Scroll is a multimodal AI activity generator designed to turn idle screen time into real-world activity.
Instead of asking users to fill out a long form, they can simply:
- type what is happening,
- speak naturally,
- take a photo of their surroundings,
- or combine those inputs.
The application extracts a structured understanding of the situation, including things such as:
- number of people,
- available time,
- activity goal,
- desired intensity,
- indoor or outdoor setting,
- available space,
- ground surface,
- visible environmental features,
- equipment,
- and safety constraints.
Before generating anything, the user gets a Context Review screen where every interpreted detail can be checked and edited.
Then the system creates exactly three missions tailored to that context.
Each mission contains:
- a title,
- short description,
- duration,
- intensity,
- player count,
- equipment requirements,
- an explanation of why it fits,
- step-by-step instructions,
- and safety guidance.
The user picks one, optionally listens to the mission using ElevenLabs text-to-speech, starts the mission timer, goes and does it, and comes back only when the activity is complete.
Afterward they can leave feedback and, optionally, share their experience with the public Real Outventures section.
The complete flow is:
Text / Voice / Photo
→ Context Understanding
→ Context Review
→ 3 Missions
→ Choose One
→ Listen
→ Start
→ Get Off the Screen
→ Complete
→ Reflect
→ Share
The important part is that the AI is not trying to keep the user chatting.
Its job is to understand the moment quickly enough that the next interaction happens away from the screen.
Demo
Live Application
👉 https://before-we-scroll.vercel.app/
Source Code
👉 https://github.com/sheda3838/before-we-scroll
Full Engineering Build Log
The build log contains the complete development journey, including model benchmarks, multimodal testing, timer architecture, voice integration, MongoDB persistence, GridFS storage, and the final product redesign.
I Took It Outside
Building an application for the Touch Grass challenge without actually taking it outside felt wrong.
So instead of stopping once the app deployed successfully, I created five different real-world test cases.
I intentionally varied the input method and the kind of reasoning required.
The five tests covered:
- Photo + Text — visual context and indirect time reasoning
- Photo + Voice — combining spoken details with visual surroundings
- Photo + Text — environment-aware exploration
- Text Only — complex constraint following
- Voice Only — conversational speech understanding
I also completed missions and added real feedback rather than only checking whether the API returned JSON.
Test 1 — Five Minutes Before Futsal
This became one of my favorite tests.
I uploaded a real photo taken inside an indoor futsal arena.
The photo showed the venue and several people, but I deliberately did not tell the AI:
- how many people were there,
- whether we were indoors,
- what the surface was,
- how large the space was,
- what features surrounded us,
- or what equipment was available.
The only text I gave it was:
“We have five minutes left before our game starts. Tell us something active we can do while we wait.”
Notice that I did not explicitly say:
“Create a five-minute mission.”
The system had to understand that five minutes left before the game means roughly five minutes of usable activity time.
What it understood
The Context Review identified:
- 4 people
- 5 available minutes
- activity goal: active waiting before a game
- indoors
- large space
- artificial turf
- indoor sports arena
- safety netting
- overhead lighting
- metal frame structure
It also correctly avoided adding equipment that was not clearly available.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/q0hf2nktwrkcce2dhzbk.jpeg)
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/j6xsl2odlglglmvqqk4d.png)
That was exactly the sort of multimodal behavior I wanted from the project.
The image supplied the physical world.
The sentence supplied the intent and time constraint.
The AI joined the two.
Missions generated
It produced three five-minute ideas:
- Netting Geometry Challenge
- Arena Soundscape Mapping
- Lighting Shadow Play
I selected Lighting Shadow Play, which used the arena's overhead lighting and turf to create a quick movement activity.
What impressed me most was that this was not a generic “do jumping jacks for five minutes” answer.
It was built from features that were actually visible in the environment.
My feedback afterward was:
“The mission matched our space really well and gave us a fun, active way to use the few minutes we had before the game.”
Test 2 — A Backyard, Two Rackets, Seven Minutes
For the second test I changed the input method.
I uploaded a photo of a grassy residential yard and used voice input instead of typing.
I spoke naturally:
“My friend and I have about seven minutes to kill before we head out. We’ve got two rackets and a shuttlecock with us — give us something quick and fun to do in this space.”
This test divided the information between two sources.
The voice provided:
- two people,
- roughly seven minutes,
- two rackets,
- a shuttlecock,
- and the desire for something quick and fun.
The photo provided:
- the outdoor setting,
- grassy surface,
- available space,
- walls,
- trees,
- gate,
- and surrounding features.
The goal was to see whether the system would combine both sources rather than treating them separately.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/g211oyw3ozywuy870npb.jpeg)
Missions generated
One of the suggestions was:
The Gate-to-Tree Rally
It transformed a normal badminton rally into a small precision challenge using boundaries already present in the yard.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/32xllzonoecele57cdnk.png)
I already had a real image showing the activity happening in the same space, which made this one especially satisfying to test.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/8wcck8c6phnnf319o80a.png)
My feedback:
“The mission matched the space well and made good use of the rackets and shuttlecock we already had. It turned a simple rally into a quick, fun challenge.”
Test 3 — Turning a Park Into an Exploration Mission
The third test happened in a much larger outdoor environment.
The image showed an open grassy historic area with paths, trees and surrounding landmarks.
This time I supplied:
“I’m here with two kids and we’ve got a football with us. We have about 15 minutes before we need to leave, and we want to do something fun that gets us moving around and exploring this place.”
Here I wanted to see whether the system would avoid generating just another football drill.
The word “exploring” mattered.
The physical environment mattered.
And the activity needed to remain suitable for a mixed group.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/dyesa0cvd9jcax369umz.jpeg)
Missions generated
The three generated ideas included:
- Cannonball Guardian
- The Historic Path Finder
- Pass the History
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/icfaldapmc89h2jm0877.png)
I chose:
The Historic Path Finder
Instead of simply passing the football back and forth, the mission encouraged us to move through the location and use the ball as part of an exploration challenge.
That was a great example of the AI using the character of the location, rather than treating every grassy area as the same place.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/3vha6ok3fl15h9jiyqnt.jpeg)
My feedback:
“This mission fit the location really well and made the walk feel more engaging. Using the football while exploring different points around the fort made the activity simple, fun, and easy to follow.”
Test 4 — No Image, Just Constraints
For Test 4 I removed vision entirely.
The input was only text:
“Four of us are stuck inside for about 12 minutes while we wait for dinner. We want something competitive and energetic, but the room isn’t very big, we can’t run, and we don’t have any equipment.”
This was deliberately designed to contain several interacting constraints:
- four people,
- 12 minutes,
- indoors,
- limited space,
- competitive,
- energetic,
- no running,
- no equipment.
Some of those requirements even pull in opposite directions.
“Energetic” normally encourages movement.
“No running” restricts it.
“Competitive” often introduces equipment.
“No equipment” removes that option.
So this test was less about vision and more about whether the mission generator could respect all of the constraints simultaneously.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/mdq24rg71mjxvyh1xbdo.jpeg)
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/0yz5rn9k2pbjnolkil55.jpeg)
The generated activities stayed within the available space and did not suddenly invent balls, cones, cards, or other equipment.
My feedback:
“The missions followed the constraints really well. They stayed competitive and active without requiring running, extra space, or any equipment.”
Test 5 — Talking Like a Normal Person
The last test was voice only.
No image.
No carefully structured sentence.
I intentionally spoke more like someone would actually speak:
“Okay, so it’s just me and my friend right now. We’ve probably got maybe twenty minutes before we need to head off. We’re both pretty tired, so nothing too intense, but we also don’t want to just sit around. We don’t have anything with us either. Give us something chill but still fun.”
This test actually exercises two AI stages.
First:
ElevenLabs Speech-to-Text
has to correctly transcribe casual speech, including words such as:
- “okay”
- “probably”
- “maybe”
- “pretty tired”
Then the language model has to find the useful meaning inside that transcript.
The important context is:
- two people,
- around 20 minutes,
- low intensity,
- no equipment,
- something fun,
- environment unspecified.
A good system should also know what not to infer.
Because I never supplied a photo or location, it should not confidently invent a park, beach, garden or indoor court.
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/xm5fo8r4x1fr2z7fzlux.jpeg)
(https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/43hqytmn0c5peuc6rznx.jpeg)
My feedback:
“The app understood the casual voice input surprisingly well and suggested activities that matched our limited energy, time, and lack of equipment.”
What I Learned From Testing It Outside
The most useful lesson was that multimodal AI becomes much more interesting when each input carries a different part of the truth.
In the futsal test:
Image
→ people + arena + turf + environmental features
Text
→ only five minutes left + desire to stay active
In the backyard test:
Image
→ garden + grass + physical boundaries
Voice
→ two people + seven minutes + rackets + shuttlecock
The model's job is not simply to “look at a photo.”
It has to merge different pieces of context into one useful representation.
Another important result was what happened when information was missing.
Before We Scroll's context schema intentionally supports unknown values.
If the system does not know something, I prefer:
null
over a confident hallucination.
The Context Review exists specifically because AI interpretation should remain transparent and correctable.
Code
The entire project is available here:
Before We Scroll
Less scrolling. More doing.
Before We Scroll turns text, voice, and photos of your surroundings into tailored real-world missions designed to get you away from the screen and into the moment
Quick Links
- Live Production App: https://before-we-scroll.vercel.app/
- GitHub Repository: https://github.com/sheda3838/before-we-scroll
- Detailed Build Log: BUILD_LOG.md
Project Overview
The Problem
When people have 15 or 30 minutes of free time, they default to scrolling feeds on autopilot. Even when standing in a park, sitting on a balcony, or hanging out in a living room, the phone screen wins by default—not because physical surroundings lack interesting things to do, but because deciding what to do takes effort.
The Solution
Before We Scroll acts as a catalyst between screen fatigue and real-world engagement. By taking in what you have around you—through typed intent, spoken thoughts, or a quick snapshot of your environment—it extracts a structured understanding…
Or directly:
👉 github.com/sheda3838/before-we-scroll
The repository also contains a detailed engineering journal:
How I Built It
Before We Scroll is a full-stack Next.js 16 App Router application written in TypeScript.
Frontend
- Next.js 16
- React 19
- TypeScript
- Tailwind CSS 4
- Lucide React
- responsive mobile-first interface
- light and dark themes
AI
- Google GenAI SDK
- Gemini 3.1 Flash Lite
- Gemma 4 models used during model evaluation and benchmarking
- ElevenLabs Speech-to-Text
- ElevenLabs Text-to-Speech
Persistence
- MongoDB
- MongoDB GridFS for optional completion photos
Deployment
- Vercel
The production app is:
The AI Pipeline
The project uses AI at several distinct stages instead of making one giant prompt do everything.
1. Context Extraction
POST /api/context/analyze
Text and/or image input is converted into a structured MissionContext.
The structure includes:
- people count
- duration
- activity goal
- intensity
- age group
- environment type
- indoor/outdoor
- space size
- surface
- visible features
- obstacles
- available equipment
- constraints
The user can review and edit everything before proceeding.
2. Mission Generation
POST /api/mission/generate
The confirmed context becomes the source of truth.
A single AI request returns exactly three complete mission objects.
Each contains:
- title
- description
- duration
- intensity
- players
- required equipment
- why it fits
- steps
- safety notes
No second AI call is necessary when a user selects a mission.
3. Voice Input
POST /api/transcribe
The browser records audio using MediaRecorder.
The temporary audio blob is sent server-side to ElevenLabs Speech-to-Text.
The transcript is placed back into the normal input flow.
Audio is not permanently stored.
4. Mission Narration
POST /api/tts
ElevenLabs Text-to-Speech generates mission narration.
Before the mission starts, users can listen to the complete briefing.
During the active mission, the app can read only the actionable steps and safety information.
Audio is cached temporarily in the browser so replaying does not repeatedly call the API.
5. The Timer
One thing I deliberately avoided was a timer based only on decrementing React state.
Instead the mission stores:
startedAtexpectedEndAt
Remaining time is calculated from the current timestamp.
That means:
- refreshing the page does not reset the mission,
- switching tabs does not desynchronize it,
- and background browser throttling does not destroy the countdown.
6. Feedback and Real Outventures
After completion, users can submit:
- a 1–5 rating,
- whether the activity matched their context,
- instruction accuracy,
- difficulty,
- an optional comment,
- and an optional completion photo.
Completion records are persisted to MongoDB.
Images are stored using GridFS, rather than being embedded as huge base64 strings inside normal documents.
Public sharing is controlled separately for feedback and photos.
The public endpoint exposes only a sanitized DTO containing the fields needed by the Real Outventures feed.
Session IDs, private context and other internal information are not returned.
Why I Benchmarked the Models
I originally expected to use Gemma much more heavily throughout the application.
Instead of choosing a model based on reputation alone, I benchmarked it.
For context extraction I tested:
Gemma 4 26B A4B
- average latency: approximately 30.35 seconds
- median: approximately 28.77 seconds
- strong structured extraction quality
Gemma 4 31B
- average latency: approximately 71.06 seconds
- median: approximately 71.52 seconds
- no meaningful accuracy improvement over the smaller variant
Gemini 3.1 Flash Lite
- average latency: approximately 2.88 seconds
- median: approximately 1.12 seconds
- 100% successful responses in the controlled test
- strong schema adherence
I also separately benchmarked mission-generation quality.
The final production path uses Gemini 3.1 Flash Lite for the interactive context and mission-generation workflows because the latency difference dramatically changed the usability of the product.
Gemma remained extremely valuable during development because it gave me an open-weight reference point against which I could evaluate speed, structured output quality and creative mission generation.
The detailed benchmark methodology is available in the build log:
👉 Model benchmarking and architecture decisions
Why Does Open Innovation Matter?
This project changed significantly because I could experiment rather than treating one model as a black box that simply had to be accepted.
Working with open-weight models such as Gemma gave me a meaningful architectural baseline.
I could ask:
- How much quality do I actually gain from a larger model?
- Is the latency appropriate for an interactive product?
- Which stages truly need creative reasoning?
- Which stages are primarily structured extraction?
- Could the mission layer eventually be self-hosted or swapped without redesigning the complete application?
The answer was not “one model should do everything.”
The architecture became better because models could be evaluated as components.
The same principle applies to the rest of the project.
MongoDB gives me control over the application's data model.
GridFS gives me direct binary persistence without tying the MVP to another storage provider.
The application code itself is publicly available so the architecture, experiments and trade-offs can be inspected rather than hidden behind a demo.
To me, that is the useful part of open innovation:
being able to understand, compare, replace and improve the pieces instead of treating the technology as magic.
Prize Categories
The project uses:
- ElevenLabs — Speech-to-Text input and Text-to-Speech mission narration
- MongoDB / GridFS — completion persistence and optional photo storage
- Google AI / Gemma — open-weight model evaluation and benchmarking during development
Final Result
Before We Scroll started with a very simple question:
What if the moment before another scroll could become the moment you actually did something?
Now the application can:
listen to you,
look at your surroundings,
understand your constraints,
create something realistic to do,
read the instructions aloud,
start a timer,
and then get out of your way.
The most rewarding part of this challenge was not seeing the build pass.
It was taking the application outside and seeing AI suggestions turn into actual movement, games, exploration and time spent with other people.
That feels much closer to what I want AI to do:
not always give us another reason to stay on a screen, but sometimes give us a good reason to leave it.
Try Before We Scroll
🌐 Live: https://before-we-scroll.vercel.app/
💻 Code: https://github.com/sheda3838/before-we-scroll
📖 Build Log: BUILD_LOG.md
Top comments (0)