Sky: Let's Go Outside, Make a Plan, and Have Some Company

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

After a day of work or study, going outside can sound good while still feeling like another task. You need to choose a place, work out the time and budget, and find the motivation to leave. Going alone can make that first step harder.

I built Sky, a mobile-first AI companion with three connected goals: invite you outside, help you make a manageable plan, and keep you company during the outing.

1. A small invitation to go outside

Sky uses the interests you choose to help suggest an outing you might actually enjoy. If you enable daily invitations, it can send a personal suggestion around your study/work routine. A short nearby walk after work can be enough; the experience does not need to become a big weekend trip.

These invitations are optional. You choose the schedule and whether to save a location for nearby suggestions. Sky is instructed to respect rest and refusal, and you can disable invitations or follow-up reminders.

2. A plan you can act on

You can tell Sky how much time you have, what you enjoy and what you want to spend. With permission to use your location, it searches for real nearby venue candidates through HERE and helps turn a choice into a saved outing plan.

The plan includes a destination, transport, a schedule, estimated costs and a Google Maps directions link. You can change it through conversation. Unknown prices remain unknown, so an estimate does not quietly become a made-up fact.

For example, you might say: “I have 45 minutes after work and want a quiet walk nearby.” Then: “Actually, make it 30 minutes.” These are example prompts illustrating the intended experience.

3. Company along the way

Sky stays available for conversation during the outing. You can talk about what you are doing, share a photo with a question, or ask to change the plan. Voice input and spoken replies let the conversation use less typing. Optional memories help carry explicitly shared preferences into later conversations; you can inspect, edit, delete or disable them.

The companionship responds to what you share. Sky does not assume it can hear your surroundings or know what you are looking at. Voice and location also depend on browser permissions and a visible page; continuous screen-lock operation is not established.

For Touch Grass, the intended outcome is simple: a person leaves with a realistic plan and has a little company when they want it. The screen should be a short bridge into the outing. This is the product goal; I have not yet measured whether Sky increases time spent outside.

Demo

Video: Watch the Sky demo on Google Drive

The linked video presents the Sky demo. The screenshots below show the conversation, outing planner and memory controls using simulated account and API data. They illustrate the interface; the live Gemma checks are described separately below.

Code

Repository: Lsdfs/double-fun

The repository README covers setup, configuration and current limitations. For the model integration and test evidence, see docs/gemma-integration.md and the JSON reports in artifacts/.

How I Built It

Sky uses a TypeScript/Vite frontend, a Python/FastAPI backend and SQLite account storage. Its core agent uses Google's open-weight gemma-4-26b-a4b-it, served through the Gemini API. This prototype needs internet access and provider quota.

Gemma connects all three parts of the experience: it personalizes outing invitations, creates and revises plans through tools, and handles the companion conversation. It also extracts explicitly stated preferences for optional memory.

HERE supplies venue candidates, and the backend validates structured plans and calculates their known costs. Private provider keys stay on the backend. Speech and web verification use separately configured providers. The core chat route is configured as CHAT_PROVIDERS=gemma, with no automatic closed-model fallback; response metadata identifies the provider and model used.

Architecture and agent flow

The browser manages the interface, permissions, microphone, image preparation and audio playback. FastAPI handles authentication, account-scoped data, AI requests, place tools and the invitation scheduler. SQLite stores account information, conversation history, memories and saved plans.

For a planning request, the flow is:

1. The browser sends the user's message and, when available, a fresh location sample to the authenticated backend.
2. The backend builds the agent context from the conversation, selected interests, enabled memories, active plan and current local time. Country, timezone, weekday and study/work routine are refreshed for each generation.
3. Gemma can request a place-search tool. HERE returns venue candidates; the agent uses those results to choose a destination rather than inventing a venue.
4. Gemma calls create_outing_plan or update_outing_plan with structured arguments. The backend validates the plan and destination, calculates totals and saves the result for that account.
5. A successful lone plan operation can return its short confirmation from the same model generation. Failed operations or mixed tool batches continue through the normal agent loop, so a success message is not returned before the operation succeeds.

The model integration is in backend/app/gemma.py, with plan handling in backend/app/outing_plans.py. GemmaProvider shares the Gemini REST transport and native function-calling format. The core configuration is:

CHAT_PROVIDERS=gemma
GEMMA_CHAT_MODEL=gemma-4-26b-a4b-it

This is hosted inference: there is no local model download or GPU server in the prototype. HERE venue lookup, speech and web verification remain separate integrations.

Personalization and scheduled invitations

Long-term memory is an optional background step after a response. It extracts explicitly stated preferences, profile facts and dated plans, retaining source quotes. The backend rejects unsupported quotes and filters memory access by the authenticated account. Users can inspect, edit and delete facts, or disable extraction and use of memory. Chat history and saved memories have separate deletion controls.

Invitations use two AI stages: generate venue queries from interests and relevant context, then select a returned HERE candidate and write a personal invitation. Venue queries contain attributes rather than private conversation text. The backend validates the selected coordinates for the directions link. If a usable candidate or valid AI output is unavailable, it skips the invitation instead of sending a generic fallback.

The scheduler uses the account's local timezone and study/work routine. Busy-day invitations wait until after the configured routine, with a 1.5 km search radius and instructions to favor a gentle outing of around 45 minutes including return travel; free-day searches use a 5 km radius. Those are planning constraints, not guarantees of travel time or venue availability.

Nearby invitations use an explicitly saved location anchor, which expires after seven days. It is separate from live GPS. Invitation rounds are claimed before generation and delivery to avoid duplicates across workers. Optional follow-ups have daily limits and stop for the day when the user returns to the app, sends a message or taps a notification. Delivery requires a running backend and a valid push subscription.

Voice, photos and browser permissions

The configured voice path uses OpenAI Realtime transcription over WebRTC. The backend exchanges session information while keeping private keys on the server. Transcribed questions enter the same conversation flow as typed messages. Spoken replies use streamed PCM audio at 24 kHz; cancellation stops playback and the request.

Photos are resized in the browser to a maximum edge of 1280 pixels and converted to JPEG to remove original metadata. The backend validates image format and size, with a 2 MB limit. Image inputs use a separately configured vision-capable provider; they are not evidence that Gemma performed image understanding. Original image data is not retained in the history or memory database.

GPS requires permission and a fresh, sufficiently accurate sample for nearby lookup. Hiding the page pauses tracking and stops microphone use. These choices limit accidental ongoing sharing, but they also mean the prototype cannot promise uninterrupted background or screen-lock conversation. Phone microphone, playback and permissions need testing over HTTPS.

Making planning more practical

One useful finding came from testing the full conversation flow. The initial creation turn consumed 11,849 input tokens including the memory step, and the next turn reached the tested account's per-minute free-tier token limit. I reduced repeated instructions and tool declarations, and made successful plan operations return their confirmation without another prose-only generation.

In a separate optimized creation run, input tokens dropped to 4,747 — about 59.9% lower. Creation succeeded, but the next revision hit a provider-busy 503. A later isolated check completed revision and companionship turns. It began with a seeded plan, so these separate successes do not establish uninterrupted conversation under the tested quota.

The shared system instructions were reduced from 1,925 to 839 words, and serialized tool declarations from 3,249 to 1,954 characters. The plan format is declared once, successful tool responses avoid duplicating the full plan, and successful lone plan operations avoid an extra prose-only model call. Interests, relevant memories, local context and the active plan remain available to the agent.

Temporary provider_busy 503 responses are retried on the same model at most twice, with waits of one and two seconds plus up to half a second of jitter. Only the failed generation is retried; completed place searches and plan writes are not replayed. Exhausted retries return a sanitized, retryable error. The browser does not automatically resend the whole chat turn, and provider errors are not saved as assistant replies. These retries do not bypass quota limits.

What I verified

The earlier native-tool check also verified three cases with real Gemma requests: create a plan, revise allowances, and preserve an unknown taxi fare. All passed using synthetic venue data. The artifacts document both successes and limits; they do not establish real HERE results, phone audio performance or outdoor use.

The integration notes also record 140 backend tests, 14 frontend tests and a successful production build. Those checks were not rerun for this write-up. Provider-busy retries were verified with simulated failures. Phone audio, real venue results and an outdoor trial still need separate verification.

Saved check	Observed result	Scope
Native plan creation	Total 50,000 VND; model request 4.28 seconds	Real Gemma, synthetic venues
Allowance revision	Total 10,000 VND; model request 28.00 seconds	Real Gemma, synthetic venues
Unknown taxi fare	Transport allowance and total remained null; 4.27 seconds	Real Gemma, synthetic venues
Optimized create turn	4,747 input tokens; 11.54 seconds including the background task	Authenticated test flow with synthetic HERE results; following revision failed with 503
Isolated revision / companionship	HTTP 200; 2,753 / 2,781 input tokens, including memory extraction	Initial plan seeded synthetically; separate follow-up run

These timings measure different scopes and should not be read as voice latency. The reports are artifacts/gemma-verification.json, artifacts/gemma-free-tier-check.json, artifacts/gemma-free-tier-optimized-check.json and artifacts/gemma-optimized-followup-check.json. Regression coverage includes plan validation, preservation of unknown costs, refreshed local context, rejection of premature success confirmations and retries without duplicate tool operations. Full details and the scripts for reproducing the bounded checks are in docs/gemma-integration.md.

Why Does Open Innovation Matter?

Gemma is central to Sky's behavior: the invitation, planning and conversation all use an open-weight core. The live checks demonstrate that it can call planning tools, revise structured output and preserve unknown costs.

Open weights give this project a future path to independently serving or adapting the model, subject to its terms and the infrastructure required. The provider adapter keeps model transport separate from plan validation, so those application rules do not depend on a particular hosted endpoint.

This prototype uses Google's hosted API and sends model inputs to Google. Independent hosting and fine-tuning are future work; Sky currently needs internet access. I have not measured an advantage over a closed model. What I can show is an open-weight model doing the core planning work, with saved evidence of its results and limitations.

Prize Categories

Best Use of Gemma — Gemma powers the core agent, including outing planning and revision. The saved live reports identify gemma-4-26b-a4b-it and its native tool operations.

Author: Lsdfs (@lsdfs)