<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Aarushi Tandon</title>
    <description>The latest articles on DEV Community by Aarushi Tandon (@aarushitandon).</description>
    <link>https://dev.to/aarushitandon</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4024520%2F7b3892b3-0fd5-47fb-bffa-f19ff2a1318b.jpg</url>
      <title>DEV Community: Aarushi Tandon</title>
      <link>https://dev.to/aarushitandon</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aarushitandon"/>
    <language>en</language>
    <item>
      <title>sidequestmaxxing :p (Turn Your City Into an Open-World Game)</title>
      <dc:creator>Aarushi Tandon</dc:creator>
      <pubDate>Sun, 11 Oct 2026 11:19:00 +0000</pubDate>
      <link>https://dev.to/aarushitandon/sidequestmaxxing-p-turn-your-city-into-an-open-world-game-36dd</link>
      <guid>https://dev.to/aarushitandon/sidequestmaxxing-p-turn-your-city-into-an-open-world-game-36dd</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week1-2026-10-05"&gt;Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Sidequestmaxxing&lt;/em&gt; is built around a simple idea: what if you treated real life like an open-world video game?&lt;/p&gt;

&lt;p&gt;The term comes from a Gen Z lifestyle trend of collecting side quests instead of focusing exclusively on the main quest. Your main quest might be your degree, career, or daily responsibilities. Side quests are everything else: exploring a street you've never walked down, finding a hidden café, learning to identify birds, visiting a local art installation, striking up a conversation with a stranger, or picking up a random niche skill just because it sounds interesting. In short sidequestmaxxing means treating real-life daily routines like an open-world video game, where optional hobbies, spontaneous trips, and random experiences take priority over a rigid "main storyline" of work or school. And as a GENZ myself, i gotta start sidequestmaxxing and up my lore xD&lt;/p&gt;

&lt;p&gt;These small, low-stakes adventures make everyday life feel less repetitive and give you a reason to explore beyond your usual routine. I wanted to take that idea literally and build a game engine for real life.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sidequestmaxxing turns your neighbourhood into an open-world map, with an AI Dungeon Master that gives you one real-world quest at a time.&lt;/strong&gt; It is my submission to the Touch Grass Open Source AI Challenge, built around one central principle: AI should give people more reasons to leave their screens, not more reasons to stare at them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft01pm2wklfenbtnqlmhm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft01pm2wklfenbtnqlmhm.png" alt="Intro Image" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How it works
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Set up your quest.&lt;/strong&gt; Tell Sidequestmaxxing how many minutes you have, what mood you're in, and what kinds of activities interest you. You can also configure your Dungeon Master's persona and the radius you're willing to explore.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Your AI Dungeon Master builds an adventure.&lt;/strong&gt; A small open-weight language model, designed to be LoRA fine-tuned into a Dungeon Master, generates exactly one quest at a time. The system combines your preferences with real-world context, including the time of day, weather, remaining daylight, and nearby places discovered through OpenStreetMap. The quest is grounded in actual locations, with the goal of choosing an adventure that fits your available time and is practical to complete on foot.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Listen, then leave your phone behind.&lt;/strong&gt; Your quest briefing is spoken aloud, with a choice of four Dungeon Master voices. Once you're ready, enter Walk Mode. The interface fades to near-black, making the phone less tempting to use while you are outside. The app is designed to brief you before the adventure, not demand your attention throughout it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Complete the quest in the real world.&lt;/strong&gt; Go explore. It could be an unfamiliar street, a nearby landmark, or another small adventure based on the places around you. The point is to experience something outside your normal routine rather than complete another task on a screen.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Verify your adventure privately.&lt;/strong&gt; After returning, you can submit a proof photo for verification. On supported devices, Gemma 3n can run directly in the browser using WebGPU to assess whether the photo matches the quest. The photo stays on your device: the application has no server endpoint for uploading it. If on-device model support is unavailable, alternative verification options include a check-in or honour mode.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Turn adventures into progression.&lt;/strong&gt; Completing quests earns XP and advances seven skill categories: Explorer, Naturalist, Social, Foodie, Creative, Fitness, and Mindful. You can unlock badges, track recent adventures, and build a Chronicle of your week, turning small experiences into a campaign log of your life. A personal Journal lets you preserve notes and memories from your quests.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq7igluvau0scov5moeqr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq7igluvau0scov5moeqr.png" alt="Diagram of the six-step quest loop" width="800" height="287"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The design principle: the screen should be the shortest part of the experience
&lt;/h3&gt;

&lt;p&gt;The Touch Grass challenge is not just about building an AI application with an outdoor theme. It is about using AI to help people disengage from their devices and reconnect with the physical world.&lt;/p&gt;

&lt;p&gt;That principle shapes the entire experience. Sidequestmaxxing generates one quest rather than an endless feed of recommendations. It speaks the instructions aloud so you do not need to keep checking your screen. Walk Mode reduces visual distractions. Your Journal stores entries locally on your device, and supported on-device photo verification means your proof does not have to be uploaded to a server.&lt;/p&gt;

&lt;p&gt;Even the XP system reflects this philosophy: it includes a &lt;strong&gt;1.2x multiplier when screen usage stays below 10% of the quest duration&lt;/strong&gt;. The goal is to reward actually going outside, not spending more time interacting with the app.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why open-source AI?
&lt;/h3&gt;

&lt;p&gt;Open-weight models make it possible to shape the Dungeon Master's behaviour around a specific purpose instead of treating it as a generic chatbot. OpenStreetMap provides real-world place data, while the broader stack uses tools that make the system easier to inspect and build upon.&lt;/p&gt;

&lt;p&gt;The project also explores a second role for open-weight AI: running vision verification locally in the browser. Instead of sending a personal photo to a remote vision API, a supported device can process it locally. This is a practical example of how open models can support both customisation and privacy.&lt;/p&gt;

&lt;p&gt;Not every component is open source. Voice generation currently uses ElevenLabs, with browser speech synthesis as a fallback. The project combines open and hosted components where each serves a purpose, while keeping the core experience focused on getting people outdoors.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who is it for?
&lt;/h3&gt;

&lt;p&gt;Anyone who keeps saying they want to go outside, try something new, learn a random skill, or explore their own city, but ends up following the same routine instead.&lt;/p&gt;

&lt;p&gt;Sidequestmaxxing makes starting smaller and easier. You do not need to plan an entire day, organise a trip, or commit to a new hobby. You just need a little time and one reason to step out.&lt;/p&gt;

&lt;p&gt;The adventure does not have to be extraordinary. It just has to be something you would not normally have done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your city is the game. Your phone is just the quest log.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Built for the Touch Grass Open Source AI Challenge.&lt;/p&gt;




&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/eVARmWV4YEc" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live Demo:&lt;/strong&gt; &lt;a href="https://sqm-web.onrender.com/" rel="noopener noreferrer"&gt;Sidequestmaxxing&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  I took it outside
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Examples of sidequests:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fag4i4tdo2cee7ilaf5ni.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fag4i4tdo2cee7ilaf5ni.png" alt="Quest 1" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzwmwovkl25i5tlso6vda.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzwmwovkl25i5tlso6vda.png" alt="Quest 2" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Journal It:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwbow2e3rj6nvlnd866j0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwbow2e3rj6nvlnd866j0.png" alt="Journal" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Earn Points:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjghpuauxuqkgr5jsf57.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjghpuauxuqkgr5jsf57.png" alt="Earn Points" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/aarushitandon0" rel="noopener noreferrer"&gt;
        aarushitandon0
      &lt;/a&gt; / &lt;a href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;
        sidequestmaxxing
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div&gt;
&lt;a rel="noopener noreferrer" href="https://github.com/aarushitandon0/sidequestmaxxing/apps/web/public/icon.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Faarushitandon0%2Fsidequestmaxxing%2FHEAD%2Fapps%2Fweb%2Fpublic%2Ficon.png" width="96" alt="Sidequestmaxxing logo: a parchment-coloured ridgeline under a gold sun, with a dotted trail running across it"&gt;&lt;/a&gt;
&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Sidequestmaxxing&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;Your city is the game. Your phone is just the quest log.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;An open-weight Dungeon Master reads your neighbourhood, writes you one real side quest
speaks it into your earbuds, and then gets out of the way so you can go outside and do it.&lt;/p&gt;
&lt;p&gt;Built for the DEV &lt;strong&gt;Touch Grass&lt;/strong&gt; challenge (Hacktoberfest Open-Source AI Challenge, Week 1).&lt;/p&gt;
&lt;/div&gt;
---
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;What "sidequestmaxxing" means&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;Side quest maxxing is Gen Z slang for treating real life the way you treat an open-world game.
Instead of grinding only the main quest (your career, your degree, your to-do list), you
deliberately seek out, collect and fully dive into random, low-stakes, spontaneous detours:
the market three streets over, the hill you have never walked up, the mural behind the bus depot.
Daily routines, hobbies and small unexpected adventures get treated as optional side quests,
which builds skills, sparks joy, and makes life more interesting.&lt;/p&gt;
&lt;p&gt;This…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;github.com/aarushitandon0/sidequestmaxxing&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;Sidequestmaxxing combines open-weight AI, real-world map data, and deterministic safety checks to turn everyday life into a game. The core idea is simple: &lt;strong&gt;the AI handles the planning so you can put your phone away and go explore.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The open models
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Where it runs&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen-class open-weight model&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;llama.cpp&lt;/code&gt; on a private Render service, CPU-only&lt;/td&gt;
&lt;td&gt;Generates quests as the Dungeon Master&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 3n&lt;/td&gt;
&lt;td&gt;Player's browser via WebGPU&lt;/td&gt;
&lt;td&gt;Verifies proof photos on supported devices&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BGE-small-en-v1.5&lt;/td&gt;
&lt;td&gt;API container via &lt;code&gt;fastembed&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Finds relevant places and detects repetitive quests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is no hosted LLM API in the quest-generation path.&lt;/p&gt;

&lt;h3&gt;
  
  
  How a quest is generated
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fzlq70jxw2es1hfk38r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fzlq70jxw2es1hfk38r.png" alt="Diagram of the three Render services and their fallbacks" width="800" height="553"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The app has three services: a React and TypeScript PWA (&lt;code&gt;sqm-web&lt;/code&gt;), a FastAPI backend (&lt;code&gt;sqm-api&lt;/code&gt;), and a private &lt;code&gt;llama.cpp&lt;/code&gt; inference service (&lt;code&gt;sqm-llm&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjlabmduy5edx9iabr65.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjlabmduy5edx9iabr65.png" alt="Diagram of the quest generation pipeline" width="800" height="600"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Context Builder:&lt;/strong&gt; Checks weather, daylight, and safety conditions before generating a quest. Severe weather, extreme heat, or insufficient daylight can trigger indoor-only mode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scout:&lt;/strong&gt; Uses OpenStreetMap data and vector search to find real nearby places, then filters them by distance, activity type, and available time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory Check:&lt;/strong&gt; Compares candidates with recent adventures to reduce repetitive quests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quest Master:&lt;/strong&gt; Generates exactly one structured quest grounded in the supplied places. It cannot invent a location or assign its own XP.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validators:&lt;/strong&gt; Twelve checks evaluate grounding, time limits, formatting, and safety. One retry is allowed before the deterministic fallback takes over.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fallback Engine:&lt;/strong&gt; Generates quests from tested templates, even when the model or network is unavailable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Game State:&lt;/strong&gt; The server assigns quest IDs, rarity, and XP only after validation succeeds.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Safety and security
&lt;/h3&gt;

&lt;p&gt;Testing the first version revealed that obvious keyword filters were not enough. In an expanded adversarial suite, &lt;strong&gt;34 of 104 test cases initially bypassed the existing validators&lt;/strong&gt;. These included unsafe suggestions involving private property, surveillance, water, and thunderstorms. All 34 are now rejected.&lt;/p&gt;

&lt;p&gt;I also found a prompt-injection risk in OpenStreetMap place names. Because map data can be edited by anyone, malicious instructions could be disguised as a location name. I added a sanitisation layer that removes instruction-like text while preserving legitimate names. Place grounding relies on stable IDs, so sanitisation does not break the connection to real locations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Privacy by design
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftf1etn7wy89kj02ip054.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftf1etn7wy89kj02ip054.png" alt="Diagram of what stays on the device and what crosses the network" width="800" height="440"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Photo verification:&lt;/strong&gt; No image-upload endpoint exists. On supported devices, Gemma 3n processes proof photos directly in the browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Location privacy:&lt;/strong&gt; Coordinates are rounded to approximately 110-metre precision and are not stored as raw coordinates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Local journal:&lt;/strong&gt; Entries and photos stay in device storage, with photo metadata stripped before storage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private observability:&lt;/strong&gt; Sentry traces record errors and timings without capturing prompts or completions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graceful fallback:&lt;/strong&gt; Cached briefings can be replayed without connectivity, and the deterministic quest engine can generate quests without model inference. The complete application still requires network access for some features.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Training and evaluation
&lt;/h3&gt;

&lt;p&gt;The training pipeline uses real seeded places, geographic train/test splits, and deliberately difficult cases, including missing candidates and indoor-only conditions. Candidate examples are filtered through the same validators used in production before LoRA fine-tuning.&lt;/p&gt;

&lt;p&gt;The intended evaluation measures the quantized model actually served in production, with a target of at least a ten-percentage-point improvement in full validator pass rate or a clear latency and cost advantage.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test results as of October 10, 2026
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;API tests&lt;/td&gt;
&lt;td&gt;837 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frontend tests&lt;/td&gt;
&lt;td&gt;119 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playwright tests&lt;/td&gt;
&lt;td&gt;10 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strict type checking&lt;/td&gt;
&lt;td&gt;54 files, no issues&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secret scanning&lt;/td&gt;
&lt;td&gt;No secrets found in the tree or Git history&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A custom Content Security Policy test also runs the production build in Chromium, checks all eleven routes, and verifies that the map and real OpenStreetMap tiles load.&lt;/p&gt;

&lt;h3&gt;
  
  
  The design principle
&lt;/h3&gt;

&lt;p&gt;The model is responsible for generating the adventure, not controlling the entire system. Deterministic code handles safety checks, validation, and progression, while open-weight models provide customisation and the option of on-device photo verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Everything comes back to one rule: &lt;strong&gt;the screen should be the shortest part of the experience.&lt;/strong&gt;
&lt;/h2&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. I can shape the model, not just prompt it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Sidequestmaxxing is designed around an open-weight Dungeon Master that can be adapted using LoRA, quantised into GGUF, and served through &lt;code&gt;llama.cpp&lt;/code&gt;. This gives me control over the model weights and deployment, rather than tying the experience to a single hosted LLM provider. The same validation rules are shared across training, evaluation, and production to keep quest generation grounded and safe.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Privacy is built into the architecture.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On supported devices, Gemma 3n can verify proof photos directly in the browser using WebGPU. The API has no image-upload endpoint, so photos do not need to leave the device for verification. This makes local inference a practical privacy advantage, not just a promise in a privacy policy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The model can run without a GPU.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quantised GGUF models make CPU inference possible, including on modest hardware. With a configurable inference endpoint, the backend can also be pointed at another compatible model server instead of being permanently tied to one provider.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Open data makes neighbourhood exploration more accessible.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;OpenStreetMap provides real-world place data, Open-Meteo supplies weather information, and BGE-small enables local CPU-based embeddings. These components reduce dependence on proprietary data providers and per-request AI fees, while making it easier to inspect and adapt the system.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Open innovation matters because it gives us control over how AI is customised, where it runs, and how personal data is handled.&lt;/strong&gt; For Sidequestmaxxing, those freedoms help build an experience that gets people off their phones and into the world.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Best Use of Render
&lt;/h3&gt;

&lt;p&gt;All three services are deployed through a single &lt;code&gt;render.yaml&lt;/code&gt; Blueprint, as described in System Architecture.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-web&lt;/code&gt;&lt;/strong&gt;: Hosts the static Progressive Web App (PWA).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-api&lt;/code&gt;&lt;/strong&gt;: Runs the FastAPI backend in Docker. It uses the repository root as its build context so it can access the &lt;code&gt;/prompts&lt;/code&gt; directory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-llm&lt;/code&gt;&lt;/strong&gt;: Runs as a private service with no public URL and a 10 GB persistent disk mounted at &lt;code&gt;/data&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The API does not hard-code the model service's host or port. Instead, it reads &lt;code&gt;LLM_HOSTPORT&lt;/code&gt;, which Render injects for service-to-service communication with &lt;code&gt;sqm-llm&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;On its first boot, the LLM service downloads the GGUF model specified by &lt;code&gt;MODEL_URL&lt;/code&gt; to the persistent disk and verifies its integrity against &lt;code&gt;MODEL_SHA256&lt;/code&gt;. On subsequent boots, the existing model file is reused, avoiding repeated downloads.&lt;/p&gt;

&lt;p&gt;The entire deployment is managed through one Blueprint, with the configuration and deployment process documented end to end.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Gemma
&lt;/h3&gt;

&lt;p&gt;Gemma 3n runs entirely in the browser using WebGPU and MediaPipe's LLM Inference task. Its &lt;code&gt;.litertlm&lt;/code&gt; model file is stored in the player's Origin Private File System (OPFS), so the model and its inference process remain on the player's device.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Local inference:&lt;/strong&gt; The model runs in the browser rather than on the backend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No image uploads:&lt;/strong&gt; The API has no image-upload endpoint, &lt;code&gt;UploadFile&lt;/code&gt;, &lt;code&gt;File()&lt;/code&gt; handler, or multipart upload handler under &lt;code&gt;apps/api/app&lt;/code&gt;. A test explicitly verifies this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Capability detection:&lt;/strong&gt; The application checks for &lt;code&gt;navigator.gpu&lt;/code&gt;, a compatible GPU adapter, and OPFS support before attempting local inference.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graceful fallback:&lt;/strong&gt; Devices that cannot run the model can use check-in or honour-mode verification instead of encountering a broken experience.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This design keeps player photos off the server while allowing the application to work on devices that do not support local inference.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of ElevenLabs
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;POST /v1/tts&lt;/code&gt; endpoint acts as a server-side proxy for ElevenLabs text-to-speech. The browser never receives the API key, and clients cannot submit arbitrary text for synthesis. The endpoint only accepts the briefing or recap already stored for the device's quest.&lt;/p&gt;

&lt;p&gt;The endpoint uses three safeguards:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Character limit:&lt;/strong&gt; Caps the amount of text processed per request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-device rate limit:&lt;/strong&gt; Restricts the number of requests a device can make per hour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Global daily budget:&lt;/strong&gt; Tracks character usage in MongoDB using the &lt;code&gt;usage&lt;/code&gt; collection and keys in the format &lt;code&gt;tts:YYYY-MM-DD&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the API key is missing or the daily character budget has been exhausted, &lt;code&gt;/v1/tts&lt;/code&gt; returns &lt;code&gt;503 Service Unavailable&lt;/code&gt;. The browser then uses its built-in &lt;code&gt;speechSynthesis&lt;/code&gt; API to read the same text.&lt;/p&gt;

&lt;p&gt;All four Dungeon Master personas remain available through this fallback, although the voice quality may be lower. The experience remains functional even when the external TTS service is unavailable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of MongoDB Atlas
&lt;/h3&gt;

&lt;p&gt;MongoDB Atlas supports semantic place discovery, repeat-quest detection, geospatial queries, memory expiration, and duplicate-completion prevention.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Vector Search indexes&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;places_vec&lt;/code&gt;&lt;/strong&gt;: A 384-dimensional cosine-similarity index used by the Scout for semantic place search. Results can be filtered by &lt;code&gt;cell5&lt;/code&gt; and &lt;code&gt;kind&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memories_vec&lt;/code&gt;&lt;/strong&gt;: A 384-dimensional cosine-similarity index used for repeat-quest checks, with filtering by &lt;code&gt;device_id&lt;/code&gt; and &lt;code&gt;ts&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Additional indexes&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Geospatial index:&lt;/strong&gt; A &lt;code&gt;2dsphere&lt;/code&gt; index on &lt;code&gt;places.loc&lt;/code&gt; supports location-based queries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TTL index:&lt;/strong&gt; Automatically expires documents in &lt;code&gt;memories&lt;/code&gt; after 90 days.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unique completion index:&lt;/strong&gt; A unique index on &lt;code&gt;completions.quest_id&lt;/code&gt; enforces one completion per quest at the database level, preventing concurrent requests from creating duplicate completions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These indexes combine semantic retrieval, geographic relevance, memory management, and database-enforced consistency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Sentry Agent Tracing
&lt;/h3&gt;

&lt;p&gt;Sentry tracing provides end-to-end visibility into the quest-generation pipeline without recording sensitive quest content.&lt;/p&gt;

&lt;p&gt;Each stage has its own span under a single &lt;code&gt;quests.generate&lt;/code&gt; transaction. These stages include the Context Builder, Scout, memory check, Quest Master, validators, and fallback engine. Model calls emit &lt;code&gt;gen_ai.invoke_agent&lt;/code&gt; and &lt;code&gt;gen_ai.chat&lt;/code&gt; spans with token counts and latency information.&lt;/p&gt;

&lt;p&gt;Three tags make requests searchable without requiring anyone to inspect span contents:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;source&lt;/code&gt;&lt;/strong&gt;: Identifies the generation path, such as &lt;code&gt;llm&lt;/code&gt;, &lt;code&gt;fallback&lt;/code&gt;, &lt;code&gt;empty&lt;/code&gt;, or &lt;code&gt;cached:*&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;retry&lt;/code&gt;&lt;/strong&gt;: Indicates whether the request was retried (&lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;validator_failures&lt;/code&gt;&lt;/strong&gt;: Records &lt;code&gt;none&lt;/code&gt; when validation succeeds, or the failing validator IDs, such as &lt;code&gt;V2&lt;/code&gt; or &lt;code&gt;V5,V9&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prompts and completions are deliberately excluded from traces. With &lt;code&gt;send_default_pii=False&lt;/code&gt; and no draft text attached to spans, a trace can reveal that validator &lt;code&gt;V2&lt;/code&gt; failed and how long a model call took, without exposing the generated quest or the player's surroundings.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Tinker
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;training/&lt;/code&gt; directory contains a complete, tested pipeline for LoRA supervised fine-tuning (SFT) and teacher-sample generation.&lt;/p&gt;

&lt;p&gt;The workflow includes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Context generation:&lt;/strong&gt; Samples contexts from the real seeded places and splits the data by area so that the test set contains unseen locations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Teacher sampling:&lt;/strong&gt; Generates draft quests through Tinker and filters them using the same validator code used in production, imported directly rather than duplicated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dataset preparation:&lt;/strong&gt; Deduplicates samples and balances categories, rarity levels, and Dungeon Master personas.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LoRA fine-tuning:&lt;/strong&gt; Sweeps three learning rates over three epochs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model selection:&lt;/strong&gt; Selects the model using validation pass rate rather than training loss alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reproducibility:&lt;/strong&gt; Uses fixed seeds and records run metadata in &lt;code&gt;training/runs/&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the complete methodology and evaluation results, see Training and Evaluation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of SerpApi
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;scripts/fetch_events.py&lt;/code&gt; script uses SerpApi to retrieve Google's &lt;code&gt;events_results&lt;/code&gt; and incorporate relevant local events into the quest-generation pipeline.&lt;/p&gt;

&lt;p&gt;Each listing passes through several filtering and validation stages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Input sanitization:&lt;/strong&gt; Sanitizes event strings using the same sanitizer that protects against unsafe OpenStreetMap place names.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time validation:&lt;/strong&gt; Runs each listing through validator V13, which rejects events that have already ended, start after the quest window, or have times that cannot be parsed without guessing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Content filtering:&lt;/strong&gt; Applies the project's existing banned-term checks and filters out indoor or ticketed events.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The event-fetching and validation pipeline is covered by three test files:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;test_fetch_events.py&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;test_serp_events.py&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;test_event_time.py&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This ensures that event-based quests use sanitized listings with valid, relevant timing information.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;The screen is the shortest part. Go live outside and Touch Grass :p.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
    </item>
    <item>
      <title>Tiny Me: A Little Version of Me That Lives on My Mom’s Laptop</title>
      <dc:creator>Aarushi Tandon</dc:creator>
      <pubDate>Sun, 04 Oct 2026 14:07:46 +0000</pubDate>
      <link>https://dev.to/aarushitandon/tinyme-a-little-version-of-me-that-lives-on-my-friends-and-familys-laptops-597e</link>
      <guid>https://dev.to/aarushitandon/tinyme-a-little-version-of-me-that-lives-on-my-friends-and-familys-laptops-597e</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-weekend-2026-10-01"&gt;Hacktoberfest Weekend Challenge: Build for a Friend&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;I built &lt;strong&gt;Tiny Me for my mom.&lt;/strong&gt; I don't live at home anymore, so I can't always be there when she gets stuck on her laptop. When I'm home, she can simply call out, "Aarushi, come look at this." But when I'm away, even something as simple as finding a downloaded file, opening the right setting, or figuring out which button to press can become frustrating. My friend &lt;strong&gt;Aditi&lt;/strong&gt; made me realize this isn't just a problem for parents. She's incredibly smart and is studying law, but being good at law doesn't mean she should have to be good at navigating confusing computer interfaces. &lt;strong&gt;Being good at what you do shouldn't require being good at computers.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So I wondered: &lt;strong&gt;what if a small version of me could live on their laptop?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's &lt;strong&gt;Tiny Me&lt;/strong&gt;, a local desktop helper for people who know exactly what they want to accomplish but don't always know how to make their computer do it. Press a hotkey and say what you're trying to do in plain language, like "Find my train ticket for tomorrow" or "Show me how to search for trains." Tiny Me understands what's on the screen, identifies the next control, and draws a circle around exactly where to click. &lt;strong&gt;You click it.&lt;/strong&gt; It checks whether the step worked, then guides you to the next one.&lt;/p&gt;

&lt;p&gt;It's like having a tech savvy friend sitting beside you saying, &lt;strong&gt;"Yep. Click this."&lt;/strong&gt; Except that friend lives on your laptop.&lt;/p&gt;

&lt;p&gt;But Tiny Me is deliberately &lt;strong&gt;not&lt;/strong&gt; an autonomous computer use agent. &lt;strong&gt;It is designed to teach, not take over.&lt;/strong&gt; The goal isn't to let AI use your computer for you. It's to make your computer understandable. The user stays in control and performs the actions themselves.&lt;/p&gt;

&lt;p&gt;Tiny Me also knows when to stop. If a workflow reaches a password, OTP, card number, payment, or other sensitive input, it simply says, &lt;strong&gt;"This part is yours. I'll wait."&lt;/strong&gt; It doesn't type, click, or attempt to bypass that boundary.&lt;/p&gt;

&lt;p&gt;Everything runs locally on the user's Windows machine. Screen understanding, control detection, and the language model stay on the device, so screen content doesn't need to leave the laptop.&lt;/p&gt;

&lt;p&gt;At its heart, Tiny Me comes from a very simple problem: &lt;strong&gt;I can't always be there to help my mom. So I built something that can.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who it is for.&lt;/strong&gt; My mom, my friend Aditi, and anyone who is brilliant at what they do but shouldn't have to become a computer expert just to use their own laptop. &lt;br&gt;
Everything runs on the user's Windows machine. Reading the screen, finding controls and the language model are all local. No screen content leaves the laptop.&lt;/p&gt;
&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/aarushitandon0/tinyme/blob/main/SS/07_demo_walkthrough.mp4" rel="noopener noreferrer"&gt;Watch the full demo walkthrough (MP4)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The demo runs against a local booking website included in the repo (&lt;code&gt;demo_site/&lt;/code&gt;), and shows:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Press &lt;code&gt;Ctrl+Alt+H&lt;/code&gt; and type a goal.&lt;/li&gt;
&lt;li&gt;Tiny Me circles the next control and explains the step in one line.&lt;/li&gt;
&lt;li&gt;The user clicks it.&lt;/li&gt;
&lt;li&gt;Tiny Me checks whether the step worked.&lt;/li&gt;
&lt;li&gt;The circle moves to the next control.&lt;/li&gt;
&lt;li&gt;At a sign-in page, Tiny Me stops and hands control back.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+P&lt;/code&gt; stops it at any time.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3nk10r600l0d29fvb3y9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3nk10r600l0d29fvb3y9.png" alt="TinyMe home screen" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Press the hotkey, then describe what you're trying to do in plain language.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklwtlvvxc7g73mjmt56p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklwtlvvxc7g73mjmt56p.png" alt="TinyMe guiding the user step by step" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Tiny Me circles the control and waits for the user to click it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9wqjwnrzt2ypg3j2otc2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9wqjwnrzt2ypg3j2otc2.png" alt="TinyMe handing control back at a sensitive step" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;At a sensitive step like sign-in, Tiny Me stops instead of interacting.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1cnbntmi726bbdahm3j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1cnbntmi726bbdahm3j.png" alt="TinyMe example goals" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Example goals.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lem5o415f33jh8egadd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lem5o415f33jh8egadd.png" alt="TinyMe helping find a file" width="800" height="507"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Finding a downloaded file, one circled click at a time.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;All screenshots are in the repo's &lt;a href="https://github.com/aarushitandon0/tinyme/tree/main/SS" rel="noopener noreferrer"&gt;SS folder&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/aarushitandon0" rel="noopener noreferrer"&gt;
        aarushitandon0
      &lt;/a&gt; / &lt;a href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;
        tinyme
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Tiny Me&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;A Windows desktop assistant that watches one screen, decides what to click next, and draws a circle around it.&lt;/p&gt;
&lt;p&gt;Tiny Me is built for a single non-technical user. She presses a hotkey, types what she wants in plain words, and the application either &lt;strong&gt;shows her how&lt;/strong&gt; by circling the exact control to click and waiting until she clicks it, or &lt;strong&gt;does it for her&lt;/strong&gt; while she watches. At any screen involving a password, a one-time code, a card number or a payment, it stops and hands control back.&lt;/p&gt;
&lt;p&gt;Everything runs on the local machine. Optical character recognition, UI inspection and the language model are all local processes. No screen content is transmitted. The application functions with networking disabled.&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Table of contents&lt;/h2&gt;
&lt;/div&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#1-design-constraints" rel="noopener noreferrer"&gt;Design constraints&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#2-system-architecture" rel="noopener noreferrer"&gt;System architecture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#3-the-per-step-pipeline" rel="noopener noreferrer"&gt;The per-step pipeline&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#4-module-reference" rel="noopener noreferrer"&gt;Module reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#5-the-element-list-the-models-entire-world" rel="noopener noreferrer"&gt;The element list&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#6-the-model-contract" rel="noopener noreferrer"&gt;The model contract&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#7-deterministic-success-checking" rel="noopener noreferrer"&gt;Deterministic success checking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#8-the-guard" rel="noopener noreferrer"&gt;The guard&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#9-coordinate-systems-and-dpi" rel="noopener noreferrer"&gt;Coordinate systems and DPI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#10-threading-model" rel="noopener noreferrer"&gt;Threading model&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/aarushitandon0/tinyme#11-the-overlay" rel="noopener noreferrer"&gt;The&lt;/a&gt;…&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;github.com/aarushitandon0/tinyme&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How to Run
&lt;/h3&gt;

&lt;p&gt;Requires Windows 10 (version 2004 or later) or Windows 11, Python 3.11, and &lt;a href="https://ollama.com" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/aarushitandon0/tinyme
&lt;span class="nb"&gt;cd &lt;/span&gt;tinyme
python &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv
.venv&lt;span class="se"&gt;\S&lt;/span&gt;cripts&lt;span class="se"&gt;\a&lt;/span&gt;ctivate
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="s2"&gt;".[dev]"&lt;/span&gt;
playwright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
ollama pull gemma4:e2b
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To run the demo, open two terminals in the repository folder.&lt;/p&gt;

&lt;p&gt;Terminal 1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; http.server 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Terminal 2:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; app.main
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;TinyMe starts with no visible window.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+H&lt;/code&gt; opens the prompt. Type a goal and choose "Show me how" or "Do it for me."&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+P&lt;/code&gt; stops TinyMe at any time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The demo targets &lt;code&gt;http://localhost:8000/demo_site/index.html&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Optional settings go in a &lt;code&gt;.env&lt;/code&gt; file at the repo root: &lt;code&gt;TINYME_MODEL&lt;/code&gt; (Ollama model tag), &lt;code&gt;TINYME_LANGUAGE&lt;/code&gt;, and &lt;code&gt;TINYME_TELEMETRY&lt;/code&gt; (off unless set to &lt;code&gt;1&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;The core idea: &lt;strong&gt;the language model chooses a control, never a coordinate.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tiny Me builds a numbered list of the on-screen controls, and asks the model to pick one number from it. The model answers &lt;code&gt;17&lt;/code&gt;, not &lt;code&gt;click(834, 492)&lt;/code&gt;. The app can then check that &lt;code&gt;17&lt;/code&gt; is a real, interactable control. A model that generates pixel coordinates can produce plausible but wrong ones. A number from a list the app built can be validated.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The model proposes. The app decides.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Success is checked in code.&lt;/strong&gt; After each step, a watcher polls the screen about once per second and checks window titles or classes, visible text, pixel changes, or browser structure. The model can suggest which check to run, but Python runs it. I deliberately don't ask the model "did that work?" A model judging its own action can easily talk itself into success.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The safety guard runs before the model.&lt;/strong&gt; Before every model call and every automated action, the guard checks for sensitive screens. If one is present, nothing is sent to the model and nothing is clicked or typed. In the browser, the guard uses page structure, so a password field counts as a password field whatever its label says. Screenshots stay in memory and are never written to disk.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Everything runs locally:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma 4 (&lt;code&gt;gemma4:e2b&lt;/code&gt;)&lt;/strong&gt; via &lt;strong&gt;Ollama&lt;/strong&gt;, for choosing the control&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RapidOCR&lt;/strong&gt; on &lt;strong&gt;ONNX Runtime&lt;/strong&gt;, for screen text&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Windows UI Automation&lt;/strong&gt;, for discovering controls&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python&lt;/strong&gt; for orchestration, guard, validation and execution&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Playwright&lt;/strong&gt; for the browser demo&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen3.5-4B fine-tuned on Tinker&lt;/strong&gt; as a swappable alternative picker (see below)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The Tinker experiment
&lt;/h3&gt;

&lt;p&gt;I fine-tuned &lt;code&gt;Qwen/Qwen3.5-4B&lt;/code&gt; with LoRA (rank 16, 3 epochs, 80 training examples, about $0.17) on Tinker. The held-out test was the whole &lt;code&gt;demo_site&lt;/code&gt; app, which the model never saw in training, with 22 labelled examples.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;System&lt;/th&gt;
&lt;th&gt;Correct control&lt;/th&gt;
&lt;th&gt;Valid answer format&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Untuned Qwen3.5-4B&lt;/td&gt;
&lt;td&gt;0/22 (0%)&lt;/td&gt;
&lt;td&gt;0/22 (0%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fine-tuned Qwen3.5-4B&lt;/td&gt;
&lt;td&gt;18/22 (81%)&lt;/td&gt;
&lt;td&gt;22/22 (100%)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The untuned model's 0/22 is mostly a format failure: its replies didn't parse. Fine-tuning fixed the format but not the choice. In 19 of 22 examples the tuned model still picked the wrong element. For example, asked to "pay for my ticket," it picked a heading when the right answer was "not on this screen."&lt;/p&gt;

&lt;p&gt;That's a modest result, and it's a useful failure. It says the next problem is teaching the model when to act, which control to pick, and when no safe action exists, not only following the output format.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;For Tiny Me, local and open models aren't a preference; they're part of the product. If I'm asking my friends and my mom to let software look at their screen, I don't want that to depend on trusting a closed hosted API.&lt;/p&gt;

&lt;p&gt;With open weights:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the model runs on the laptop, and screen content stays there,&lt;/li&gt;
&lt;li&gt;sensitive controls are blocked before any model call,&lt;/li&gt;
&lt;li&gt;the code is inspectable,&lt;/li&gt;
&lt;li&gt;and I could swap the model, fine-tune it, and measure where it fails.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A closed API would have hidden the 0/22 and the 3/22 behind a single "it works" demo. Open weights let me train a model, test it on an app it hadn't seen, and report that it mostly still fails.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Best Use of Gemma&lt;/strong&gt;&lt;br&gt;
Gemma 4 (&lt;code&gt;gemma4:e2b&lt;/code&gt;) is the default model, run fully locally through Ollama. It reads the numbered list of on-screen controls and picks one. Nothing in the runtime path is a hosted API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Best Use of Tinker&lt;/strong&gt;&lt;br&gt;
I fine-tuned Qwen3.5-4B on Tinker and tested it on a whole held-out app. Valid answer format went from 0/22 to 22/22, and correct control selection from 0/22 to 18/22. &lt;/p&gt;

&lt;h2&gt;
  
  
  What's Next?
&lt;/h2&gt;

&lt;p&gt;The next step is to put Tiny Me in front of the people I built it for: friends, family, and my mom. I want to find out where they hesitate, which instructions make sense right away, what confuses them, whether they remember the steps next time, where Tiny Me should step back, and most importantly, &lt;strong&gt;after using Tiny Me, do they need it less?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's the metric I care about. I don't want to build something that makes people feel they can't use their own computer without it. I want the opposite: a small version of your tech-savvy friend that says "Here. Click this." and, eventually, you don't need to ask.&lt;/p&gt;

&lt;h1&gt;
  
  
  devchallenge #weekendchallenge #hf26challenge
&lt;/h1&gt;

</description>
      <category>devchallenge</category>
      <category>weekendchallenge</category>
      <category>hf26challenge</category>
    </item>
  </channel>
</rss>
