<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Aarushi Tandon</title>
    <description>The latest articles on DEV Community by Aarushi Tandon (@aarushitandon).</description>
    <link>https://dev.to/aarushitandon</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4024520%2F7b3892b3-0fd5-47fb-bffa-f19ff2a1318b.jpg</url>
      <title>DEV Community: Aarushi Tandon</title>
      <link>https://dev.to/aarushitandon</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aarushitandon"/>
    <language>en</language>
    <item>
      <title>What the Doc?!: A Local Gemma That Fixes Messy Word Formatting While You Rest</title>
      <dc:creator>Aarushi Tandon</dc:creator>
      <pubDate>Sun, 25 Oct 2026 19:57:07 +0000</pubDate>
      <link>https://dev.to/aarushitandon/what-the-doc-a-local-gemma-that-fixes-messy-word-formatting-while-you-rest-42b9</link>
      <guid>https://dev.to/aarushitandon/what-the-doc-a-local-gemma-that-fixes-messy-word-formatting-while-you-rest-42b9</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week3-2026-10-19"&gt;Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frk5uzj408u3454foihg2.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frk5uzj408u3454foihg2.gif" alt="What the Doc?! intro: relaxing in a hammock while the laptop does the work" width="600" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Have you ever made one tiny change in a Word document, and the whole document decided to rearrange itself?&lt;/p&gt;

&lt;p&gt;A page break shows up out of nowhere. Paragraphs shift. One paragraph is suddenly centred when everything else is justified. And there is a big blank gap that you cannot delete, no matter what you click.&lt;/p&gt;

&lt;p&gt;Fixing the formatting ends up taking longer than writing the document. I know this because it happens to me every time I write a research paper.&lt;/p&gt;

&lt;p&gt;So for Sloth Day, I built something that does that annoying work for me.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What the Doc?!&lt;/strong&gt; is a privacy-first formatting assistant for Word (&lt;code&gt;.docx&lt;/code&gt;) files. It uses &lt;strong&gt;Gemma, an open-weight model, running locally on my own laptop through Ollama&lt;/strong&gt; to find out &lt;em&gt;why&lt;/em&gt; your formatting looks wrong, and then proposes targeted fix.&lt;/p&gt;

&lt;p&gt;Here is all you have to do:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Upload&lt;/strong&gt; your document.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Point&lt;/strong&gt; at the problem. Pick one paragraph, or check the whole document.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Describe&lt;/strong&gt; it in your own words, like "there's a weird gap here" or "this paragraph doesn't line up".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wait.&lt;/strong&gt; Gemma looks at the formatting evidence and works out the cause.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compare&lt;/strong&gt; the before and after previews side by side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Accept&lt;/strong&gt; the change and download the fixed file.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpeifnuzupaaawi4lbcwp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpeifnuzupaaawi4lbcwp.png" alt="A small paper workshop: messy documents go in on one belt and come out tidy on the other" width="800" height="501"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The idea is simple: I want to write my research paper, not figuring out why changing one thing messed up the rest of the document. Hacktoberfest's &lt;em&gt;Slow and Steady&lt;/em&gt; theme seemed like the perfect excuse to automate something I've always hated doing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this is harder than it looks
&lt;/h3&gt;

&lt;p&gt;The problem with Word formatting is that &lt;strong&gt;you can see the symptom, but you can't see the cause.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A gap above a heading could come from five different things in the file: spacing typed directly on that paragraph, spacing inherited from a style, a "keep with next" setting, a manual page break, or a new section starting. They all look exactly the same on the page. Each one is fixed from a different menu. So you end up clicking around until something works, and often you make it worse.&lt;/p&gt;

&lt;p&gt;What the Doc?! reads the actual settings behind the paragraph, including where each value came from, and hands that evidence to Gemma. Gemma's job is to say which setting is the real cause and what one change would fix it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who gets to be lazier
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Students&lt;/strong&gt; writing reports, theses and research papers the night before they're due.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anyone&lt;/strong&gt; who has ever opened a document someone else formatted and given up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Me&lt;/strong&gt;, mostly. I wanted my evenings back.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  What it saves
&lt;/h3&gt;

&lt;p&gt;It saves you the investigation. The slow part of fixing formatting is not clicking the right button. It's &lt;em&gt;finding out&lt;/em&gt; which button is the right one. That's the part you hand over.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;Here is the full walkthrough, with two real examples:&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/ZTS-tGmwjHE" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;h3&gt;
  
  
  Example 1: The unwanted page break
&lt;/h3&gt;

&lt;p&gt;The first document has a page break that has pushed part of the text onto the next page and left an awkward empty space behind. The lines should flow together, but Word has split them.&lt;/p&gt;

&lt;p&gt;I choose &lt;strong&gt;the entire document&lt;/strong&gt;, describe the problem, and let Gemma investigate. It finds the cause, the app checks that the fix is safe, and then it shows me the before and after side by side. On the left, the original with the gap. On the right, the text flowing properly again.&lt;/p&gt;

&lt;p&gt;None of the actual words are changed. Only the one formatting setting that caused the problem. I click &lt;strong&gt;Accept Changes&lt;/strong&gt;, download the fixed file, and I'm done.&lt;/p&gt;

&lt;h3&gt;
  
  
  Example 2: The paragraph that won't line up
&lt;/h3&gt;

&lt;p&gt;The second document looks fine, except for one paragraph that is aligned differently from everything around it.&lt;/p&gt;

&lt;p&gt;This time I already know where the problem is, so I select &lt;strong&gt;just that paragraph&lt;/strong&gt; with the Text option. Gemma compares its settings with the paragraphs around it, sees that this one has an alignment the others don't, and suggests putting it back in line with the rest.&lt;/p&gt;

&lt;p&gt;Again, I check the before and after, click &lt;strong&gt;Accept Changes&lt;/strong&gt;, and download. Two formatting headaches gone, and I barely moved.&lt;/p&gt;

&lt;h3&gt;
  
  
  Try it on the same files
&lt;/h3&gt;

&lt;p&gt;The repo has nine prepared test documents in &lt;a href="https://github.com/aarushitandon0/what-the-doc/tree/main/test-documents" rel="noopener noreferrer"&gt;&lt;code&gt;test-documents/&lt;/code&gt;&lt;/a&gt;, each with a known problem and a suggested complaint to type. Some of them are meant to be refused, so you can check that the tool knows when to leave things alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/aarushitandon0" rel="noopener noreferrer"&gt;
        aarushitandon0
      &lt;/a&gt; / &lt;a href="https://github.com/aarushitandon0/what-the-doc" rel="noopener noreferrer"&gt;
        what-the-doc
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;p&gt;
  &lt;a rel="noopener noreferrer" href="https://github.com/aarushitandon0/what-the-doc/frontend/public/assets/workshop.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Faarushitandon0%2Fwhat-the-doc%2FHEAD%2Ffrontend%2Fpublic%2Fassets%2Fworkshop.png" alt="A small paper workshop: documents going in on one belt, coming out tidy on the other" width="460"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;What the Doc?!&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;&lt;b&gt;Point at the formatting problem. Get a targeted, reviewable repair.&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;
  &lt;a href="https://youtu.be/ZTS-tGmwjHE" rel="nofollow noopener noreferrer"&gt;Watch the demo&lt;/a&gt; ·
  &lt;a href="https://github.com/aarushitandon0/what-the-doc#the-problem" rel="noopener noreferrer"&gt;The problem&lt;/a&gt; ·
  &lt;a href="https://github.com/aarushitandon0/what-the-doc#system-architecture" rel="noopener noreferrer"&gt;Architecture&lt;/a&gt; ·
  &lt;a href="https://github.com/aarushitandon0/what-the-doc#run-it-locally" rel="noopener noreferrer"&gt;Run it locally&lt;/a&gt; ·
  &lt;a href="https://github.com/aarushitandon0/what-the-doc/CAPABILITIES.md" rel="noopener noreferrer"&gt;What it repairs&lt;/a&gt; ·
  &lt;a href="https://github.com/aarushitandon0/what-the-doc/API.md" rel="noopener noreferrer"&gt;HTTP API&lt;/a&gt;
&lt;/p&gt;




&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;The problem&lt;/h2&gt;
&lt;/div&gt;

&lt;p&gt;You change one line in a Word document and the whole thing rearranges itself. A page
break appears from nowhere. Paragraphs shift. A heading floats half a page from the
text it belongs to. There is a blank gap you cannot delete, and none of the settings
you click seems to be the one responsible.&lt;/p&gt;
&lt;p&gt;The symptom is visible; the cause is not. A gap above a heading can be direct spacing
a value inherited from a style, a &lt;code&gt;keep_with_next&lt;/code&gt; chain, an explicit break element or a
section start: five different things in the file that look identical on the page, each
fixed from a different dialog. So you work through the ribbon by elimination, and…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/aarushitandon0/what-the-doc" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;The repo includes the Python backend, the React frontend, 259 backend tests and 24 frontend tests, plus nine test documents and detailed docs (&lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/START_HERE.md" rel="noopener noreferrer"&gt;START_HERE.md&lt;/a&gt;, &lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/CAPABILITIES.md" rel="noopener noreferrer"&gt;CAPABILITIES.md&lt;/a&gt;, &lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/API.md" rel="noopener noreferrer"&gt;API.md&lt;/a&gt;, &lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/DEPLOYMENT.md" rel="noopener noreferrer"&gt;DEPLOYMENT.md&lt;/a&gt;).&lt;/p&gt;

&lt;h3&gt;
  
  
  How to run it yourself
&lt;/h3&gt;

&lt;p&gt;I haven't put this on a public server, so here's how to run the whole thing on your own computer. Everything runs locally and nothing costs money.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What you need&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.11 or newer&lt;/li&gt;
&lt;li&gt;Node.js 20 or newer&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ollama.com" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.libreoffice.org" rel="noopener noreferrer"&gt;LibreOffice&lt;/a&gt; (for the before and after previews)&lt;/li&gt;
&lt;li&gt;About 8 GB of RAM for the model. No GPU needed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Option A: Run it step by step&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Get the code&lt;/span&gt;
git clone https://github.com/aarushitandon0/what-the-doc.git
&lt;span class="nb"&gt;cd &lt;/span&gt;what-the-doc

&lt;span class="c"&gt;# 2. Set up Python&lt;/span&gt;
python &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv
&lt;span class="c"&gt;# Windows:      .venv\Scripts\Activate.ps1&lt;/span&gt;
&lt;span class="c"&gt;# macOS/Linux:  source .venv/bin/activate&lt;/span&gt;
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements.txt

&lt;span class="c"&gt;# 3. Download the open-weight Gemma model (one time)&lt;/span&gt;
ollama pull gemma4:e2b
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start the backend in one terminal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Windows PowerShell&lt;/span&gt;
&lt;span class="nv"&gt;$env&lt;/span&gt;:GEMMA_MODEL &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"gemma4:e2b"&lt;/span&gt;
python api.py &lt;span class="nt"&gt;--cors&lt;/span&gt; http://localhost:5173

&lt;span class="c"&gt;# macOS/Linux&lt;/span&gt;
&lt;span class="nv"&gt;GEMMA_MODEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;gemma4:e2b python api.py &lt;span class="nt"&gt;--cors&lt;/span&gt; http://localhost:5173
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Start the frontend in a second terminal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cd &lt;/span&gt;frontend
npm ci
npm run dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open &lt;strong&gt;&lt;a href="http://localhost:5173" rel="noopener noreferrer"&gt;http://localhost:5173&lt;/a&gt;&lt;/strong&gt;, upload a file from &lt;code&gt;test-documents/&lt;/code&gt;, and try it.&lt;/p&gt;

&lt;p&gt;LibreOffice is found automatically in the usual install locations. If it isn't, set &lt;code&gt;SOFFICE_PATH&lt;/code&gt; to your &lt;code&gt;soffice&lt;/code&gt; file. You can check that everything is connected at &lt;code&gt;http://127.0.0.1:8000/api/health&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Option B: One Docker container&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The repo includes a &lt;code&gt;Dockerfile&lt;/code&gt; that packs everything into one container: Ollama, the Gemma model, LibreOffice, the API and the interface.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker build &lt;span class="nt"&gt;-t&lt;/span&gt; what-the-doc &lt;span class="nb"&gt;.&lt;/span&gt;
docker run &lt;span class="nt"&gt;-p&lt;/span&gt; 10000:10000 what-the-doc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open &lt;strong&gt;&lt;a href="http://localhost:10000" rel="noopener noreferrer"&gt;http://localhost:10000&lt;/a&gt;&lt;/strong&gt;. The first build takes a while, because it downloads the model (about 4.6 GB).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Option C: Host it on Render&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you do want it online, the repo has a ready-made &lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/render.yaml" rel="noopener noreferrer"&gt;&lt;code&gt;render.yaml&lt;/code&gt;&lt;/a&gt;. In Render, choose &lt;strong&gt;New, then Blueprint&lt;/strong&gt;, pick the repo, and it sets up the same Docker container with a health check. It needs the 8 GB plan because of the model. The full steps are in &lt;a href="https://github.com/aarushitandon0/what-the-doc/blob/main/DEPLOYMENT.md" rel="noopener noreferrer"&gt;DEPLOYMENT.md&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A note on speed:&lt;/strong&gt; Gemma runs on the CPU, so one diagnosis takes about one to two minutes. That's normal. It's not stuck. It's just being a sloth.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;Here is a short clip of the repair flow from start to finish:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdaa8cc6yvs3mapynz122.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdaa8cc6yvs3mapynz122.gif" alt="What the Doc?! demo clip" width="600" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The open-source pieces
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Piece&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Gemma (&lt;code&gt;gemma4:e2b&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The open-weight model that reads the evidence and decides the cause. 4.6B parameters, quantised, runs on a normal laptop CPU.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Ollama&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Runs Gemma locally and serves it on &lt;code&gt;localhost&lt;/code&gt;. No account, no API key.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;LibreOffice&lt;/strong&gt; (headless)&lt;/td&gt;
&lt;td&gt;Turns the original and fixed documents into page images for the before and after previews.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;FastAPI + Uvicorn&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The small Python backend.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;React + Vite&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The interface.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;lxml, pypdfium2&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Reading and editing the Word file's XML, and turning PDFs into images.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  How it fits together
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0wwuk7kn8mz6mevjxyll.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0wwuk7kn8mz6mevjxyll.png" alt="Architecture: a React interface over a FastAPI backend, over the pipeline, over the diagnosis and repair layers, over the core modules" width="800" height="520"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The design has two halves, and they never mix:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma does the thinking.&lt;/strong&gt; It figures out &lt;em&gt;what&lt;/em&gt; is wrong and &lt;em&gt;which&lt;/em&gt; setting to change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Plain, tested code does the editing.&lt;/strong&gt; It decides if that change is allowed, makes it on a copy, and checks the result.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Gemma never touches the file. It never sees a file path, a shell or raw XML. It can only fill in a small, strict JSON form, and anything outside that form is rejected.&lt;/p&gt;

&lt;h3&gt;
  
  
  What Gemma actually sees
&lt;/h3&gt;

&lt;p&gt;A &lt;code&gt;.docx&lt;/code&gt; file is really a zip of XML. Raw XML is hard for a small model to reason about, so the app does the hard reading first.&lt;/p&gt;

&lt;p&gt;For each paragraph, it works out the &lt;strong&gt;real&lt;/strong&gt; value of every setting and &lt;strong&gt;where it came from&lt;/strong&gt;: typed directly on the paragraph, inherited from a style, or set by the document defaults.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjnzqs6jksswpdo6q0es.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjnzqs6jksswpdo6q0es.png" alt="How each formatting value is resolved: direct formatting, then the style chain, then document defaults, then application default" width="800" height="376"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It also shows Gemma how other paragraphs with the same style are formatted. So instead of a meaningless number, Gemma sees something like: "this heading has 48pt of space above it, typed directly, while every other heading of this style gets 12pt from the style." That makes the cause easy to spot, even for a small model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three things that all look like "a page break"
&lt;/h3&gt;

&lt;p&gt;This was the trickiest part of the whole project. In a Word file, several completely different things can make text jump to a new page:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4p6rm9v78hq4p398esso.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4p6rm9v78hq4p398esso.png" alt="Four things that look like a page break: a paragraph setting, a manual break, a section start, and a break that only exists in the rendered layout" width="800" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;They look the same on screen, but each one needs a different fix. And some page breaks are there &lt;strong&gt;on purpose&lt;/strong&gt;, like one before a new chapter. So the app finds and labels each kind separately, and removing a break is always a choice the person makes, not something the tool assumes.&lt;/p&gt;

&lt;h3&gt;
  
  
  The safety checks
&lt;/h3&gt;

&lt;p&gt;Every fix goes through the same steps:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo3n0gnytvoodanstfyfx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo3n0gnytvoodanstfyfx.png" alt="The repair flow: upload, read evidence, describe the problem, local Gemma answers, schema check, plan check, patch and render, verify, awaiting review, then a human accepts or rejects" width="800" height="762"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Gemma's answer must match a strict format. If it makes up a paragraph, cites evidence it was never shown, or tries to edit something it wasn't asked about, it's rejected.&lt;/li&gt;
&lt;li&gt;The change is checked again against the allowed list of edits.&lt;/li&gt;
&lt;li&gt;The fix is applied to a &lt;strong&gt;copy&lt;/strong&gt;. Your original file is never written to.&lt;/li&gt;
&lt;li&gt;Both versions are rendered, and the app checks that every word, every bold and italic, and every other paragraph is exactly the same as before.&lt;/li&gt;
&lt;li&gt;It &lt;strong&gt;stops and waits for you.&lt;/strong&gt; Download is blocked until you click Accept.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frp26ke2gtjlga72hknc3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frp26ke2gtjlga72hknc3.png" alt="The eight job states: queued and running, then needs clarification, no supported repair, failed, or awaiting review, which a person then accepts or rejects" width="800" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Gemma is also allowed to say "I don't know" or "there's nothing here to fix." That counts as a good answer. A tool that always changes &lt;em&gt;something&lt;/em&gt; is how documents get worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;For this project, open isn't a bonus. It's the reason the idea works at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Your documents stay on your machine.&lt;/strong&gt; People format things that are private: research before it's published, job applications, medical letters, contracts, coursework. With a closed API, every one of those would be uploaded to someone else's server. With Gemma running through Ollama, the document never leaves your laptop. No account, no API key, no upload.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. It costs nothing to run.&lt;/strong&gt; A formatting fix is a small, everyday annoyance. Nobody wants to pay per request or keep a credit card on file to fix a paragraph. Local inference means you can run it as many times as you like for free.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. It runs on an ordinary laptop.&lt;/strong&gt; Gemma's small size means I didn't need a GPU or a cloud server. A student with a regular laptop can run this.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. I could see and control everything.&lt;/strong&gt; Because the model, the runtime and the code are all open, I could decide exactly what Gemma sees, force its output into a strict JSON format using Ollama's schema support, and test it repeatedly on my own machine. I can also swap in a larger Gemma later without changing anything else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Anyone can check it.&lt;/strong&gt; The tool is editing your files, so you should be able to see exactly how. Every rule, every check and every test is in the public repo.&lt;/p&gt;

&lt;p&gt;A closed API might give faster answers. But it would ask you to trust it with your documents and your money, and for a tool that edits your files, I didn't think that was a fair trade.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Lazy Payoff
&lt;/h2&gt;

&lt;p&gt;Let's compare.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fixing it yourself:&lt;/strong&gt;&lt;br&gt;
Notice the problem. Open the Paragraph dialog. Check spacing. Check indents. Check the style. Turn on hidden formatting marks. Look for a page break. Check the section settings. Change something. Watch something else move. Undo. Search online. Try again. On a bad day, this is easily &lt;strong&gt;20 to 30 minutes&lt;/strong&gt; for a single annoying gap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;With What the Doc?!:&lt;/strong&gt;&lt;br&gt;
Upload, point, type one sentence, then do nothing while Gemma works. Look at the before and after. Click Accept. That's about &lt;strong&gt;a minute of actual effort&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Yes, Gemma is slow on a CPU. A diagnosis takes a minute or two. But this is a sloth-themed week, and I think that's the best part. The tool works slowly and steadily so that &lt;strong&gt;you&lt;/strong&gt; don't have to work at all. Make a cup of tea. Stretch. Lie on the floor. Come back to a fixed document.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Gemma&lt;/strong&gt;: Gemma (&lt;code&gt;gemma4:e2b&lt;/code&gt;) is the core of the project. It runs locally through Ollama, reads the formatting evidence and decides the repair, with its output held to a strict JSON schema.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Sentry Agent Tracing&lt;/strong&gt;: The Gemma diagnosis pipeline, the background jobs and the review and download checks are instrumented with Sentry spans. It's built to send no document content at all.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;So that's &lt;strong&gt;What the Doc?!&lt;/strong&gt; A small tool for a very common frustration. Writing a document is already enough work. Fixing its mystery formatting shouldn't become a second project.&lt;/p&gt;

&lt;p&gt;The AI handles the annoying investigation. You stay in control of your file. And you get to relax while it works.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Less formatting. More resting.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Thanks for reading!&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
    </item>
    <item>
      <title>sidequestmaxxing :p (Turn Your City Into an Open-World Game)</title>
      <dc:creator>Aarushi Tandon</dc:creator>
      <pubDate>Sun, 11 Oct 2026 11:19:00 +0000</pubDate>
      <link>https://dev.to/aarushitandon/sidequestmaxxing-p-turn-your-city-into-an-open-world-game-36dd</link>
      <guid>https://dev.to/aarushitandon/sidequestmaxxing-p-turn-your-city-into-an-open-world-game-36dd</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week1-2026-10-05"&gt;Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Sidequestmaxxing&lt;/em&gt; is built around a simple idea: what if you treated real life like an open-world video game?&lt;/p&gt;

&lt;p&gt;The term comes from a Gen Z lifestyle trend of collecting side quests instead of focusing exclusively on the main quest. Your main quest might be your degree, career, or daily responsibilities. Side quests are everything else: exploring a street you've never walked down, finding a hidden café, learning to identify birds, visiting a local art installation, striking up a conversation with a stranger, or picking up a random niche skill just because it sounds interesting. In short sidequestmaxxing means treating real-life daily routines like an open-world video game, where optional hobbies, spontaneous trips, and random experiences take priority over a rigid "main storyline" of work or school. And as a GENZ myself, i gotta start sidequestmaxxing and up my lore xD&lt;/p&gt;

&lt;p&gt;These small, low-stakes adventures make everyday life feel less repetitive and give you a reason to explore beyond your usual routine. I wanted to take that idea literally and build a game engine for real life.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sidequestmaxxing turns your neighbourhood into an open-world map, with an AI Dungeon Master that gives you one real-world quest at a time.&lt;/strong&gt; It is my submission to the Touch Grass Open Source AI Challenge, built around one central principle: AI should give people more reasons to leave their screens, not more reasons to stare at them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft01pm2wklfenbtnqlmhm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft01pm2wklfenbtnqlmhm.png" alt="Intro Image" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How it works
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Set up your quest.&lt;/strong&gt; Tell Sidequestmaxxing how many minutes you have, what mood you're in, and what kinds of activities interest you. You can also configure your Dungeon Master's persona and the radius you're willing to explore.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Your AI Dungeon Master builds an adventure.&lt;/strong&gt; A small open-weight language model, designed to be LoRA fine-tuned into a Dungeon Master, generates exactly one quest at a time. The system combines your preferences with real-world context, including the time of day, weather, remaining daylight, and nearby places discovered through OpenStreetMap. The quest is grounded in actual locations, with the goal of choosing an adventure that fits your available time and is practical to complete on foot.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Listen, then leave your phone behind.&lt;/strong&gt; Your quest briefing is spoken aloud, with a choice of four Dungeon Master voices. Once you're ready, enter Walk Mode. The interface fades to near-black, making the phone less tempting to use while you are outside. The app is designed to brief you before the adventure, not demand your attention throughout it.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Complete the quest in the real world.&lt;/strong&gt; Go explore. It could be an unfamiliar street, a nearby landmark, or another small adventure based on the places around you. The point is to experience something outside your normal routine rather than complete another task on a screen.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Verify your adventure privately.&lt;/strong&gt; After returning, you can submit a proof photo for verification. On supported devices, Gemma 3n can run directly in the browser using WebGPU to assess whether the photo matches the quest. The photo stays on your device: the application has no server endpoint for uploading it. If on-device model support is unavailable, alternative verification options include a check-in or honour mode.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Turn adventures into progression.&lt;/strong&gt; Completing quests earns XP and advances seven skill categories: Explorer, Naturalist, Social, Foodie, Creative, Fitness, and Mindful. You can unlock badges, track recent adventures, and build a Chronicle of your week, turning small experiences into a campaign log of your life. A personal Journal lets you preserve notes and memories from your quests.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq7igluvau0scov5moeqr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq7igluvau0scov5moeqr.png" alt="Diagram of the six-step quest loop" width="800" height="287"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The design principle: the screen should be the shortest part of the experience
&lt;/h3&gt;

&lt;p&gt;The Touch Grass challenge is not just about building an AI application with an outdoor theme. It is about using AI to help people disengage from their devices and reconnect with the physical world.&lt;/p&gt;

&lt;p&gt;That principle shapes the entire experience. Sidequestmaxxing generates one quest rather than an endless feed of recommendations. It speaks the instructions aloud so you do not need to keep checking your screen. Walk Mode reduces visual distractions. Your Journal stores entries locally on your device, and supported on-device photo verification means your proof does not have to be uploaded to a server.&lt;/p&gt;

&lt;p&gt;Even the XP system reflects this philosophy: it includes a &lt;strong&gt;1.2x multiplier when screen usage stays below 10% of the quest duration&lt;/strong&gt;. The goal is to reward actually going outside, not spending more time interacting with the app.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why open-source AI?
&lt;/h3&gt;

&lt;p&gt;Open-weight models make it possible to shape the Dungeon Master's behaviour around a specific purpose instead of treating it as a generic chatbot. OpenStreetMap provides real-world place data, while the broader stack uses tools that make the system easier to inspect and build upon.&lt;/p&gt;

&lt;p&gt;The project also explores a second role for open-weight AI: running vision verification locally in the browser. Instead of sending a personal photo to a remote vision API, a supported device can process it locally. This is a practical example of how open models can support both customisation and privacy.&lt;/p&gt;

&lt;p&gt;Not every component is open source. Voice generation currently uses ElevenLabs, with browser speech synthesis as a fallback. The project combines open and hosted components where each serves a purpose, while keeping the core experience focused on getting people outdoors.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who is it for?
&lt;/h3&gt;

&lt;p&gt;Anyone who keeps saying they want to go outside, try something new, learn a random skill, or explore their own city, but ends up following the same routine instead.&lt;/p&gt;

&lt;p&gt;Sidequestmaxxing makes starting smaller and easier. You do not need to plan an entire day, organise a trip, or commit to a new hobby. You just need a little time and one reason to step out.&lt;/p&gt;

&lt;p&gt;The adventure does not have to be extraordinary. It just has to be something you would not normally have done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your city is the game. Your phone is just the quest log.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Built for the Touch Grass Open Source AI Challenge.&lt;/p&gt;




&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/eVARmWV4YEc" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live Demo:&lt;/strong&gt; &lt;a href="https://sqm-web.onrender.com/" rel="noopener noreferrer"&gt;Sidequestmaxxing&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  I took it outside
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Examples of sidequests:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fag4i4tdo2cee7ilaf5ni.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fag4i4tdo2cee7ilaf5ni.png" alt="Quest 1" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzwmwovkl25i5tlso6vda.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzwmwovkl25i5tlso6vda.png" alt="Quest 2" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Journal It:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwbow2e3rj6nvlnd866j0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwbow2e3rj6nvlnd866j0.png" alt="Journal" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Earn Points:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjghpuauxuqkgr5jsf57.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgjghpuauxuqkgr5jsf57.png" alt="Earn Points" width="592" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/aarushitandon0" rel="noopener noreferrer"&gt;
        aarushitandon0
      &lt;/a&gt; / &lt;a href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;
        sidequestmaxxing
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div&gt;
&lt;a rel="noopener noreferrer" href="https://github.com/aarushitandon0/sidequestmaxxing/apps/web/public/icon.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Faarushitandon0%2Fsidequestmaxxing%2FHEAD%2Fapps%2Fweb%2Fpublic%2Ficon.png" width="96" alt="Sidequestmaxxing logo: a parchment-coloured ridgeline under a gold sun, with a dotted trail running across it"&gt;&lt;/a&gt;
&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Sidequestmaxxing&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;Your city is the game. Your phone is just the quest log.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;An open-weight Dungeon Master reads your neighbourhood, writes you one real side quest
speaks it into your earbuds, and then gets out of the way so you can go outside and do it.&lt;/p&gt;
&lt;p&gt;Built for the DEV &lt;strong&gt;Touch Grass&lt;/strong&gt; challenge (Hacktoberfest Open-Source AI Challenge, Week 1).&lt;/p&gt;
&lt;/div&gt;
---
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;What "sidequestmaxxing" means&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;Side quest maxxing is Gen Z slang for treating real life the way you treat an open-world game.
Instead of grinding only the main quest (your career, your degree, your to-do list), you
deliberately seek out, collect and fully dive into random, low-stakes, spontaneous detours:
the market three streets over, the hill you have never walked up, the mural behind the bus depot.
Daily routines, hobbies and small unexpected adventures get treated as optional side quests,
which builds skills, sparks joy, and makes life more interesting.&lt;/p&gt;
&lt;p&gt;This…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/aarushitandon0/sidequestmaxxing" rel="noopener noreferrer"&gt;github.com/aarushitandon0/sidequestmaxxing&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;Sidequestmaxxing combines open-weight AI, real-world map data, and deterministic safety checks to turn everyday life into a game. The core idea is simple: &lt;strong&gt;the AI handles the planning so you can put your phone away and go explore.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  The open models
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Where it runs&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen-class open-weight model&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;llama.cpp&lt;/code&gt; on a private Render service, CPU-only&lt;/td&gt;
&lt;td&gt;Generates quests as the Dungeon Master&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 3n&lt;/td&gt;
&lt;td&gt;Player's browser via WebGPU&lt;/td&gt;
&lt;td&gt;Verifies proof photos on supported devices&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BGE-small-en-v1.5&lt;/td&gt;
&lt;td&gt;API container via &lt;code&gt;fastembed&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Finds relevant places and detects repetitive quests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is no hosted LLM API in the quest-generation path.&lt;/p&gt;

&lt;h3&gt;
  
  
  How a quest is generated
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fzlq70jxw2es1hfk38r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3fzlq70jxw2es1hfk38r.png" alt="Diagram of the three Render services and their fallbacks" width="800" height="553"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The app has three services: a React and TypeScript PWA (&lt;code&gt;sqm-web&lt;/code&gt;), a FastAPI backend (&lt;code&gt;sqm-api&lt;/code&gt;), and a private &lt;code&gt;llama.cpp&lt;/code&gt; inference service (&lt;code&gt;sqm-llm&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjlabmduy5edx9iabr65.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmjlabmduy5edx9iabr65.png" alt="Diagram of the quest generation pipeline" width="800" height="600"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Context Builder:&lt;/strong&gt; Checks weather, daylight, and safety conditions before generating a quest. Severe weather, extreme heat, or insufficient daylight can trigger indoor-only mode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scout:&lt;/strong&gt; Uses OpenStreetMap data and vector search to find real nearby places, then filters them by distance, activity type, and available time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory Check:&lt;/strong&gt; Compares candidates with recent adventures to reduce repetitive quests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quest Master:&lt;/strong&gt; Generates exactly one structured quest grounded in the supplied places. It cannot invent a location or assign its own XP.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validators:&lt;/strong&gt; Twelve checks evaluate grounding, time limits, formatting, and safety. One retry is allowed before the deterministic fallback takes over.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fallback Engine:&lt;/strong&gt; Generates quests from tested templates, even when the model or network is unavailable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Game State:&lt;/strong&gt; The server assigns quest IDs, rarity, and XP only after validation succeeds.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Safety and security
&lt;/h3&gt;

&lt;p&gt;Testing the first version revealed that obvious keyword filters were not enough. In an expanded adversarial suite, &lt;strong&gt;34 of 104 test cases initially bypassed the existing validators&lt;/strong&gt;. These included unsafe suggestions involving private property, surveillance, water, and thunderstorms. All 34 are now rejected.&lt;/p&gt;

&lt;p&gt;I also found a prompt-injection risk in OpenStreetMap place names. Because map data can be edited by anyone, malicious instructions could be disguised as a location name. I added a sanitisation layer that removes instruction-like text while preserving legitimate names. Place grounding relies on stable IDs, so sanitisation does not break the connection to real locations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Privacy by design
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftf1etn7wy89kj02ip054.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftf1etn7wy89kj02ip054.png" alt="Diagram of what stays on the device and what crosses the network" width="800" height="440"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Photo verification:&lt;/strong&gt; No image-upload endpoint exists. On supported devices, Gemma 3n processes proof photos directly in the browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Location privacy:&lt;/strong&gt; Coordinates are rounded to approximately 110-metre precision and are not stored as raw coordinates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Local journal:&lt;/strong&gt; Entries and photos stay in device storage, with photo metadata stripped before storage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private observability:&lt;/strong&gt; Sentry traces record errors and timings without capturing prompts or completions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graceful fallback:&lt;/strong&gt; Cached briefings can be replayed without connectivity, and the deterministic quest engine can generate quests without model inference. The complete application still requires network access for some features.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Training and evaluation
&lt;/h3&gt;

&lt;p&gt;The training pipeline uses real seeded places, geographic train/test splits, and deliberately difficult cases, including missing candidates and indoor-only conditions. Candidate examples are filtered through the same validators used in production before LoRA fine-tuning.&lt;/p&gt;

&lt;p&gt;The intended evaluation measures the quantized model actually served in production, with a target of at least a ten-percentage-point improvement in full validator pass rate or a clear latency and cost advantage.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test results as of October 10, 2026
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;API tests&lt;/td&gt;
&lt;td&gt;837 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frontend tests&lt;/td&gt;
&lt;td&gt;119 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playwright tests&lt;/td&gt;
&lt;td&gt;10 passed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strict type checking&lt;/td&gt;
&lt;td&gt;54 files, no issues&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secret scanning&lt;/td&gt;
&lt;td&gt;No secrets found in the tree or Git history&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A custom Content Security Policy test also runs the production build in Chromium, checks all eleven routes, and verifies that the map and real OpenStreetMap tiles load.&lt;/p&gt;

&lt;h3&gt;
  
  
  The design principle
&lt;/h3&gt;

&lt;p&gt;The model is responsible for generating the adventure, not controlling the entire system. Deterministic code handles safety checks, validation, and progression, while open-weight models provide customisation and the option of on-device photo verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Everything comes back to one rule: &lt;strong&gt;the screen should be the shortest part of the experience.&lt;/strong&gt;
&lt;/h2&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. I can shape the model, not just prompt it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Sidequestmaxxing is designed around an open-weight Dungeon Master that can be adapted using LoRA, quantised into GGUF, and served through &lt;code&gt;llama.cpp&lt;/code&gt;. This gives me control over the model weights and deployment, rather than tying the experience to a single hosted LLM provider. The same validation rules are shared across training, evaluation, and production to keep quest generation grounded and safe.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Privacy is built into the architecture.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On supported devices, Gemma 3n can verify proof photos directly in the browser using WebGPU. The API has no image-upload endpoint, so photos do not need to leave the device for verification. This makes local inference a practical privacy advantage, not just a promise in a privacy policy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The model can run without a GPU.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quantised GGUF models make CPU inference possible, including on modest hardware. With a configurable inference endpoint, the backend can also be pointed at another compatible model server instead of being permanently tied to one provider.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Open data makes neighbourhood exploration more accessible.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;OpenStreetMap provides real-world place data, Open-Meteo supplies weather information, and BGE-small enables local CPU-based embeddings. These components reduce dependence on proprietary data providers and per-request AI fees, while making it easier to inspect and adapt the system.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Open innovation matters because it gives us control over how AI is customised, where it runs, and how personal data is handled.&lt;/strong&gt; For Sidequestmaxxing, those freedoms help build an experience that gets people off their phones and into the world.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Best Use of Render
&lt;/h3&gt;

&lt;p&gt;All three services are deployed through a single &lt;code&gt;render.yaml&lt;/code&gt; Blueprint, as described in System Architecture.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-web&lt;/code&gt;&lt;/strong&gt;: Hosts the static Progressive Web App (PWA).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-api&lt;/code&gt;&lt;/strong&gt;: Runs the FastAPI backend in Docker. It uses the repository root as its build context so it can access the &lt;code&gt;/prompts&lt;/code&gt; directory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;sqm-llm&lt;/code&gt;&lt;/strong&gt;: Runs as a private service with no public URL and a 10 GB persistent disk mounted at &lt;code&gt;/data&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The API does not hard-code the model service's host or port. Instead, it reads &lt;code&gt;LLM_HOSTPORT&lt;/code&gt;, which Render injects for service-to-service communication with &lt;code&gt;sqm-llm&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;On its first boot, the LLM service downloads the GGUF model specified by &lt;code&gt;MODEL_URL&lt;/code&gt; to the persistent disk and verifies its integrity against &lt;code&gt;MODEL_SHA256&lt;/code&gt;. On subsequent boots, the existing model file is reused, avoiding repeated downloads.&lt;/p&gt;

&lt;p&gt;The entire deployment is managed through one Blueprint, with the configuration and deployment process documented end to end.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Gemma
&lt;/h3&gt;

&lt;p&gt;Gemma 3n runs entirely in the browser using WebGPU and MediaPipe's LLM Inference task. Its &lt;code&gt;.litertlm&lt;/code&gt; model file is stored in the player's Origin Private File System (OPFS), so the model and its inference process remain on the player's device.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Local inference:&lt;/strong&gt; The model runs in the browser rather than on the backend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No image uploads:&lt;/strong&gt; The API has no image-upload endpoint, &lt;code&gt;UploadFile&lt;/code&gt;, &lt;code&gt;File()&lt;/code&gt; handler, or multipart upload handler under &lt;code&gt;apps/api/app&lt;/code&gt;. A test explicitly verifies this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Capability detection:&lt;/strong&gt; The application checks for &lt;code&gt;navigator.gpu&lt;/code&gt;, a compatible GPU adapter, and OPFS support before attempting local inference.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graceful fallback:&lt;/strong&gt; Devices that cannot run the model can use check-in or honour-mode verification instead of encountering a broken experience.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This design keeps player photos off the server while allowing the application to work on devices that do not support local inference.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of ElevenLabs
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;POST /v1/tts&lt;/code&gt; endpoint acts as a server-side proxy for ElevenLabs text-to-speech. The browser never receives the API key, and clients cannot submit arbitrary text for synthesis. The endpoint only accepts the briefing or recap already stored for the device's quest.&lt;/p&gt;

&lt;p&gt;The endpoint uses three safeguards:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Character limit:&lt;/strong&gt; Caps the amount of text processed per request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-device rate limit:&lt;/strong&gt; Restricts the number of requests a device can make per hour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Global daily budget:&lt;/strong&gt; Tracks character usage in MongoDB using the &lt;code&gt;usage&lt;/code&gt; collection and keys in the format &lt;code&gt;tts:YYYY-MM-DD&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the API key is missing or the daily character budget has been exhausted, &lt;code&gt;/v1/tts&lt;/code&gt; returns &lt;code&gt;503 Service Unavailable&lt;/code&gt;. The browser then uses its built-in &lt;code&gt;speechSynthesis&lt;/code&gt; API to read the same text.&lt;/p&gt;

&lt;p&gt;All four Dungeon Master personas remain available through this fallback, although the voice quality may be lower. The experience remains functional even when the external TTS service is unavailable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of MongoDB Atlas
&lt;/h3&gt;

&lt;p&gt;MongoDB Atlas supports semantic place discovery, repeat-quest detection, geospatial queries, memory expiration, and duplicate-completion prevention.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Vector Search indexes&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;places_vec&lt;/code&gt;&lt;/strong&gt;: A 384-dimensional cosine-similarity index used by the Scout for semantic place search. Results can be filtered by &lt;code&gt;cell5&lt;/code&gt; and &lt;code&gt;kind&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;memories_vec&lt;/code&gt;&lt;/strong&gt;: A 384-dimensional cosine-similarity index used for repeat-quest checks, with filtering by &lt;code&gt;device_id&lt;/code&gt; and &lt;code&gt;ts&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Additional indexes&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Geospatial index:&lt;/strong&gt; A &lt;code&gt;2dsphere&lt;/code&gt; index on &lt;code&gt;places.loc&lt;/code&gt; supports location-based queries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TTL index:&lt;/strong&gt; Automatically expires documents in &lt;code&gt;memories&lt;/code&gt; after 90 days.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unique completion index:&lt;/strong&gt; A unique index on &lt;code&gt;completions.quest_id&lt;/code&gt; enforces one completion per quest at the database level, preventing concurrent requests from creating duplicate completions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These indexes combine semantic retrieval, geographic relevance, memory management, and database-enforced consistency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Sentry Agent Tracing
&lt;/h3&gt;

&lt;p&gt;Sentry tracing provides end-to-end visibility into the quest-generation pipeline without recording sensitive quest content.&lt;/p&gt;

&lt;p&gt;Each stage has its own span under a single &lt;code&gt;quests.generate&lt;/code&gt; transaction. These stages include the Context Builder, Scout, memory check, Quest Master, validators, and fallback engine. Model calls emit &lt;code&gt;gen_ai.invoke_agent&lt;/code&gt; and &lt;code&gt;gen_ai.chat&lt;/code&gt; spans with token counts and latency information.&lt;/p&gt;

&lt;p&gt;Three tags make requests searchable without requiring anyone to inspect span contents:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;source&lt;/code&gt;&lt;/strong&gt;: Identifies the generation path, such as &lt;code&gt;llm&lt;/code&gt;, &lt;code&gt;fallback&lt;/code&gt;, &lt;code&gt;empty&lt;/code&gt;, or &lt;code&gt;cached:*&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;retry&lt;/code&gt;&lt;/strong&gt;: Indicates whether the request was retried (&lt;code&gt;0&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;validator_failures&lt;/code&gt;&lt;/strong&gt;: Records &lt;code&gt;none&lt;/code&gt; when validation succeeds, or the failing validator IDs, such as &lt;code&gt;V2&lt;/code&gt; or &lt;code&gt;V5,V9&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prompts and completions are deliberately excluded from traces. With &lt;code&gt;send_default_pii=False&lt;/code&gt; and no draft text attached to spans, a trace can reveal that validator &lt;code&gt;V2&lt;/code&gt; failed and how long a model call took, without exposing the generated quest or the player's surroundings.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of Tinker
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;training/&lt;/code&gt; directory contains a complete, tested pipeline for LoRA supervised fine-tuning (SFT) and teacher-sample generation.&lt;/p&gt;

&lt;p&gt;The workflow includes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Context generation:&lt;/strong&gt; Samples contexts from the real seeded places and splits the data by area so that the test set contains unseen locations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Teacher sampling:&lt;/strong&gt; Generates draft quests through Tinker and filters them using the same validator code used in production, imported directly rather than duplicated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dataset preparation:&lt;/strong&gt; Deduplicates samples and balances categories, rarity levels, and Dungeon Master personas.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LoRA fine-tuning:&lt;/strong&gt; Sweeps three learning rates over three epochs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model selection:&lt;/strong&gt; Selects the model using validation pass rate rather than training loss alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reproducibility:&lt;/strong&gt; Uses fixed seeds and records run metadata in &lt;code&gt;training/runs/&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the complete methodology and evaluation results, see Training and Evaluation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best Use of SerpApi
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;scripts/fetch_events.py&lt;/code&gt; script uses SerpApi to retrieve Google's &lt;code&gt;events_results&lt;/code&gt; and incorporate relevant local events into the quest-generation pipeline.&lt;/p&gt;

&lt;p&gt;Each listing passes through several filtering and validation stages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Input sanitization:&lt;/strong&gt; Sanitizes event strings using the same sanitizer that protects against unsafe OpenStreetMap place names.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time validation:&lt;/strong&gt; Runs each listing through validator V13, which rejects events that have already ended, start after the quest window, or have times that cannot be parsed without guessing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Content filtering:&lt;/strong&gt; Applies the project's existing banned-term checks and filters out indoor or ticketed events.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The event-fetching and validation pipeline is covered by three test files:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;test_fetch_events.py&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;test_serp_events.py&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;test_event_time.py&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This ensures that event-based quests use sanitized listings with valid, relevant timing information.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;The screen is the shortest part. Go live outside and Touch Grass :p.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
    </item>
    <item>
      <title>Tiny Me: A Little Version of Me That Lives on My Mom’s Laptop</title>
      <dc:creator>Aarushi Tandon</dc:creator>
      <pubDate>Sun, 04 Oct 2026 14:07:46 +0000</pubDate>
      <link>https://dev.to/aarushitandon/tinyme-a-little-version-of-me-that-lives-on-my-friends-and-familys-laptops-597e</link>
      <guid>https://dev.to/aarushitandon/tinyme-a-little-version-of-me-that-lives-on-my-friends-and-familys-laptops-597e</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-weekend-2026-10-01"&gt;Hacktoberfest Weekend Challenge: Build for a Friend&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;I built &lt;strong&gt;Tiny Me for my mom.&lt;/strong&gt; I don't live at home anymore, so I can't always be there when she gets stuck on her laptop. When I'm home, she can simply call out, "Aarushi, come look at this." But when I'm away, even something as simple as finding a downloaded file, opening the right setting, or figuring out which button to press can become frustrating. My friend &lt;strong&gt;Aditi&lt;/strong&gt; made me realize this isn't just a problem for parents. She's incredibly smart and is studying law, but being good at law doesn't mean she should have to be good at navigating confusing computer interfaces. &lt;strong&gt;Being good at what you do shouldn't require being good at computers.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So I wondered: &lt;strong&gt;what if a small version of me could live on their laptop?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's &lt;strong&gt;Tiny Me&lt;/strong&gt;, a local desktop helper for people who know exactly what they want to accomplish but don't always know how to make their computer do it. Press a hotkey and say what you're trying to do in plain language, like "Find my train ticket for tomorrow" or "Show me how to search for trains." Tiny Me understands what's on the screen, identifies the next control, and draws a circle around exactly where to click. &lt;strong&gt;You click it.&lt;/strong&gt; It checks whether the step worked, then guides you to the next one.&lt;/p&gt;

&lt;p&gt;It's like having a tech savvy friend sitting beside you saying, &lt;strong&gt;"Yep. Click this."&lt;/strong&gt; Except that friend lives on your laptop.&lt;/p&gt;

&lt;p&gt;But Tiny Me is deliberately &lt;strong&gt;not&lt;/strong&gt; an autonomous computer use agent. &lt;strong&gt;It is designed to teach, not take over.&lt;/strong&gt; The goal isn't to let AI use your computer for you. It's to make your computer understandable. The user stays in control and performs the actions themselves.&lt;/p&gt;

&lt;p&gt;Tiny Me also knows when to stop. If a workflow reaches a password, OTP, card number, payment, or other sensitive input, it simply says, &lt;strong&gt;"This part is yours. I'll wait."&lt;/strong&gt; It doesn't type, click, or attempt to bypass that boundary.&lt;/p&gt;

&lt;p&gt;Everything runs locally on the user's Windows machine. Screen understanding, control detection, and the language model stay on the device, so screen content doesn't need to leave the laptop.&lt;/p&gt;

&lt;p&gt;At its heart, Tiny Me comes from a very simple problem: &lt;strong&gt;I can't always be there to help my mom. So I built something that can.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who it is for.&lt;/strong&gt; My mom, my friend Aditi, and anyone who is brilliant at what they do but shouldn't have to become a computer expert just to use their own laptop. &lt;br&gt;
Everything runs on the user's Windows machine. Reading the screen, finding controls and the language model are all local. No screen content leaves the laptop.&lt;/p&gt;
&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/aarushitandon0/tinyme/blob/main/SS/07_demo_walkthrough.mp4" rel="noopener noreferrer"&gt;Watch the full demo walkthrough (MP4)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The demo runs against a local booking website included in the repo (&lt;code&gt;demo_site/&lt;/code&gt;), and shows:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Press &lt;code&gt;Ctrl+Alt+H&lt;/code&gt; and type a goal.&lt;/li&gt;
&lt;li&gt;Tiny Me circles the next control and explains the step in one line.&lt;/li&gt;
&lt;li&gt;The user clicks it.&lt;/li&gt;
&lt;li&gt;Tiny Me checks whether the step worked.&lt;/li&gt;
&lt;li&gt;The circle moves to the next control.&lt;/li&gt;
&lt;li&gt;At a sign-in page, Tiny Me stops and hands control back.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+P&lt;/code&gt; stops it at any time.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3nk10r600l0d29fvb3y9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3nk10r600l0d29fvb3y9.png" alt="TinyMe home screen" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Press the hotkey, then describe what you're trying to do in plain language.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklwtlvvxc7g73mjmt56p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fklwtlvvxc7g73mjmt56p.png" alt="TinyMe guiding the user step by step" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Tiny Me circles the control and waits for the user to click it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9wqjwnrzt2ypg3j2otc2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9wqjwnrzt2ypg3j2otc2.png" alt="TinyMe handing control back at a sensitive step" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;At a sensitive step like sign-in, Tiny Me stops instead of interacting.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1cnbntmi726bbdahm3j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi1cnbntmi726bbdahm3j.png" alt="TinyMe example goals" width="800" height="546"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Example goals.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lem5o415f33jh8egadd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0lem5o415f33jh8egadd.png" alt="TinyMe helping find a file" width="800" height="507"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Finding a downloaded file, one circled click at a time.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;All screenshots are in the repo's &lt;a href="https://github.com/aarushitandon0/tinyme/tree/main/SS" rel="noopener noreferrer"&gt;SS folder&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/aarushitandon0" rel="noopener noreferrer"&gt;
        aarushitandon0
      &lt;/a&gt; / &lt;a href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;
        tinyme
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Tiny Me&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;A Windows desktop assistant that watches one screen, decides what to click next, and draws a circle around it.&lt;/p&gt;
&lt;p&gt;Tiny Me is built for a single non-technical user. She presses a hotkey, types what she wants in plain words, and the application either &lt;strong&gt;shows her how&lt;/strong&gt; by circling the exact control to click and waiting until she clicks it, or &lt;strong&gt;does it for her&lt;/strong&gt; while she watches. At any screen involving a password, a one-time code, a card number or a payment, it stops and hands control back.&lt;/p&gt;
&lt;p&gt;Everything runs on the local machine. Optical character recognition, UI inspection and the language model are all local processes. No screen content is transmitted. The application functions with networking disabled.&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Table of contents&lt;/h2&gt;
&lt;/div&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#1-design-constraints" rel="noopener noreferrer"&gt;Design constraints&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#2-system-architecture" rel="noopener noreferrer"&gt;System architecture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#3-the-per-step-pipeline" rel="noopener noreferrer"&gt;The per-step pipeline&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#4-module-reference" rel="noopener noreferrer"&gt;Module reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#5-the-element-list-the-models-entire-world" rel="noopener noreferrer"&gt;The element list&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#6-the-model-contract" rel="noopener noreferrer"&gt;The model contract&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#7-deterministic-success-checking" rel="noopener noreferrer"&gt;Deterministic success checking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#8-the-guard" rel="noopener noreferrer"&gt;The guard&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#9-coordinate-systems-and-dpi" rel="noopener noreferrer"&gt;Coordinate systems and DPI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/aarushitandon0/tinyme#10-threading-model" rel="noopener noreferrer"&gt;Threading model&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/aarushitandon0/tinyme#11-the-overlay" rel="noopener noreferrer"&gt;The&lt;/a&gt;…&lt;/li&gt;
&lt;/ol&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/aarushitandon0/tinyme" rel="noopener noreferrer"&gt;github.com/aarushitandon0/tinyme&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How to Run
&lt;/h3&gt;

&lt;p&gt;Requires Windows 10 (version 2004 or later) or Windows 11, Python 3.11, and &lt;a href="https://ollama.com" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/aarushitandon0/tinyme
&lt;span class="nb"&gt;cd &lt;/span&gt;tinyme
python &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv
.venv&lt;span class="se"&gt;\S&lt;/span&gt;cripts&lt;span class="se"&gt;\a&lt;/span&gt;ctivate
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="s2"&gt;".[dev]"&lt;/span&gt;
playwright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
ollama pull gemma4:e2b
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To run the demo, open two terminals in the repository folder.&lt;/p&gt;

&lt;p&gt;Terminal 1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; http.server 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Terminal 2:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; app.main
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;TinyMe starts with no visible window.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+H&lt;/code&gt; opens the prompt. Type a goal and choose "Show me how" or "Do it for me."&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Ctrl+Alt+P&lt;/code&gt; stops TinyMe at any time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The demo targets &lt;code&gt;http://localhost:8000/demo_site/index.html&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Optional settings go in a &lt;code&gt;.env&lt;/code&gt; file at the repo root: &lt;code&gt;TINYME_MODEL&lt;/code&gt; (Ollama model tag), &lt;code&gt;TINYME_LANGUAGE&lt;/code&gt;, and &lt;code&gt;TINYME_TELEMETRY&lt;/code&gt; (off unless set to &lt;code&gt;1&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;The core idea: &lt;strong&gt;the language model chooses a control, never a coordinate.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tiny Me builds a numbered list of the on-screen controls, and asks the model to pick one number from it. The model answers &lt;code&gt;17&lt;/code&gt;, not &lt;code&gt;click(834, 492)&lt;/code&gt;. The app can then check that &lt;code&gt;17&lt;/code&gt; is a real, interactable control. A model that generates pixel coordinates can produce plausible but wrong ones. A number from a list the app built can be validated.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The model proposes. The app decides.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Success is checked in code.&lt;/strong&gt; After each step, a watcher polls the screen about once per second and checks window titles or classes, visible text, pixel changes, or browser structure. The model can suggest which check to run, but Python runs it. I deliberately don't ask the model "did that work?" A model judging its own action can easily talk itself into success.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The safety guard runs before the model.&lt;/strong&gt; Before every model call and every automated action, the guard checks for sensitive screens. If one is present, nothing is sent to the model and nothing is clicked or typed. In the browser, the guard uses page structure, so a password field counts as a password field whatever its label says. Screenshots stay in memory and are never written to disk.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Everything runs locally:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma 4 (&lt;code&gt;gemma4:e2b&lt;/code&gt;)&lt;/strong&gt; via &lt;strong&gt;Ollama&lt;/strong&gt;, for choosing the control&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;RapidOCR&lt;/strong&gt; on &lt;strong&gt;ONNX Runtime&lt;/strong&gt;, for screen text&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Windows UI Automation&lt;/strong&gt;, for discovering controls&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python&lt;/strong&gt; for orchestration, guard, validation and execution&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Playwright&lt;/strong&gt; for the browser demo&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Qwen3.5-4B fine-tuned on Tinker&lt;/strong&gt; as a swappable alternative picker (see below)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The Tinker experiment
&lt;/h3&gt;

&lt;p&gt;I fine-tuned &lt;code&gt;Qwen/Qwen3.5-4B&lt;/code&gt; with LoRA (rank 16, 3 epochs, 80 training examples, about $0.17) on Tinker. The held-out test was the whole &lt;code&gt;demo_site&lt;/code&gt; app, which the model never saw in training, with 22 labelled examples.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;System&lt;/th&gt;
&lt;th&gt;Correct control&lt;/th&gt;
&lt;th&gt;Valid answer format&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Untuned Qwen3.5-4B&lt;/td&gt;
&lt;td&gt;0/22 (0%)&lt;/td&gt;
&lt;td&gt;0/22 (0%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fine-tuned Qwen3.5-4B&lt;/td&gt;
&lt;td&gt;18/22 (81%)&lt;/td&gt;
&lt;td&gt;22/22 (100%)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The untuned model's 0/22 is mostly a format failure: its replies didn't parse. Fine-tuning fixed the format but not the choice. In 19 of 22 examples the tuned model still picked the wrong element. For example, asked to "pay for my ticket," it picked a heading when the right answer was "not on this screen."&lt;/p&gt;

&lt;p&gt;That's a modest result, and it's a useful failure. It says the next problem is teaching the model when to act, which control to pick, and when no safe action exists, not only following the output format.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;For Tiny Me, local and open models aren't a preference; they're part of the product. If I'm asking my friends and my mom to let software look at their screen, I don't want that to depend on trusting a closed hosted API.&lt;/p&gt;

&lt;p&gt;With open weights:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the model runs on the laptop, and screen content stays there,&lt;/li&gt;
&lt;li&gt;sensitive controls are blocked before any model call,&lt;/li&gt;
&lt;li&gt;the code is inspectable,&lt;/li&gt;
&lt;li&gt;and I could swap the model, fine-tune it, and measure where it fails.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A closed API would have hidden the 0/22 and the 3/22 behind a single "it works" demo. Open weights let me train a model, test it on an app it hadn't seen, and report that it mostly still fails.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Best Use of Gemma&lt;/strong&gt;&lt;br&gt;
Gemma 4 (&lt;code&gt;gemma4:e2b&lt;/code&gt;) is the default model, run fully locally through Ollama. It reads the numbered list of on-screen controls and picks one. Nothing in the runtime path is a hosted API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Best Use of Tinker&lt;/strong&gt;&lt;br&gt;
I fine-tuned Qwen3.5-4B on Tinker and tested it on a whole held-out app. Valid answer format went from 0/22 to 22/22, and correct control selection from 0/22 to 18/22. &lt;/p&gt;

&lt;h2&gt;
  
  
  What's Next?
&lt;/h2&gt;

&lt;p&gt;The next step is to put Tiny Me in front of the people I built it for: friends, family, and my mom. I want to find out where they hesitate, which instructions make sense right away, what confuses them, whether they remember the steps next time, where Tiny Me should step back, and most importantly, &lt;strong&gt;after using Tiny Me, do they need it less?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's the metric I care about. I don't want to build something that makes people feel they can't use their own computer without it. I want the opposite: a small version of your tech-savvy friend that says "Here. Click this." and, eventually, you don't need to ask.&lt;/p&gt;

&lt;h1&gt;
  
  
  devchallenge #weekendchallenge #hf26challenge
&lt;/h1&gt;

</description>
      <category>devchallenge</category>
      <category>weekendchallenge</category>
      <category>hf26challenge</category>
    </item>
  </channel>
</rss>
