DEV Community: Shashank Kumar Singh

Roam: turn a remembered place into your next walk with local Gemma

Shashank Kumar Singh — Mon, 12 Oct 2026 06:36:07 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

Roam is a map-backed collection of place memories that helps someone decide where to go, then close the tab and actually go there.

The starting idea was simple: a photo of a park, a café, a trail, or a workshop is more useful when it comes with somebody's experience of being there. A long feed can make me want to travel without helping me take the first step. I wanted the map, the memory, and a practical plan in one place.

Roam combines a readable social feed with photography and nearby discovery. There is no signup wall. In the local prototype, someone can attach a photo, speak a short experience, review the transcript, confirm the place, and ask Gemma to help turn it into a memory. Another person can save that place and ask for a visit plan that fits the time they have.

The important design decision is that AI should help express the experience, not replace it. Saying “I went here and liked the quiet” must not turn into a fabricated review of food, facilities, or an event that the person never mentioned.

The intended audience is local explorers: people looking for a small outdoor visit, a walk, a café stop, or something nearby worth remembering. Workshops and cultural memories are supported too, but a memory is not a claim that an event is still running.

Demo

Watch the narrated Roam demo — 97 seconds, 1080p · Download MP4

Dated supplement — 12 October 2026: This video was added after the Week 1 deadline of 12:29 PM IST. The original submission was published at 12:06:07 PM IST before the cutoff. The video is an edited walkthrough of captured local application screens, with narration by ElevenLabs' Hope voice. It adds presentation material and does not claim new features or an additional prize category completed during the entry period. The README records the post-deadline addition.

Open the hosted Roam preview

The Render preview supports browsing, the map, place search, and nearby discovery. It is intentionally read-only: public posting is disabled while private photo storage is being connected, and the hosted preview does not yet have a GPU/Gemma service. Free Render can take time to wake after inactivity.

The full Gemma-assisted workflow runs locally with Ollama. These are screenshots of the implemented local prototype, not a claim that the cloud preview currently performs inference.

Local feed with clearly labelled fictional example memories and illustrative photography. The example photo is not evidence of an actual visit to the named park.

Gemma generated a plan from example memories, with those examples identified in the UI. Plans enforce a time budget and cite their source memories.

For this Misu example, retrieval supplied a restaurant category and mapped cuisine. The interface explicitly says no matching Wikipedia background was found; it does not invent the missing story.

To reproduce the local AI demo:

git clone https://github.com/Bitshank-2338/roam-touch-grass.git
cd roam-touch-grass
npm ci
ollama pull gemma3:4b
npm start

Use Node 24.14 or newer, keep Ollama running, and open http://127.0.0.1:4180. Add your own Geoapify key in a private .env for the selected map provider; the repository includes the configuration example. Model weights and credentials are not bundled.

Code

Public source and setup instructions

The application source is MIT licensed. Model weights, map data, fonts, and photographs retain their respective licenses or terms. This submission repository is a clean snapshot of the project built during the challenge window; it excludes operational secrets, private browser identities, user uploads, and private development history.

How I Built It

A small web stack, with local Gemma at the centre

The frontend uses plain JavaScript and CSS, Leaflet, and a forest-green “Trail R” identity. The backend uses Node.js, Express, Sharp for photo processing, and Ollama for Gemma 3 4B.

SQLite is the local default. An asynchronous PostgreSQL backend is implemented, and the hosted preview now connects to Tiger Data. The app already has a private S3-compatible media adapter tested against local MinIO; connecting Supabase Storage is the next deployment step.

The flow is:

Photo + reviewed voice transcript + confirmed place
                 |
       server-side source retrieval
                 |
     Gemma selects existing facts / assists fields
                 |
        editable memory in private review
                 |
        approved feed and map contribution
                 |
       source-based, time-bounded visit plan

Browser speech recognition supplies a transcript for review; Gemma works on that reviewed text. Recognition can depend on the browser provider's online service. “Listen” uses the device's available speech voices, with Stop and cancellation when navigating away.

Grounding instead of a bigger hallucination

One challenge was expanding a one- or two-line experience without pretending to know more about the user's visit.

For short notes, the original experience remains verbatim. Roam retrieves place context and asks Gemma to select known fact IDs. Application code attaches the exact retrieved source text, rather than accepting arbitrary model-authored facts. The source metadata is signed so a client cannot simply attach invented attribution.

When suitable background is absent, Roam provides labelled visit guidance and an explicit limitation. It does not manufacture opening hours, admission fees, venue history, or workshop schedules.

Nearby discovery also needed correction: a road near the user is not automatically an interesting destination. Retrieval now prioritises named parks, viewpoints, cafés, restaurants, museums, and cultural places. Wikipedia/Wikidata background requires geographic checks.

Anonymous does not mean ownership-free

The server issues a random private browser identity. A recovery export restores ownership and saved memories; it is not a device fingerprint. Clearing browser storage without recovery loses that ownership.

Local posting requires a recent location receipt and confirmation of the public pin. This is a client claim, not proof of attendance. Photos are decoded, resized, stripped of metadata, and re-encoded before storage.

Reactions are real, with one heart per browser per post. Replies have moderation states. Trending uses approved, non-demo contributions, duplicate filtering, and independent-browser thresholds. Gemma explains an existing ranking with cited post IDs; it cannot invent a trending venue or change the ranking.

Safety has an explicit unfinished boundary

The application implements Checking, Needs review, Blocked, and Published states. Pending photos remain private; edits retain the previous approved version while the new revision is checked.

The ShieldGemma integration and evaluation harness are prepared, but the actual ShieldGemma models have not been installed or evaluated on the future GPU server. Automatic approval stays disabled. Contract tests verify failure handling with controlled responses; those tests are not a substitute for real model evaluation. New video uploads are disabled in this release.

What I verified

54 automated tests pass, with syntax checks passing.
Coverage includes ownership/recovery, stale moderation decisions, private media, exact source attribution, reactions/comments, speech cancellation, and trend grounding.
Isolated PostgreSQL/MinIO integration checks cover migration, access denial, interrupted media operations, and persistence.
Live Geoapify search, named nearby discovery, tiles, and walking routes were checked.
Real local Gemma produced sourced place context and time-bounded plans.
Microphone capture and audible summary playback were confirmed on the intended device.
Render health, search, feed, and trends pass. Tiger Data's provider-usage ledger and trend snapshot survived a Render redeploy.

These checks do not establish a completed public launch. Private cloud media, GPU inference, real moderation evaluation, and final Figma repairs remain pending.

Why Does Open Innovation Matter?

Gemma's open weights let the useful part of Roam run on a machine I control. A personal memory does not need to be sent to a proprietary paid inference API just to get help expressing it.

The model is also replaceable. Retrieval, source validation, identity, and publication decisions live in application code, so the model does not become the only place where the product's rules exist. I can inspect those rules, test them, and change the inference provider without rewriting the social application.

This is not an entirely offline app: maps and source retrieval need internet, and browser voice recognition may do so too. The local model makes the AI step independently runnable, while the UI states those limits honestly.

The best outcome for Roam is a short interaction: find a place, understand why someone remembers it, make a realistic plan, and spend the next hour outside.

My Agent Session

I developed Roam with Codex assistance, including implementation, test orchestration, and this submission write-up. This post was prepared and published by the agent under my instruction. I am not publishing the raw session because it includes private operational setup.

Inspiration included Leaf Peep by Souvik Biswas, which helped shape the idea of using an open model to encourage an actual outing. Roam's focus is the place-memory and social-discovery workflow.

Prize Categories

Best Use of Gemma: local Gemma 3 4B assists memory drafting, selects grounded place facts, and generates plans from cited memories.
Best Use of Render: the deployed web frontend/backend preview runs on Render, with public posting deliberately gated while durable photo storage is completed.

Credits

Ollama; Google's Gemma and ShieldGemma under their model terms; Geoapify; Leaflet; OpenStreetMap and OpenMapTiles; Wikipedia/Wikidata; Phosphor icons; Outfit and DM Sans; and illustrative Unsplash photography. The Trail R brand imagery was generated with imagegen. Example stories are fictional and are excluded from real trends.

Why I chose Ollama over cloud AI for my screen-reading assistant (and what I learned)

Shashank Kumar Singh — Wed, 06 May 2026 11:38:43 +0000

When I started building Clicky — a Windows AI assistant that reads your screen and answers out loud — I had to make a fundamental choice: cloud AI or local AI?

I chose local (Ollama). Here's exactly why, and what I learned the hard way.

The problem with cloud AI for a screen assistant

Clicky's core loop is:

Take a screenshot of your screen
Record your voice question
Send both to an LLM
Speak the answer back

Step 3 is where cloud AI becomes a problem. You're sending a screenshot of your screen to a remote server every single time someone presses the hotkey.

Think about what's on your screen:

Passwords in password managers
Emails with sensitive info
Code with API keys
Personal documents

I wasn't comfortable sending that to OpenAI servers (or any server) by default. And I couldn't expect users to be comfortable with it either.

What Ollama actually gives you

Ollama lets you run LLMs locally. Pull a model, it runs on your GPU/CPU, responses never leave your machine.

For Clicky, this means:

Zero privacy risk — screenshots stay on your PC
Zero cost per query — no API bills
Zero internet required — works offline
Zero rate limits — query as fast as your hardware allows

The tradeoff: you need a decent GPU and ~4–8GB RAM for a usable model. But most modern gaming PCs qualify.

Models I tested

Model	VRAM needed	Quality	Speed
Llama 3.1 8B	6GB	Excellent	Fast
Mistral 7B	5GB	Great	Fast
Phi-3 mini	3GB	Good	Very fast
Llama 3.2 3B	2GB	OK	Blazing

I use Llama 3.1 8B as the default. It handles screen content, code, and general questions well.

The Ollama HTTP API is dead simple

import httpx
import base64

async def ask_ollama(screenshot_path: str, question: str) -> str:
    with open(screenshot_path, "rb") as f:
        img_b64 = base64.b64encode(f.read()).decode()

    payload = {
        "model": "llama3.1",
        "messages": [
            {
                "role": "user",
                "content": question,
                "images": [img_b64]
            }
        ],
        "stream": False
    }

    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.post(
            "http://localhost:11434/api/chat",
            json=payload
        )
        return response.json()["message"]["content"]

That's it. No SDK, no auth, no API key. Just a POST to localhost.

When I kept cloud AI as a fallback

Despite going local-first, I built a CompanionManager class that routes to different providers:

class CompanionManager:
    def get_provider(self):
        if self.settings.provider == "ollama":
            return OllamaProvider()
        elif self.settings.provider == "openai":
            return OpenAIProvider(api_key=self.settings.openai_key)
        elif self.settings.provider == "claude":
            return ClaudeProvider(api_key=self.settings.claude_key)
        else:
            return OllamaProvider()  # default

Power users who want GPT-4V or Claude's vision can switch in settings. But the default is Ollama, zero config.

What I learned

1. Ollama model management is the real UX problem.
Users don't want to open a terminal and run ollama pull llama3.1. I had to build a model-picker UI that shows which models are installed and lets you pull new ones from inside Clicky.

2. First response latency matters more than throughput.
For a voice assistant, you want the first token fast. Streaming responses (even over localhost) made Clicky feel much snappier.

3. GPU vs CPU is night and day.
On CPU only, Llama 3.1 8B takes 15–30 seconds. On a GTX 1080, it's 2–4 seconds. I added a GPU detection check and recommend CPU-only users use Phi-3 mini instead.

4. Context window size limits what you can do with screenshots.
A 1080p screenshot as base64 JPEG at q=70 is ~200–400KB. Some models handle this fine; others choke. I compress to 768px max width before encoding.

The verdict

For a privacy-sensitive app like a screen assistant, local AI is the right default. Cloud AI makes sense as an opt-in for users who want more power and don't mind the tradeoff.

Ollama made this possible without asking users to do anything complicated. Install Clicky, install Ollama, pull a model — done.

Download Clicky free (Windows): https://clicky.foo

Article 1 (what it does): https://dev.to/bitshank2338/i-built-a-free-ai-that-reads-my-screen-and-answers-out-loud-no-api-key-runs-offline-with-ollama-15np

Article 2 (full technical breakdown): https://dev.to/bitshank2338/how-i-built-a-screen-aware-ai-assistant-in-python-full-stack-breakdown-pyqt6-whisper-ollama-1354

What model are you running locally? Let me know in the comments.

How I built a screen-aware AI assistant in Python – full stack breakdown (PyQt6 + Whisper + Ollama)

Shashank Kumar Singh — Wed, 06 May 2026 11:35:22 +0000

Three months ago I started building Clicky — a Windows AI assistant that reads your screen and answers out loud. Here's the full technical breakdown of every piece.

TL;DR: PyQt6 system tray → Ctrl+Alt+Space hotkey → screenshot + Whisper STT → Ollama/OpenAI/Claude → edge-tts speaks answer back. Open source, free, no API key needed.

Architecture overview

User presses Ctrl+Alt+Space
        ↓
GlobalHotkey listener (pynput)
        ↓
Screenshot all monitors (mss)
        ↓
Whisper.cpp transcribes audio
        ↓
CompanionManager routes to AI provider
        ↓
Ollama (local) / OpenAI / Claude / Copilot
        ↓
edge-tts speaks answer + arrow overlay on screen

1. System tray + hotkey (PyQt6 + pynput)

The app lives in the system tray — no window, zero friction.

from pynput import keyboard

def on_activate():
    QMetaObject.invokeMethod(companion, "start_listening", Qt.QueuedConnection)

hotkey = keyboard.GlobalHotKeys({'<ctrl>+<alt>+<space>': on_activate})
hotkey.start()

The key trick: QMetaObject.invokeMethod with Qt.QueuedConnection — this crosses the thread boundary safely from pynput's background thread into Qt's main thread.

2. Screen capture (mss)

import mss, base64
from PIL import Image

def capture_all_screens():
    with mss.mss() as sct:
        for monitor in sct.monitors[1:]:  # skip monitor[0] (all combined)
            shot = sct.grab(monitor)
            img = Image.frombytes("RGB", shot.size, shot.bgra, "raw", "BGRX")
            # encode as JPEG base64 for vision API
            buffer = io.BytesIO()
            img.save(buffer, format="JPEG", quality=75)
            yield base64.b64encode(buffer.getvalue()).decode()

Quality 75 JPEG keeps the payload under API limits while preserving readability.

3. Speech-to-text (Whisper.cpp)

I use the whisper-cpp Python bindings — runs on CPU, no GPU needed.

from whispercpp import Whisper

w = Whisper.from_pretrained("base.en")

def transcribe(audio_path: str) -> str:
    result = w.transcribe(audio_path)
    return w.extract_text(result)[0].strip()

The base.en model is 142MB and transcribes ~10s of audio in ~2s on a mid-range CPU. Fast enough to feel instant.

4. AI provider routing

This was the trickiest part — supporting 4 providers with one interface:

class CompanionManager:
    def get_provider(self):
        match self.config["provider"]:
            case "ollama":   return OllamaProvider()
            case "openai":   return OpenAIProvider()
            case "claude":   return ClaudeProvider()
            case "copilot":  return GitHubCopilotProvider()

    async def ask(self, question: str, screenshots: list[str]) -> str:
        provider = self.get_provider()
        # Identity questions skip screenshots (avoids vision API refusals)
        if is_identity_question(question):
            screenshots = []
        return await provider.complete(question, screenshots, self.system_prompt)

The is_identity_question() filter was a fun challenge — vision APIs refuse to identify people in images. So I detect "who is X" patterns with regex and strip the screenshots before sending.

5. Local AI with Ollama

import httpx

async def complete(self, question, images, system_prompt):
    payload = {
        "model": "qwen2.5vl:3b",  # vision model
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question, "images": images}
        ],
        "stream": False
    }
    async with httpx.AsyncClient(timeout=60) as client:
        r = await client.post("http://localhost:11434/api/chat", json=payload)
        return r.json()["message"]["content"]

6. Text-to-speech (edge-tts)

Microsoft's neural TTS — free, no API key, sounds great:

import edge_tts, asyncio

async def speak(text: str):
    communicate = edge_tts.Communicate(text, voice="en-US-AriaNeural")
    await communicate.save("/tmp/response.mp3")
    # play with pygame
    pygame.mixer.music.load("/tmp/response.mp3")
    pygame.mixer.music.play()

7. Packaging (PyInstaller + Inno Setup)

# build.bat
pyinstaller clicky.spec --noconfirm
# Then Inno Setup builds Setup-Clicky.exe
iscc installer.iss

The .spec file needs explicit hidden imports for everything dynamic:

hiddenimports=["ai.ollama_bootstrap", "ui.setup_wizard", "whispercpp", ...]

Lessons learned

Thread safety with PyQt6 is non-negotiable — never call Qt UI methods from background threads. Use QMetaObject.invokeMethod or signals.
Whisper base.en is the sweet spot — tiny is too inaccurate, small is too slow on CPU.
Vision APIs hate face identification — build the filter early, not after your first support ticket.
PyInstaller + Ollama = packaging nightmare — Ollama runs as a separate process, not bundled. The setup wizard that auto-installs it saved countless support issues.

What's next

macOS port (the hotkey system is the main blocker)
Plugin system for custom AI skills
Sub-200ms response time optimization

The full source is on GitHub and it's MIT licensed. If you build something on top of it, let me know.

📥 Download: https://clicky.foo
⭐ GitHub: https://github.com/Bitshank-2338/clicky-windows

I built a free AI that reads my screen and answers out loud – no API key, runs offline with Ollama

Shashank Kumar Singh — Wed, 06 May 2026 11:15:16 +0000

I kept hitting ChatGPT's $20/mo wall. So I built my own.

Clicky is a free, open-source AI assistant for Windows that floats next to your cursor, reads your screen, and speaks the answer back out loud. No copy-paste. No alt-tab. No subscription.

The problem I was solving

Every time I hit an API limit or saw that $20/mo charge, I thought — I have a decent laptop, Ollama runs locally, why am I paying for this?

But existing local AI setups require you to open a terminal, type prompts, copy-paste context... it's friction. I wanted something that just worked — press a hotkey, ask a question, get an answer.

How Clicky works

Press Ctrl+Alt+Space anywhere on Windows
It captures all your screens and starts listening
You ask your question out loud
It reads the screen context, queries the AI, and speaks the answer back

That's the entire flow. No window switching. No typing.

Tech stack

Python + PyQt6 — system tray app, always running in background
Whisper.cpp — fast local speech-to-text (runs on CPU)
Ollama — local LLM inference (llama3.2:3b for text, qwen2.5vl:3b for vision)
edge-tts — Microsoft's neural TTS, free and offline-capable
PyInstaller — packages everything into a single Windows .exe
Inno Setup — builds the one-click installer

AI provider flexibility

Clicky isn't locked to Ollama. It supports:

Provider	Cost	Requires
Ollama (local)	Free forever	~4GB RAM
OpenAI GPT-4o	Pay per use	API key
Anthropic Claude	Pay per use	API key
GitHub Copilot	Free with Student Pack	GitHub account

What it can do

Read whatever is on your screen (YouTube, code, emails, docs, anything)
Search the web and answer instantly
Point at exact UI elements with a glowing arrow overlay
Click buttons on your behalf just by asking
Answer questions about code you're looking at without copy-pasting

Setup in v1.1.1

The latest version ships with a first-run setup wizard — it detects if Ollama is missing, downloads and installs it automatically, then pulls the right models. One click from installer to working AI.