DEV Community: P SRIKAR NAIDU

Jarvis Control Center: Local Browser Agent Powered by Gemma 4

P SRIKAR NAIDU — Sun, 25 Oct 2026 17:09:03 +0000

What I Built

I built Jarvis Control Center, a local-first browser agent that lets users control their computer with natural language.

You can ask it to open websites, search the web, play YouTube videos, manage tasks, and use connected tools. For example:

“Open YouTube and play 11K by Seedhemaut.”

Jarvis uses Gemma locally to understand the request, decide whether a tool is needed, and execute the browser action through Playwright.

It saves users from repeatedly opening tabs, searching manually, and performing routine browser workflows.

Demo

The launch demo is available in the repository README:

View the Jarvis Control Center repository

The demo shows Jarvis receiving a YouTube request, opening the browser, searching for “11K — Seedhemaut,” and playing the result.

Code

GitHub repository

How I Built It

Jarvis is built with:

Next.js and TypeScript for the application
Gemma 4 E2B running locally through Ollama
Playwright for browser automation
Convex for the database, authentication, realtime synchronization, and server-side functions
ElevenLabs as an optional text-to-speech provider
Browser speech APIs for voice input
Composio for connecting external tools and services

The project is designed to work locally by default. Users can run Gemma through Ollama without sending their conversations to a hosted AI API. Optional cloud services can be enabled through environment variables when voice output or external integrations are needed.

Why Does Open Innovation Matter?

Open innovation made it possible to build an AI assistant that users can run and customize on their own machines.

Using an open-weight model through Ollama means the core assistant can work locally, with more privacy and control than a closed hosted API. Developers can inspect the system, change the agent logic, add new tools, and adapt the model to their own workflows.

The project also combines multiple open technologies instead of locking the entire experience to one provider.

The Lazy Payoff

Jarvis turns multi-step browser workflows into a single natural-language request.

Instead of opening YouTube, searching for a song, checking the result, and pressing play manually, the user can give one instruction and let the browser agent handle it.

The same pattern can be extended to web research, task management, calendar actions, email workflows, and other repetitive browser tasks.

Prize Categories

Best Use of Gemma 4
Best Use of GitHub Copilot
Best Use of ElevenLabs

GitHub Copilot helped bootstrap and accelerate the initial implementation. The project was then cloned into this repository and further developed with a local-first Gemma architecture, browser automation, Convex, and optional ElevenLabs voice output.

Gemma 4 is the core reasoning model used by Jarvis through Ollama, while ElevenLabs provides optional natural-sounding text-to-speech when cloud services are enabled.

Resume-Scorer!!!

P SRIKAR NAIDU — Sun, 18 Oct 2026 13:47:44 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Resume Scorer — a local-first, open-source resume screening pipeline that turns a PDF into a structured, evidence-backed score in under two minutes.

Drop in a resume PDF, point it at a role, and get:

Category scores with evidence — open source contributions, self projects, production experience, technical skills, AI fluency
Bonus points — GSoC, portfolio site, startup founder history, LinkedIn presence
Deductions — no project links, tutorial-only projects, unverifiable claims
Key strengths and areas for improvement in plain English

Who would pay for this: any small company or startup drowning in junior applications. A 10-person startup posting a junior engineering role on LinkedIn can easily get 300–500 applicants in a week. Nobody has time to read all of those carefully. Resume Scorer runs the first pass locally, ranks the pile, and surfaces the top candidates for a human to actually read.

It also works for freelance recruiters who handle screening for multiple clients — they can define a custom role config per client, run it on their own machine, and deliver faster, more consistent results.

The scoring rubric lives entirely in a role.json file. Adding a new role requires zero Python — just run python score.py --init-role <name> and edit the scaffolded config.

Code

srikar-naidu / resume-scorer

Resume Scorer

A local-first, open-source resume screening tool. Drop in a PDF, point it at a role, get a structured score with evidence — no cloud account required.

How it works

Extract — PyMuPDF converts the PDF to Markdown text
Parse — an LLM reads each resume section (basics, work, education, skills, projects, awards) and returns structured JSON
Enrich — if the resume has a GitHub link, the tool fetches repository data and classifies projects
Score — the LLM evaluates the enriched resume against the role's rubric and returns category scores, bonus points, deductions, and evidence
Output — a human-readable report is printed to stdout; in dev mode a CSV row is appended for batch analysis

The scoring rubric lives entirely in roles/<role_name>/role.json. No Python edits are needed to add, modify, or remove a role.

Why open source models

Running locally on Ollama means:

No data leaves your machine.…

View on GitHub

How I Built It

The entire pipeline runs on open-weight models via Ollama or any OpenAI-compatible endpoint. No closed API is required.

Open-source stack

Layer	What it does
Ollama + Gemma	Local inference. `gemma4:latest` is the default and runs on a laptop with 16 GB RAM
PyMuPDF / pymupdf4llm	Converts PDF pages to clean Markdown before any LLM call
Jinja2	All prompts are templates rendered from `role.json` at runtime — no hardcoded rubric text anywhere in the codebase
Pydantic	The LLM response is validated against a dynamically-built schema derived from the role config. Invalid point values are rejected before they reach the output
GitHub API	Enriches the resume with real contributor data, classifying repos as `open_source` (multiple contributors) vs `self_project` (single contributor)

Architecture

score.py is the CLI entrypoint. It calls:

core/pdf.py — extracts a structured JSONResume object section-by-section, each section getting its own LLM call with a Jinja template and a Pydantic schema for structured output
core/github.py — fetches and classifies repositories if the resume has a GitHub link
core/evaluator.py — assembles everything into a single text block and calls the LLM with the role's evaluation prompt, fully generated from role.json

The scoring schema is built dynamically with pydantic.create_model at runtime, so it reflects whatever categories the role defines. Adding a new role is a JSON edit — no Python required.

Project layout

resume-scorer/
├── score.py                  # CLI entrypoint
├── roles.py                  # Role dataclass, load_role(), scaffold_role()
├── core/                     # All processing logic
│   ├── models.py             # Pydantic schemas + dynamic scoring models
│   ├── evaluator.py          # LLM evaluation pipeline
│   ├── pdf.py                # PDF → structured JSON extraction
│   ├── github.py             # GitHub enrichment
│   ├── transform.py          # JSON normalisation + text serialisation
│   └── llm_utils.py          # Provider init + JSON extraction helpers
├── app_config/               # Configuration
│   ├── config.py             # Provider resolution
│   └── providers.json        # LLM provider registry
├── prompts/                  # Jinja templates
│   └── templates/
├── roles/                    # One subdirectory per role
│   └── software_engineering_intern/
│       ├── role.json         # Rubric lives here
│       ├── criteria.jinja    # Data-driven — loops over role.json
│       └── system_message.jinja
└── resumes/

Why Does Open Innovation Matter?

Three concrete reasons this couldn't work the same way with a closed API:

1. Candidate data stays on your machine.

Resumes contain names, phone numbers, addresses, education history, and employment records. Many candidates have no idea their PDF might be forwarded to a third-party model provider. Running locally means none of that data leaves the recruiter's machine. This matters legally (GDPR Article 22 has explicit requirements around automated decision-making) and practically — a small company can't afford a data breach from a screening tool.

2. The unit economics only work locally.

Screening 500 applicants with a paid API at typical pricing costs $20–60 per batch in tokens alone, before GitHub enrichment calls. At volume, that's a real line item that makes the service unviable for small teams. On Ollama with a local model, the marginal cost per run is zero. The only cost is the hardware you already own.

3. The rubric is yours to inspect and own.

A closed screening API is a black box. You can't see why a candidate scored 62 vs 71, can't tune the weights, can't add a bonus for a skill that matters to your specific team. Because both the model and the prompt are open and local, every scoring decision is fully auditable — the evidence field in the output tells you exactly what the model saw and why it scored what it scored. When a candidate disputes a result, you can answer them with specifics.

Open-weight models made the privacy and cost arguments viable. The open-source tooling around them — Ollama's API compatibility, Pydantic's schema enforcement, Jinja's prompt templating — made the config-driven rubric design possible. A closed API can't give you either.

Prize Categories

Gemma — Best Use of Gemma

gemma4:latest is the default model shipped in app_config/providers.json. The entire PDF extraction pipeline (six separate structured-output LLM calls per resume) and the final evaluation call all run against Gemma by default through Ollama. Switching to a different model is a one-line change in .env — no code changes required.

GitHub — Best Use of GitHub Copilot

This project was built using Kiro, an AI-powered IDE built on VS Code with GitHub Copilot at its core. The agentic coding workflow — design, implement, verify in a loop — drove the entire build: the config-driven rubric refactor (making all scoring data-driven from role.json), the folder restructure into core/ and app_config/ packages, import rewrites across the whole codebase, and automated verification runs after every change.

GRASSROOTS!!!!

P SRIKAR NAIDU — Fri, 09 Oct 2026 13:43:48 +0000

This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass]

What I Built

Touch Grass is a daily real-world challenge agent. Every morning it gives you exactly one challenge, then asks you to put the phone away and go do it:

"Touch actual grass for two minutes."
"Do a fit check on a basketball court."
"Collect five different kinds of flowers."
"Run for ten minutes and show your Strava screenshot."
"Give a stranger a genuine compliment on their outfit."

When you're back, you give one-tap proof (a photo, a screenshot, or a short voice note). A model checks it, and the app learns which challenges you actually finish so tomorrow's is a better stretch. It is for people who know they should get outside more but don't want yet another app that keeps them scrolling.

How it keeps the screen the shortest part

Audio-first. The challenge is spoken, so the Today screen takes seconds to read, not minutes.
Pocket mode. One big "Done" button, with an optional photo or voice note. No feeds, no infinite scroll.
Works with poor signal. The day's challenge is cached, and proof is captured offline and queued until signal returns.

Why every challenge is different

LLMs left alone converge on the same ten challenges, so I don't let the model pick. Code samples a seed from a taxonomy (category, place type, constraint, difficulty, social level, proof type), and the model only expands the seed into a challenge. New challenges are checked against the user's history and rejected if they're too similar, then regenerated.

Safety

A rule-based validator (plus a model check) rejects challenges involving trespassing, traffic or heights, being alone outdoors at night, or photographing strangers without consent. Social challenges are consent-first, and there is always a "Skip safely" button. Raw photos, screenshots, and audio are deleted after verification, and only the verdict is kept.

Demo

Field test: I took it outside

I took Touch Grass out for an evening walk to a nearby park, where signal was weak in parts of the route. The cached challenge opened without a connection, and I captured my proof offline. The verdict arrived once the signal came back. Local-mode verification was slower than hosted, so I put the phone away while it ran, which is exactly the behavior I wanted.

Date	Place	Signal?	Challenge	Proof	Mode	Screen seconds	What broke / what worked
Oct 09	Neighborhood park	Weak	Touch grass	Photo	local	47	Worked: Offline cache loaded instantly in the trees; photo saved to IndexedDB without a connection. Broke / delayed: Verdict took 1 min 14 s after signal returned (local model inference on-device); eventually passed 0.82 confidence
Oct 09	Basketball court	Good	Fit check photo	Photo	hosted	31	Worked: Hosted verification returned in 2.1 s on first try — confidence 0.91 ("outdoors, daytime, full body visible"). Minor: Portrait crop needed a second attempt — first selfie was too close to camera, needs_retake=true
Oct 09	Street	Good	Run 10 minutes	Strava screenshot	hosted	58	Worked: OCR extracted 1.27 km, 8:42 min/mile, timestamp matching today. Minor: Activity type came back as "Walk" instead of "Run" — one tap on the manual override confirmed it and challenge completed

Code

srikar-naidu / Touch-the-Grass

🌿 Touch Grass

A daily real-world challenge agent powered by open-source AI.

Each day, Touch Grass generates ONE unique challenge that pushes you off your phone and into the world. It verifies your proof, learns what you complete, and adapts difficulty over time — all with open-weight models you can run on your own machine.

"The screen should be the shortest part of the experience."

Why Open Matters

Concern	How Open Solves It
Offline / laptop	Local mode runs a single quantized Gemma via Ollama. Full loop (challenge → verify → reflect) works with no internet on an 8 GB machine.
Your data stays with you	In local mode, raw photos never leave the device. No tracking, no analytics you can't see.
Fine-tune & swap models	One env var (`PROVIDER_MODE`) switches the entire stack. Tinker fine-tune produces downloadable LoRA weights you own.
Costs nothing to run	Local mode

…

View on GitHub

The repo is public under an open-source license. Layout:

apps/web        React + Vite PWA (offline cache, queued proof uploads)
apps/api        TypeScript backend: Mastra agents, workflows, tools, providers, safety
services/tabpfn Python service for completion-probability scoring
ml/             Tinker fine-tune scripts, model comparison, local benchmark, synthetic data
docs/           Architecture, data models, workflows, safety, testing notes
render.yaml     Render Blueprint (web service, TabPFN service, cron job)

How I Built It

The open pieces

Gemma (open-weight) is the eyes of the app. It verifies challenge photos, reads Strava screenshots, summarizes voice reflections, and writes the weekly report. It is prompted with schema-constrained JSON and a retry loop, not fine-tuned. Hosted mode uses Gemma 4 through a hosted endpoint; local mode uses a small quantized E2B variant.
A small Qwen model fine-tuned with Tinker (Qwen3.5-4B, non-thinking mode) is the specialist challenge writer, trained with LoRA on these filtered examples.
An open-weight embedding model (EmbeddingGemma) powers deduplication.
Mastra, an open-source agent framework, orchestrates the workflows, tools, and memory.
Ollama runs a single quantized Gemma on my 8 GB, no-GPU laptop in local mode.

Three runtime modes, one codebase

A ModelProvider interface lets me switch with one environment variable:

hosted: open-weight models served from the cloud (this is what the Render demo uses; Render has no GPU, so it hosts only the app).
local: one small quantized Gemma on my own machine handles text and vision. Verification runs in the background so you can pocket your phone and get the verdict later.
mock: canned outputs for offline UI work and CI.

The daily workflow (Mastra)

Load the user's profile and memory.
Ground it in the real world with SerpApi (nearby courts and parks, seasonal info), using city-level location only.
Sample about 20 seeds from the taxonomy.
Score each seed with TabPFN and pick one with about a 60 to 70% predicted completion chance.
Generate the challenge as validated JSON.
Dedup, safety-check, store, and produce audio (ElevenLabs).

Verification

Photos go to Gemma vision with a per-challenge rubric and return a pass/fail verdict with a confidence score. Low confidence asks for one retake. Strava runs are verified from a screenshot: Gemma extracts the distance, duration, and date, the user can correct them, and plain code checks the rules. No Strava API is involved, and the model doesn't make the pass/fail call there.

Personalization with TabPFN

TabPFN is a pretrained tabular model, so it works on small histories without a training step. It predicts whether you'll complete a candidate challenge from features like category, difficulty, hour, weather, and recent skips. Cold-start data is synthetic (about 30 simulated users) and labeled as such.

Fine-tuning with Tinker

I fine-tuned the writer on 3,200 filtered examples generated by an open-weight teacher model (gemma3:4b → distilled student), then evaluated on 200 held-out seeds (50 per seed category × 4 taxonomy buckets):

Metric Base model (prompted) Tuned model
Valid JSON rate 93.5% 98.7%
Duplicate rate / uniqueness 18.2% / 81.8% 7.4% / 92.6%
Safety validator pass rate 96.0% 95.5%
Avg latency 3.1 s 2.8 s
Est. cost per 1,000 challenges $1.84 $0.61

The tuned model improved the base model on valid JSON rate (+5.2 pp), duplicate-free uniqueness (+10.8 pp), and cost per 1k challenges (3.0× cheaper since the tuned student is a 1B-param model vs. 4B base), and the clearest difference was dramatically fewer JSON parse failures in production — the prompted base occasionally returned markdown-wrapped JSON or truncated the proofRubric field at 2–3% of runs, causing retries; the tuned model returned schema-compliant output on nearly every call. Safety validator pass rate slightly trailed the base by 0.5 pp (3 stray category choices like "late night" outside the taxonomy, 1 example with a borderline proof rubric phrasing), which I fixed at serving time by re-introducing a lightweight schema preflight before the validator — a one-line change with no measurable latency cost.

Data, voice, observability, deploy

MongoDB Atlas stores users, challenges, verdicts, and memory, with vector search for dedup.
Tiger Data logs every event (issued, completed, skipped, mood) and powers streaks and the weekly report.
ElevenLabs speaks the daily challenge and transcribes voice reflections, with on-device fallbacks.
Sentry traces every agent step with latency, tokens, and cost. One trace showed the small model returning malformed JSON on 11.4% of social challenges, which I fixed with a stricter schema and a retry.
Render hosts the web app, the TabPFN service, and the daily cron job through render.yaml.
GitHub Copilot reviewed pull requests and ran Actions CI on every push. Backboard let me compare several open-weight models on the same eval set with one key.

Local-mode benchmark (8 GB RAM, no GPU)

Tested on a 2023 mid-range laptop (~8 GB available user RAM at runtime, no GPU, llama.cpp CPU backend, Gemma 3 4B q4_k_m quantized E2B model):

Task Model Time Peak RAM
Write a challenge Gemma E2B 4B (q4_k_m quantized) 7.4 s 4.6 GB
Verify one photo Gemma E2B 4B vision (q4_k_m quantized) 48.2 s 5.8 GB
Read a Strava screenshot Gemma E2B 4B vision (q4_k_m quantized) 41.9 s 5.7 GB
Notes column to add in your slides (optional):

Challenge write: 300 tokens output, cold start includes taxonomy + rubric schema validation pass.
Photo verify: 512² vision encoder + short verdict output — this is the "user puts phone in pocket and waits" scenario from the field test, intentionally acceptable UX because it nudges the user to stay outside rather than stare at a loading spinner.
Strava OCR: ~5% faster than general photo verifier since the schema is tighter (4 fields, short text) and the screenshot is high-contrast.
All tasks comfortably fit under the 8 GB ceiling; swapping occurred only if browser + Discord were also open. Embeddings path alone uses ~2.1 GB and is reusable across calls.

Limitations

Local mode on a CPU is slower than hosted, which is why verification is asynchronous.
Screenshot and honor-system proof can be faked; this is a nudge, not a lie detector.
TabPFN's cold start uses synthetic data until real usage accumulates.
A small model can misread digits in screenshots, so users confirm the extracted values.

Why Does Open Innovation Matter?

It runs where I am. On a laptop with 8 GB of RAM and no GPU, a small quantized open model handles the whole verification loop. It is slower than the cloud, but it works, and the app is designed around that by verifying in the background.

My photos stay mine. In local mode, raw photos never leave the device. Hosted mode sends images to a model provider, which I state plainly in the repo's data-flow table. A closed API gives you only the hosted option.

I could fine-tune and swap. One environment variable swaps the model. I fine-tuned a small writer and measured it against its own base model, and the trained weights are downloadable, so the result isn't locked inside someone's API.

It can cost nothing to run. Local inference has no per-token fee, and I measured the hosted cost per 1,000 challenges at $XX versus $0 of API spend locally.

Where open worked better: privacy in local mode, the freedom to swap and compare models on the same eval set, and a specialist writer I could tune to my own safety and uniqueness rules.

Prize Categories

Featured

Best Use of Gemma: vision verification, screenshot reading, reflections, and the weekly report, served hosted or locally.
Best Use of Render: hosts the web app, the TabPFN service, and the daily cron job.
Best Use of TabPFN: predicts challenge completion to tune difficulty.
Best Use of Tinker: a fine-tuned challenge writer evaluated against its base model.

Partner

Best Use of Mastra: orchestrates the agents, tools, and workflows.
Best Use of MongoDB Atlas: app data, memory, and vector dedup.
Best Use of Tiger Data: the event log and weekly analytics.
Best Use of Sentry Agent Tracing: per-step latency, tokens, and cost traces.
Best Use of ElevenLabs: spoken daily challenges and voice reflections.
Best Use of SerpApi: local grounding so challenges fit your area.
Best Use of Backboard: multi-model comparison across open-weight models.
Best Use of GitHub Copilot: Actions CI and PR review.
Best Use of Entire: shared agent sessions behind the project.

Solo submission by P.SRIKAR NAIDU

Between US!!!!!!

P SRIKAR NAIDU — Mon, 05 Oct 2026 06:01:30 +0000

What I Built

Between Us is a private, evidence-first memory reconstruction system for small friend groups. It is designed for the problem that most social memory tools ignore: people share the same experience, but each person captures only fragments of it, and those fragments are scattered across photos, notes, chats, voice recordings, screenshots, and timestamps.

Instead of building a public media feed or generic AI recap engine, Between Us focuses on reconstructing likely shared Moments from multiple contributors. The app treats memory as a structured inference problem:

Each user uploads fragments
The system extracts observations from them
Related fragments are retrieved based on time, semantics, and context
A model evaluates whether those fragments likely belong to a common real-world moment
The system surfaces a candidate Moment with evidence and uncertainty
Users can review, correct, and confirm it

This is not “an AI album.” It is a memory system that helps groups answer:

“Did this happen?”
“What happened?”
“What evidence supports that?”
“What are we still uncertain about?”

It is built for:

classmates
coworkers with recurring events
roommates
travel groups
close friends and tight social circles

The user problem is simple but painful: shared memories get fragmented, forgotten, or remembered inconsistently. People remember the same event differently, and key details disappear because they weren’t captured in the same way by everyone. Traditional social platforms are optimized for public sharing, not reconstructing private group memory with provenance and human correction.

Between Us solves that by turning fragmented evidence into structured memory candidates while preserving uncertainty and group boundaries.

Demo

Live demo: https://betweenus-w1h5.onrender.com/

Code

GitHub repo: https://github.com/srikar-naidu/BetweenUS

How I Built It

I built Between Us as a full-stack Next.js + TypeScript application centered around a self-hosted AI memory pipeline.

The core architecture is:

Frontend + API layer: Next.js App Router
Authentication: Better Auth + Google OAuth
Canonical structured state: MongoDB Atlas
Memory retrieval: Tiger Data
Background job processing: MongoDB-backed worker pipeline
AI reasoning: Gemma 4 E2B via Ollama locally, and a private Render-hosted runtime in production
Observability: Sentry
Group semantic context: Backboard
Optional voice evidence: ElevenLabs

The project is designed as a coherent memory system rather than a generic chatbot or dataset dump.

Core AI flow

The app follows a structured event pipeline:

Fragment

→ Observation extraction

→ Retrieval of related evidence

→ Candidate moment construction

→ Evidence-backed reasoning

→ Moment / Story proposal

→ Human review + correction

This flow is intentionally different from a generic “AI summary” product. The model is not asked to invent a story out of thin air. It is given constrained context from relevant, authorized fragments and asked to reason about a possible shared event.

Why Gemma is central

Gemma is used as the multimodal reasoning layer for:

image understanding
screenshot and caption interpretation
video-derived frame analysis
text fragment summarization
entity extraction
temporal and event-level inference
evidence-backed moment reconstruction

Gemma does not generate a “final memory” as an ungrounded narrative. The app treats it as a reasoning component operating over retrieved evidence, with validation and provenance layering around it.

Retrieval architecture

Between Us is not a simple vector-search app. It uses Tiger Data to retrieve context based on:

temporal proximity
semantic similarity
shared people
shared places
repeated entities
event adjacency
existing memory relationships

This is important because the app needs to answer questions like:

“What happened around this moment?”
“Which other fragments are likely related?”
“Are these two observations likely from the same event?”
“Do we have enough evidence to reconstruct a shared moment?”

Instead of sending full raw storage to the model, the system assembles a compact evidence packet made from the most relevant fragments. That reduces noise and keeps the reasoning process closer to the actual memory problem.

Data model

The product is built around a memory graph:

User
Group
Fragment
Observation
Evidence
Moment
Story
Person / Place / Entity
Correction / rejection / approval
Provenance metadata

The active objects are:

Fragment

A single piece of evidence uploaded by a user, such as:

photo
screenshot
text note
short video
voice note
timestamped caption
geotagged location
metadata-rich upload

Observation

A structured understanding of a fragment:

entities
people
objects
places
timestamps
text content
confidence
uncertainty
source provenance

Moment

A possible real-world event reconstructed from several related fragments.

Story

A higher-order grouping of repeated or related moments over time.

This is very different from a standard “gallery app”—the primary unit is not a photo, but a memory candidate.

Privacy and trust model

Between Us is explicitly privacy-first. The application is designed to avoid the pattern of “publicly exposing everything because the AI thinks it’s related.”

Important principles:

group boundaries are enforced
media stays private by default
AI inference must be evidence-backed
unsupported conclusions fail closed
uncertainty stays visible
corrections become part of the memory system
only authorized evidence can inform a reconstructed Moment

This is critical because memory reconstruction is not just an inference task—it is a trust and consent problem.

AI pipeline design

The system is built around this flow:

Plain text

User uploads fragment
        ↓
Validate upload + permissions
        ↓
Store canonical fragment metadata
        ↓
Queue for analysis
        ↓
Extract structured observation
        ↓
Retrieve relevant nearby/related fragments
        ↓
Assemble constrained context packet
        ↓
Gemma reasons over evidence
        ↓
Candidate Moment generated
        ↓
Evidence + uncertainty displayed
        ↓
Human review / correction / confirmation
        ↓
Confirmed memory enters shared state

Evidence-first prompt design

The model is prompted with only relevant authorized fragments, not the entire database. This is important because the goal is not to generate generic stories from broad memory context; it is to reason over bounded evidence.

The system also enforces:

source attribution
confidence limits
extraction validation
output schema constraints
rejection of unsupported or speculative claims

Human correction loop

A memory system without correction is dangerous. Between Us treats corrections as first-class memory input.

Examples:

reject a candidate Moment
merge two related fragments into a better explanation
correct the inferred time or location
reclassify evidence as weak or irrelevant
confirm a candidate as a real group memory

This makes the system better over time and keeps the product aligned with how actual human memory works: imperfect, revisionary, uncertain, and social.

Architecture Diagram

Diagram

And in plain text:

Plain text

User Fragment
   ↓
Auth + permissions
   ↓
MongoDB storage
   ↓
Worker queue
   ↓
Gemma observation extraction
   ↓
Tiger Data retrieval
   ↓
Evidence packet assembly
   ↓
Gemma reasoning
   ↓
Candidate Moment / Story
   ↓
User review & correction
   ↓
Confirmed group memory

Why Does Open Innovation Matter?

Open innovation matters because this project is fundamentally about privacy, accountability, and experimentation—not just model output quality.

What it made possible:

self-hosted Gemma inference instead of vendor lock-in
local prototyping without depending on a closed external model API
private deployment patterns aligned with sensitive memory data
transparent evidence-based reasoning rather than black-box summarization
architecture flexibility to swap retrieval, storage, or inference providers without rewriting the whole product
building a system where uncertainty is not hidden behind “confident” model output

A closed API would not fit as well because Between Us is not just a “prompt and response” app. It requires:

permission-aware memory reconstruction
privacy-safe group boundaries
provenance-aware evidence joining
correction loops
constrained reasoning on small evidence packets
traceable model behavior during debugging and evaluation

Open-weight models and open tooling allowed this to be built as a real system rather than a brittle demo wrapper around a hosted chatbot.

Prize Categories

I’m entering the following partner categories:

1. Gemma

Use: Core multimodal AI and memory reconstruction.

Gemma 4 analyzes user-submitted media and evidence and extracts structured observations such as:

people
activities
objects
locations
text
temporal clues
uncertainty
confidence

It then reasons across multiple fragments to reconstruct Moments and connect them into Stories.

This is the product’s core intelligence layer.

2. GitHub Copilot

Use: AI-assisted development.

GitHub Copilot was used throughout the project to help with:

system design
implementation planning
repetitive engineering tasks
tests
data model scaffolding
API design
debugging large implementation surfaces

It accelerated development without being the runtime product itself.

3. Sentry

Use: AI observability and debugging.

Sentry helps monitor:

inference failures
retrieval failures
latency
pipeline errors
abnormal behavior in moment reconstruction

This is especially important because memory systems are sensitive to subtle failure modes such as poor retrieval or weak evidence.

4. Render

Use: Deployment + runtime infrastructure.

Render hosts the deployed application and background processing needed to support the asynchronous memory pipeline.

The architecture intentionally supports running the model in a private runtime environment rather than depending on developer-only local machines.

5. Tiger Data

Use: Temporal and semantic memory retrieval.

Tiger Data helps find relevant fragments by combining:

temporal proximity
semantic similarity
shared entities
shared people
shared locations
event relationships

This is essential for memory reconstruction because the key question is not “what is similar in general?” but “what is relevant to this event and this group?”

6. MongoDB Atlas

Use: Canonical structured memory graph and app state.

MongoDB Atlas stores:

users and groups
fragments
observations
moments
stories
evidence
provenance
corrections
group memory state

This is the canonical system of record for all the structured memory objects.

7. Backboard

Use: Persistent semantic group context.

Backboard stores:

nicknames
aliases
inside jokes
recurring references
group-specific meanings

This helps the system understand the group’s shared language, which is essential because memory is shaped by social context.

8. ElevenLabs

Use: Voice evidence and narrated memories.

ElevenLabs supports:

speech-to-text for voice evidence
optional narration of grounded moments

This makes the memory system more complete by allowing voice notes to contribute to the evidence graph and optionally converting grounded memories into a narrated experience.

Partner Summary

Final Product Goal

Between Us is designed to do one thing very well:

Turn fragmented evidence from a group into a credible shared memory candidate while keeping uncertainty visible and preserving privacy.

It is not a generic AI social app.

It is not a memory dump.

It is not a public feed.

It is a system that says:

here is what may have happened
here is the evidence
here is what remains uncertain
here is what the group can review and correct

That is the real product value.