DEV Community: Thilakasiri H.A.B.D

I Couldn't Remember the Last Time I Did Nothing, So I Built a Sloth to Do It With Me. You just have to do nothing. 🦥

Thilakasiri H.A.B.D — Thu, 22 Oct 2026 11:57:28 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

Be honest with yourself for a second. Can you actually remember the last time you sat back in your chair and did nothing?

I don't mean watching a video while eating lunch. I don't mean listening to a podcast on a walk, or scrolling through your phone on the couch. I mean genuinely sitting back, looking at the ceiling or out a window, with no screen in your hand and no noise in your ears.

If you can't remember, neither could I.

Lately, every tiny two-minute gap in my day gets eaten by my phone. Waiting for a project to build? I grab my phone. Waiting for water to boil? Phone. Lying in bed exhausted at midnight? Still scrolling. We tell ourselves that picking up our phone is "taking a break," but it really isn't. It's just feeding more noise to an already tired brain.

The funny thing is, when we finally try to rest using a meditation or wellness app, those apps make us work too. They give you strict breathing counts (inhale for 4, hold for 7, exhale for 8), tell you how to sit, and track your "mindfulness streak." If you miss a day, the app turns red and makes you feel guilty. If your mind wanders, you feel like you're failing at relaxing.

While thinking about the Slow and Steady theme this week, I started looking into why "doing nothing" feels so hard for us, and why it's so important for our health.

It comes down to one simple truth: rest isn't something you can force. It's something you stop getting in the way of.

Most of us treat relaxation like a transaction: "Okay, I did this breathing exercise for five minutes, why am I not calm yet?!" It's just like trying to force yourself to sleep when you have insomnia. The harder you try and the more you check the clock, the more awake you stay.

Your brain and body already know how to settle down if you just give them space. When you finally put the phone down and stop taking in new information, your brain's natural resting state kicks in. In the background, it starts sorting through your day, letting stress fade out, and connecting random ideas. You don't need another productivity technique. You just need permission to sit back and let your thoughts float by like leaves on a river.

So I built SlothMode, a calm, 100% local laptop app where you cannot fail.

You put your phone down, lean back, and do nothing. While you rest, a sleepy little three-toed sloth named Momo (our own character, though definitely a friendly distant cousin of DEV's sloth mascot, Sloan!) hangs out with you.

Here is how it works:

No techniques or rules. You don't have to close your eyes, sit still, or count your breaths. Fidgeting, stretching, or staring at the wall are all completely fine.
Only your phone matters. Your laptop webcam checks once per second for one thing only: are you holding your phone? Leaving your phone face-down on the desk is totally fine.
No guilt, ever. If you do pick up your phone, nothing turns red. No alarm goes off and you don't lose a streak. The timer just pauses quietly. Momo opens one sleepy eye, lets out a slow yawn, rolls over, and waits for you: "No rush. Momo will wait."
The Moss Meter. Real sloths move so slowly that green algae grows on their fur. In SlothMode, you rest so slowly that soft green moss grows in patches on Momo. After 20 minutes of rest in a day, a tiny pink flower pops up on his head.
Rest Moments and Scenes. You don't always have 15 minutes free. On the home screen, you can tap quick buttons for real moments in your day, like "Before a meeting" (2 min), "Want to scroll?" (3 min), "After a hard talk" (5 min), or "Before sleep" (15 min), while Momo rests in a Forest Morning, Rainy Day, Seaside, or Starry Night window.
Momo's Leaf Book. Have you ever noticed that the second you sit down to rest, your brain suddenly remembers three chores you forgot to do, or comes up with a random project idea? Usually, you grab your phone to write it down and accidentally spend 30 minutes scrolling. In SlothMode, after your rest ends, Momo softly asks: "Did anything float by?" Local Gemma tidies your rough notes into leaves in your Leaf Book (Idea, To-do, or Thought), or lets you click "Let go" because "Some ideas are just nice to have had."

(Note: SlothMode is a rest tool, not medical help.)

Demo

🎥 Direct link: Slothmode - Its ok to do nothing (YouTube)

A couple of small details I loved building (and that you can spot in the walkthrough above):

Code

Bin4yi / slothMode

SlothMode

A laptop app for doing nothing, where you can't fail. Rest with Momo the sloth. The webcam checks one thing only: is a phone in your hand?

Built for DEV Hacktoberfest Week 3, "Slow and Steady". The AI at the core is open: MediaPipe for the camera check and Gemma (open weights, via Ollama) for Momo's few words. Both run on your laptop.

What it feels like

Press Do nothing, or tap a rest moment ("Before a meeting", "Want to scroll?", "Before sleep" and so on). No menus before resting.
Momo meditates, then sleeps, in a scene with a soft looping sound. Momo picks the scene himself: forest morning, rainy day, seaside sunset or starry night, by time of day and the rest moment. Before sleep always rests under the stars, slightly dimmed.
Pick up your phone and the timer pauses quietly. Momo yawns and waits: "No rush.…

View on GitHub

🔗 Repository: https://github.com/Bin4yi/slothMode

Licensed under the **MIT License.

How I Built It

I built SlothMode with Vite, React, and TypeScript. I wanted the whole app to feel like a quiet forest at golden hour, so I used soft greens and warm sand colors (#F4EFE3, #6E9A5B, #2F4A3A, #E7D3B0), rounded Nunito type, and slow 600–1200ms transitions so nothing on screen ever jumps or rushes.

1. A webcam check that moves like a sloth (MediaPipe)

An app about resting shouldn't make your laptop fan spin like a jet engine.

I used MediaPipe Tasks Vision (EfficientDet-Lite Object Detector and Pose Landmarker) in the browser, slowed down to just 1 frame per second.
Just detecting a "cell phone" wasn't enough because my phone sits on my desk while I work. So I combined the phone box with the wrist coordinates from the Pose model. It only counts as "phone in hand" if the phone is near your wrist for 3 checks in a row.
If you get up from your chair to stretch or look out a real window, the app notices nobody is in the frame after 10 seconds and switches to away. The timer keeps running and Momo snacks on a leaf. And if you don't want to turn on your camera at all, there is a "No camera, just a timer" option right on the setup screen.

2. Momo's sleepy voice (Gemma on Ollama)

I used Gemma running locally through Ollama on localhost for four small jobs:

The Sloth Guide: Most guided sessions talk way too much. In a 15-minute SlothMode session, one Gemma call at the very beginning pre-generates just 2 or 3 short lines (max 12 words each, like "Nothing to fix right now. Your mind can wander. Momo's does too."). Making them at the start means nothing loads during the quiet part of your break. The browser's speechSynthesis reads them softly at a slow pace.
Sloth Thought of the Day: One short line generated once a day on the home screen.
Sorting the Leaf Book: When you jot down a thought after resting, Gemma turns it into clean JSON ({ title, type, first_step }) and saves it in IndexedDB. For ideas, it adds one tiny, low-pressure first step.
Weekly Look-Back: Once a week, Gemma looks over your leaf titles and points out a simple pattern (like "The week's notes include a thought, a todo list, and an idea for a balcony garden"). It only states facts from your notes and never tries to give life advice.

If someone opens the app without Ollama running, it quietly falls back to 30 pre-written offline lines without throwing any errors.

Why Does Open Innovation Matter?

To put it simply: nobody wants an app watching them through their webcam if that video goes to the cloud.

If you're leaning back with your eyes closed, or writing down private thoughts that floated into your head while resting, you shouldn't have to wonder if your camera frames or journal entries are sitting on someone else's server.

Running MediaPipe and Gemma on your own laptop solves that completely:

Frames never leave your computer. The camera check happens in browser memory once a second and the frame is thrown away right after. Nothing is saved or uploaded.
No subscriptions or paywalls. Many wellness apps ruin the mood by locking sounds or sessions behind a paywall. Because open models run locally on your own hardware, SlothMode costs nothing to run and stays free.
Works offline. Turning off Wi-Fi is honestly the best way to take a real break. Since Gemma runs on localhost, the whole app works offline.

The Lazy Payoff

I was juggling two big deadlines this week, which meant I badly needed this app myself. Here is what I noticed from using it and checking the logs:

Quiet fans: Running MediaPipe at 1 FPS instead of 30 FPS cut the vision CPU load by over 90%, so my laptop stayed completely silent during sessions.
Real Gemma timings: In my console logs, pre-generating a session's lines with local Gemma took around 1.4s at the start of a break, and tidying a rough note into a Leaf Book entry took about 0.9s.
Real breaks: Over four days of building and testing, I ended up taking 65+ minutes of actual phone-free rest. Even better, whenever a random "don't forget to fix that bug" thought popped into my head, I knew I could drop it in the Leaf Book after the timer ended, so I didn't reach for my phone once.

Prize Categories

Overall Winner ("Slow and Steady"): Instead of automating chores just so we can cram more work into our day, SlothMode rewards actual laziness and protects the quiet downtime our brains need.
Best Use of Gemma: Local Gemma gives Momo his sleepy voice during sessions, writes the daily thought on the home screen, turns messy post-rest notes into clean JSON leaves, and finds weekly patterns in the Leaf Book, all on localhost.

Thanks for reading. Now close this tab, put your phone face-down, lean back, and go do nothing for 5 minutes. Momo will wait. 🌿🦥

RepCheck: An AI Helper That Watches Workout Videos for Online Coaches (MediaPipe + Gemma)

Thilakasiri H.A.B.D — Sun, 18 Oct 2026 16:11:14 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

RepCheck is a helper for online personal trainers.

Online coaches get workout videos from their clients every day. Watching each one closely takes hours, so feedback often ends up as just "looks good 👍". The client doesn't improve, and the coach burns out.

RepCheck fixes that in four simple steps:

Film it. The client films one set on their phone and uploads it.
We measure it. MediaPipe finds the body in every frame. Python code counts each rep and measures depth, speed, tempo and left/right balance.
We check it. Clear, fixed rules turn those numbers into a 0–100 form score and one or two simple tips. The rules use the client's profile: a beginner gets fewer cues, and pain, age 65+ or an injury switches on a careful mode with no load increases.
The AI explains it. An open model (built for Gemma, running locally with Ollama) reads only the numbers, never the video, and drafts friendly feedback, a progress note and next week's plan.

Then the most important part: the coach decides. Nothing reaches the client until the coach reads the draft, edits it and clicks Approve. The client can also get it as a voice note.

Clients also get a GitHub-style training page: a contribution heatmap where every workout adds a square, plus streaks and badges.

Who pays? Online coaches with 20–100 remote clients. They already spend their evenings watching videos. My price idea is about $10 per month per coach for up to 20 clients. That only works because the AI runs locally and costs almost nothing per video (more on that below).

Demo

Code

Bin4yi / repCoach

RepCheck

▶️ Watch the 2-minute demo on YouTube

AI helper for online personal trainers. Clients film a set; MediaPipe finds body points, Python code measures and judges form, and Gemma (open-weight, run locally with Ollama) turns only the numbers into feedback, a progress summary and a next-week plan. The coach reviews, edits and approves everything. The AI only drafts.

Run it

Code: https://github.com/Bin4yi/repCoach

git clone https://github.com/Bin4yi/repCoach.git && cd repCoach
cp .env.example .env                         # set COACH_PASSWORD
docker compose up --build                    # backend, frontend, mongo, ollama (pulls Gemma)
docker compose exec backend python -m scripts.seed_demo_data   # 3 demo clients, ~1 year of data
# open http://localhost:3000/login  (coach)   and   http://localhost:3000/c/demo-kasun  (client)

No Docker? Quick local demo with an in-memory DB (no Ollama needed, template messages are used):

cd backend && python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt   # Linux/Mac: .venv/bin/pip
python scripts/download_model.py
MONGODB_URI=mongomock:// DEMO_SEED_ON_START=true COACH_PASSWORD=demo uvicorn app.main:app --port 8000
cd

…

View on GitHub

Backend: Python, FastAPI, MediaPipe Tasks (PoseLandmarker), OpenCV, NumPy/SciPy, MongoDB (Motor).
Frontend: Next.js, TypeScript, Tailwind, Recharts.
Tests: 128 backend tests (pose maths, rep counting, judgement rules, the LLM layer with a mocked Ollama, API tests, heatmap and streak logic), plus a frontend type-check, build and tests.
Run it: docker compose up starts the backend, frontend, MongoDB and Ollama, then pulls Gemma.

How I Built It

The big idea: MediaPipe measures. Code judges. Gemma explains.

I split the work so that each tool does only what it is good at.

1. Measuring (open-source vision). MediaPipe PoseLandmarker finds 33 body points in every frame. My code smooths the jitter, fills short gaps, and checks the video quality. Is the whole body in view? Is the camera angle the one the client chose? Is there only one person in the shot? If not, the client gets a friendly "please film again from the side" message.

2. Counting and scoring (plain code, no AI). Reps are found as valleys in the knee angle (squat) or elbow angle (push-up, curl). Each rep gets its depth, its down time and up time, and its speed. The form score comes from simple weighted rules that I documented, so the same video always gets the same score.

I tested it on real, openly licensed footage, not just synthetic data, and real footage found real bugs:

A tracking glitch at the bottom of a squat split one rep into two. Fixed: two valleys now count as separate reps only if the signal climbs back up between them.
Setting up and re-racking the weight looked like extra reps. Fixed: shallow dips at the start or end of a set are dropped.
My "knees caving in" check gave false alarms in deep squats. Fixed: it now uses the knee-width to ankle-width ratio.

After these fixes, a real side-view squat set counts 6 of 6 reps and a real push-up set counts 6 of 6, with no false knee-cave flag on a front-view goblet squat.

3. Explaining (open-weight LLM). Gemma runs locally through Ollama and uses structured JSON output. It never sees the video, the images, or even the client's name, only the numbers. Then my code checks every answer:

No invented numbers. Every number in the AI's text must exist in the input data. If not, it regenerates once, then falls back to a plain template.
No diet or calorie talk, and client messages stay under 120 words.
Pain means pain. If the client reported pain, the message must tell them to stop and talk to their coach, and it may not suggest more weight.
Weight increases are capped in code at +5% per week (0% in careful mode), even if the AI suggests more.

4. Storing and sharing.

MongoDB stores only numbers and text. The raw video is deleted right after processing, and the skeleton overlay video is deleted when the coach approves.
ElevenLabs turns the approved message into an optional voice note.
render.yaml deploys the backend and frontend to Render. Ollama runs on the coach's own machine or a GPU box.

Why Does Open Innovation Matter?

For this product, open models aren't just a nice extra. They're what makes it possible.

Privacy of body videos. People don't want videos of their bodies sent to a big AI company. With MediaPipe and a local Gemma, the video never leaves the server, and it's deleted after measuring. Only numbers and text are kept.
Near-zero cost per video. A closed vision API would have to read dozens of video frames for every upload. RepCheck sends the AI a tiny JSON of numbers instead, and the model runs locally, so the cost per video is basically electricity. That is what makes a low monthly price for coaches realistic.
Accuracy by design. An LLM looking at a video can "see" a knee angle that isn't there. Here, measuring is done by an open pose model plus deterministic code, and the LLM only explains numbers it was given, with a check that it didn't invent any.
No lock-in. Switching to a newer Gemma means changing one setting (OLLAMA_MODEL). A coach can run everything on their own hardware.
I could fix it. When real footage exposed rep-counting bugs, I could dig into the landmark signals myself and fix them. With a closed "send video, get feedback" API, I would have been stuck.

Prize Categories

Gemma. The writing layer (session feedback, progress summary, weekly plan) is built for Gemma through Ollama, with structured output and code-side safety checks.
MongoDB Atlas. Clients, sessions, weight logs and plans are stored in MongoDB (MONGODB_URI points at Atlas).
ElevenLabs. The coach can send the approved message as a voice note. The promo video's narration was also made with ElevenLabs.

FieldDay: say any game, play it outside, and let your phone be the referee

Thilakasiri H.A.B.D — Fri, 09 Oct 2026 06:37:11 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

FieldDay is a web app that turns your phone into a referee for real games outside.

You say a game, like "highest throw battle, 3 rounds, 2 players". FieldDay makes the game, checks
that it is safe, and reads you the rules. Then you lean the phone on your bag and go play.

The phone watches with the camera. It sees your body and the ball. It measures how high you throw
and how high you jump. It keeps the score and shouts like a sports announcer: "Two point four
metres! Massive!"

The screen is not the game. The field is the game. The phone just watches.

What you can do with it:

Say any game. Speak or type it. The AI turns your words into a game with real rules.
12 quick games: Sky Toss, Jump Battle, Freeze Tag Statue, Reaction Race, Squat Storm, Boss Raid and more.
Battles with friends on one phone: turn battles, Red vs Blue teams, Chaos Mode, King of the Hill, and Boss Raid, where everyone jumps, squats and punches together to beat a monster. When the referee shouts "FREEZE!", everyone must stand still.
Quests that move you around the park: "Race to the biggest tree." "Find something red." The phone checks your photo, and the photo never leaves the phone.
Ghost challenges: your scores go into a QR code. A friend scans it and plays against your "ghost", with no internet needed.
Touch Grass meter: after every game it tells you how long you played and how little you looked at the screen: "You played 34 minutes. Screen time: 4%."
Safety first: it says no to unsafe ideas (throwing at people, roads, climbing, water) and offers a safer game. Kids Mode uses a calm voice, and every 15 minutes it reminds you to drink water.

Who is it for? Friends, families, kids and school groups who want to play outside but end up on their phones. It's also for places like a park with no signal. FieldDay works with zero internet.

Demo

Try it: https://github.com/Bin4yi/FieldDay.git

Home	Say a game	Play	Quests

No ball or park nearby? Pick Demo camera in the app. A simulated player throws and jumps, and you can watch the referee work.

Code

Bin4yi / FieldDay

FieldDay

Say any game. Play it outside. The phone is the referee.

FieldDay turns a phone into a voice-first referee for real outdoor games, battles and quests Say a game ("highest throw battle, 3 rounds, 2 players"), lean the phone on a bag, and play with your body and a ball. The phone watches (pose + ball tracking), listens (bounce and catch sounds) measures (throw height, jump height, reaction time), keeps score, and shouts like a football announcer, a wrestling host, a calm coach, a robot or a pirate.

The AI never writes code. It only fills in a Game Spec: JSON built from fixed blocks (trackers, events, measures, rules). A small, fully tested rules engine runs the spec.

Open Mode works with zero internet: the brain (Gemma), speech to text (Whisper), body and ball tracking (MediaPipe) all run on the phone.

Two hackathons

DEV Hacktoberfest Week 1…

View on GitHub

The git tag v0.1-devto is my submission for this challenge.

How I Built It

The main idea is simple: the AI never writes code. It only fills in a Game Spec, a small piece of JSON built from fixed blocks: trackers, events, measures and rules. A small, well-tested rules engine runs that spec. So the AI can be creative, but it can't break the game.

The open-source AI inside FieldDay, all running on the phone:

Part	Open model / tool	What it does
The brain	Gemma (open weights), run in the browser with MediaPipe LLM Inference	Turns your words into a Game Spec and makes quests
The ears	Whisper tiny (MIT), run with Transformers.js and ONNX Runtime Web	Speech to text, so you can just say your game
The eyes	MediaPipe Pose Landmarker + EfficientDet object detector (Apache-2.0)	Finds your body and the ball in every frame

How a game works:

You speak. Whisper turns your voice into text on the phone.
Gemma designs the game. It writes a Game Spec. I check the spec with strict zod schemas. If something is wrong, Gemma gets one chance to fix it. If it still fails, a built-in template takes over, so you always get a game.
Safety check. Every game passes a safety check and a "can the camera see this?" check. For example, the camera can't measure spin or heart rate, so FieldDay offers the closest game it can referee.
Field Check. Before you play, FieldDay checks that the phone is steady, the light is good, your whole body is in view, the ball is found and the space is clear.
The camera referees. Pose and ball tracking turn into events like jump, ball apex, catch and freeze broken. Heights come from your body size, so it doesn't need a tape measure (about ±10% after a quick calibration).
The referee talks. Short, hand-written lines in 5 styles (football announcer, wrestling host, calm coach, robot, pirate) are spoken with the phone's own voice.

No Gemma model on your phone, or no WebGPU? Open Mode still works. An offline rules designer
turns your words into a game from the templates, and the same safety check runs.

Built with: Vite + React (a PWA that works offline), Zustand, Dexie (IndexedDB), zod, and a
pnpm monorepo with separate packages for the engine, vision, brain, quests and UI. It has 165 unit
tests and 26 Playwright browser tests, including a test that plays a whole game in airplane mode.

Some bugs I had to fix along the way:

Freeze never fired. The body points shake a tiny bit every frame, and that looked like movement. Now I compare against a frame from about 200 ms earlier.
Punches everywhere. Raising your arms looked like a punch. Now a punch needs a straight arm at shoulder height that moves fast.
A falling ball counted as a catch. Now the ball must also slow down at your hand.

Why Does Open Innovation Matter?

For FieldDay, open models are not a nice extra. Without them, the app doesn't work.

Parks don't have Wi-Fi. The best places to play often have weak or no signal. A closed AI API needs the internet for every request. Gemma, Whisper and MediaPipe run on the phone, so FieldDay works in airplane mode. I tested this: a typed game becomes a playable, refereed game with the network turned off.
Your video stays on your phone. The camera watches kids and families playing. With open models, the video goes to the model on the phone and nowhere else. Nothing is uploaded. That would be very hard to promise with a cloud API.
It's free. No API key, no bill and no limits. A school can use it with a whole class, and nobody pays per game.
It's fast. A referee has to react right away. A call to a server and back is too slow for a "FREEZE!".
I can see inside it and change it. Open weights and open tools let me test, measure and fix every part, from the pose points to the game rules.

A closed API would have made FieldDay another app that only works at home on Wi-Fi. Open AI let me
build one that works outside, which is the whole point of touching grass.

My Agent Session

I built FieldDay with an AI coding agent (Claude Code) as my pair programmer. I planned the
features, made the decisions and tested the results, and the agent helped me write and test the
code.

Prize Categories

Best Use of Gemma. Gemma is FieldDay's brain in Open Mode: it designs every game and quest on the phone, offline.

Thanks for reading. Now put your phone down and go play! 🌱

Mathaka (Memory): “Did I take my medicine?” - answered by open AI on an old phone

Thilakasiri H.A.B.D — Sun, 04 Oct 2026 11:30:06 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

In many Sri Lankan homes, the hardest medicine question is not when. It is "did I already take it?"

I built this for my grandmother. She has mild dementia, and she often forgets her pills. Some days she misses a dose. Other days she isn't sure, so she asks us again and again: "Did I take my medicine?" We aren't always home to answer. I wanted her phone to give her that answer: kindly, the same way every time, and without guessing.

We tried pill reminder apps, but none of them fit her:

They only say when. The alarm rings and she takes the pills. Ten minutes later she can't remember doing it. Then she either skips the dose or takes it twice, and both are dangerous.
They are made for young people. Small text, menus, swipes and settings confuse her. One wrong tap and the reminder is gone.
They don't speak her language. Most are in English only. She reads and speaks Sinhala.
They don't know her pills. Her clinic gives loose tablets in small paper envelopes, often with no label. An app that shows drug names doesn't help when she can't match a name to a white tablet.
They need the internet and an account, and they send her health data to someone's cloud. Her phone is old and cheap, and the signal at home isn't always good.

So I built Mathaka (මතකය, "memory" in Sinhala). It is a medicine helper for older people with memory loss. It works in Sinhala, Tamil and English. It is one Android app with two roles:

On the elder's phone (this can be an old, cheap phone):

One gentle reminder, in the family's own recorded voice if they want.
Only the pills for this time of day, shown as big pictures with what they are for ("sugar medicine"), never a drug name.
A big button: "Did I take my medicine?" The app answers from its own log, in the same calm words every time: "Yes, Ammi. You took your morning medicine at 8:00. The next one is at lunch time."
A double-dose guard. If she tries to take it again, nothing is logged twice, and the app tells her kindly that she already took it.
Huge text, 7:1 contrast, a high-contrast theme, a real clock, "Say it again", and no swipes or double taps.

On the family's phone:

Set up the medicines once, by scanning the prescription and photographing each pill.
See the day at a glance: doses taken, missed doses, low stock, the clinic date and a week of history.
Get alerts over home Wi-Fi, or by plain SMS when there is no internet.
Print a "fridge card" with every pill on paper, for days when the phone is off.

There is also an optional Arduino pill box. The slot that is due lights up, and pressing it means "I took it".

Home	This slot's pills	"Did I take it?"	Family dashboard

Demo

The 4.5-minute video shows the elder phone, the family phone, how the AI works, the pill measuring, the Arduino box and the test results.

Try it in a browser: clone the repo, then run npm install and npm run web. Demo mode fills in a sample family and a week of history.
Try the Arduino box with no hardware: open the Wokwi ESP32 simulator. Paste in firmware/esp32/mathaka_box/mathaka_box.ino and diagram.json, then type LIGHT 1 in the serial monitor.

Code

Bin4yi / Mathaka

Mathaka (මතකය)

A dementia-friendly medicine helper for older people in Sri Lanka. Android first, Sinhala / Tamil / English. All AI runs on the phones, with open-weight models. There is no cloud.

A reminder is not remembering. Alarm apps say when. Mathaka also answers whether you already took it, and stops a second dose.

Two open AIs. Gemma can see. Laya decides but cannot speak. Grandma only hears words her family wrote.

Status: prototype for the DEV Hacktoberfest "Build for a Friend" challenge. Not a medical device. See Honest limits.

Demo video (4.5 min)

▶ Watch the demo (MP4). It covers the elder flow, the family phone, the on-device AI, pill vision, the Arduino box and the evaluation The video is made from HTML with HyperFrames. Its source is in videos/mathaka-promo.

Screenshots

Real screens from the web build (npm run web…

View on GitHub

It's an Expo / React Native app (TypeScript), a shared pure-TypeScript core, an ESP32 Arduino sketch, an optional Python laptop helper, and an evaluation folder with frozen test sets. It has 124 automated tests.

How I Built It

My rule was: heavy AI once, light AI every day, and no cloud anywhere.

1. Setup happens once, on the family phone, with Gemma

A family member photographs the prescription. Gemma 3 4B (vision, Q4_K_M GGUF) runs on the phone through llama.rn, which is llama.cpp for React Native. It reads the prescription into rows. It also reads the number printed on each pill and writes a short description for the family.

Clinic books are handwritten, so I don't trust the model blindly:

Every answer must fit a JSON schema. It's checked with zod and retried once.
A person confirms every row before anything is used.
Gemma is never loaded on the elder's phone. The code refuses.

2. The pill is measured, not guessed

You put each pill on a printed mat with four ArUco markers and a grey patch. Then plain TypeScript (no big model) does the work:

finds the markers and flattens the perspective
converts pixels to millimetres
fixes the colour using the grey patch
cuts out the pill and measures its size, shape, colour, two-colour halves and score line

For the demo pill it measured 19.2 × 9.2 mm, oblong, white, with a score line, in about 200 ms. The same code runs on the phone, in the browser and in the tests.

3. One file moves the setup to the elder's phone

A second person must approve the whole setup. Then it is saved as one file (.mtk, a zip with a SHA-256 hash for every part). You can send it by Bluetooth, a USB cable or WhatsApp-to-yourself. The elder's phone rejects any file that is broken, changed, older or not approved. There's no account and no server.

4. The AI that is not allowed to speak

This is the part I care about most. No AI ever writes the words the elder hears. Every sentence is a fixed template in Sinhala, Tamil or English, filled in with facts from her own log. A unit test blocks words like "take another".

For understanding what she means (like "mama beheth biwwada?", Singlish for "did I take my medicine?"), I tested Laya. It's an open, non-generative decision model: it can only pick one option from a short list and give a probability. It cannot write a sentence.

5. Alerts without a cloud

When a dose is missed, the family finds out:

over home Wi-Fi (the two phones are paired once with a QR code)
by plain SMS if Wi-Fi fails, with no mobile data needed
by retrying in the background until the alert gets through

6. The Arduino pill box

An ESP32 with 4 LEDs and 4 buttons talks to the phone over Bluetooth LE (or USB serial). The phone sends LIGHT 1, and the box sends back PRESS 1. The box never decides anything. A press goes through the same double-dose guard as the screen.

What the tests say

I froze the test sets before measuring.

Test	Result
"Did I take it?" answers (40 hand-written cases)	40 / 40
Double doses stopped	4 / 4
Unsafe sentences said to the elder	0
Pill matching, known pills found (synthetic photos)	100% (colour alone: 62%)
Pill matching, new pills rejected	100% (colour alone: 92%)

The honest part: Laya, without training, acted wrongly on about 1 in 4 Sinhala and Singlish questions. So it stays switched off. The elder's phone keeps big buttons, which can't misunderstand her. I'd rather ship buttons than a confident wrong answer to someone with dementia. I've prepared a fine-tune on Sinhala care phrases as the next step.

Why Does Open Innovation Matter?

Privacy that is real. Medicine photos, the dose log and a family member's recorded voice never leave the two phones. With open weights I can check that, instead of trusting a privacy policy.
It works in a village. After a one-time model download, everything runs offline. Alerts go by plain SMS. A cloud API stops working when the signal does.
No monthly bill for a pensioner. Running the model costs nothing per use, and the elder's phone can be old and cheap.
I could measure the risk. Because Laya is open, I could test exactly how often it would act wrongly before letting it near a vulnerable person. That test is the reason it's switched off.
I could change what matters for us: Sinhala and Tamil templates, a pill pipeline built for loose clinic pills in paper envelopes, and later a model fine-tuned for Singlish that I can give back.

My Agent Session

I built Mathaka with Claude Code as my pair programmer, working on Windows with no Android emulator. We used the web build and headless Chrome to test the screens. The demo video was also made with code: HyperFrames HTML scenes, an offline Kokoro voice and MusicGen music, all in videos/mathaka-promo.

Prize Categories

Best Use of Gemma: Gemma 3 4B vision runs on the family phone through llama.rn. It reads prescriptions and pill imprints and writes pill descriptions, always behind a JSON schema and a human check.
Best Use of Arduino: the optional pill box is an Arduino sketch for the ESP32 (Bluetooth LE + USB serial), with a Wokwi diagram so anyone can try it without hardware.

Honest limits

It's a prototype. It is not a medical device, and doctors haven't tested it.
A pill photo can't safely identify a medicine. Mathaka uses it only as an extra check, after the schedule and the box slot.
The pill numbers come from synthetic photos so far. I'm collecting real ones.
It doesn't replace a carer. It gives the carer one less worry.

Credits

Thanks to the Gemma, llama.cpp / llama.rn, Laya, js-aruco2, Expo and HyperFrames teams. The photos in the video are from Wikimedia Commons: Ayena (public domain), Stevepb (CC0), trokilinochchi (CC BY 2.0) and Edwiyanto (CC BY-SA 4.0).